We all agree that reftest and the current manifest solution works great. It can be extended and the tools can support a wide range of flexibility. There is also a need to have a better solution for identifying and conditionally running our other tests thanks to things like mobile and e10s.

A few things to think about:
1) reftest has its own internal parser with bugs and glitches. It works well because we all understand it, so we don’t need to change it
2) the reftest manifests are getting more complex with e10s, mobile, 64 bit and new os versions, and preferences. This results in them being harder to read and what happens if a specific test has to do different things based on different conditions?
3) what is an ideal manifest for mochitest? Mochitests are really just a listing of test files and don’t need the ==, != type of operators, nor the conditional clauses.
4) how would this work for xpcshell? xpcshell is a list of directories, followed by a list of files including the head and tail files (in appropriate order). I do not see a way to make the reftest format work for xpcshell
5) would having >1 manifest type be a problem? so lets say we keep the reftest* suites with the reftest manifest, could we use a different format for mochitest and xpcshell that would meet their needs better, maybe json, xml, yaml ?
6) would the ability to add layers (think css) to existing or future manifest formats help alleviate some concerns here? It would sure solve the problem of munging the manifests with fennec specific stuff. One idea is we could keep reftest manifest format, but read it into an object and overlay the conditions and rules with another manifest file. This overlay file could be the same format or a different format.

Those are just some questions and things to think about.

- Run N times, and take the average or minimum time
- Have the harness time how long the benchmark takes to run, or, expect the benchmark to output its own metrics

Also, the reftest manifest format is not a good fit for any of our other test suites. We could certainly wedge them in there, like jsreftest has done, at the expense of making them harder to read and write, but I don’t think that’s a great solution. We could also invent a new manifest format for Mochitest, but that means we’re not much better than the current state, where you have to learn a new syntax for each harness’ manifest.

I agree that simply changing reftest to accomplish other goals isn’t a great reason. If we can remove some of the complexity of the reftest manifests by converting them to a new format, then I think we’ll have a more compelling argument.

It’s not hard to add new features like “temporarily set this pref” to reftest.js. That was the answer when we wanted to move jstests to use reftest.js a few months ago, and I don’t see why the answer would be different for mochitests.

JSON is not inherently more flexible, but it is harder to work with in a text editor.

I’m not sure what you mean by “per-branch info”. Tests live in the same hg repository as code, so when code branches, the test manifest branches with it.