Having just joined the Auto-Tools team they thought that they would start me on something fun! Here’s the problem that I was presented with:
- each type of unit test uses it’s own manifest file format
- there is a different manifest file reader for each type of manifest
- we hack each manifest separately when we want to expand functionality (like skipping a test per-branch, or adding a pref to a test)
Here are some examples:
- canvas/reftest.list
- Regress/jstests.list
- xpcshell example Makefile
- mochitest example Makefile
- sanity/crashtests.list
So, some tests use Makefiles and some use lists and we don’t have a standardized means of adding extra information to any given test. Some test lists are generated on the fly (say as built up in the web browser for mochitest), some are checked in and static. Changing which tests mochitest and xpcshell export currently involves Makefile hacking, something that is pretty unpleasant.
Auto-tools proposes that we settle on a universal manifest format for all tests. Most likely this would be json (we toyed with yaml but that would add extra dependency overhead and bug 517950 has requested that we remove yaml from talos, so it doesn’t seem to have many fans).
We’ve collected proposed formats to the Universal Manifest Project Wiki. One thing that we don’t want to do here is design in a bubble. While there are benefits to the auto-tools team in terms of code re-use, centralizing bug fixes and such the biggest consumer of these tests are developers. We want to provide all these benefits without sacrificing developer productivity. Our goal is to keep our test harnesses as simple and as easy-to-use as possible while making them extensible and flexible for whatever the future holds. Feedback is both requested and highly appreciated.
I can’t really make any sense out of the proposed examples, some comments on just what they are describing might help, but being that unreadable makes them harder to hack on imo.
Note that I’ve been working in the background on improving the state of mochitest makefiles in bug 370750, should I abandon that now?
Comment by Mossop — May 7, 2010 @ 3:01 pm
@Mossop Good point, I just chucked ideas presented by auto-tools members into a wiki and didn’t really put any sort of extra info. I’ll have to flesh that out.
I wouldn’t totally abandon makefile cleanup just yet, we’ll still need to get more people on board before we can push through a universal manifest solution.
Comment by alice — May 7, 2010 @ 3:40 pm
reftests, jstests, and crashtests all use the same manifest format, parsed by reftest.js. I think it’s a pretty good manifest format, and I don’t understand why you want to invent a new one. Why not just convert mochitest to use the reftest manifest format?
Comment by Jesse Ruderman — May 7, 2010 @ 3:56 pm
What I like about the current reftest manifest format is that it’s concise and scannable. Moving to JSON would lose that.
Comment by Jesse Ruderman — May 7, 2010 @ 3:59 pm
@Mossop, yeah the project is just getting off the ground, so that wiki page will get more useful as we get further along in the design. I think the work you’re doing in bug 370750 is valid as it solves one of the problems we have now (specifically, that it sucks to edit mochitests in the makefiles). The universal manifest is the way forward to help us be more intentional and flexible with these tests as our harnesses and our platform changes. For instance we can use these manifests to mark and filter our mochitests. So we can create filters of “these tests fail on fennec” and we can either ignore those or run those, depending on what we want to do.
Comment by Clint Talbert — May 7, 2010 @ 4:03 pm
@jruderman The interest is in having a more flexible format, one that could allow for more per-test information (preferences, per-branch info, etc) that is not supported in the current simple reftest/jstest format. At the moment this information is handle in inconsistent ways depending on the test, making it both hard to add and then also hard to track after the fact (ie, what prefs run with this test?).
Comment by alice — May 7, 2010 @ 4:04 pm
Reftest already supports conditional tests, which you can use for Firefox vs Fennec.
It’s not hard to add new features like “temporarily set this pref” to reftest.js. That was the answer when we wanted to move jstests to use reftest.js a few months ago, and I don’t see why the answer would be different for mochitests.
JSON is not inherently more flexible, but it is harder to work with in a text editor.
I’m not sure what you mean by “per-branch info”. Tests live in the same hg repository as code, so when code branches, the test manifest branches with it.
Comment by Jesse Ruderman — May 8, 2010 @ 12:21 am
Don’t forget for xpcshell tests, it is critical that these can be run one at a time, or per-sub directory (it’d be nice to do that for other types of test as well!).
Comment by Standard8 — May 8, 2010 @ 2:31 am
What Jesse said — the reftest format can be extended in arbitrary ways, and we already have a lot of manifests in that format, and it’s easy to work with. What’s not to like?
Comment by Robert O'Callahan — May 9, 2010 @ 8:25 pm
The reftest format is a bit of a PITA to parse, and it starts to get pretty ugly when you add more complexity to it. I do like its simplicity in base form, but when you start layering more things (fails-if, asserts-if, etc) on top of it it gets unwieldy.
Also, the reftest manifest format is not a good fit for any of our other test suites. We could certainly wedge them in there, like jsreftest has done, at the expense of making them harder to read and write, but I don’t think that’s a great solution. We could also invent a new manifest format for Mochitest, but that means we’re not much better than the current state, where you have to learn a new syntax for each harness’ manifest.
I agree that simply changing reftest to accomplish other goals isn’t a great reason. If we can remove some of the complexity of the reftest manifests by converting them to a new format, then I think we’ll have a more compelling argument.
Comment by Ted Mielczarek — May 11, 2010 @ 7:00 am
So, in bug https://bugzilla.mozilla.org/show_bug.cgi?id=553351, we’d like to be able to have little benchmarks that get run. I imagine there could be a few different ways of running these little things, which the manifest should support. Things like:
- Run N times, and take the average or minimum time
- Have the harness time how long the benchmark takes to run, or, expect the benchmark to output its own metrics
Comment by Chris — May 12, 2010 @ 6:17 am
Guess I am a bit late to the game here.
We all agree that reftest and the current manifest solution works great. It can be extended and the tools can support a wide range of flexibility. There is also a need to have a better solution for identifying and conditionally running our other tests thanks to things like mobile and e10s.
A few things to think about:
1) reftest has its own internal parser with bugs and glitches. It works well because we all understand it, so we don’t need to change it
2) the reftest manifests are getting more complex with e10s, mobile, 64 bit and new os versions, and preferences. This results in them being harder to read and what happens if a specific test has to do different things based on different conditions?
3) what is an ideal manifest for mochitest? Mochitests are really just a listing of test files and don’t need the ==, != type of operators, nor the conditional clauses.
4) how would this work for xpcshell? xpcshell is a list of directories, followed by a list of files including the head and tail files (in appropriate order). I do not see a way to make the reftest format work for xpcshell
5) would having >1 manifest type be a problem? so lets say we keep the reftest* suites with the reftest manifest, could we use a different format for mochitest and xpcshell that would meet their needs better, maybe json, xml, yaml ?
6) would the ability to add layers (think css) to existing or future manifest formats help alleviate some concerns here? It would sure solve the problem of munging the manifests with fennec specific stuff. One idea is we could keep reftest manifest format, but read it into an object and overlay the conditions and rules with another manifest file. This overlay file could be the same format or a different format.
Those are just some questions and things to think about.
Comment by Joel Maher — May 12, 2010 @ 8:18 am