Having just joined the Auto-Tools team they thought that they would start me on something fun! Here’s the problem that I was presented with:
- each type of unit test uses it’s own manifest file format
- there is a different manifest file reader for each type of manifest
- we hack each manifest separately when we want to expand functionality (like skipping a test per-branch, or adding a pref to a test)
Here are some examples:
- canvas/reftest.list
- Regress/jstests.list
- xpcshell example Makefile
- mochitest example Makefile
- sanity/crashtests.list
So, some tests use Makefiles and some use lists and we don’t have a standardized means of adding extra information to any given test. Some test lists are generated on the fly (say as built up in the web browser for mochitest), some are checked in and static. Changing which tests mochitest and xpcshell export currently involves Makefile hacking, something that is pretty unpleasant.
Auto-tools proposes that we settle on a universal manifest format for all tests. Most likely this would be json (we toyed with yaml but that would add extra dependency overhead and bug 517950 has requested that we remove yaml from talos, so it doesn’t seem to have many fans).
We’ve collected proposed formats to the Universal Manifest Project Wiki. One thing that we don’t want to do here is design in a bubble. While there are benefits to the auto-tools team in terms of code re-use, centralizing bug fixes and such the biggest consumer of these tests are developers. We want to provide all these benefits without sacrificing developer productivity. Our goal is to keep our test harnesses as simple and as easy-to-use as possible while making them extensible and flexible for whatever the future holds. Feedback is both requested and highly appreciated.