Any comments and suggestions on what needs addition or further clarification would be great.  I’m mostly going by the questions that are directed at me the most frequently.

Again, if you use Standalone Talos for local testing please update to latest.

• new tscroll test
• updated version of dromaeo
• tests for group opacity perf
• ability to dump all output to stdout using the ‘–screen’ option
• a bunch of orange fixes for failing to initialize browsers/timeouts

If you use Standalone talos for local testing please download and update to latest.

I’m still working from the stance that script doesn’t belong in manifest files so the format handles && and || cases without code. With the help of my auto-tools chums we searched the code base for difficult examples to ensure that this was still flexible enough to handle the expressions that we currently support.

Also, having worked on the conversions from old to new I have to say that it was really easy to get this going - admittedly, I’d done some reading on JSON and was immersed in the problem set - but I think that it is a good indication that this could make reading and writing manifests a lot nicer.

I’m once again happy to get any comments on the current proposal. If things look good it will be time to get everyone together for a Brown Bag to hash this out and finalize. I’ll get that in the works once I see if I have to take this back to the drawing board.

1. simple case

old

 == 399209-1.html 399209-1-ref.html
 == 399209-2.html 399209-2-ref.html
 == 399384-1.html 399384-1-ref.html

proposed

{   "tests" :
    [
        {  "==" : ['399209-1.html', '399209-1-ref.html']  },
        {  "==" : ['399209-2.html', '399209-2-ref.html']  },
        {  "==" : ['399384-1.html', '399384-1-ref.html']  }
    ]
}

2. ref test with two fails-if clauses and one random-if clause

old

fails-if(MOZ_WIDGET_TOOLKIT=="windows") fails-if(MOZ_WIDGET_TOOLKIT=="cocoa") random-if(MOZ_WIDGET_TOOLKIT!="cocoa"&&MOZ_WIDGET_TOOLKIT!="windows") != 399636-quirks-html.html 399636-quirks-ref.html # windows failure bug 429017, mac failure bug 429019

proposed

{   "tests" :
    [
        {  "!=" : ['399636-quirks-html.html', '399636-quirks-ref.html'],
           "fails" : { 'if' : ['windows']},
           "fails" : { 'if' : ['cocoa']},
           "random : { 'if' : ['!cocoa', '!windows']}  }
    ]
}

3. ref test with two random-if clauses

old

random-if(MOZ_WIDGET_TOOLKIT=="gtk2") random-if(MOZ_WIDGET_TOOLKIT=="cocoa") == mirroring-02.html mirroring-02-ref.html

proposed

{   "tests" :
    [
        {   "==" : ['mirroring-02.html', 'mirroring-02-ref.html'],
            "random" : { 'if' : ['gtk2']},
            "random" : { 'if' : ['cocoa']} }
    ]
}

4. ref test with one fails-if clause and one skip-if clause

old

fails-if(!haveTestPlugin) skip-if(!prefs.getBoolPref("dom.ipc.plugins.enabled")) == pluginproblemui-direction-1.html pluginproblemui-direction-ref.html

proposed

{   "tests" :
    [
        {   "==" : ['pluginproblemui-direction-1.html', 'pluginproblemui-direction-ref.html'],
            "fails" : { 'if' : ['!haveTestPlugin']},
            "skip" : { 'prefs' : ["dom.ipc.plugins.enabled" : "False"] }  }
    ]
}

5. js test with one fails-if clause and one skip-if clause

old

fails-if(!xulRuntime.shell&&!isDebugBuild) skip-if(!xulRuntime.shell&&isDebugBuild) script regress-455464-04.js # bug xxx - hangs reftests in debug, ### bug xxx - NS_ERROR_DOM_NOT_SUPPORTED_ERR in opt

proposed

{   "tests" :
    [
        {  "script"  : "regress-455464-04.js",
           "fails" : { 'if' : ['!shell', '!isDebugBuild']},
           "skip" : { 'if' : ['!shell', 'isDebugBuild']}  }
    ]
}

6. js test with one random-if clause and one asserts-if(count) clause

old

random-if(!xulRuntime.shell&&xulRuntime.OS=="WINNT") asserts-if(!xulRuntime.shell&&xulRuntime.OS=="WINNT",1) script regress-344804.js # bug 524732

proposed

{   "tests" :
    [
        {  "script" :  'regress-344804.js',
           "random" : { 'if' : ['!shell', 'WINNT']},
           "asserts"  : { 'if' :  ['!shell, 'WINNT'], "count" : 1} }
    ]
}

7. js test with one random-if clause, one fails-if clause and one skip-if clause

old

fails-if(xulRuntime.OS=="WINNT") random-if(xulRuntime.OS=="Linux"&&!XPCOMABI.match(/x86_64/)) skip-if(xulRuntime.OS=="Linux"&&XPCOMABI.match(/x86_64/)) script regress-3649-n.js # No test results on windows, sometimes no test results on 32 bit linux, hangs os/consumes ram/swap on 64bit linux.

proposed

{   "tests" :
    [
        {  "script" :  'regress-3649-n.js'
           "fails" : { 'if' : ['WINNT']},
           "random" : { 'if' : ['Linux', '!x86_64']},
           "skip" : { 'if' : ['Linux', 'x86_64']}  }
    ]
}

8. ref test with one asserts(min,max) clause

old

asserts(0-1) == background-draw-nothing-malformed-images.html background-draw-nothing-ref.html

proposed

{   "tests" :
    [
        {  "==" : ['background-draw-nothing-malformed-images.html', 'background-draw-nothing-ref.html'],
           "asserts" : {'min' : 0, 'max' : 1}
    ]
}

9. ref test with one asserts-if(count) clause

old

asserts-if(MOZ_WIDGET_TOOLKIT=="gtk2",1) == 355548-3.xml 355548-3-ref.xml # bug 456899

proposed

{   "tests" :
    [
        {  "==" : [' 355548-3.xml', '355548-3-ref.xml],
           "asserts" : { 'if' : ['gtk2'], 'count' : 1}  }
    ]
}

10. load test with one asserts-if(count) and one asserts(count) clause

old

asserts(10) asserts-if(MOZ_WIDGET_TOOLKIT=="windows",8) load 265986-1.html

proposed

{   "tests" :
    [
        {  "load" : '265986-1.html',
           "asserts" : { 'if' : ['windows'], 'count' : 8},
           "asserts" : { 'count' : 10}  }
    ]
}

Why aren’t we just going to take one of our existing manifest formats and roll it out to the rest of the tests?  Why re-invent the wheel?
The most suggestions were for adopting the reftest manifest format across the board.  While the reftest manifests are excellent at what they are designed for we are hitting the point where they can’t handle the growth in terms of 32 vs. 64 bit tests, qt vs. gtk vs. android (which all end up in the same ‘linux’ bucket), shell tests vs. full browser, etc.  Here’s some of the worst offenders that are already in use:

./js/src/tests/js1_5/extensions/jstests.list:
random-if(xulRuntime.OS=="WINNT"&&!isDebugBuild) skip-if(xulRuntime.OS=="Linux") script regress-342960.js # slow
./js/src/tests/js1_5/Regress/jstests.list:
fails-if(xulRuntime.OS=="WINNT") random-if(xulRuntime.OS=="Linux"&&!XPCOMABI.match(/x86_64/)) skip-if(xulRuntime.OS=="Linux"&&XPCOMABI.match(/x86_64/)) script regress-3649-n.js # No test results on windows, sometimes no test results on 32 bit linux, hangs os/consumes ram/swap on 64bit linux.
./js/src/tests/js1_6/extensions/jstests.list:
fails-if(!xulRuntime.shell&&!isDebugBuild) skip-if(!xulRuntime.shell&&isDebugBuild) script regress-455464-04.js # bug xxx - hangs reftests in debug, ### bug xxx - NS_ERROR_DOM_NOT_SUPPORTED_ERR in opt
./layout/reftests/bugs/reftest.list:
fails-if(MOZ_WIDGET_TOOLKIT=="windows") fails-if(MOZ_WIDGET_TOOLKIT=="cocoa") random-if(MOZ_WIDGET_TOOLKIT!="cocoa"&&MOZ_WIDGET_TOOLKIT!="windows") != 399636-quirks-html.html 399636-quirks-ref.html # windows failure bug 429017, mac failure bug 429019
./modules/plugin/test/reftest/reftest.list:
fails-if(!haveTestPlugin) skip-if(!prefs.getBoolPref("dom.ipc.plugins.enabled")) == pluginproblemui-direction-1.html pluginproblemui-direction-ref.html

It’s all pretty unreadable, and things are just going to get worse as more mobile platforms come online and we have to handle exceptions for each one.

Adding tests is currently easy and I don’t want to lose that.
Totally in agreement here! There may end up being a few more keystrokes, but overall things really aren’t changing much for the simplest case. Something like this:
#old syntax
== file1.html file1-ref.html
#new syntax
{"type" : "=="
"files" : ["file1.html", "file1-ref.html"]
}

What’s the benefit here?

• Greater granularity

Make it possible to target specific sections of code for test runs.  Specialized manifests would be easy to create around an area of interest. For instance, if I make a manifest that includes all content tests, then I can do something like:
> python runtests.py --manfiest=mycontentmanfiest.json
and it can run only the tests that I want based on my patch.

• Better benchmarks

Enable us to easily benchmark things - run all known [orange] tests, run specific performance intensive tests several times for profiling diagnostic, etc.

• Finer filtering

Want to enable a test on maemo only? Want to never run a test on android? We want to make this simpler and easier without having to do a bunch of inline javascript hacking.

Why are you picking on reftests?
We chose refest to be used as a proof of concept.  We wanted to roll the new manifest format out to reftest first to ensure that everything worked the way that we wanted (kick the tires a bit) and then slowly convert over our other test harnesses.

What about inline js calls like for isDebugBuild or checks against MOZ_WIDGET_TOOLKIT?
We no longer want to support js code in manifest files. We believe that that sort of code should live outside the manifest. Support would be built into the manifest for the states that these calls check (ie, lists of available platforms to test against, lists of build types, native theme types, etc). This way every test would use the same check, and if the check needs to be updated to support a new platform/arch/build/theme/etc it can be done in a central location. This will come in handy for the mobile platforms which would all currently end up identified as ‘linux’.

Where’s the new format?
This is probably the biggest question.  Truth is, we aren’t finalized here yet.  We are pretty sure that json is the way to go for the flexibility that we want - but we are still very open to suggestions.  Right now this is what we are considering:

{   "tests" :
    [
        {    "type": "==",
             "files": ["file1.html", file2.html"],
             "skip-if": ["maemo", "android"],
             "prefs": {
                          pref1 : bool1
                          pref2 : bool2
                      },
        },
        {    "type" : "js",
             "files" : ["jsfile.html"],
             "random-if" : ["winnt"],
             "skip-if": ["isdebug"],
        }
        {    "group" : "withTestplugin",
             "plugins": ["plugin1.xpi"],
             "prefs": {
                          enablePlugin : True
                      }
             "tests" :
             [
                 {    "type" : "!=",
                      "files" : ["file1.html", "file6.html"],
                 }
             ]
         }
    ]
}

This format would allow you to easily set prefs either per-test or per-group of tests.  There is also a means to install/enable plugins and then group tests together that use that plugin.  A lot has been borrowed from reftest manifest (skip-if, random-if) as that is stuff that works - we want to take concepts that work and keep them around.  What do you like about it?  What don’t you like?  What would you like included as an option?  Would you prefer xml?  yaml?  We are in very early stages here and the end result should be a manifest format that everyone is happy with.

• each type of unit test uses it’s own manifest file format
• there is a different manifest file reader for each type of manifest
• we hack each manifest separately when we want to expand functionality (like skipping a test per-branch, or adding a pref to a test)

Here are some examples:

• canvas/reftest.list
• Regress/jstests.list
• xpcshell example Makefile
• mochitest example Makefile
• sanity/crashtests.list

So, some tests use Makefiles and some use lists and we don’t have a standardized means of adding extra information to any given test.  Some test lists are generated on the fly (say as built up in the web browser for mochitest), some are checked in and static.  Changing which tests mochitest and xpcshell export currently involves Makefile hacking, something that is pretty unpleasant.

Auto-tools proposes that we settle on a universal manifest format for all tests.  Most likely this would be json (we toyed with yaml but that would add extra dependency overhead and bug 517950 has requested that we remove yaml from talos, so it doesn’t seem to have many fans).

We’ve collected proposed formats to the Universal Manifest Project Wiki.  One thing that we don’t want to do here is design in a bubble.  While there are benefits to the auto-tools team in terms of code re-use, centralizing bug fixes and such the biggest consumer of these tests are developers.  We want to provide all these benefits without sacrificing developer productivity.  Our goal is to keep our test harnesses as simple and as easy-to-use as possible while making them extensible and flexible for whatever the future holds.  Feedback is both requested and highly appreciated.

• Bug 480413 - design test to monitor browser shut down time
• Bug 483097 - have talos use firefox-bin to run the browser on linux/mac

See Standalone Talos documentation for download and installation instructions.

With a clearer head I approached the bugs on Monday morning and resolved both of them. Then it was just a question of waiting for an appropriate downtime. Though I’m as excited about 3.5b4 as anyone, it really gets in the way of arranging Talos code changes. Finally, yesterday afternoon we were able to shut down everything and check things in.

I’m glad to say that this landing was successful. There’s a good chance that Tshutdown will clean up the issues in Bug 478603 - intermittent orange on Windows mozilla-central talos Ts and Tdhtml tests (”failed to initialize browser”), as we’ll start to record the longer shutdown times instead of freezing up and reporting orange.

It’ll be a few more days till we have enough data for Tshutdown to look like anything but a scatter graph, but you can check out the reported results at graphs-new.mozilla.org. There’s actually more to Tshutdown than a single test. You’ll see “Tp3 Shutdown”, “Tp3 Nochrome Shutdown”, “Tp3 Fast Shutdown” and “Ts Shutdown”. Basically, we record the shutdown time after running the given test. Post Ts results in a ‘clean’ shutdown time as we run Ts with an empty, new profile; post Tp3 results in a ‘dirty’ shutdown time as the browser has just completed cycling 10 times through 400 web pages. The post Tp3 results will also show greater variance because we only run Tp3 once per full test cycle (the test does take a good hour or more to complete depending on the platform) and we only have that single value to report, with Ts we rapidly open and close the browser 20 times so we have a data set that we can average to get a more consistent value.

I’m very pleased to get the whole mess put to bed.  There were non-threadsafe python libraries (subprocess, I’m looking at you) to deal with, twisted banana errors (I kid you not), and a whole mess of timing issues.  You can’t build what you don’t monitor and shutdown is an import part of our user’s experience - hopefully we’ll be able to start to trim down our shutdown time now that we are reporting results from all our active Talos boxes.

Mostly it sounds like a bunch of Buildbot nonsense that shouldn’t really interest anyone outside of the Release Engineering team, but it has huge benefits to sheriffs and developers.  With Talos buildbot now supporting sendchange we can push builds through the Talos testing infrastructure provided only a correctly formatted link to the build in question.  I’ve already used this to force a re-test of a build that had failed; it immediately failed again on different Talos test boxes and proved that the issue was in the build and not in Talos.

If a build is on stage and downloadable we can make Talos test it as many times as we want.  We still don’t have the means to push any build we like through, but retesting pretty much any build on staging can be very useful when trying to narrow down a regression range or figure out if a regression is ‘real’.  If you are in a situation where you would like to retest a build contact the Release Engineering team member on buildduty (identified as nick-buildduty on irc) and they’ll get it going for you.

Thanks, catlee!

1. graphs-new.mozilla.org will become graphs.mozilla.org
2. The current graphs.mozilla.org will become graphs-old.mozilla.org
3. No new data will be sent to graphs-old.mozilla.org
4. graphs-old.mozilla.org will remain up so that older data can be viewed/searched

We’ve been seeding the new graph server with data for a month now and I think that the majority of people are already using it as their main means of viewing performance data; for the most part the switch over should be painless.

What may be slightly more controversial is that there is no current plan to migrate any data from the old graph server to the new.  There’s some very good reasons for this:

• Data in the old graph server was generated by using throttled test slaves, we no longer throttle slaves so the numbers would not be comparable
• We are close to rolling out a new Tp test page set, we did not test with this new page set before so the numbers in the old graph server would not be comparable
• Most of the numbers on the old graph server were collected before we rolled out reboot-every-test-slave-post-every-test - the numbers have greater variance and there are large swaths of data that aren’t trustworthy or useful in anyway (basically, long periods of time when the box in question was in serious need of a reboot)

Instead of banging our heads against shifting data from the old, poorly thought out schema to the new, super fast schema we’re going to consider designing a system whereby we can pick up old builds of interest and push them through the current testing harness. That way we save ourselves headaches and get data that is actually comparable to current results.

Once all the dependent bugs filed against graph server migration have been fixed we’ll roll all this out.  I’m hoping that that will be in the next week or two.  Right now I’m more interested if anyone has any strong feelings about what builds are ‘interesting’ enough that we should come up with the means to re-test them with our current test harness.  Any favorites out there?  Top ten?