happiest unalice ever

I’ve come up with a proposed universal manifest format for unit tests based upon the many good comments and suggestions that I received after my last two blog posts. The simplest case is still very simple and the more complicated cases are now easier to read and understand. I’ve put together a list of examples from the current manifests and then converted them to the new manifest. We are still not totally finalized here, but I think that this is a good working format that can be refined.

I’m still working from the stance that script doesn’t belong in manifest files so the format handles && and || cases without code. With the help of my auto-tools chums we searched the code base for difficult examples to ensure that this was still flexible enough to handle the expressions that we currently support.

Also, having worked on the conversions from old to new I have to say that it was really easy to get this going - admittedly, I’d done some reading on JSON and was immersed in the problem set - but I think that it is a good indication that this could make reading and writing manifests a lot nicer.

I’m once again happy to get any comments on the current proposal. If things look good it will be time to get everyone together for a Brown Bag to hash this out and finalize. I’ll get that in the works once I see if I have to take this back to the drawing board.

1. simple case

old

 == 399209-1.html 399209-1-ref.html
 == 399209-2.html 399209-2-ref.html
 == 399384-1.html 399384-1-ref.html

proposed

{   "tests" :
    [
        {  "==" : ['399209-1.html', '399209-1-ref.html']  },
        {  "==" : ['399209-2.html', '399209-2-ref.html']  },
        {  "==" : ['399384-1.html', '399384-1-ref.html']  }
    ]
}

2. ref test with two fails-if clauses and one random-if clause

old

fails-if(MOZ_WIDGET_TOOLKIT=="windows") fails-if(MOZ_WIDGET_TOOLKIT=="cocoa") random-if(MOZ_WIDGET_TOOLKIT!="cocoa"&&MOZ_WIDGET_TOOLKIT!="windows") != 399636-quirks-html.html 399636-quirks-ref.html # windows failure bug 429017, mac failure bug 429019

proposed

{   "tests" :
    [
        {  "!=" : ['399636-quirks-html.html', '399636-quirks-ref.html'],
           "fails" : { 'if' : ['windows']},
           "fails" : { 'if' : ['cocoa']},
           "random : { 'if' : ['!cocoa', '!windows']}  }
    ]
}

3. ref test with two random-if clauses

old

random-if(MOZ_WIDGET_TOOLKIT=="gtk2") random-if(MOZ_WIDGET_TOOLKIT=="cocoa") == mirroring-02.html mirroring-02-ref.html

proposed

{   "tests" :
    [
        {   "==" : ['mirroring-02.html', 'mirroring-02-ref.html'],
            "random" : { 'if' : ['gtk2']},
            "random" : { 'if' : ['cocoa']} }
    ]
}

4. ref test with one fails-if clause and one skip-if clause

old

fails-if(!haveTestPlugin) skip-if(!prefs.getBoolPref("dom.ipc.plugins.enabled")) == pluginproblemui-direction-1.html pluginproblemui-direction-ref.html

proposed

{   "tests" :
    [
        {   "==" : ['pluginproblemui-direction-1.html', 'pluginproblemui-direction-ref.html'],
            "fails" : { 'if' : ['!haveTestPlugin']},
            "skip" : { 'prefs' : ["dom.ipc.plugins.enabled" : "False"] }  }
    ]
}

5. js test with one fails-if clause and one skip-if clause

old

fails-if(!xulRuntime.shell&&!isDebugBuild) skip-if(!xulRuntime.shell&&isDebugBuild) script regress-455464-04.js # bug xxx - hangs reftests in debug, ### bug xxx - NS_ERROR_DOM_NOT_SUPPORTED_ERR in opt

proposed

{   "tests" :
    [
        {  "script"  : "regress-455464-04.js",
           "fails" : { 'if' : ['!shell', '!isDebugBuild']},
           "skip" : { 'if' : ['!shell', 'isDebugBuild']}  }
    ]
}

6. js test with one random-if clause and one asserts-if(count) clause

old

random-if(!xulRuntime.shell&&xulRuntime.OS=="WINNT") asserts-if(!xulRuntime.shell&&xulRuntime.OS=="WINNT",1) script regress-344804.js # bug 524732

proposed

{   "tests" :
    [
        {  "script" :  'regress-344804.js',
           "random" : { 'if' : ['!shell', 'WINNT']},
           "asserts"  : { 'if' :  ['!shell, 'WINNT'], "count" : 1} }
    ]
}

7. js test with one random-if clause, one fails-if clause and one skip-if clause

old

fails-if(xulRuntime.OS=="WINNT") random-if(xulRuntime.OS=="Linux"&&!XPCOMABI.match(/x86_64/)) skip-if(xulRuntime.OS=="Linux"&&XPCOMABI.match(/x86_64/)) script regress-3649-n.js # No test results on windows, sometimes no test results on 32 bit linux, hangs os/consumes ram/swap on 64bit linux.

proposed

{   "tests" :
    [
        {  "script" :  'regress-3649-n.js'
           "fails" : { 'if' : ['WINNT']},
           "random" : { 'if' : ['Linux', '!x86_64']},
           "skip" : { 'if' : ['Linux', 'x86_64']}  }
    ]
}

8. ref test with one asserts(min,max) clause

old

asserts(0-1) == background-draw-nothing-malformed-images.html background-draw-nothing-ref.html

proposed

{   "tests" :
    [
        {  "==" : ['background-draw-nothing-malformed-images.html', 'background-draw-nothing-ref.html'],
           "asserts" : {'min' : 0, 'max' : 1}
    ]
}

9. ref test with one asserts-if(count) clause

old

asserts-if(MOZ_WIDGET_TOOLKIT=="gtk2",1) == 355548-3.xml 355548-3-ref.xml # bug 456899

proposed

{   "tests" :
    [
        {  "==" : [' 355548-3.xml', '355548-3-ref.xml],
           "asserts" : { 'if' : ['gtk2'], 'count' : 1}  }
    ]
}

10. load test with one asserts-if(count) and one asserts(count) clause

old

asserts(10) asserts-if(MOZ_WIDGET_TOOLKIT=="windows",8) load 265986-1.html

proposed

{   "tests" :
    [
        {  "load" : '265986-1.html',
           "asserts" : { 'if' : ['windows'], 'count' : 8},
           "asserts" : { 'count' : 10}  }
    ]
}