Comments on: Universal Manifest Format For Unit Tests (part 2)

By: Robert Kaiser

Robert Kaiser — Wed, 19 May 2010 16:18:37 +0000

What would be really good is to have some possibility to say “if the pref foo is (not) true/2/’bar’” in the skip-if or random-if values.
Reftests can currently do that, and we have somewhat strangely done things in other tests for that, like bailing out early from a test when a pref is (not) matched. It would be nice to have a generic way to set this.

By: David Baron

David Baron — Sun, 16 May 2010 22:17:34 +0000

I think the skip, fails, etc. conditions should remain JS. That keeps it clear what’s an && vs. an ||.

We can use significantly shorter variable names (the current MOZ_WIDGET_TOOLKIT is for backwards compatibility). However, I’m wary of making them too short. Currently things are based on widget because that’s the most likely cause of platform-specific failures in reftests, which mostly test layout and graphics. But if we start using this for other tests, that will likely change. And once we have a qt build (for mobile) there’s a real difference between “this fails with gtk widgets” and “this fails on Linux”.

I’d prefer something like:
fails: true
or
fails: widget == “cocoa”

By: alice

alice — Sat, 15 May 2010 20:22:55 +0000

@Boris - the indentation is not required by json, I’ve included it in the examples to make things more readable.

By: Rob Sayre

Rob Sayre — Sat, 15 May 2010 14:42:29 +0000

I like this plan. YAML is nicer for humans to write than JSON, has plenty of parser support, and it has comments. I would go with that instead.

By: Robert O'Callahan

Robert O'Callahan — Sat, 15 May 2010 11:41:56 +0000

How about replacing the token ‘files’ with the type, use an extended JSON that allows the value to be an array and allows comments, and don’t indent array elements, so one could write

[
{"==": ["file1.html", "file1-ref.html"]},
{”==”: ["file2.html", "file2-ref.html"]},
// {”==”: ["file3.html", "file3-ref.html"]},
{”==”: ["file4.html", "file4-ref.html"]},
]

This would keep common cases on one line without indentation, which I think would be a big help.

By: Boris

Boris — Sat, 15 May 2010 02:23:21 +0000

So for the common case this is a lot more typing than the current reftest format, actually. Worse yet, there’s indentation involved, which can very easily get pretty wacky unless one is careful (which increases the time it takes to add entries even more). Do we have any plan for handling that?

By: Joel Maher

Joel Maher — Sat, 15 May 2010 01:35:03 +0000

This is really cool. I think the && and || stuff is in logic like this:
“skip-if”: ["maemo", "android"],

But this is ambiguous as I could assume it is an || condition, yet how would we solve an && condition. I found this exact problem while toying with filtering for fennec mochitests (https://elvis314.wordpress.com/2010/04/27/filtering-mochitests-for-remote-testing/) I ended up coding my implementation to be && which is not always the most useful.

One thing that I would like to know is how would mochitest format look like? Currently this is a os.walk of test_* files in a directory tree. So with no manifest other than entries in a Makefile and using a format above, would we have:
{ “type” : “mochitest”,
“files” : ["mochifile.html"],
“skip-if”: ["isdebug"],
}

By: Jesse Ruderman

Jesse Ruderman — Sat, 15 May 2010 01:02:45 +0000

I like your idea of replacing “MOZ_WIDGET_TOOLKIT==’cocoa’” with “cocoa”. It would save typing, prevent errors, and make it easier to read manifest entries for tests that have complex conditions. But that’s orthogonal to the syntax of the manifest format.

I like the idea of allowing groups of tests to share a common pref or skip-condition. We could add that to the reftest manifest format using indentation, like in Python or YAML, without changing the syntax for the normal case.

I also like the idea of using a common –filter or –manifest option across all of our test suites. But again, that has nothing to do with the syntax.

How would your manifest format would handle conditions that require && or ||, like the test for bug 399636 requires? Are you planning to allow || and && and ! in “fails-if” and “random-if” conditions, but not arbitrary JavaScript?