The addons performance testing system has been up and live for a few weeks now. With so many more eyes then mine on the system I’ve seen a bunch of bug filings - which is awesome. With each bug fixed the Talos system works better for both addons and for the general Firefox performance testing.
Here’s what’s already fixed and rolled out:
- Bug 639898 - Tracking bug for talos performance testing on firebug (non-clean state)
- Bug 647954 - When extracting add-ons files should be written in binary mode
- Bug 648113 - documentation for how addons are perf tested/how to home test (See the generated documentation here, along with updates to the MDC documentation here.)
Here’s what’s fixed but waiting on deployment (which will probably happen early next week):
- Bug 648229 - Compatibility info of the add-ons isn’t properly considered during performance testing
- Bug 648732 - Testing add-ons unpacked creates unrealistic results
- Bug 648749 - Talos should capture error console output in the logs
- Bug 648978 - install.rdf parsing in Talos is very rudimentary and fails for some add-ons
- Bug 650709 - Suspiciously high performance impact reported for Personas Plus and SimilarWeb on Windows 7
- Bug 650947 - Make sure that NoScript doesn’t break startup test page
There are still more bug fixes in the works. Next on my list is Bug 648225 - Performance of platform-dependent add-ons is not tested, which will improve the testing system’s simulation of the real world.
In terms of future plans, we are going forward with a second quarter goal of completing an on-demand addon testing service. Basically, this would allow an addon author to request that their addon be tested at any time, instead of waiting for our weekly tests of the 100 most downloaded addons. This will gain us greater coverage of more addons along with a means to double or triple check results. Did your addon perform poorly? Retest! Are you suspicious of the results? Retest! Did the addon fail to download or install? Retest! If you want to follow along the bugs that will lead to this system are:
- Bug 643506 -Move 7 automation and tools machines to new Addons Testing Secure buildbot pool
- Bug 617762 - set up new vm to act as talos addon tester master
- Bug 648114 - means of pushing sendchanges from amo to addons perf test pool
Once the on-demand system is in place, we’ll be working to introduce a greater variety of tests. The ts (test startup) was an easy test to begin with, but it can be of limited meaning for a lot of addons. I’d like us to cover far more of the available Talos tests, concentrating on the tp (test pageload) tests. Tp is interesting because it uses a set of collected, local web pages (100 culled from the Alexa top 300 list of worldwide top used sites) that are then cycled through ten times. With a given addon installed and active (for some meaning of ‘active’ which will be different for different addons) this will give a greater idea of how real world page load time is impacted. As a side benefit of Tp, Talos will monitor the memory footprint and CPU usage during this test. By comparing an addon run to a no-addon run we’ll be able to observe memory and CPU usage differences.
I believe that we are also going to need to put effort into provided some testing hooks/prefs for Talos to use. As in Bug 459965 - Add standardized support for first-run pages to install.rdf. Talos doesn’t react well to first run pages and it would be great to have the means to disable them with a single pref, instead of a customized pref per-addon - especially since the standard use case is that users do not see the first-run page on a regular basis, as they only see it post-installation and never again. I believe that there are probably some other settings like this that would standardize creating a Talos testing environment, and thus make addon-testing more applicable to the type of bulk testing that Talos does.
Testing addons has been a whole new area of Talos testing, and it has its own unique set of challenges. I’ve spent most of my time at Mozilla concentrating on automated browser performance testing and the addons world is still quite new to me. Each addon effects the browser in its own way; while I’ve grown accustomed to standardizing tests across browsers versions, platforms and machines this is definitely a new horizon. The Talos tests are just one way to look at performance impact, but not the final word. It may never be appropriate for a given type of addons, but I most definitely want to work with addon developers to try and get the best coverage that we can.