During a long downtime last Friday morning I unthrottled all the WinXP, Vista and Ubuntu boxes testing Firefox3.0. I had forcast a 25-30 minute time savings in each full Talos test cycle but, alas, that was not to be. After crunching the numbers, it looks like we save 10 minutes per full test cycle for each operating system. Now, that was initially pretty disappointing. After further consideration I think that 10 minutes would be a very welcome time savings to anyone babysitting the tree waiting on performance results; in fact, any decrease in Talos machine cycle time is pretty much a good thing.
Now that the machines have had some time to cycle and get us a base of results I went over the appropriate Firefox3.0 graph links to see how the numbers look. From examinations of Tp and Ts:
- Ubuntu results slightly increased in variance post un-throttling
- WinXP results slightly decreased in variance post un-throttling
- Vista variance was unchanged
With the time savings, and knowing that the Linux numbers are slightly degraded, I believe that unthrottling should be rolled out across all Talos boxes testing all branches. While it isn’t going to solve all our problems it’s a step in the direction of a better testing harness. Feel free to comment here or in the bug. I’d like to get as much feed back as I can, as this will change the reported results of a large amount of our test boxes.