Today we’d like to announce SunSpider 1.0, an update of the suite that fixes a number of bugs and aims to further increase test accuracy and repeatability. This update comprises fixes to test validation, better interaction with power management, and a backlog of minor fixes to the harness and tests.
We address both problems by adding validation checks to 23 out of the 26 SunSpider tests. The validation checks are intended to incur minimal overhead to the running-time of the tests: <2% overhead was our goal. The tests not covered by validation are ones where results depend on timezone or random number generation; on those tests we expect different users and implementations to get different results. We also do not perform validation on the exact results of certain math functions like Math.sin(), since ECMAScript 5 permits these functions to return implementation-dependent results.
The changes to include test validation and prevent dead code elimination are covered by WebKit bugs 38446, 63863, 114852, and 114895.
Short-running tests of the sort used in SunSpider require special care from the test harness. The SunSpider 1.0 release further improves the repeatability of test execution times by eliminating the delay that previous versions used between test executions, thus reducing the chances of interference from the operating system’s power management logic.
The original SunSpider 0.9 test harness employed a 500ms delay between tests. The intent was to give the browser a chance to complete any asynchronous activities before doing another round of measurement. But performance improved, and the power management facilities in modern operating systems became more sophisticated. Running a test that took on average 10ms with a 500ms delay between test executions meant that the machine could enter a lower clock-rate, power-saving state for each test execution. If power management did kick in, the machine would run slower, and the SunSpider score would be penalized. Whether power management kicked in or not depended on a variety of factors outside the harness’s control. Paradoxically, this would lead to slower or noisier machines exhibiting higher performance. A slower machine would be active for a higher fraction of the test time slice thereby reducing the likelihood of interference from power management. A noisier machine—for example a machine also busy doing other work while SunSpider testing was in progress—would also have a chance of getting a better result because the noisiness would also inhibit power management.
SunSpider 1.0 eliminates the delay between test executions. In addition to reducing interference from power management facilities, it also has two other useful side-effects:
- The SunSpider test suite now runs up to twice as fast. Typical SunSpider tests require less than 10ms to run on fast machines, so the 10ms delay between test executions meant that most of the running time of SunSpider was spent idling. Removing this delay means that the tests finish faster, by spending less time waiting in between individual tests.
The change to improve test repeatability when power management is in effect is covered by bug 114458.
Minor Tweaks in the Harness
SunSpider 1.0 also includes a handful of additional fixes and features in the command-line and in-browser test harnesses:
- bug 47045: The harness doesn’t actually close() its test documents.
- bug 60937: Avoid using an undeclared variable named ‘name’, which is a DOM API.
- bug 71799: Extend SunSpider driver to be able to run Kraken.
- bug 80783: Add –instruments option to SunSpider to profile with Instruments.
Give it a try!
We invite you to try out our new SunSpider 1.0 benchmark suite, report your findings, and provide feedback if you encounter unexpected issues. It is still possible, though not recommended, to run previous versions of the harness from the all versions page.