As any browser vendor will quickly point out, accurately measuring the performance of a browser is extremely complex. On the surface, testing performance seems quite easy… visit a few sites with one browser and then again with a different browser, and simply time how long it took to load the page.
In reality, it’s much more complex than that.
Many things need to be taken into account when comparing the page load performance of different browsers. For example, due to the constantly changing nature of the Internet it is not easy to tell if the exact same content was delivered to each browser for each test. ISP’s, routers, and cable modems often cache their content, meaning that the page being loaded isn’t always coming all the way from the web server. The amount of network traffic can easily change between tests. All of these things (and more) can dramatically effect page load times.
Unfortunately, tools to accurately benchmark browser page load times don’t exist. All of the existing browser benchmarking tools available today are either narrow in their scope (SunSpider, Celtic Kane), inaccurate (iBench), or don’t consider important factors such as network latency, network congestion, and caching.
In the absence of effective benchmarking tools the Internet Explorer Team created a real world test which took the above mentioned factors into consideration and created a level playing field for all browsers tested. The results when comparing Internet Explorer 8 page load times to Firefox and Chrome were captured on video:
The performance video visually compares page load times between IE8, FF, and Chrome. Of the top 25 most popular sites in the world, IE8 wins 48% of the time, Chrome wins 38% 36% of the time, and FF only wins 16% of the time. Of course, they didn’t cherry pick the sites they tested. They chose the top 25 sites as reported by ComScore in December 2008.
We encourage (and expect) people to run their own tests to measure page load times and for the browser industry at large to create a test that can accurately measure page load times – as seen by the user – across browsers. To assist people in running their own tests the Internet Explorer Team has also created a whitepaper which describes techniques that can be used to contend with some of the complex issues mentioned above.
For more information on Internet Explorer 8, visit www.microsoft.com/ie8.
UPDATED 6:16pm Pacific Time: Corrected percentage numbers to accurately reflect data from whitepaper.
