Internet Explorer 10 May Score Well with RoboHornet, but It’s Even Better for Web Browsing in the Real World

Internet Explorer 10 May Score Well with RoboHornet, but It’s Even Better for Web Browsing in the Real World

  • Comments 18
  • Likes

Yesterday Google released its latest micro-benchmark, RoboHornet, in which Internet Explorer 10 scores rather well.  While we appreciate the gesture, members of our engineering team took a look at the benchmark and found that RoboHornet isn’t all that representative of the performance users might encounter on real-world sites. Like all micro-benchmarks, RoboHornet is a lab test that only focuses on specific aspects of browser performance.

We decided to take the RoboHornet micro-benchmark and run it in the context of a real-world scenario. Using modern browser capabilities like CSS3 Animations, CSS3 Transforms, CSS3 Text Shadows, custom WOFF fonts, Unicode, Touch, and more, we created a site that looks a little bit like the familiar Matrix. We then ran the RoboHornet micro-benchmark in the context of this real website. While running both the Matrix and RoboHornet micro-benchmark at the same time, Chrome slows to a crawl and stops animating the screen, because it wasn’t designed to handle a benchmark load in the context of a real-world scenario. Meanwhile Internet Explorer 10 remains responsive, continues animating the screen, and finishes the test in less than half the time that Chrome does.  You can check out my demo of RoboHornet Pro on a Samsung Series 7 PC running Windows 8 below.

This is a great example of why we have consistently said real-world performance matters when evaluating a browser. We have made RoboHornet Pro available on IE Test Drive, so you can check it out for yourself.

Internet Explorer is built from the ground up to perform incredibly well on web sites, not lab micro-benchmarks. Third-party results like those from Strangeloop Networks last week showing that that Internet Explorer 10 is 8% faster than Chrome 20 at loading web pages from the top 2,000 retail websites reinforce that we’re taking the right approach as we strive to build the fastest web browser available.

For sure, more micro-benchmarks will be released in the future. While we will look to see how we stack up on other browser vendors’ tests, our focus will remain on real-world site performance. After all, Internet Explorer is built for web browsing - in the real world.

Roger Capriotti

Director, Internet Explorer Marketing

18 Comments
You must be logged in to comment. Sign in or Join Now
  • Nathan
    63 Posts

    @trolls, "thank you very much for crafty trolling against Microsoft corp" - Eric Schmidt, "Here is your free Android phone running Chrome. Not to mention, it has capability to steal your privacy, personal data and every key stroke you make.. all in the name of openness. And remember, we only sell your privacy to the highest bidder, we don't undersell you!" (interestingly, most of the open-source/*nix loving guys thinks Google as their god. But its a shame how cunningly they confused the open-source community with their hypocritical openness concept). Here are the comments of some careful people I met:

    "Whenever I login to Gmail account, it stops at an intermediate screen, asking me to provide my phone number and it takes 2 steps to skip that part (a small text 8px skip link followed by a popup with Are you sure Skip button). There is no option like, don't show this message again! I guess someday they come to my door and force me to give my phone number" (verified and its true)

    "When I tried to upload a large video file on youtube, they asked me to increase my video upload limit in 'one simple step', next screen, provide your phone number to Google!". (verified and its true)

    Do you really need Chrome with WebGL support, at the cost of selling your personal data?

    @Fed44, "przemoli" or "przemo_li" is known for trolling against MSFT. There are tons of hardware accelerated graphics benchmarks in which IE9 and IE10 outperforming Chrome (like those on ie.microsoft.com/.../SiteMap under graphics demo). These trolls used to bitch about IE7's box model back in 2004, then it was getInnerHTML for select-control, then SVG vs VML yada yada. When everything got fixed and IE is outperforming their gods, they started bitchin about WebGL, the only thing left. which is yet a debatable issue.

    In 1997, when there was no such "CRAPPY" browser like Chrome, IE was the first to implement DX filters. In 2004 the DX filters equivalent standards (for CSS and HTML) were proposed to W3C. The proposal went through long journey and tons of iterations, discussions and debates. In 2011 Aug, the DX filters equivalent HTML5 standards draft were signed as candidate recommendation at W3C. IE9 was released in 2011 Apr and implemented most of those standards. Today IE10 RTM has removed DX filters and support all standards introduced in the recommendation. In between 2004 and 2011, there are zillions of viral blog posts, bashing comments, whining against IE team by the web developers, because they were not implementing the draft yet to become a standard (although they were giving the equivalent support via DX filters with 1 extra line for each CSS rule). End users didn't had issues. It was those lazy ass bitching devs who do not want to write extra code for with -ms prefix but they wont have such problem in writing -webkit prefixed code.

    Firefox (with Google disabled from the "addressbar") is the browser of my choice on Windows, Fedora and Mac. On Windows, I use IE's InPrivate for secure browsing (bank, paypal etc.). Its good to HTML5 is maturing and all browser vendors are complying with it with minor conflict. I consider just "installing" Chrome on any machine (pc, mac, phone) is letting google start messing with your personal life.

  • @abm

    We both agree on (relatively) recent MS activity in supporting w3c standards. Great job, great job and great job.

    Nor I complain about MS work in performance department. The more the merrier.

    I do complain about WinRT lack of APIs "fairness". JIT engine for JS will be available only in IE. MS do not trust others to use needed APIs in good manner. (And as I have stated earlier such problem is not present on Win8 both in metro and classic).

    And I do complain a bit about lack of support for WebGL. And if MS decide to not support WebGL, than at least MS should commit itself to making GPU safety, so projects like WebGL can be safe for everyone. MS have all that is needed here to end this conundrum. Large influence on hardware manufacturers, drivers implementations, API (webgl on Win use DX in Chome and FF!!), and OS (and web browser!). Nobody but Apple have strictier control over 3D stack. (And even Apple needs are satisfied only after MS is pleased in desktop GPU).

    But getting back on topic.

    MS should provide feedback to RoboHornet via normal channels, not via blog posts. Should contribute code back to RoboHornet. (Btw what is license of RoboHornet Pro?) And if feel like should continue criticizing it.

    @Fed44

    Buffer overflow in GPU driver? Have not see any reports about such regarding WebGL. And they should not be treated any differently than buffer overflows in any other software. Its not deal breaker. Its flow to get rid off.

    On the other hand ALL sales department would be ready to sell their respectable mothers to be able to provide users with interactive 3D representations of their stuff.

    (Real world example. IIRC Nike offered preview of its shoes that way! But its possible for everything, watches? cars? houses? etc.)

    There is whole subset of populace whose primary cognitive tool is spatial sense. Presenting 3D for them is best way to present information them.

    So safe way of using 3D on the web is beneficiary.

    Though I do agree that current safeguards are not enough. There should be more, and MS can (and should) take part in creating such mechanisms

    .

    @Leila

    All WebGL vendors use black lists and white lists, and have as much free hand there as they want.

    SL5 only differ in the set of shader functionality exposed (eg no loops). And have ability to white list apps separately same as WebGL implementation in Safari ....

  • Leila
    12 Posts

    Besides, the main consumption for Silverlight5 is Windows desktop, RT and windows phone space. Silverlight on Web is not even supported in IE10-metro browser in Windows 8! So Microsoft is NOT promoting the Silverlight5 for web. However, they do support Flash natively on IE10-metro...

  • Leila
    12 Posts

    @amtiskaw, the way WebGL works is the issue. Microsoft has blacklisted the problematic Radeon and Nvidia drivers (especially the latest ones) to mitigate the issue in Silverlight5. The difference in SL and WGL is that they have control over whitelisting and blacklisting of trusted graphics-card driver in SL.

  • @Fed44

    Again, if that's the case, and 3D can't be secured on the web, why did Microsoft add the capability to Silverlight 5?

  • Fed44
    17 Posts

    @przemoli

    If you are able to get a single buffer overflow in some GPU driver you have directly bypassed 3 sandboxes and gone straight to system level permissions...

    Apart from that Websites are not to be trusted and should not be able to degrade your experience in any way. Also while I am sure there are some cool applications, such a feature would be more the exception than the rule.

  • abm
    268 Posts

    @przemoli, IE desktop and metro on Windows 8 does have very slight performance differences. IE desktop in all cases outperforms IE metro by some milliseconds in stress testing.

    The GPGPU and vectorized assembly code is not confined to RT. If you compile the C++ code on VS2012 on Windows 7, you will get the same results.

    The results of ECMA test Nathan mention (and you quoted) do reflect the support and completeness of features... Last time I checked, in http://test262.ecmascript.org/ test, IE10 fails 3 out of some 10,000 test cases while other browsers were failing 20+, 50+, 100+ tests.. I guess Microsoft has done a great job in implementing the HTML5, CSS3 and ES5.1. The most important thing is the completeness of the features. Take an example of Placeholder attribute of HTML5 input fields and the corresponding CSS properties:

    Tested on IE10 RP, -ms-input-placeholder{} psuedo selector supports all the 13 properties listed on this test page newilk.com/.../Placeholder_styling (I mostly upload the benchmarks on it where other outperforms IE ...  but things are changing dramatically with IE10). Chrome doesn't support padding in placeholder while FF is short of line-height. Newer version of Safari supports 5 out of 13 properties (while 10 properties were supported in its previous version). As of Opera 11, the placeholder styling is dropped.

    As a developer, its good to see IE supporting latest standards so eagerly.

  • @Fed44

    Thx for link.

    As for HW acceleration in Win7 and IE9.

    Firefox use exactly same APIs, so I (and Mizilla) can not see how MS can perform better. Unless there are some undocumented aspects of those APIs known only to MS. Unlikely scenario.

    As for the 3D GPU security. Yes its very low. But it do not affect other areas of computer in any form but DoS.

    MS for too long did nothing to remedy this problem :| Now they are getting better and better (with better architecture for gpu drivers!), but there is still lots of ground to cover. ("virtualized" GPU for example!)

    So using WebGL may mean need for computer restart or GPU reset.

    Quality of experience rather than security per se. But another area where MS should lead development.

    (Silverlight 5 have capabilities similar to WebGL, unfortunately MS do not invest more in SL, so there is no public word on developments in that regard.)

    All in all. IE9 an 10 do not provide accelerated 3D that can be used by web developers. In other areas MS do fine, just they could cooperate more...

  • Fed44
    17 Posts

    @przemoli

    While IE 9 and IE 10 do not offer WebGL Apis for complicated 3D renderings, they still use Hardware Acceleration under the hood to perform the actual rendering in a way that most browsers at least for a while could not replicate (not sure what the status is today, all other browser change every second day...). If I recall correctly Microsoft was not happy with the WebGL specs from a security perspective and while I haven't look at the spec, I know from past experience that messing with DirectX can affect the rest of your system...

    Also the reason they probably aren't interested in supporting RobotHornet per se is because they have their own testing lab which has been described in detail at: blogs.msdn.com/.../internet-explorer-performance-lab-reliably-measuring-browser-performance.aspx

    "For sure, more micro-benchmarks will be released in the future. While we will look to see how we stack up on other browser vendors’ tests, our focus will remain on real-world site performance."

  • @Roger Capriotti

    Can you add info, why MS decided that cooperation  with RobotHornet benchmark is not possible. Why this code could not be submitted as pull request?

    I do not forbid MS to clean, limit and freeze part of RoboHornet benchmark for use in IE Test Drive. But idea behind RoboHornet is about cooperation around various needs of various parties.

    I can not omit comparison with mozilla folks, who engaged RoboHornet maintainers and provided valuable insights, which in turn may help overcome RoboHornet pitfalls.

    Will MS submitt their test cases? Offer expertise and/or reviews for tests?

    (RoboHornet project aim at providing JS engine vendors with ability to comment early on in development process on tests for next release!)

  • @Nathan

    Your response would be more persuasive if you haven't used words such as "every CRAPPY browser", "G-crap". Also you would score your points better using proper naming. ECMA currently do not provide benchmark for JS. They provide test suite for JS ES5. Benchmark would check performance, while test suite is there to ensure that every JS engine support one and the same JS syntax.

    Also you may want to know that IE9 and IE10 do not accelerate 3D graphics on your GPU in a way that is available for code that is using DX. While Chrome, FF, Opera, Safari (and others) tap into moder GPU resources allowing such acceleration. Hardware is there, but not available.

    Pls add link to official MS statement about availability of IE10 on Win7.

    @abm

    Any comparision on WinRT is pointless. MS stated that some APIs available to IE will not be available to other apps. It will hurt most JS engines which relay on JIT compilers the most, but all areas of UX in web browsers will suffer.

    MS have lost several cases of antitrust investigations in the past on the basis of limiting APIs just for their own apps. So maybe this situation is not permanent. But for now MS IE is the king in the town for WinRT. Just because MS castrated other apps potential.

    Win8 do not have such limitations. So comparisions on Win8 are fair game.

  • abm
    268 Posts

    @bobnaier, its easy. You can actually run these benchmarks on your computer (normally by clicking a link). You have the URL to benchmark from this blog post. Download it, review the code and then test it on IE10 and Chrome21 on Windows 8 RTM. Also, test the other benchmarks that Nathan mentioned. There is a significant difference in performance. The basic reason for improved graphics rendering is the hardware accelerated graphics of IE9 and IE10 and Windows RT as a whole is using hardware accelerated graphics (WinRT on 60fps! That I read somewhere right after MSFT acquire Perceptive Pixel). Above all, if you are a C++ developer, you must have known that Visual Studio 2012 now complies the vectorized assembly code giving some major goose bumps to the performance.. stackoverflow.com/.../movl-multi-data-instruction-and-assembly-optimization-comparison... It has also incorporated GPGPU -- general purpose programing on graphical processing unit (in addition to consuming CPU). and this time Windows, IE teams and 100k employees at MSFT are very aggressive to bring about "significant difference in performance"..

    for that matter I can safely say that, its more than a mere spin ;-)

  • It's not about hating, it's about not really caring much. I'm reading this because of a post elsewhere, and as I suspected, it's just another typical microsoft spin.

  • Nathan
    63 Posts

    To all the IE haters. IE10 is outperforming every crappy browser out there in ECMA (ECMA is like W3C for JavaScript) benchmarks and Apple's SunSpider benchmark. So yeah, besides Google's habit of harvesting your personal data, your activity via Chrome and all the G-crap, profiling you and selling you to the highest bidders; GChrome sucks!

    @Tommyinoz, would you like to compare your benchmarks with mine (consists of 2Dim and 3Dim graphics and text animation). Bring your Garbage-Collector and I will bring Internet Explorer 10 RTM (and soon to released on Windows 7) and lets compare the real stats. Are you still using 11 years old Windows XP (which is definitely irrelevant)? Shame on you!

    To all the IE haters again... if you really need an alternate browser, use Firefox and disable the google-search from the address-bar because it sucks everything you do in it. After all its Google's primary way of income..selling you for dime while others are selling real products they are giving everything for free and selling YOU out!

  • boards
    1 Posts

    IE is really not relevant anymore. It's not vailable anywhere I work. It's not on my iPad, iPhone, Android, Linux or OS X, so who cares? Releasing your irrelevant tests is not going to make IE cross-platform, which is really to only issue these days for most of us.

    Even if you work on Windows at work still, it's very likely you use OS X at home (and an iPad, and an iPhone etc...) and you would like to have all the wonderfull syncing going on to track what you're doing. IE is the missing link. Not just bad, under performing and less standard then the others, it's nowhere other then on Windows, which less and less people use.

  • Tommyinoz
    46 Posts

    Interesting to note that this test works flawlessly on iPhone and iPad (both on Safari and Chrome), but doesn't work at all on IE9.  IE9  displays a message telling me to use IE10 or greater.  So I took Microsoft's advice and the only  browser that I had greater than IE10 installed on my machine was Firefox, so I used that instead. Firefox ran the test successfully but it had jerky animation. But I mainly use iPad at home for web browsing and I happy that this test worked well on iPad.

  • I don't know what universe you and your Microsoft colleagues are living in if you think that Matrix demo represents a "real-world scenario", but it certainly isn't the one the rest of us inhabit. Given a choice between a browser that did well on the things RoboHornet tests, and one runs this Matrix nonsense well, I know what I'd choose.

    Of course, this is all transparent posturing. Apparently Microsoft were invited to participate in developing the RoboHornet benchmark, but declined, and are now trying to discredit it. I guess you were too busy writing FUD blog posts and living out teenage demo-scene fantasies from a decade ago. How about joining the rest us in the present and helping to improve the web, without this kind of rubbish.

  • Tommyinoz
    46 Posts

    IE failed all of my WebGL tests.  Given that there are so many games and educational stuff out there that use WebGL, I think this is rather disappointing.  

    I use Chrome on my iPad, iPhone, Linux PC, iMac and all of my Windows machines.  IE is the odd one out, I can't even upgrade to IE 9 on Windows XP!  So is this the future of IE and Windows? The upgrade cycle trap where you have to upgrade the OS to get a more updated web browser?  Bugger that, just install Chrome or Firefox, problem solved.

    As everyone probably already knows, there was a big new zero-day vulnerability affecting Internet Explorer last week.  The Windows update to fix this vulnerability was installed on all our PC's in the office last Monday.  Of course, for the changes to take effect, every single PC and server had to be rebooted.  This is the only web browser I know that requires the whole OS to be restarted when it is updated.    

    Sorry, but IE is too much of a hassle.