NSS Labs report touting the benefits of IE8?
by Steve Ragan - Mar 23 2009, 16:15
NSS Labs report used to tout the benefits of Internet Explorer 8. Image: Microsoft.
On Thursday, as Microsoft pushed out Internet Explorer 8 to the masses, a report by NSS Labs was making the rounds in the news. The report says that Internet Explorer 8 blocks two to four times as many malicious Web sites as any of the other browsers on the market today. Yet, the news online surrounding this report is misleading to say the least, and the report itself appears biased and questionable at best.
According to an IE8 press releases, Microsoft said: “...91 percent of adults in the U.S. are concerned about online fraud and identity theft in today’s economic climate, and 37 percent are less likely to shop online because they would have to give their personal information. Internet Explorer 8 offers the best security protections among leading browsers: a study released today by NSS Labs indicates that Internet Explorer 8 blocks two to four times as many malicious sites as other browsers on the market today.”
The NSS Labs report shows that IE8 is effective on its own 69 percent of the time when dealing with socially-engineered Malware. Firefox v3.07 earned a 30 percent effectiveness rate, while Safari (v.3) earned 24 percent, Google’s Chrome (v. 1.0.154) earned 16 percent, Opera (v. 9.64) earned 5 percent, and IE7 earned 4 percent.
However, the results are far from conclusive. There were only 492 sites tested in the first place, a rather small testing sample. In addition, the test relied on standard protections offered by the browser, depending on the browser used; no two will have the exact same set of default protections.
In addition, the report itself is suspect as it is funded by Microsoft. Funding aside, the timing of the report and the breakdown of testing is something that one needs to take a long hard look at.
NSS Labs started with a set of more than 1,779 URLs, which it ultimately narrowed down to a testing set of just 492 sites. These sites were confirmed, by manual testing, to be malicious. Yet, not a single tested URL is presented for comparison. This lack of peer-review opportunity leads some to argue the NSS test was completely biased.
The NSS test used two points to grade the tested browsers. A browser would pass if it blocked the Malware from being downloaded and issued a warning. Failure means that there was no warning and the Malware downloaded and installed. The sites tested, all 492 of them, were related to “socially engineered Malware,” meaning the site that hosted the Malware often finds its victims from Phishing or other Social Engineering attacks.
Yet, at no time does the NSS report take into account layered security, it does not mention the add-ons or options available for Opera, Firefox, or Safari, each of which scored poorly on the test. For example, Firefox alone offers various add-ons for layered security, Opera and Safari both have extensive options for layered defense that are configured by the user for protection. None of these extras were tested.
With regard to the size of the sample set, there is no clear answer as to why it was so small. Millions of Phishing sites host or link to “socially engineered Malware”. Relying on a test set as small as the one used by NSS offers a slanted test at worse, or at best inconclusive results. There is just no way to fairly test each of the browsers without including all of their features or, in the case of Firefox, the most popular and recommended add-ons, and letting the test access several thousand sites -- if not more.
Moreover, Safari’s score should not be counted; this is because a new version of Safari was released before the NSS Labs test. Safari 4 was released to the public on February 24, 2009. NSS Labs says that it conducted its testing from February 26 to March 10, 2009. Why then, would it intentionally pick the older browser when testing features against those of Internet Explorer 8?
Layered security, which was neither discussed nor tested in the NSS study, is the ultimate lesson to be learned. Each tested browser has a different set of default security features. Thanks to various control options and add-ons, each one of them will secure the end user differently. The only way to protect the victim from “socially engineered Malware” is to ensure they are covered with layers.
In most of the cases where a browser failed the test by NSS, anti-Virus software or other added features to a browser could have prevented exploitation. However, the lack of published URLs used in testing makes this point impossible to prove.
However, in the past, layered security has proven time and time again to be the best defense; things like NoScript for Firefox, the use of the block lists by Opera, anti-Virus integration in Safari, or sandboxing by Chrome, offer various layers of protection for the end user. None of these features are listed in the NSS report as being tested. If they were tested, why is there no breakdown on the results?
Perhaps it is unwise to use a test you paid for as proof of why your browser is better than those of your rivals. In an interview last week, Microsoft said it hopes we (the users) will look past the sponsorship issues. Yet, how can we? It is impossible not to question them, considering the testing methods and the hands-down victory that IE8 obtained as a result.
The test in itself would have been better if there had been more time spent testing malicious URLs, and the layered abilities of the browsers were taken into account with the results documented. Something else that would have added value is the ability to re-create the test outside of the environment used by NSS.
The published results, paid for and used by Microsoft, can be read here.
Comments