Today, I was scanning through various industry blogs when I stumbled upon an entry from Kaspersky Labs. What was interesting was that under the veil of improving testing quality, the blog openly admitted that the organization in question had been trying to play tricks on competing organizations just to position itself more favorably among the media.
The organization explained that it deliberately created clean files and added fake detections in order to “show” that other vendors copied it. This was a risky decision. Across the industry, research organizations share a level of trust and participate in sample-sharing programs in order to protect customers, which for Trend Micro, is what always comes first. (I should just add here, that Trend Micro was not one of those companies affected by this, as we always QA our own detections and never rely on those of another vendor).
Aside from the organization’s cheap prank, we were very pleased that the other resounding message that came from the blog post was that it finally understood and supported the message Trend Micro has been promoting for a long time now—the need for change in testing methodologies to include real-world testing such as those delivered by NSS Labs.
The need to change testing methodologies was also a primary reason for the foundation of the Anti-Malware Testing Standards Organization (AMTSO), which aims to come up with more realistic and useful benchmarks.
This story really shows just how influential the media is on the antivirus industry in that even a respected vendor should manipulate detection rates just so it can positively position itself with the press rather than focus on its customers.
But another more positive learning is also that the path that AMTSO is taking is the right way. Pure detection rates based on numbers or one-to-one comparisons are yesterday’s methods when verifying the value and performance of a security solution.
Customers need holistic reviews, giving them a real-world scenario-based feedback about what different solutions can offer them instead of pure “I detect more then you” headlines. I am glad to see that testing organizations like NSS Labs, AV-Comparatives, and AV-Test meanwhile understood and started to pick up these principles.
If you're new here, you may want to subscribe to our RSS feed. Thanks for visiting!
February 2nd, 2010 at 4:17 pm
I hope you're not referring to the IOBit/Malwarebytes fiasco of last year. In my opinion, MalwareBytes was entirely justified in setting their trap. Such intentional errors are not confined to the Security Software industry but have been employed for centuries by cartographers, lexicographers, and encyclopedists.
February 3rd, 2010 at 4:14 am
Could you please supply some links to the mentioned blogs?
February 4th, 2010 at 8:11 am
Hi, Andrew, thanks for the comment. As for the answer–no, I am explicitly referring to this blog post by Kaspersky. My overall point is that the race for "as many detections as possible" does not help users at all and actually damages the reputation of AV vendors.
Personally, I find it unethical to create FA’s intentionally. Consider the following scenario: one customer of a follower of the said AV vendor's FA detection might face a production outage due to this FA.
Aside from the legal implications that are already apparent here, this damages not only the customer but also the company that might have followed the said detection.
And … even if they committed to only have released 20 of these “bombs,” who can guarantee that it was not 20,000 or that they are not doing this again next week? No one knows.
So my point here is that this unethical behavior definitively damages the trust in AV companies in general, in tools like Virus Total and all kinds of detection statistics.