Earlier this month, the Anti-Malware Testing Standards Organization (AMTSO) published new guidelines on the delicate topic of testing anti-malware products. Since then, many experts in the anti-malware industry have been commenting on the said guidelines. Many of these comments (including some from myself), have been summed up by tech blogger Kevin Townsend here.
After reading the article, you will appreciate that everyone has the right to express different points of view. However, we all came to the same conclusion—measurement and comparison are indispensable and essential in guiding competition. Competition in the anti-malware industry is truly ethical and based on conducting research on and mapping out user needs and requirements. Only with these in hand can we offer customers the most effective solutions to solve their problems.
All industries undergo testing and benchmarking to gauge how much their products have improved over previous versions, and how they compare to the competition. Without testing, improvements or innovation will not be possible, as nobody will see the need to change the way things are done.
Trend Micro particularly approves of the latter. As early as 2004 or 2005, our researchers had recognized that the detection rate on a static collection of files could no longer be considered a viable benchmark. At the same time, improving file scanning would not solve the new generation of malware threats.
That has since proven to be true. Malware threats change and evolve so quickly that everyone is forced to admit that waiting for new patterns is no longer good enough. File scanning became only part of a greater whole, a module of a greater threat protection concept.
Since threat protection is a concept expressed in a program suite, it is only fair to test the whole product, not independent parts of it. For some environments, it may be important to figure out how effectively each module works (e.g., intrusion detection, file scanning, email scanning, anti-phishing, etc.). For end users, however, what matters is that the threat is stopped preferably before it arrives on any of their devices.
Testing bodies (individuals and companies alike) saw that change was needed. In 2007, they started to discuss (at AVAR where Andreas Marx presented) and develop new concepts and methodologies for whole-product testing. Some of the pioneers in this area were NSS Labs and Dennis Technology Labs. Back in 2009, both showed the first results of new approaches that focused on how whole product testing can be conducted.
Whenever you see a new set of anti-malware software testing results, keep the methodology used in mind. See if you can identify any of the top ten testing mistakes frequently made by testers and prepare to question the conclusions in the report.