We’ve recently had the honor of meeting up with our partners and customers in the industry and of sharing insights during our annually held convention Direction 2010, held in Tokyo, Japan last July 7. For a panel discussion, I invited representatives from AV-Comparatives, AV-Test, and NSS Labs to Japan to discuss the need for new metrics and methodologies in testing security products against real-world threats. We all agreed that new tests are needed, tests that better reflect attacks on real customer environments.
Figure 1. There is a need to reform security product testing methodologies in response to the nature of threats today.
Live-attack scenarios set against the intended implementation of security solutions provide the best representation of how well a product performs. Testers should also be able to make evident that the harmful effects of malware can be blocked at several points prior to execution. For instance, different solutions are able to:
- Block malware from arriving at the endpoint (e.g., Web filtering; Web reputation services)
- Stop malware files from executing on the endpoint (e.g., signature-based scanning of files)
- Interrupt malware doing bad things upon execution (e.g., behavior monitoring)
- Protect vulnerabilities from being exploited (e.g., disable access to known vulnerabilities until patched)
For the longest time, traditional testing has only been measuring detection (#2). This simplistic approach penalizes innovative solution makers and discounts the value of blocking a threat early on or in mid-execution. Another aspect to consider is the speed by which new threats are addressed (given that we are now talking about one new threat every 1.5 seconds).
The customer benefit in these new tests is that they really could see which product protects best against real threats. They don’t need to look at artificial “shootouts” against a million old malware samples being checked through a manual scan. And not only do the new tests show how effective a product is in blocking Web-based threats, they also show how fast the security vendor is providing protection against a new threat—time to protect.
As malware become more and more aggressive and as malware distribution switches from self-replication to spamming campaigns, this will become extremely important. Malware look different now compared with five years ago so the tests should be different as well.
I’m really happy to see the test labs move toward this direction in order to provide more meaningful tests.