Delta Testing, a specialist in anti-malware testing methodologies and research, has released a new report highlighting shortcomings in the detection rates of leading players in the advanced threat solution market. Based on independent tests carried out by Delta Testing, the New Approach to Assessing Advanced Threat Solutions report shows a significant discrepancy between the seven vendors tested, with FireEye achieving a 99.14 per cent detection rate compared to Fidelis 5.17 at the other end of the scale.
Unlike established information security tests that rely on legacy malware such as common viruses and worms or malware repositories, Delta Testing runs advance malware that has never before been detected by a product. This malicious code is sourced from Delta’s network of incident response teams, security researchers, and organisations that have been breached.
Featured Download: Social media access at work. Do your employees know the rules?
By using malicious code already in the wild yet recently created insofar as no signatures currently exist, products are exposed to an unpredictable environment. This approach ensures that products that don’t have the ability to detect unknown malware using dynamic analysis can be easily identified. The approach of using actual malicious code along with actual packet captures is a clear advantage over using synthetic malware that, while lab-appropriate, doesn’t mimic real-life examples.
In the advanced threat protection solutions test, packet captures were stripped of customer data and replayed through the latest published software version of the advanced malware protection products. All of the products were allowed time to analyse the packets in their virtual environments or perform lookups in their cloud or threat intelligence. Each product’s interface was subsequently checked to see if an alert had been triggered to determine detection. Detection rates were calculated as a percentage by dividing the number of detections against a total number of samples sent.
The findings are separated by vendor, appliance(s), version(s), and result (percent):
AhnLab
MDS 2000
Version 2.1.2.27 ( Build 75r)
6.9%
Checkpoint
2200 Appliance, 4600 Appliance
Version R77, Version R77
24.14%
Fidelis Cybersecurity Solutions
XPS Edge 200, XPS Sensor, XPS Command, Post
Version: 7.6.5, Version: 7.6.5, Version: 7.6.5
5.17%
FireEye
NX 7500
Version 7.4.0
99.14%
McAfee
Advanced Threat Defence Network Security Platform
Version 3.0.2.36.34869, Version 8.0.5.9
12.94%
Vendor A
Appliance with Cloud Based Sandbox
Version 6.0.0
21.55%
TrendMicro
Deep Discovery Advisor, Deep Discovery Inspector
Version 2.95.0.1104, Version 3.5
33.91%
Commenting on the findings, Mark Thomas, commercial director, Delta Testing says, “It would appear that most mainstream vendors are on par when it comes to previously known attacks. However, as soon as you move towards a methodology that exposes the products to fresh malware samples taken from actual customer environments, detection rates start to show significant variations based on the technology and detection methodology used by the vendor. This clearly demonstrated FireEye’s leadership in detecting advanced/unknown threats.”
“While only one malware sample tested was missed by all seven products, the fact that no vendor scored 100 per cent in identifying all of the samples shows that some attacks are still getting through the net. As such, in order to meet the shortfall, organisations need to consider a holistic approach to security which doesn’t just rely on detection but addresses threat prevention and response tactics too.”
The full report can be viewed and downloaded here.
About Delta Testing
Delta Testing specialises in anti-malware testing methodologies and research and is committed to creating the industry’s finest real world test systems to combat advanced attacks. The company comprises of several in-house malware specialists headed by a Chief Researcher with over 20 years industry experience. Together the team has tested hundreds of products from global vendors, giving the company unrivalled expertise. Delta Testing is based in West Glamorgan, UK.
