INTELLIGENCE-LED TESTING

PREDICTIVE MALWARE RESPONSE

MARCH 2018

The test exposed systems protected by this older version of CylancePROTECT to very impactful threats discovered and reported widely after May 2015. In this way the test shows to what extent the product was able to predict how future threats would appear. This “Predictive Advantage” (PA), the advantage that users of the product have against future adversaries, is presented in this report.

Contents

1. Predictive Advantage by Threat Family 06
2. Predictive Advantage by Individual Campaign 07
3. Legitimate Software Handling 09
4. Conclusions 10 Appendix A: FAQs 11 Appendix B: Sample Validation 12 Appendix C: Product Versions 12

Introduction

A common criticism of computer security products is that they can only protect against known threats. When new attacks are detected and analysed security companies produce updates based on this new knowledge, which can then be applied to endpoint, network and cloud security software and services.

But in the time between detection of the attack and application of the corresponding updates, systems are vulnerable to compromise. Almost by definition at least one victim, the so-called ‘patient zero’, has to experience the threat before new protection systems can be deployed. While the rest of us benefit from patient zero’s misfortune, patient zero has potentially suffered catastrophic damage to its operations.

MINORITY REPORT

Security companies have, for some years, developed advanced detection systems, often labelled as using ‘AI’, ‘machine learning’ or some other technical-sounding term. The basic idea is that past threats are analysed in deep ways to identify what future threats might look like. Ideally the result will be a product that can detect potentially bad files or behaviour before the attack is successful.

Executive Summary

It is possible to test claims of this type of predictive capability by taking an old version of a product, denying it the ability to update or query cloud services, and then exposing it to threats that were created, detected and analysed months or even years after its own creation. It’s the equivalent of sending an old product forward in time and seeing how well it works with future threats.

11 months up to 33 months, with an average PA of 25 months. In other words, in some cases it was able to recognise and protect against threats that would not appear in real life for up to two years and nine months into the future. Generally speaking it was effective, without updates, against threats just over two years into the future.

This is exactly what we did in this test. Using CylancePROTECT’s AI model from May 2015 we collected serious threats dating from February 2016 all the way through to November 2017.

Malware campaigns can run over a period of time, with those in control making changes to the malware to add features or evade detection. For this reason we used different variants for each ‘family’ of attack. For example, we used five different versions of the Cerber ransomware attack, with samples dating from December 2016 through to February 2018.

CylancePROTECT’s Predictive Advantage (PA) varied, depending on the threat.

1. Predictive Advantage by Threat Family

Predictive Advantage (PA) is the time difference between the creation of the model and the first time a threat is seen by victims and security companies protecting those victims.

The model represented in this test was created in May 2015. This is the same model as that deployed in the real world with CylancePROTECT’s agent, version 1300.

The graphs below shows the different PA values for the individual threats, which are grouped into their own families.

2. Predictive Advantage by Individual Campaign

Predictive Advantage (months)

For this reason we used different variants for each ‘family’ of attack. Variants within one family group may appear in the real world at different times as a campaign develops. For example, the GoldenEye campaign:

CAMPAIGN: GoldenEye

CAMPAIGN: Reyptson

CAMPAIGN: Locky

CAMPAIGN: Petya

3. Legitimate Software Handling

We tested popular business and consumer applications, and visited 50 highly popular websites, according to Alexa’s index, and found only one sub-optimum case. In this case a spreadsheet viewing utility was quarantined. Subsequently we discovered that this utility was bundled with potentially unwanted code so users may well be advised not to install it.

As such, there were no ‘false positives’ and no other types of sub-optimum handling of legitimate files.

4. Conclusions

This test was designed to examine Cylance's claim that the Artificial Intelligence (AI) technology at the heart of its endpoint protection product is self-contained, in terms of being effective without relying on regular updates or cloud queries. It was also intended to determine whether or not an AI model created some months and even years in the past could identify and handle threats that subsequently attacked systems on the internet.

Out of 45 threats, 43 were detected and prevented from compromising the system with an average PA of 25 months. The threats used in the test were discovered in the wild at dates ranging from 11 months to two years and nine months (33 months) after the creation of the AI model.

A full methodology for this test is available from our website.

Not only does the data demonstrate that CylancePROTECT (agent v1300, model May 2015) was capable of preventing threats that did not exist at the time the AI model was 'trained', but it provides an insight into how far ahead in time it could be effective without new knowledge. In practical terms, this indicates that regular updates to the product are not always needed, although we would expect Cylance to develop and deploy newly-trained models over time, simply because product development is an ongoing process and machine learning continues to take into account new threats to predict future ones.

The test was conducted without internet or other access to back-end systems. Threats and legitimate applications were independently located and verified by SE Labs.

Appendix A: FAQs

1. Did SE Labs provide Cylance with access to the threats used before the test started? No, threats were identified, collected and verified before testing, and only then made available to Cylance.

2. Is the data shown in this report the full data from the test, or a selection of it? The full set of data collected during testing of the featured model is represented in this report. We have not excluded any sub-optimum results.

3. So did Cylance persuade you to drop any sub-optimum results? No, all results gathered during testing are represented in this report. We did not drop any sub-optimum results for any reason.

4. Isn’t this kind of offline testing unrealistic, given that most systems are connected to the internet in the real world? This report’s conclusions show that the model under test would have been able to protect against theoretical threats that subsequently became real. It demonstrates the model’s potential against unknown threats. Disconnecting from the internet was necessary to show that Cylance was not enhancing its product’s abilities using online updates.