Expect the Unexpected: The Unintended Consequences of Machine Learning

by Roark Pollock

February 21, 2017

access_time 5 min read

The history of technology is littered with examples of serious unintended consequences following the introduction of any new technology. It often surprises people that new technologies might have nefarious purposes in addition to the positive purposes for which they are brought to market but it happens all the time.

For example, Train robbers using dynamite (“You think you used enough Dynamite there, Butch?”) or spammers using email. More recently the use of SSL to hide malware from security controls has become more common just because the legitimate use of SSL has made this technique more useful. Because new technology is often appropriated by bad actors, we have no reason to think this will not be true about the new generation of machine learning tools that have reached the market.

To what effect will these tools be misused? There are probably a couple of ways that attackers could use machine learning to their advantage. At a minimum, malware writers will test their new malware against the new class of advanced threat protection products in a bid to modify their code so that it is less likely to be flagged as malicious. The effectiveness of protective security controls always has a half-life due to adversarial learning. An understanding of machine learning defenses will help attackers become more proactive in reducing the effectiveness of machine learning based defenses. An example would be an attacker flooding a network with fake traffic with the hope of “poisoning” the machine learning model being built from that traffic. The goal of the attacker would be to fool the defender’s machine learning tool into misclassifying traffic or to create such a high level of false positives that the defenders would dial back the fidelity of the alerts.

Machine learning will likely also be used as an offensive tool by attackers. For example, some researchers predict that attackers will utilize machine learning techniques to hone their social engineering attacks (e.g., spear phishing). The automation of the effort it takes to customize a social engineering attack is particularly troubling given the effectiveness of spear phishing. The ability to automate mass customization of these attacks is a potent economic incentive for attackers to adopt the techniques. Expect breaches of this type that deliver ransomware payloads to increase sharply in 2017.

The need to automate tasks is a major driver of investment decisions for both attackers and defenders. Machine learning promises to automate detection and response and increase the operational tempo. While the technology will increasingly become a standard component of defense in depth strategies, it is not a magic bullet. It should be understood that attackers are actively working on evasion methods around machine learning based detection products while also using machine learning for their own offensive purposes. This arms race will require defenders to increasingly achieve incident response at machine speed, further exacerbating the need for automated incident response capabilities.