Is Building Secure ML Possible?

Chapter 2. Known Security Risks of ML Systems

In this chapter, I will give a brief overview of some of the security risks that we currently know about. This is not a complete list of everything that you should be thinking about, but an introduction to a few of the most well-known and potentially harmful risks. Various organizations have published lists of ML security risks, including NIST, Microsoft, and the MITRE Corporation. However, at the time of writing, there is no complete consensus on what risks exist or the terminology to describe them.

This chapter aims to bring together the current state of knowledge of ML security risks. I will also show how these risks manifest in the different phases of an ML lifecycle. In the following discussions, an “attacker” is any person or organization that is attempting to cause harm or disruption to an ML system.

Adversarial Training Data

This security risk, also known as a data poisoning attack, occurs when an attacker interferes with a model’s training data. In many machine learning systems, a model is not simply trained once from a static dataset; it has a feedback loop where new data is collected, then used to retrain the model. This feedback loop means an attacker can contaminate the training data to either disrupt the system or influence the predictions of the retrained model. This is a risk during the data ingestion phase, but the effects are seen in the model training step.

Microsoft experienced a very high-profile attack of this ...