Managing risk in machine learning models
The O’Reilly Data Show Podcast: Andrew Burt and Steven Touw on how companies can manage models they cannot fully explain.
Sculpture (source: Pxhere.com)
In this episode of the Data Show, I spoke with Andrew Burt, chief privacy officer at Immuta, and Steven Touw, co-founder and CTO of Immuta. Burt recently co-authored a white paper on managing risk in machine learning models, and I wanted to sit down with them to discuss some of the proposals they put forward to organizations that are deploying machine learning.
Some high-profile examples of models gone awry have raised awareness among companies for the need for better risk management tools and processes. There is now a growing interest in ethics among data scientists, specifically in tools for monitoring bias in machine learning models. In a previous post, I listed some of the key considerations organization should keep in mind as they move models to production, but the report co-authored by Burt goes far beyond and recommends lines of defense, including a description of key roles that are needed.
Learn faster. Dig deeper. See farther.
Here are some highlights from our conversation:
Privacy and compliance meet data science
Andrew Burt: I would say the big takeaway from our paper is that lawyers and compliance and privacy folks live in one world and data scientists live in another with competing objectives. And that can no longer be the case. They need to talk to each other. They need to have a shared process and some shared terminology so that everybody can communicate.
One of the recommendations that we make, taken from some of the model risk management frameworks, is to create what we call lines of defense: these are basically different lines of reviewers who conduct periodic reviews from the creation testing phase to validation to an auditing phase. And the members of those lines of review need to be made up of teams with multiple expertise. So, there needs to be data owners, the people responsible for the data being piped into the models; there needs to be compliance personnel who are thinking about legal and ethical obligations; there needs to be data scientists; and there needs to be subject domain experts.
… We also dive into how you should be thinking about de-risking and monitoring your input data. How you should be thinking about monitoring and de-risking your output data and using output data for models. … And then, I think really importantly, is this idea of thinking about what it means for a model to fail, and having a concrete plan for what that means, how to correct it if it fails, and how to pull it from production if you need to.
Explainability and GDPR
Steven Touw: I gave a talk, “How Does the GDPR Impact Machine Learning?”, at Strata Data London. A lot of people are concerned about language in GDPR, which states that you must be able to explain how the model came to that conclusion. I think people are kind of overreacting to this a little bit, and we need to inject some common sense in there along the lines of: at the end of the day, you can explain what data went in; you can explain the logic of what you’re trying to solve and why; and you don’t have to explain every neuron in the neural net and how it was correlated to every other piece.
I think the GDPR is actually doing a good thing. It’s enabling consumers to understand how the decisions are being made about them, but they don’t have to understand everything in the weeds about it. Because the whole point of machine learning is that it can do things that we can’t as humans. So, that’s why we use it, and there are cases where it makes sense to trust the model rather than humans to get things done and, potentially and hopefully, done more accurately.
Related resources:
- “How to build analytic products in an age when data privacy has become critical”
- “We need to build machine learning tools to augment machine learning engineers”
- “The ethics of artificial intelligence”
- “Interpreting predictive models with Skater: Unboxing model opacity”
- “Toward the Jet Age of machine learning”
- “Building tools for the AI applications of tomorrow”
- “The current state of applied data science”