Chapter 9. Monitoring and Observability for Models
Managing production systems is somewhere between an art and a science. Add the complexities of ML to this hybrid discipline, and it looks less like a science and more like an art. What we do today is very much a frontier, rather than a well-defined space. Despite that, this chapter outlines what we know about how to monitor, observe, and alert for ML production systems, and makes suggestions for developing the practice within your own organization.
What Is Production Monitoring and Why Do It?
This chapter is about how to monitor systems that are doing ML, rather than using ML to monitor systems. The latter is sometimes called AIOps; we are focusing on the former.
With that out of the way, let’s talk about production monitoring generically, without the complexities of ML, so we can make things easier to understand—and where better to begin than with a definition? Monitoring, at the most basic level, provides data about how your systems are performing; that data is made storable, accessible, and displayable in some reasonable way. Observability is an attribute of software, meaning that when correctly written, the emitted monitoring data—usually extended or expanded in some way, with labeling or tagging—can be used to correctly infer the behavior of the system.1
Why would you care? It turns out there are lots ...
Get Reliable Machine Learning now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
