Reliable Machine Learning

Chapter 10. Continuous ML

Up until now, our discussions of ML systems have sometimes centered on the idea that a model is something we train and then deploy, almost as though this is something that happens once and only once. A slightly deeper view is to draw a distinction between models that are trained once and deployed versus those that are trained in a more continuous fashion, which we will refer to as continuous ML systems. Typical continuous ML systems receive new data in a streaming or periodic batch fashion and use this to trigger training an updated model version that is pushed to serving.

Clearly, there are major differences from an MLOps perspective between a model that is trained once versus a model that is updated in a continuous manner. Moving to continuous ML raises the stakes for automated verification. It introduces the potential for headaches around feedback loops and model reactions to changes in the external world. Managing continuous data streams, responding to model failures and corruptions, and even seemingly trivial tasks like introducing new features for the model to train on all increase system complexity.

Indeed, on the surface, it may seem like creating a continuous ML system might be a terrible idea. After all, in doing so, we expose our system to a set of changes that is inherently unknowable in advance because of potential changes in the real external world, and thus may cause unexpected or undesired system behavior. If we remember that in ML, data ...