Building Machine Learning Powered Applications

Chapter 5. Train and Evaluate Your Model

In the previous chapters we’ve covered how to identify the right problem to tackle, make a plan to tackle it, build a simple pipeline, explore a dataset, and generate an initial set of features. These steps have allowed us to gather enough information to begin training an adequate model. An adequate model here means a model that is a good fit for the task at hand and that has good chances of performing well.

In this chapter, we will start by briefly going over some concerns when choosing a model. Then, we will describe best practices to separate your data, which will help evaluate your models in realistic conditions. Finally, we’ll look at methods to analyze modeling results and diagnose errors.

The Simplest Appropriate Model

Now that we are ready to train a model, we need to decide which model to start with. It may be tempting to try every possible model, benchmark them all, and pick the one with the best results on a held-out test set according to some metrics.

In general, this is not the best approach. Not only is it computationally intensive (there are many sets of models and many parameters for each model, so realistically you will only be able to test a suboptimal subset), it also treats models as predictive black boxes and entirely ignores that ML models encode implicit assumptions about the data in the way they learn.

Different models make different assumptions about the data and so are suited for different tasks. In addition, ...