Deep Learning at Scale

Chapter 11. Scaling Experiments: Effective Planning and Management

Plans are worthless, but planning is everything.

—Dwight D. Eisenhower

In this chapter, you will learn how to plan and manage scaling your training workload in an iterative and efficient manner by making informed choices along the way. The chapter is divided into two main sections, “Planning for Experiments and Execution” and “Techniques to Scale Your Experiments”.

The first of these sections focuses on the problem framing side of things and discusses how to plan your experiments to incrementally build up capabilities while keeping the entropy to a minimum. This section also provides some guiding principles on setting up projects and environment and system configurations to support smoother iterative development. You will learn about some tooling for versioning and card/summary systems to facilitate fluid iteration.

The following section dives into a set of deep learning techniques that are helpful in setting the direction and roadmap for experiment planning for iterative improvement of your model. In this section, you will learn about various approaches that are useful in accelerating model development, such as hyperparameter optimization, AutoML, transfer learning, meta learning, contrastive learning, and mixture of experts. Practical exercises demonstrating the use of some of these techniques are provided in “Hands-On Exercises”.

We’ll start by briefly considering the iterative nature of model development, ...