Optimization: Gradient Descent and Deep Learning (ML Foundations Series)

• Capstone ML class: Serves as the final class in the 14-part "ML Foundations" series by Jon Krohn, blending material from the fields of linear algebra, calculus, probability, statistics and algorithms.
• Gradient descent mastery: Develop a deep understanding of the essential theory behind the ubiquitous gradient descent approach to optimization, along with hands-on experience applying it using PyTorch and TensorFlow.
• Latest optimization techniques: Learn about state-of-the-art optimizers, such as Adam and Nadam, widely used for training deep neural networks, while also receiving guidance on next steps in your ML journey.

The Machine Learning Foundations series of online trainings provides a comprehensive overview of all of the subjects — mathematics, statistics, and computer science — that underlie contemporary machine learning techniques, including deep learning and other artificial intelligence approaches. Extensive curriculum detail can be found at the course’s GitHub repo.

All of the classes in the ML Foundations series bring theory to life through the combination of vivid full-color illustrations, straightforward Python examples within hands-on Jupyter notebook demos, and comprehension exercises with fully-worked solutions.

The focus is on providing you with a practical, functional understanding of the content covered. Context will be given for each topic, highlighting its relevance to machine learning. You will be better positioned to understand cutting-edge machine learning papers and you will be provided with resources for digging even deeper into topics that pique your curiosity.

There are 14 classes in the series, organized into four subject areas:

Linear Algebra (three classes)

• Linear Algebra for Machine Learning: Intro
• Linear Algebra for Machine Learning, Level II: Matrix Tensors
• Linear Algebra for Machine Learning, Level III: Eigenvectors

Calculus (four classes)

• Calculus for Machine Learning: Intro
• Calculus for Machine Learning, Level II: Automatic Differentiation
• Calculus for Machine Learning, Level III: Partial Derivatives
• Calculus for Machine Learning, Level IV: Gradients & Integrals

Probability and Statistics (four classes)

• Intro to Probability Theory
• Probability II and Information Theory
• Intro to Statistics
• Statistics II: Regression and Bayesian

Computer Science (three classes)

• Intro to Data Structures and Algorithms
• DSA II: Hashing, Trees, and Graphs
• Optimization

Each of the four subject areas are fairly independent, however theory within a given subject area generally builds over the 3-4 classes — topics in later classes of a given subject area often assume an understanding of topics from earlier classes. Work through the individual classes based on your particular interests or your existing familiarity with the material.

(Note that at any given time, only a subset of the ML Foundations classes will be scheduled and open for registration.)

This class, Optimization, is the final class in the 14-part Machine Learning Foundations series. It builds upon the material from each of the other classes in the series — on linear algebra, calculus, probability, statistics, and algorithms — in order to provide a detailed introduction to training ML models. Through the measured exposition of theory paired with interactive examples, you’ll develop a working understanding of all of the essential theory behind the ubiquitous gradient descent approach to optimization as well as how to apply it yourself — both at a granular, matrix operations level and a quick, abstract level — with TensorFlow and PyTorch. You’ll also learn about the latest optimizers, such as Adam and Nadam, that are widely used for training deep neural networks. Now well-equipped with all the foundational knowledge underlying ML, at the end of class you’ll be left with guidance on where to proceed from here on your ML journey.