Deep Learning for Coders with fastai and PyTorch

Deep Learning for Coders with fastai and PyTorch

by Jeremy Howard, Sylvain Gugger
Released July 2020
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492045526

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Deep learning is often viewed as the exclusive domain of math PhDs and big tech companies. But as this hands-on guide demonstrates, programmers comfortable with Python can achieve impressive results in deep learning with little math background, small amounts of data, and minimal code. How? With fastai, the first library to provide a consistent interface to the most frequently used deep learning applications.

Authors Jeremy Howard and Sylvain Gugger, the creators of fastai, show you how to train a model on a wide range of tasks using fastai and PyTorch. You’ll also dive progressively further into deep learning theory to gain a complete understanding of the algorithms behind the scenes.

  • Train models in computer vision, natural language processing, tabular data, and collaborative filtering
  • Learn the latest deep learning techniques that matter most in practice
  • Improve accuracy, speed, and reliability by understanding how deep learning models work
  • Discover how to turn your models into web applications
  • Implement deep learning algorithms from scratch
  • Consider the ethical implications of your work
  • Gain insight from the foreword by PyTorch cofounder, Soumith Chintala

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Who This Book Is For
    2. What You Need to Know
    3. What You Will Learn
    4. O’Reilly Online Learning
    5. How to Contact Us
  2. Foreword
  3. I. Deep Learning in Practice
  4. 1. Your Deep Learning Journey
    1. Deep Learning Is for Everyone
    2. Neural Networks: A Brief History
    3. Who We Are
    4. How to Learn Deep Learning
      1. Your Projects and Your Mindset
    5. The Software: PyTorch, fastai, and Jupyter (And Why It Doesn’t Matter)
    6. Your First Model
      1. Getting a GPU Deep Learning Server
      2. Running Your First Notebook
      3. What Is Machine Learning?
      4. What Is a Neural Network?
      5. A Bit of Deep Learning Jargon
      6. Limitations Inherent to Machine Learning
      7. How Our Image Recognizer Works
      8. What Our Image Recognizer Learned
      9. Image Recognizers Can Tackle Non-Image Tasks
      10. Jargon Recap
    7. Deep Learning Is Not Just for Image Classification
    8. Validation Sets and Test Sets
      1. Use Judgment in Defining Test Sets
    9. A Choose Your Own Adventure Moment
    10. Questionnaire
      1. Further Research
  5. 2. From Model to Production
    1. The Practice of Deep Learning
      1. Starting Your Project
      2. The State of Deep Learning
      3. The Drivetrain Approach
    2. Gathering Data
    3. From Data to DataLoaders
      1. Data Augmentation
    4. Training Your Model, and Using It to Clean Your Data
    5. Turning Your Model into an Online Application
      1. Using the Model for Inference
      2. Creating a Notebook App from the Model
      3. Turning Your Notebook into a Real App
      4. Deploying Your App
    6. How to Avoid Disaster
      1. Unforeseen Consequences and Feedback Loops
    7. Get Writing!
    8. Questionnaire
      1. Further Research
  6. 3. Data Ethics
    1. Key Examples for Data Ethics
      1. Bugs and Recourse: Buggy Algorithm Used for Healthcare Benefits
      2. Feedback Loops: YouTube’s Recommendation System
      3. Bias: Professor Latanya Sweeney “Arrested”
      4. Why Does This Matter?
    2. Integrating Machine Learning with Product Design
    3. Topics in Data Ethics
      1. Recourse and Accountability
      2. Feedback Loops
      3. Bias
      4. Disinformation
    4. Identifying and Addressing Ethical Issues
      1. Analyze a Project You Are Working On
      2. Processes to Implement
      3. The Power of Diversity
      4. Fairness, Accountability, and Transparency
    5. Role of Policy
      1. The Effectiveness of Regulation
      2. Rights and Policy
      3. Cars: A Historical Precedent
    6. Conclusion
    7. Questionnaire
      1. Further Research
    8. Deep Learning in Practice: That’s a Wrap!
  7. II. Understanding fastai’s Applications
  8. 4. Under the Hood: Training a Digit Classifier
    1. Pixels: The Foundations of Computer Vision
    2. First Try: Pixel Similarity
      1. NumPy Arrays and PyTorch Tensors
    3. Computing Metrics Using Broadcasting
    4. Stochastic Gradient Descent
      1. Calculating Gradients
      2. Stepping with a Learning Rate
      3. An End-to-End SGD Example
      4. Summarizing Gradient Descent
    5. The MNIST Loss Function
      1. Sigmoid
      2. SGD and Mini-Batches
    6. Putting It All Together
      1. Creating an Optimizer
    7. Adding a Nonlinearity
      1. Going Deeper
    8. Jargon Recap
    9. Questionnaire
      1. Further Research
  9. 5. Image Classification
    1. From Dogs and Cats to Pet Breeds
    2. Presizing
      1. Checking and Debugging a DataBlock
    3. Cross-Entropy Loss
      1. Viewing Activations and Labels
      2. Softmax
      3. Log Likelihood
      4. Taking the log
    4. Model Interpretation
    5. Improving Our Model
      1. The Learning Rate Finder
      2. Unfreezing and Transfer Learning
      3. Discriminative Learning Rates
      4. Selecting the Number of Epochs
      5. Deeper Architectures
    6. Conclusion
    7. Questionnaire
      1. Further Research
  10. 6. Other Computer Vision Problems
    1. Multi-Label Classification
      1. The Data
      2. Constructing a DataBlock
      3. Binary Cross Entropy
    2. Regression
      1. Assembling the Data
      2. Training a Model
    3. Conclusion
    4. Questionnaire
      1. Further Research
  11. 7. Training a State-of-the-Art Model
    1. Imagenette
    2. Normalization
    3. Progressive Resizing
    4. Test Time Augmentation
    5. Mixup
    6. Label Smoothing
    7. Conclusion
    8. Questionnaire
      1. Further Research
  12. 8. Collaborative Filtering Deep Dive
    1. A First Look at the Data
    2. Learning the Latent Factors
    3. Creating the DataLoaders
    4. Collaborative Filtering from Scratch
      1. Weight Decay
      2. Creating Our Own Embedding Module
    5. Interpreting Embeddings and Biases
      1. Using fastai.collab
      2. Embedding Distance
    6. Bootstrapping a Collaborative Filtering Model
    7. Deep Learning for Collaborative Filtering
    8. Conclusion
    9. Questionnaire
      1. Further Research
  13. 9. Tabular Modeling Deep Dive
    1. Categorical Embeddings
    2. Beyond Deep Learning
    3. The Dataset
      1. Kaggle Competitions
      2. Look at the Data
    4. Decision Trees
      1. Handling Dates
      2. Using TabularPandas and TabularProc
      3. Creating the Decision Tree
      4. Categorical Variables
    5. Random Forests
      1. Creating a Random Forest
      2. Out-of-Bag Error
    6. Model Interpretation
      1. Tree Variance for Prediction Confidence
      2. Feature Importance
      3. Removing Low-Importance Variables
      4. Removing Redundant Features
      5. Partial Dependence
      6. Data Leakage
      7. Tree Interpreter
    7. Extrapolation and Neural Networks
      1. The Extrapolation Problem
      2. Finding Out-of-Domain Data
      3. Using a Neural Network
    8. Ensembling
      1. Boosting
      2. Combining Embeddings with Other Methods
    9. Conclusion
    10. Questionnaire
      1. Further Research
  14. 10. NLP Deep Dive: RNNs
    1. Text Preprocessing
      1. Tokenization
      2. Word Tokenization with fastai
      3. Subword Tokenization
      4. Numericalization with fastai
      5. Putting Our Texts into Batches for a Language Model
    2. Training a Text Classifier
      1. Language Model Using DataBlock
      2. Fine-Tuning the Language Model
      3. Saving and Loading Models
      4. Text Generation
      5. Creating the Classifier DataLoaders
      6. Fine-Tuning the Classifier
    3. Disinformation and Language Models
    4. Conclusion
    5. Questionnaire
      1. Further Research
  15. 11. Data Munging with fastai’s Mid-Level API
    1. Going Deeper into fastai’s Layered API
      1. Transforms
      2. Writing Your Own Transform
      3. Pipeline
    2. TfmdLists and Datasets: Transformed Collections
      1. TfmdLists
      2. Datasets
    3. Applying the Mid-Level Data API: SiamesePair
    4. Conclusion
    5. Questionnaire
      1. Further Research
    6. Understanding fastai’s Applications: Wrap Up
  16. III. Foundations of Deep Learning
  17. 12. A Language Model from Scratch
    1. The Data
    2. Our First Language Model from Scratch
      1. Our Language Model in PyTorch
      2. Our First Recurrent Neural Network
    3. Improving the RNN
      1. Maintaining the State of an RNN
      2. Creating More Signal
    4. Multilayer RNNs
      1. The Model
      2. Exploding or Disappearing Activations
    5. LSTM
      1. Building an LSTM from Scratch
      2. Training a Language Model Using LSTMs
    6. Regularizing an LSTM
      1. Dropout
      2. Activation Regularization and Temporal Activation Regularization
      3. Training a Weight-Tied Regularized LSTM
    7. Conclusion
    8. Questionnaire
      1. Further Research
  18. 13. Convolutional Neural Networks
    1. The Magic of Convolutions
      1. Mapping a Convolutional Kernel
      2. Convolutions in PyTorch
      3. Strides and Padding
      4. Understanding the Convolution Equations
    2. Our First Convolutional Neural Network
      1. Creating the CNN
      2. Understanding Convolution Arithmetic
      3. Receptive Fields
      4. A Note About Twitter
    3. Color Images
    4. Improving Training Stability
      1. A Simple Baseline
      2. Increase Batch Size
      3. 1cycle Training
      4. Batch Normalization
    5. Conclusion
    6. Questionnaire
      1. Further Research
  19. 14. ResNets
    1. Going Back to Imagenette
    2. Building a Modern CNN: ResNet
      1. Skip Connections
      2. A State-of-the-Art ResNet
      3. Bottleneck Layers
    3. Conclusion
    4. Questionnaire
      1. Further Research
  20. 15. Application Architectures Deep Dive
    1. Computer Vision
      1. cnn_learner
      2. unet_learner
      3. A Siamese Network
    2. Natural Language Processing
    3. Tabular
    4. Conclusion
    5. Questionnaire
      1. Further Research
  21. 16. The Training Process
    1. Establishing a Baseline
    2. A Generic Optimizer
    3. Momentum
    4. RMSProp
    5. Adam
    6. Decoupled Weight Decay
    7. Callbacks
      1. Creating a Callback
      2. Callback Ordering and Exceptions
    8. Conclusion
    9. Questionnaire
      1. Further Research
    10. Foundations of Deep Learning: Wrap Up
  22. IV. Deep Learning from Scratch
  23. 17. A Neural Net from the Foundations
    1. Building a Neural Net Layer from Scratch
      1. Modeling a Neuron
      2. Matrix Multiplication from Scratch
      3. Elementwise Arithmetic
      4. Broadcasting
      5. Einstein Summation
    2. The Forward and Backward Passes
      1. Defining and Initializing a Layer
      2. Gradients and the Backward Pass
      3. Refactoring the Model
      4. Going to PyTorch
    3. Conclusion
    4. Questionnaire
      1. Further Research
  24. 18. CNN Interpretation with CAM
    1. CAM and Hooks
    2. Gradient CAM
    3. Conclusion
    4. Questionnaire
      1. Further Research
  25. 19. A fastai Learner from Scratch
    1. Data
      1. Dataset
    2. Module and Parameter
      1. Simple CNN
    3. Loss
    4. Learner
      1. Callbacks
      2. Scheduling the Learning Rate
    5. Conclusion
    6. Questionnaire
      1. Further Research
  26. 20. Concluding Thoughts
  27. A. Creating a Blog
    1. Blogging with GitHub Pages
      1. Creating the Repository
      2. Setting Up Your Home Page
      3. Creating Posts
      4. Synchronizing GitHub and Your Computer
    2. Jupyter for Blogging
  28. B. Data Project Checklist
    1. Data Scientists
    2. Strategy
    3. Data
    4. Analytics
    5. Implementation
    6. Maintenance
    7. Constraints
  29. Index

Product information

  • Title: Deep Learning for Coders with fastai and PyTorch
  • Author(s): Jeremy Howard, Sylvain Gugger
  • Release date: July 2020
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492045526