Programming PyTorch for Deep Learning

Programming PyTorch for Deep Learning

by Ian Pointer
Released September 2019
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492045359

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Take the next steps toward mastering deep learning, the machine learning method that’s transforming the world around us by the second. In this practical book, you’ll get up to speed on key ideas using Facebook’s open source PyTorch framework and gain the latest skills you need to create your very own neural networks.

Ian Pointer shows you how to set up PyTorch on a cloud-based environment, then walks you through the creation of neural architectures that facilitate operations on images, sound, text,and more through deep dives into each element. He also covers the critical concepts of applying transfer learning to images, debugging models, and PyTorch in production.

  • Learn how to deploy deep learning models to production
  • Explore PyTorch use cases from several leading companies
  • Learn how to apply transfer learning to images
  • Apply cutting-edge NLP techniques using a model trained on Wikipedia
  • Use PyTorch’s torchaudio library to classify audio data with a convolutional-based model
  • Debug PyTorch models using TensorBoard and flame graphs
  • Deploy PyTorch applications in production in Docker containers and Kubernetes clusters running on Google Cloud

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Deep Learning in the World Today
    2. But What Is Deep Learning Exactly, and Do I Need a PhD to Understand It?
    3. PyTorch
      1. What About TensorFlow?
    4. Conventions Used in This Book
    5. Using Code Examples
    6. O’Reilly Online Learning
    7. How to Contact Us
    8. Acknowledgments
  2. 1. Getting Started with PyTorch
    1. Building a Custom Deep Learning Machine
      1. GPU
      2. CPU/Motherboard
      3. RAM
      4. Storage
    2. Deep Learning in the Cloud
      1. Google Colaboratory
      2. Cloud Providers
      3. Which Cloud Provider Should I Use?
    3. Using Jupyter Notebook
    4. Installing PyTorch from Scratch
      1. Download CUDA
      2. Anaconda
      3. Finally, PyTorch! (and Jupyter Notebook)
    5. Tensors
      1. Tensor Operations
      2. Tensor Broadcasting
    6. Conclusion
    7. Further Reading
  3. 2. Image Classification with PyTorch
    1. Our Classification Problem
    2. Traditional Challenges
      1. But First, Data
      2. PyTorch and Data Loaders
      3. Building a Training Dataset
      4. Building Validation and Test Datasets
    3. Finally, a Neural Network!
      1. Activation Functions
      2. Creating a Network
      3. Loss Functions
      4. Optimizing
    4. Training
      1. Making It Work on the GPU
    5. Putting It All Together
      1. Making Predictions
      2. Model Saving
    6. Conclusion
    7. Further Reading
  4. 3. Convolutional Neural Networks
    1. Our First Convolutional Model
      1. Convolutions
      2. Pooling
      3. Dropout
    2. History of CNN Architectures
      1. AlexNet
      2. Inception/GoogLeNet
      3. VGG
      4. ResNet
      5. Other Architectures Are Available!
    3. Using Pretrained Models in PyTorch
      1. Examining a Model’s Structure
      2. BatchNorm
      3. Which Model Should You Use?
    4. One-Stop Shopping for Models: PyTorch Hub
    5. Conclusion
    6. Further Reading
  5. 4. Transfer Learning and Other Tricks
    1. Transfer Learning with ResNet
    2. Finding That Learning Rate
    3. Differential Learning Rates
    4. Data Augmentation
      1. Torchvision Transforms
      2. Color Spaces and Lambda Transforms
      3. Custom Transform Classes
      4. Start Small and Get Bigger!
    5. Ensembles
    6. Conclusion
    7. Further Reading
  6. 5. Text Classification
    1. Recurrent Neural Networks
    2. Long Short-Term Memory Networks
      1. Gated Recurrent Units
      2. biLSTM
    3. Embeddings
    4. torchtext
      1. Getting Our Data: Tweets!
      2. Defining Fields
      3. Building a Vocabulary
      4. Creating Our Model
      5. Updating the Training Loop
      6. Classifying Tweets
    5. Data Augmentation
      1. Random Insertion
      2. Random Deletion
      3. Random Swap
      4. Back Translation
      5. Augmentation and torchtext
      6. Transfer Learning?
    6. Conclusion
    7. Further Reading
  7. 6. A Journey into Sound
    1. Sound
    2. The ESC-50 Dataset
      1. Obtaining the Dataset
      2. Playing Audio in Jupyter
    3. Exploring ESC-50
      1. SoX and LibROSA
      2. torchaudio
      3. Building an ESC-50 Dataset
    4. A CNN Model for ESC-50
    5. This Frequency Is My Universe
      1. Mel Spectrograms
      2. A New Dataset
      3. A Wild ResNet Appears
      4. Finding a Learning Rate
    6. Audio Data Augmentation
      1. torchaudio Transforms
      2. SoX Effect Chains
      3. SpecAugment
    7. Further Experiments
    8. Conclusion
    9. Further Reading
  8. 7. Debugging PyTorch Models
    1. It’s 3 a.m. What Is Your Data Doing?
    2. TensorBoard
      1. Installing TensorBoard
      2. Sending Data to TensorBoard
      3. PyTorch Hooks
      4. Plotting Mean and Standard Deviation
      5. Class Activation Mapping
    3. Flame Graphs
      1. Installing py-spy
      2. Reading Flame Graphs
      3. Fixing a Slow Transformation
    4. Debugging GPU Issues
      1. Checking Your GPU
      2. Gradient Checkpointing
    5. Conclusion
    6. Further Reading
  9. 8. PyTorch in Production
    1. Model Serving
      1. Building a Flask Service
      2. Setting Up the Model Parameters
      3. Building the Docker Container
      4. Local Versus Cloud Storage
      5. Logging and Telemetry
    2. Deploying on Kubernetes
      1. Setting Up on Google Kubernetes Engine
      2. Creating a k8s Cluster
      3. Scaling Services
      4. Updates and Cleaning Up
    3. TorchScript
      1. Tracing
      2. Scripting
      3. TorchScript Limitations
    4. Working with libTorch
      1. Obtaining libTorch and Hello World
      2. Importing a TorchScript Model
    5. Conclusion
    6. Further Reading
  10. 9. PyTorch in the Wild
    1. Data Augmentation: Mixed and Smoothed
      1. mixup
      2. Label Smoothing
    2. Computer, Enhance!
      1. Introduction to Super-Resolution
      2. An Introduction to GANs
      3. The Forger and the Critic
      4. Training a GAN
      5. The Dangers of Mode Collapse
      6. ESRGAN
    3. Further Adventures in Image Detection
      1. Object Detection
      2. Faster R-CNN and Mask R-CNN
    4. Adversarial Samples
      1. Black-Box Attacks
      2. Defending Against Adversarial Attacks
    5. More Than Meets the Eye: The Transformer Architecture
      1. Paying Attention
      2. Attention Is All You Need
      3. BERT
      4. FastBERT
      5. GPT-2
      6. Generating Text with GPT-2
      7. ULMFiT
      8. What to Use?
    6. Conclusion
    7. Further Reading
  11. Index

Product information

  • Title: Programming PyTorch for Deep Learning
  • Author(s): Ian Pointer
  • Release date: September 2019
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492045359