Deep Learning with PyTorch

Deep Learning with PyTorch

by Eli Stevens, Thomas Viehmann, Luca Pietro Giovanni Antiga
Released July 2020
Publisher(s): Manning Publications
ISBN: 9781617295263

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Every other day we hear about new ways to put deep learning to good use: improved medical imaging, accurate credit card fraud detection, long range weather forecasting, and more. PyTorch puts these superpowers in your hands, providing a comfortable Python experience that gets you started quickly and then grows with you as you—and your deep learning skills—become more sophisticated. Deep Learning with PyTorch will make that journey engaging and fun.

About the Technology
Although many deep learning tools use Python, the PyTorch library is truly Pythonic. Instantly familiar to anyone who knows PyData tools like NumPy and scikit-learn, PyTorch simplifies deep learning without sacrificing advanced features. It's excellent for building quick models, and it scales smoothly from laptop to enterprise. Because companies like Apple, Facebook, and JPMorgan Chase rely on PyTorch, it's a great skill to have as you expand your career options. It's easy to get started with PyTorch. It minimizes cognitive overhead without sacrificing the access to advanced features, meaning you can focus on what matters the most - building and training the latest and greatest deep learning models and contribute to making a dent in the world. PyTorch is also a snap to scale and extend, and it partners well with other Python tooling. PyTorch has been adopted by hundreds of deep learning practitioners and several first-class players like FAIR, OpenAI, FastAI and Purdue.

About the Book
Deep Learning with PyTorch teaches you to create neural networks and deep learning systems with PyTorch. This practical book quickly gets you to work building a real-world example from scratch: a tumor image classifier. Along the way, it covers best practices for the entire DL pipeline, including the PyTorch Tensor API, loading data in Python, monitoring training, and visualizing results. After covering the basics, the book will take you on a journey through larger projects. The centerpiece of the book is a neural network designed for cancer detection. You'll discover ways for training networks with limited inputs and start processing data to get some results. You'll sift through the unreliable initial results and focus on how to diagnose and fix the problems in your neural network. Finally, you'll look at ways to improve your results by training with augmented data, make improvements to the model architecture, and perform other fine tuning.

What's Inside
  • Training deep neural networks
  • Implementing modules and loss functions
  • Utilizing pretrained models from PyTorch Hub
  • Exploring code samples in Jupyter Notebooks


About the Reader
For Python programmers with an interest in machine learning.

About the Authors
Eli Stevens had roles from software engineer to CTO, and is currently working on machine learning in the self-driving-car industry. Luca Antiga is cofounder of an AI engineering company and an AI tech startup, as well as a former PyTorch contributor. Thomas Viehmann is a PyTorch core developer and machine learning trainer and consultant.

Quotes
With this publication, we finally have a definitive treatise on PyTorch. It covers the basics and abstractions in great detail.
- From the Foreword by Soumith Chintala, Cocreator of PyTorch

Deep learning divided into digestible chunks with code samples that build up logically.
- Mathieu Zhang, NVIDIA

Timely, practical, and thorough. Don’t put it on your bookshelf, but next to your laptop.
- Philippe Van Bergen, PC Consulting

Deep Learning with PyTorch offers a very pragmatic overview of deep learning. It is a didactical resource.
- Orlando Alejo Mendez Morales, Experian

Publisher resources

View/Submit Errata

Table of contents

  1. Copyright
  2. dedication
  3. contents
  4. front matter
    1. foreword
    2. preface
    3. acknowledgments
    4. about this book
      1. Who should read this book
      2. How this book is organized: A roadmap
      3. About the code
      4. Hardware and software requirements
      5. liveBook discussion forum
      6. Other online resources
    5. about the authors
    6. about the cover illustration
  5. Part 1. Core PyTorch
  6. 1 Introducing deep learning and the PyTorch Library
    1. 1.1 The deep learning revolution
    2. 1.2 PyTorch for deep learning
    3. 1.3 Why PyTorch?
      1. 1.3.1 The deep learning competitive landscape
    4. 1.4 An overview of how PyTorch supports deep learning projects
    5. 1.5 Hardware and software requirements
      1. 1.5.1 Using Jupyter Notebooks
    6. 1.6 Exercises
    7. 1.7 Summary
  7. 2 Pretrained networks
    1. 2.1 A pretrained network that recognizes the subject of an image
      1. 2.1.1 Obtaining a pretrained network for image recognition
      2. 2.1.2 AlexNet
      3. 2.1.3 ResNet
      4. 2.1.4 Ready, set, almost run
      5. 2.1.5 Run!
    2. 2.2 A pretrained model that fakes it until it makes it
      1. 2.2.1 The GAN game
      2. 2.2.2 CycleGAN
      3. 2.2.3 A network that turns horses into zebras
    3. 2.3 A pretrained network that describes scenes
      1. 2.3.1 NeuralTalk2
    4. 2.4 Torch Hub
    5. 2.5 Conclusion
    6. 2.6 Exercises
    7. 2.7 Summary
  8. 3 It starts with a tensor
    1. 3.1 The world as floating-point numbers
    2. 3.2 Tensors: Multidimensional arrays
      1. 3.2.1 From Python lists to PyTorch tensors
      2. 3.2.2 Constructing our first tensors
      3. 3.2.3 The essence of tensors
    3. 3.3 Indexing tensors
    4. 3.4 Named tensors
    5. 3.5 Tensor element types
      1. 3.5.1 Specifying the numeric type with dtype
      2. 3.5.2 A dtype for every occasion
      3. 3.5.3 Managing a tensor’s dtype attribute
    6. 3.6 The tensor API
    7. 3.7 Tensors: Scenic views of storage
      1. 3.7.1 Indexing into storage
      2. 3.7.2 Modifying stored values: In-place operations
    8. 3.8 Tensor metadata: Size, offset, and stride
      1. 3.8.1 Views of another tensor’s storage
      2. 3.8.2 Transposing without copying
      3. 3.8.3 Transposing in higher dimensions
      4. 3.8.4 Contiguous tensors
    9. 3.9 Moving tensors to the GPU
      1. 3.9.1 Managing a tensor’s device attribute
    10. 3.10 NumPy interoperability
    11. 3.11 Generalized tensors are tensors, too
    12. 3.12 Serializing tensors
      1. 3.12.1 Serializing to HDF5 with h5py
    13. 3.13 Conclusion
    14. 3.14 Exercises
    15. 3.15 Summary
  9. 4 Real-world data representation using tensors
    1. 4.1 Working with images
      1. 4.1.1 Adding color channels
      2. 4.1.2 Loading an image file
      3. 4.1.3 Changing the layout
      4. 4.1.4 Normalizing the data
    2. 4.2 3D images: Volumetric data
      1. 4.2.1 Loading a specialized format
    3. 4.3 Representing tabular data
      1. 4.3.1 Using a real-world dataset
      2. 4.3.2 Loading a wine data tensor
      3. 4.3.3 Representing scores
      4. 4.3.4 One-hot encoding
      5. 4.3.5 When to categorize
      6. 4.3.6 Finding thresholds
    4. 4.4 Working with time series
      1. 4.4.1 Adding a time dimension
      2. 4.4.2 Shaping the data by time period
      3. 4.4.3 Ready for training
    5. 4.5 Representing text
      1. 4.5.1 Converting text to numbers
      2. 4.5.2 One-hot-encoding characters
      3. 4.5.3 One-hot encoding whole words
      4. 4.5.4 Text embeddings
      5. 4.5.5 Text embeddings as a blueprint
    6. 4.6 Conclusion
    7. 4.7 Exercises
    8. 4.8 Summary
  10. 5 The mechanics of learning
    1. 5.1 A timeless lesson in modeling
    2. 5.2 Learning is just parameter estimation
      1. 5.2.1 A hot problem
      2. 5.2.2 Gathering some data
      3. 5.2.3 Visualizing the data
      4. 5.2.4 Choosing a linear model as a first try
    3. 5.3 Less loss is what we want
      1. 5.3.1 From problem back to PyTorch
    4. 5.4 Down along the gradient
      1. 5.4.1 Decreasing loss
      2. 5.4.2 Getting analytical
      3. 5.4.3 Iterating to fit the model
      4. 5.4.4 Normalizing inputs
      5. 5.4.5 Visualizing (again)
    5. 5.5 PyTorch’s autograd: Backpropagating all things
      1. 5.5.1 Computing the gradient automatically
      2. 5.5.2 Optimizers a la carte
      3. 5.5.3 Training, validation, and overfitting
      4. 5.5.4 Autograd nits and switching it off
    6. 5.6 Conclusion
    7. 5.7 Exercise
    8. 5.8 Summary
  11. 6 Using a neural network to fit the data
    1. 6.1 Artificial neurons
      1. 6.1.1 Composing a multilayer network
      2. 6.1.2 Understanding the error function
      3. 6.1.3 All we need is activation
      4. 6.1.4 More activation functions
      5. 6.1.5 Choosing the best activation function
      6. 6.1.6 What learning means for a neural network
    2. 6.2 The PyTorch nn module
      1. 6.2.1 Using __call__ rather than forward
      2. 6.2.2 Returning to the linear model
    3. 6.3 Finally a neural network
      1. 6.3.1 Replacing the linear model
      2. 6.3.2 Inspecting the parameters
      3. 6.3.3 Comparing to the linear model
    4. 6.4 Conclusion
    5. 6.5 Exercises
    6. 6.6 Summary
  12. 7 Telling birds from airplanes: Learning from images
    1. 7.1 A dataset of tiny images
      1. 7.1.1 Downloading CIFAR-10
      2. 7.1.2 The Dataset class
      3. 7.1.3 Dataset transforms
      4. 7.1.4 Normalizing data
    2. 7.2 Distinguishing birds from airplanes
      1. 7.2.1 Building the dataset
      2. 7.2.2 A fully connected model
      3. 7.2.3 Output of a classifier
      4. 7.2.4 Representing the output as probabilities
      5. 7.2.5 A loss for classifying
      6. 7.2.6 Training the classifier
      7. 7.2.7 The limits of going fully connected
    3. 7.3 Conclusion
    4. 7.4 Exercises
    5. 7.5 Summary
  13. 8 Using convolutions to generalize
    1. 8.1 The case for convolutions
      1. 8.1.1 What convolutions do
    2. 8.2 Convolutions in action
      1. 8.2.1 Padding the boundary
      2. 8.2.2 Detecting features with convolutions
      3. 8.2.3 Looking further with depth and pooling
      4. 8.2.4 Putting it all together for our network
    3. 8.3 Subclassing nn.Module
      1. 8.3.1 Our network as an nn.Module
      2. 8.3.2 How PyTorch keeps track of parameters and submodules
      3. 8.3.3 The functional API
    4. 8.4 Training our convnet
      1. 8.4.1 Measuring accuracy
      2. 8.4.2 Saving and loading our model
      3. 8.4.3 Training on the GPU
    5. 8.5 Model design
      1. 8.5.1 Adding memory capacity: Width
      2. 8.5.2 Helping our model to converge and generalize: Regularization
      3. 8.5.3 Going deeper to learn more complex structures: Depth
      4. 8.5.4 Comparing the designs from this section
      5. 8.5.5 It’s already outdated
    6. 8.6 Conclusion
    7. 8.7 Exercises
    8. 8.8 Summary
  14. Part 2. Learning from images in the real world: Early detection of lung cancer
  15. 9 Using PyTorch to fight cancer
    1. 9.1 Introduction to the use case
    2. 9.2 Preparing for a large-scale project
    3. 9.3 What is a CT scan, exactly?
    4. 9.4 The project: An end-to-end detector for lung cancer
      1. 9.4.1 Why can’t we just throw data at a neural network until it works?
      2. 9.4.2 What is a nodule?
      3. 9.4.3 Our data source: The LUNA Grand Challenge
      4. 9.4.4 Downloading the LUNA data
    5. 9.5 Conclusion
    6. 9.6 Summary
  16. 10 Combining data sources into a unified dataset
    1. 10.1 Raw CT data files
    2. 10.2 Parsing LUNA’s annotation data
      1. 10.2.1 Training and validation sets
      2. 10.2.2 Unifying our annotation and candidate data
    3. 10.3 Loading individual CT scans
      1. 10.3.1 Hounsfield Units
    4. 10.4 Locating a nodule using the patient coordinate system
      1. 10.4.1 The patient coordinate system
      2. 10.4.2 CT scan shape and voxel sizes
      3. 10.4.3 Converting between millimeters and voxel addresses
      4. 10.4.4 Extracting a nodule from a CT scan
    5. 10.5 A straightforward dataset implementation
      1. 10.5.1 Caching candidate arrays with the getCtRawCandidate function
      2. 10.5.2 Constructing our dataset in LunaDataset.__init__
      3. 10.5.3 A training/validation split
      4. 10.5.4 Rendering the data
    6. 10.6 Conclusion
    7. 10.7 Exercises
    8. Summary
  17. 11 Training a classification model to detect suspected tumors
    1. 11.1 A foundational model and training loop
    2. 11.2 The main entry point for our application
    3. 11.3 Pretraining setup and initialization
      1. 11.3.1 Initializing the model and optimizer
      2. 11.3.2 Care and feeding of data loaders
    4. 11.4 Our first-pass neural network design
      1. 11.4.1 The core convolutions
      2. 11.4.2 The full model
    5. 11.5 Training and validating the model
      1. 11.5.1 The computeBatchLoss function
      2. 11.5.2 The validation loop is similar
    6. 11.6 Outputting performance metrics
      1. 11.6.1 The logMetrics function
    7. 11.7 Running the training script
      1. 11.7.1 Needed data for training
      2. 11.7.2 Interlude: The enumerateWithEstimate function
    8. 11.8 Evaluating the model: Getting 99.7% correct means we’re done, right?
    9. 11.9 Graphing training metrics with TensorBoard
      1. 11.9.1 Running TensorBoard
      2. 11.9.2 Adding TensorBoard support to the metrics logging function
    10. 11.10 Why isn’t the model learning to detect nodules?
    11. 11.11 Conclusion
    12. 11.12 Exercises
    13. 11.13 Summary
  18. 12 Improving training with metrics and augmentation
    1. 12.1 High-level plan for improvement
    2. 12.2 Good dogs vs. bad guys: False positives and false negatives
    3. 12.3 Graphing the positives and negatives
      1. 12.3.1 Recall is Roxie’s strength
      2. 12.3.2 Precision is Preston’s forte
      3. 12.3.3 Implementing precision and recall in logMetrics
      4. 12.3.4 Our ultimate performance metric: The F1 score
      5. 12.3.5 How does our model perform with our new metrics?
    4. 12.4 What does an ideal dataset look like?
      1. 12.4.1 Making the data look less like the actual and more like the “ideal”
      2. 12.4.2 Contrasting training with a balanced LunaDataset to previous runs
      3. 12.4.3 Recognizing the symptoms of overfitting
    5. 12.5 Revisiting the problem of overfitting
      1. 12.5.1 An overfit face-to-age prediction model
    6. 12.6 Preventing overfitting with data augmentation
      1. 12.6.1 Specific data augmentation techniques
      2. 12.6.2 Seeing the improvement from data augmentation
    7. 12.7 Conclusion
    8. 12.8 Exercises
    9. 12.9 Summary
  19. 13 Using segmentationto find suspected nodules
    1. 13.1 Adding a second model to our project
    2. 13.2 Various types of segmentation
    3. 13.3 Semantic segmentation: Per-pixel classification
      1. 13.3.1 The U-Net architecture
    4. 13.4 Updating the model for segmentation
      1. 13.4.1 Adapting an off-the-shelf model to our project
    5. 13.5 Updating the dataset for segmentation
      1. 13.5.1 U-Net has very specific input size requirements
      2. 13.5.2 U-Net trade-offs for 3D vs. 2D data
      3. 13.5.3 Building the ground truth data
      4. 13.5.4 Implementing Luna2dSegmentationDataset
      5. 13.5.5 Designing our training and validation data
      6. 13.5.6 Implementing TrainingLuna2dSegmentationDataset
      7. 13.5.7 Augmenting on the GPU
    6. 13.6 Updating the training script for segmentation
      1. 13.6.1 Initializing our segmentation and augmentation models
      2. 13.6.2 Using the Adam optimizer
      3. 13.6.3 Dice loss
      4. 13.6.4 Getting images into TensorBoard
      5. 13.6.5 Updating our metrics logging
      6. 13.6.6 Saving our model
    7. 13.7 Results
    8. 13.8 Conclusion
    9. 13.9 Exercises
    10. 13.10 Summary
  20. 14 End-to-end nodule analysis, and where to go next
    1. 14.1 Towards the finish line
    2. 14.2 Independence of the validation set
    3. 14.3 Bridging CT segmentation and nodule candidate classification
      1. 14.3.1 Segmentation
      2. 14.3.2 Grouping voxels into nodule candidates
      3. 14.3.3 Did we find a nodule? Classification to reduce false positives
    4. 14.4 Quantitative validation
    5. 14.5 Predicting malignancy
      1. 14.5.1 Getting malignancy information
      2. 14.5.2 An area under the curve baseline: Classifying by diameter
      3. 14.5.3 Reusing preexisting weights: Fine-tuning
      4. 14.5.4 More output in TensorBoard
    6. 14.6 What we see when we diagnose
      1. 14.6.1 Training, validation, and test sets
    7. 14.7 What next? Additional sources of inspiration (and data)
      1. 14.7.1 Preventing overfitting: Better regularization
      2. 14.7.2 Refined training data
      3. 14.7.3 Competition results and research papers
    8. 14.8 Conclusion
      1. 14.8.1 Behind the curtain
    9. 14.9 Exercises
    10. 14.10 Summary
  21. Part 3. Deployment
  22. 15 Deploying to production
    1. 15.1 Serving PyTorch models
      1. 15.1.1 Our model behind a Flask server
      2. 15.1.2 What we want from deployment
      3. 15.1.3 Request batching
    2. 15.2 Exporting models
      1. 15.2.1 Interoperability beyond PyTorch with ONNX
      2. 15.2.2 PyTorch’s own export: Tracing
      3. 15.2.3 Our server with a traced model
    3. 15.3 Interacting with the PyTorch JIT
      1. 15.3.1 What to expect from moving beyond classic Python/PyTorch
      2. 15.3.2 The dual nature of PyTorch as interface and backend
      3. 15.3.3 TorchScript
      4. 15.3.4 Scripting the gaps of traceability
    4. 15.4 LibTorch: PyTorch in C++
      1. 15.4.1 Running JITed models from C++
      2. 15.4.2 C++ from the start: The C++ API
    5. 15.5 Going mobile
      1. 15.5.1 Improving efficiency: Model design and quantization
    6. 15.6 Emerging technology: Enterprise serving of PyTorch models
    7. 15.7 Conclusion
    8. 15.8 Exercises
    9. 15.9 Summary
  23. index

Product information

  • Title: Deep Learning with PyTorch
  • Author(s): Eli Stevens, Thomas Viehmann, Luca Pietro Giovanni Antiga
  • Release date: July 2020
  • Publisher(s): Manning Publications
  • ISBN: 9781617295263