3D Deep Learning with Python

3D Deep Learning with Python

by Xudong Ma, Vishakh Hegde, Lilit Yolyan
Released October 2022
Publisher(s): Packt Publishing
ISBN: 9781803247823

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Visualize and build deep learning models with 3D data using PyTorch3D and other Python frameworks to conquer real-world application challenges with ease

Key Features

  • Understand 3D data processing with rendering, PyTorch optimization, and heterogeneous batching
  • Implement differentiable rendering concepts with practical examples
  • Discover how you can ease your work with the latest 3D deep learning techniques using PyTorch3D

Book Description

With this hands-on guide to 3D deep learning, developers working with 3D computer vision will be able to put their knowledge to work and get up and running in no time.

Complete with step-by-step explanations of essential concepts and practical examples, this book lets you explore and gain a thorough understanding of state-of-the-art 3D deep learning. You'll see how to use PyTorch3D for basic 3D mesh and point cloud data processing, including loading and saving ply and obj files, projecting 3D points into camera coordination using perspective camera models or orthographic camera models, rendering point clouds and meshes to images, and much more. As you implement some of the latest 3D deep learning algorithms, such as differential rendering, Nerf, synsin, and mesh RCNN, you'll realize how coding for these deep learning models becomes easier using the PyTorch3D library.

By the end of this deep learning book, you'll be ready to implement your own 3D deep learning models confidently.

What you will learn

  • Develop 3D computer vision models for interacting with the environment
  • Get to grips with 3D data handling with point clouds, meshes, ply, and obj file format
  • Work with 3D geometry, camera models, and coordination and convert between them
  • Understand concepts of rendering, shading, and more with ease
  • Implement differential rendering for many 3D deep learning models
  • Advanced state-of-the-art 3D deep learning models like Nerf, synsin, mesh RCNN

Who this book is for

This book is for beginner to intermediate-level machine learning practitioners, data scientists, ML engineers, and DL engineers who are looking to become well-versed with computer vision techniques using 3D data.

Table of contents

  1. 3D Deep Learning with Python
  2. Contributors
  3. About the author
  4. About the reviewer
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Share Your Thoughts
    9. Download a free PDF copy of this book
  6. PART 1: 3D Data Processing Basics
  7. Chapter 1: Introducing 3D Data Processing
    1. Technical requirements
    2. Setting up a development environment
    3. 3D data representation
      1. Understanding point cloud representation
      2. Understanding mesh representation
      3. Understanding voxel representation
    4. 3D data file format – Ply files
    5. 3D data file format – OBJ files
    6. Understanding 3D coordination systems
    7. Understanding camera models
    8. Coding for camera models and coordination systems
    9. Summary
  8. Chapter 2: Introducing 3D Computer Vision and Geometry
    1. Technical requirements
    2. Exploring the basic concepts of rendering, rasterization, and shading
      1. Understanding barycentric coordinates
      2. Light source models
      3. Understanding the Lambertian shading model
      4. Understanding the Phong lighting model
    3. Coding exercises for 3D rendering
    4. Using PyTorch3D heterogeneous batches and PyTorch optimizers
      1. A coding exercise for a heterogeneous mini-batch
    5. Understanding transformations and rotations
      1. A coding exercise for transformation and rotation
    6. Summary
  9. PART 2: 3D Deep Learning Using PyTorch3D
  10. Chapter 3: Fitting Deformable Mesh Models to Raw Point Clouds
    1. Technical requirements
    2. Fitting meshes to point clouds – the problem
    3. Formulating a deformable mesh fitting problem into an optimization problem
    4. Loss functions for regularization
      1. Mesh Laplacian smoothing loss
      2. Mesh normal consistency loss
      3. Mesh edge loss
    5. Implementing the mesh fitting with PyTorch3D
      1. The experiment of not using any regularization loss functions
      2. The experiment of using only the mesh edge loss
    6. Summary
  11. Chapter 4: Learning Object Pose Detection and Tracking by Differentiable Rendering
    1. Technical requirements
    2. Why we want to have differentiable rendering
    3. How to make rendering differentiable
      1. What problems can be solved by using differentiable rendering
    4. The object pose estimation problem
    5. How it is coded
      1. An example of object pose estimation for both silhouette fitting and texture fitting
    6. Summary
  12. Chapter 5: Understanding Differentiable Volumetric Rendering
    1. Technical requirements
    2. Overview of volumetric rendering
    3. Understanding ray sampling
    4. Using volume sampling
    5. Exploring the ray marcher
    6. Differentiable volumetric rendering
      1. Reconstructing 3D models from multi-view images
    7. Summary
  13. Chapter 6: Exploring Neural Radiance Fields (NeRF)
    1. Technical requirements
    2. Understanding NeRF
      1. What is a radiance field?
      2. Representing radiance fields with neural networks
    3. Training a NeRF model
    4. Understanding the NeRF model architecture
    5. Understanding volume rendering with radiance fields
      1. Projecting rays into the scene
      2. Accumulating the color of a ray
    6. Summary
  14. PART 3: State-of-the-art 3D Deep Learning Using PyTorch3D
  15. Chapter 7: Exploring Controllable Neural Feature Fields
    1. Technical requirements
    2. Understanding GAN-based image synthesis
    3. Introducing compositional 3D-aware image synthesis
    4. Generating feature fields
    5. Mapping feature fields to images
    6. Exploring controllable scene generation
      1. Exploring controllable car generation
      2. Exploring controllable face generation
    7. Training the GIRAFFE model
      1. Frechet Inception Distance
      2. Training the model
    8. Summary
  16. Chapter 8: Modeling the Human Body in 3D
    1. Technical requirements
    2. Formulating the 3D modeling problem
      1. Defining a good representation
    3. Understanding the Linear Blend Skinning technique
    4. Understanding the SMPL model
      1. Defining the SMPL model
    5. Using the SMPL model
    6. Estimating 3D human pose and shape using SMPLify
      1. Defining the optimization objective function
    7. Exploring SMPLify
      1. Running the code
      2. Exploring the code
    8. Summary
  17. Chapter 9: Performing End-to-End View Synthesis with SynSin
    1. Technical requirements
    2. Overview of view synthesis
    3. SynSin network architecture
      1. Spatial feature and depth networks
      2. Neural point cloud renderer
      3. Refinement module and discriminator
    4. Hands-on model training and testing
    5. Summary
  18. Chapter 10: Mesh R-CNN
    1. Technical requirements
    2. Overview of meshes and voxels
    3. Mesh R-CNN architecture
      1. Graph convolutions
      2. Mesh predictor
    4. Demo of Mesh R-CNN with PyTorch
      1. Demo
    5. Summary
  19. Index
    1. Why subscribe?
  20. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share Your Thoughts
    3. Download a free PDF copy of this book

Product information

  • Title: 3D Deep Learning with Python
  • Author(s): Xudong Ma, Vishakh Hegde, Lilit Yolyan
  • Release date: October 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781803247823