Advanced Python Programming - Second Edition

Advanced Python Programming - Second Edition

by Quan Nguyen
Released March 2022
Publisher(s): Packt Publishing
ISBN: 9781801814010

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Write fast, robust, and highly reusable applications using Python's internal optimization, state-of-the-art performance-benchmarking tools, and cutting-edge libraries

Key Features

  • Benchmark, profile, and accelerate Python programs using optimization tools
  • Scale applications to multiple processors with concurrent programming
  • Make applications robust and reusable using effective design patterns

Book Description

Python's powerful capabilities for implementing robust and efficient programs make it one of the most sought-after programming languages.

In this book, you'll explore the tools that allow you to improve performance and take your Python programs to the next level.

This book starts by examining the built-in as well as external libraries that streamline tasks in the development cycle, such as benchmarking, profiling, and optimizing. You'll then get to grips with using specialized tools such as dedicated libraries and compilers to increase your performance at number-crunching tasks, including training machine learning models.

The book covers concurrency, a major solution to making programs more efficient and scalable, and various concurrent programming techniques such as multithreading, multiprocessing, and asynchronous programming.

You'll also understand the common problems that cause undesirable behavior in concurrent programs.

Finally, you'll work with a wide range of design patterns, including creational, structural, and behavioral patterns that enable you to tackle complex design and architecture challenges, making your programs more robust and maintainable.

By the end of the book, you'll be exposed to a wide range of advanced functionalities in Python and be equipped with the practical knowledge needed to apply them to your use cases.

What you will learn

  • Write efficient numerical code with NumPy, pandas, and Xarray
  • Use Cython and Numba to achieve native performance
  • Find bottlenecks in your Python code using profilers
  • Optimize your machine learning models with JAX
  • Implement multithreaded, multiprocessing, and asynchronous programs
  • Solve common problems in concurrent programming, such as deadlocks
  • Tackle architecture challenges with design patterns

Who this book is for

This book is for intermediate to experienced Python programmers who are looking to scale up their applications in a systematic and robust manner. Programmers from a range of backgrounds will find this book useful, including software engineers, scientific programmers, and software architects.

Table of contents

  1. Advanced Python Programming
  2. Second Edition
  3. Contributors
  4. About the author
  5. About the reviewers
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Share Your Thoughts
  7. Section 1: Python-Native and Specialized Optimization
  8. Chapter 1: Benchmarking and Profiling
    1. Technical requirements
    2. Designing your application
      1. Building a particle simulator
      2. Visualizing the simulation
    3. Writing tests and benchmarks
      1. Timing your benchmark
    4. Writing better tests and benchmarks with pytest-benchmark
    5. Finding bottlenecks with cProfile
      1. Graphically analyzing profiling results
      2. Profiling line by line with line_profiler
    6. Optimizing our code
    7. Using the dis module
    8. Profiling memory usage with memory_profiler
    9. Summary
    10. Questions
    11. Further reading
  9. Chapter 2: Pure Python Optimizations
    1. Technical requirements
    2. Using the right algorithms and data structures
      1. Lists and deques
      2. Dictionaries
        1. Hash map
      3. Sets
      4. Heaps
      5. Tries
    3. Improved efficiency with caching and memoization
      1. Joblib
    4. Efficient iteration with comprehensions and generators
    5. Summary
    6. Questions
    7. Further reading
  10. Chapter 3: Fast Array Operations with NumPy, Pandas, and Xarray
    1. Technical requirement
    2. Getting started with NumPy
      1. Creating arrays
      2. Accessing arrays
        1. Indexing and slicing
        2. Fancy indexing
      3. Broadcasting
      4. Mathematical operations
      5. Calculating the norm
    3. Rewriting the particle simulator in NumPy
    4. Reaching optimal performance with numexpr
    5. Working with database-style data with pandas
      1. pandas fundamentals
        1. Indexing Series and DataFrame objects
      2. Database-style operations with pandas
        1. Mapping
        2. Grouping, aggregations, and transforms
        3. Joining
    6. High-performance labeled data with xarray
      1. Analyzing concentration
      2. The xarray library
      3. Improved performance
      4. Plotting with xarray
    7. Summary
    8. Questions
    9. Further reading
  11. Chapter 4: C Performance with Cython
    1. Technical requirements
    2. Compiling Cython extensions
    3. Adding static types
      1. Declaring variables
      2. Declaring functions
      3. Declaring classes
    4. Sharing declarations
    5. Working with arrays
      1. C arrays and pointers
      2. Working with NumPy arrays
      3. Working with typed memoryviews
    6. Using a particle simulator in Cython
    7. Profiling Cython
    8. Using Cython with Jupyter
    9. Summary
    10. Questions
  12. Chapter 5: Exploring Compilers
    1. Technical requirements
    2. Getting started with Numba
      1. Using Numba decorators
      2. Type specializations
      3. Object mode versus native mode
      4. Numba and NumPy
        1. Universal functions with Numba
        2. Generalized universal functions
      5. JIT classes
      6. Limitations in Numba
    3. The PyPy project
      1. Setting up PyPy
      2. Running a particle simulator in PyPy
    4. Other interesting projects
    5. Summary
    6. Questions
    7. Further reading
  13. Chapter 6: Automatic Differentiation and Accelerated Linear Algebra for Machine Learning
    1. A crash course in machine learning
      1. Model parameters
      2. Loss function
      3. Loss minimization
    2. Getting JAX up and running
      1. Installing JAX
      2. Using Google Colab
    3. Automatic differentiation for loss minimization
      1. Making the dataset
      2. Building a linear model
      3. Gradient descent with automatic differentiation
    4. Just-In-Time compilation for improved efficiency
    5. Automatic vectorization for efficient kernels
      1. Data that is not linearly separable
      2. The kernel method in machine learning
      3. Automatic vectorization for kernelized models
    6. Summary
    7. Questions
    8. Further reading
  14. Section 2: Concurrency and Parallelism
  15. Chapter 7: Implementing Concurrency
    1. Technical requirements
    2. Asynchronous programming
      1. Waiting for input/output
      2. Concurrency
      3. Callbacks
      4. Futures
      5. Event loops
    3. The asyncio framework
      1. Coroutines
      2. Converting blocking code into non-blocking code
    4. Reactive programming
      1. Observables
      2. Useful operators
      3. Hot and cold observables
      4. Building a CPU monitor
    5. Summary
    6. Questions
    7. Further reading
  16. Chapter 8: Parallel Processing
    1. Technical requirements
    2. Introduction to parallel programming
      1. GPUs
    3. Using multiple processes
      1. The Process and Pool classes
      2. The Executor interface
      3. Monte Carlo approximation of pi
      4. Synchronization and locks
    4. Parallel Cython with OpenMP
    5. Automatic parallelism
      1. Getting started with Theano
      2. Profiling Theano
      3. TensorFlow
      4. Running code on a GPU
    6. Summary
    7. Questions
  17. Chapter 9: Concurrent Web Requests
    1. The basics of web requests
      1. HTML
      2. HTTP requests
      3. HTTP status code
    2. The requests module
      1. Making a request in Python
      2. Running a ping test
    3. Concurrent web requests
      1. Spawning multiple threads
      2. Refactoring request logic
    4. The problem with timeouts
      1. Support from httpstat.us and simulation in Python
      2. Timeout specifications
    5. Good practices in making web requests
      1. Consider the terms of service and data-collecting policies
      2. Error handling
      3. Update your program regularly
      4. Avoid making a large number of requests
    6. Summary
    7. Questions
    8. Further reading
  18. Chapter 10: Concurrent Image Processing
    1. Technical requirements
    2. Image processing fundamentals
      1. Python as an image processing tool
        1. Installing OpenCV and NumPy
      2. Computer image basics
        1. RGB values
        2. Pixels and image files
        3. Coordinates inside an image
      3. OpenCV API
      4. Image processing techniques
        1. Grayscaling
        2. Thresholding
    3. Applying concurrency to image processing
    4. Good concurrent image processing practices
      1. Choosing the correct way (out of many)
      2. Spawning an appropriate number of processes
      3. Processing input/output concurrently
    5. Summary
    6. Questions
    7. Further reading
  19. Chapter 11: Building Communication Channels with asyncio
    1. Technical requirements
    2. The ecosystem of communication channels
      1. Communication protocol layers
      2. Asynchronous programming for communication channels
      3. Transports and protocols in asyncio
      4. The big picture of asyncio's server client
    3. Getting started with Python and Telnet
      1. Starting a server
      2. Installing Telnet
      3. Simulating a connection channel
      4. Sending messages back to clients
      5. Closing transports
    4. Client-side communication with aiohttp
      1. Installing aiohttp and aiofiles
      2. Fetching a website's HTML code
      3. Writing files asynchronously
    5. Summary
    6. Questions
    7. Further reading
  20. Chapter 12: Deadlocks
    1. Technical requirements
    2. The concept of deadlocks
      1. The dining philosophers problem
      2. A deadlock in a concurrent system
      3. Python simulation
    3. Approaches to deadlock situations
      1. Implementing ranking among resources
      2. Ignoring locks and sharing resources
        1. An additional note about locks
        2. Concluding note on deadlock solutions
    4. The concept of livelocks
    5. Summary
    6. Questions
    7. Further reading
  21. Chapter 13: Starvation
    1. Technical requirements
    2. Understanding starvation
      1. What is starvation?
      2. Scheduling
      3. Causes of starvation
      4. Starvation's relationship to deadlock
    3. Approaching the readers-writers problem
      1. Problem statement
      2. The first readers-writers problem
      3. The second readers-writers problem
      4. The third readers-writers problem
    4. Solutions to starvation
    5. Summary
    6. Questions
    7. Further reading
  22. Chapter 14: Race Conditions
    1. Technical requirements
    2. The concept of race conditions
      1. Critical sections
      2. How race conditions occur
    3. Simulating race conditions in Python
    4. Locks as a solution to race conditions
      1. The effectiveness of locks
      2. Implementation in Python
      3. The downside of locks
        1. Turning a concurrent program into a sequential program
        2. Locks do not lock anything
    5. Race conditions in real life
      1. Security
      2. Operating systems
      3. Networking
    6. Summary
    7. Questions
    8. Further reading
  23. Chapter 15: The Global Interpreter Lock
    1. Technical requirements
    2. Introducing the GIL
      1. Analyzing memory management in Python
      2. The problem that the GIL addresses
      3. Problems raised by the GIL
    3. The potential removal of the GIL from Python
    4. Working with the GIL
      1. Implementing multiprocessing, rather than multithreading
      2. Getting around the GIL with native extensions
      3. Utilizing a different Python interpreter
    5. Summary
    6. Questions
    7. Further reading
  24. Section 3: Design Patterns in Python
  25. Chapter 16: The Factory Pattern
    1. Technical requirements
    2. Understanding design patterns
    3. Implementing the factory method
      1. Real-world examples
      2. Use cases
      3. Implementing the factory method
    4. Applying the abstract factory
      1. Real-world examples
      2. Use cases
      3. Implementing the abstract factory pattern
    5. Summary
    6. Questions
  26. Chapter 17: The Builder Pattern
    1. Technical requirements
    2. Understanding the builder pattern
    3. Real-world examples
    4. Use cases
    5. Implementing an ordering application
    6. Summary
    7. Questions
  27. Chapter 18: Other Creational Patterns
    1. Technical requirements
    2. Implementing the prototype pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    3. Implementing the singleton pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    4. Summary
    5. Questions
    6. Further reading
  28. Chapter 19: The Adapter Pattern
    1. Technical requirements
    2. Understanding the adapter pattern
    3. Real-world examples
    4. Use cases
    5. Implementation
    6. Summary
  29. Chapter 20: The Decorator Pattern
    1. Technical requirements
    2. Introducing the decorator pattern
    3. Real-world examples
    4. Use cases
    5. Implementation
    6. Summary
    7. Questions
  30. Chapter 21: The Bridge Pattern
    1. Technical requirements
    2. Real-world examples
    3. Use cases
    4. Implementation
    5. Summary
    6. Questions
  31. Chapter 22: The Façade Pattern
    1. Technical requirements
    2. Understanding the façade pattern
    3. Real-world examples
    4. Use cases
    5. Implementation
    6. Summary
    7. Questions
    8. Further reading
  32. Chapter 23: Other Structural Patterns
    1. Technical requirements
    2. Implementing the flyweight pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    3. Implementing the model-view-controller pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    4. Applying the proxy pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    5. Summary
    6. Questions
  33. Chapter 24: The Chain of Responsibility Pattern
    1. Technical requirements
    2. Understanding the Chain of Responsibility pattern
    3. Real-world examples
    4. Use cases
    5. Implementation
    6. Summary
    7. Questions
  34. Chapter 25: The Command Pattern
    1. Technical requirements
    2. Understanding the command pattern
    3. Real-world examples
    4. Use cases
    5. Implementation
    6. Summary
    7. Questions
  35. Chapter 26: The Observer Pattern
    1. Technical requirements
    2. Understanding the observer pattern
    3. Real-world examples
    4. Use cases
    5. Implementation
    6. Summary
    7. Questions
  36. Assessments
    1. Chapter 1
    2. Chapter 2
    3. Chapter 3
    4. Chapter 4
    5. Chapter 5
    6. Chapter 6
    7. Chapter 7
    8. Chapter 8
    9. Chapter 9
    10. Chapter 10
    11. Chapter 11
    12. Chapter 12
    13. Chapter 13
    14. Chapter 14
    15. Chapter 15
    16. Chapter 16
    17. Chapter 17
    18. Chapter 18
    19. Chapter 20
    20. Chapter 21
    21. Chapter 22
    22. Chapter 23
    23. Chapter 24
    24. Chapter 25
    25. Chapter 26
    26. Why subscribe?
  37. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share Your Thoughts

Product information

  • Title: Advanced Python Programming - Second Edition
  • Author(s): Quan Nguyen
  • Release date: March 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781801814010