Advanced Python Programming

Advanced Python Programming

by Dr. Gabriele Lanaro, Quan Nguyen, Sakis Kasampalis
Released February 2019
Publisher(s): Packt Publishing
ISBN: 9781838551216

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Create distributed applications with clever design patterns to solve complex problems

Key Features

  • Set up and run distributed algorithms on a cluster using Dask and PySpark
  • Master skills to accurately implement concurrency in your code
  • Gain practical experience of Python design patterns with real-world examples

Book Description

This Learning Path shows you how to leverage the power of both native and third-party Python libraries for building robust and responsive applications. You will learn about profilers and reactive programming, concurrency and parallelism, as well as tools for making your apps quick and efficient. You will discover how to write code for parallel architectures using TensorFlow and Theano, and use a cluster of computers for large-scale computations using technologies such as Dask and PySpark. With the knowledge of how Python design patterns work, you will be able to clone objects, secure interfaces, dynamically choose algorithms, and accomplish much more in high performance computing.

By the end of this Learning Path, you will have the skills and confidence to build engaging models that quickly offer efficient solutions to your problems.

This Learning Path includes content from the following Packt products:

? Python High Performance - Second Edition by Gabriele Lanaro

? Mastering Concurrency in Python by Quan Nguyen

? Mastering Python Design Patterns by Sakis Kasampalis

What you will learn

  • Use NumPy and pandas to import and manipulate datasets
  • Achieve native performance with Cython and Numba
  • Write asynchronous code using asyncio and RxPy
  • Design highly scalable programs with application scaffolding
  • Explore abstract methods to maintain data consistency
  • Clone objects using the prototype pattern
  • Use the adapter pattern to make incompatible interfaces compatible
  • Employ the strategy pattern to dynamically choose an algorithm

Who this book is for

This Learning Path is specially designed for Python developers who want to build high-performance applications and learn about single core and multi-core programming, distributed concurrency, and Python design patterns. Some experience with Python programming language will help you get the most out of this Learning Path.

Table of contents

  1. Title Page
  2. Copyright
    1. Advanced Python Programming
  3. About Packt
    1. Why Subscribe?
    2. Packt.com
  4. Contributors
    1. About the Authors
    2. Packt Is Searching for Authors Like You
  5. Preface
    1. Who This Book Is For
    2. What This Book Covers
    3. To Get the Most out of This Book
      1. Download the Example Code Files
      2. Conventions Used
    4. Get in Touch
      1. Reviews
  6. Benchmarking and Profiling
    1. Designing your application
    2. Writing tests and benchmarks
      1. Timing your benchmark
    3. Better tests and benchmarks with pytest-benchmark
    4. Finding bottlenecks with cProfile
    5. Profile line by line with line_profiler
    6. Optimizing our code
    7. The dis module
    8. Profiling memory usage with memory_profiler
    9. Summary
  7. Pure Python Optimizations
    1. Useful algorithms and data structures
      1. Lists and deques
      2. Dictionaries
        1. Building an in-memory search index using a hash map
      3. Sets
      4. Heaps
      5. Tries
    2. Caching and memoization
      1. Joblib
    3. Comprehensions and generators
    4. Summary
  8. Fast Array Operations with NumPy and Pandas
    1. Getting started with NumPy
      1. Creating arrays
      2. Accessing arrays
      3. Broadcasting
      4. Mathematical operations
      5. Calculating the norm
    2. Rewriting the particle simulator in NumPy
    3. Reaching optimal performance with numexpr
    4. Pandas
      1. Pandas fundamentals
        1. Indexing Series and DataFrame objects
      2. Database-style operations with Pandas
        1. Mapping
        2. Grouping, aggregations, and transforms
        3. Joining
    5. Summary
  9. C Performance with Cython
    1. Compiling Cython extensions
    2. Adding static types
      1. Variables
      2. Functions
      3. Classes
    3. Sharing declarations
    4. Working with arrays
      1. C arrays and pointers
      2. NumPy arrays
      3. Typed memoryviews
    5. Particle simulator in Cython
    6. Profiling Cython
    7. Using Cython with Jupyter
    8. Summary
  10. Exploring Compilers
    1. Numba
      1. First steps with Numba
      2. Type specializations
      3. Object mode versus native mode
      4. Numba and NumPy
        1. Universal functions with Numba
        2. Generalized universal functions
      5. JIT classes
      6. Limitations in Numba
    2. The PyPy project
      1. Setting up PyPy
      2. Running a particle simulator in PyPy
    3. Other interesting projects
    4. Summary
  11. Implementing Concurrency
    1. Asynchronous programming
      1. Waiting for I/O
      2. Concurrency
      3. Callbacks
      4. Futures
      5. Event loops
    2. The asyncio framework
      1. Coroutines
      2. Converting blocking code into non-blocking code
    3. Reactive programming
      1. Observables
      2. Useful operators
      3. Hot and cold observables
      4. Building a CPU monitor
    4. Summary
  12. Parallel Processing
    1. Introduction to parallel programming
      1. Graphic processing units
    2. Using multiple processes
      1. The Process and Pool classes
      2. The Executor interface
      3. Monte Carlo approximation of pi
      4. Synchronization and locks
    3. Parallel Cython with OpenMP
    4. Automatic parallelism
      1. Getting started with Theano
        1. Profiling Theano
      2. Tensorflow
      3. Running code on a GPU
    5. Summary
  13. Advanced Introduction to Concurrent and Parallel Programming
    1. Technical requirements
    2. What is concurrency?
      1. Concurrent versus sequential
      2. Example 1 – checking whether a non-negative number is prime
      3. Concurrent versus parallel
        1. A quick metaphor
    3. Not everything should be made concurrent
      1. Embarrassingly parallel
      2. Inherently sequential
        1. Example 2 – inherently sequential tasks
      3. I/O bound
    4. The history, present, and future of concurrency
      1. The history of concurrency
      2. The present
      3. The future
    5. A brief overview of mastering concurrency in Python
      1. Why Python?
    6. Setting up your Python environment
      1. General setup
    7. Summary
    8. Questions
    9. Further reading
  14. Amdahl's Law
    1. Technical requirements
    2. Amdahl's Law
      1. Terminology
    3. Formula and interpretation
      1. The formula for Amdahl's Law
        1. A quick example
      2. Implications
    4. Amdahl's Law's relationship to the law of diminishing returns
    5. How to simulate in Python
    6. Practical applications of Amdahl's Law
    7. Summary
    8. Questions
    9. Further reading
  15. Working with Threads in Python
    1. Technical requirements
    2. The concept of a thread
      1. Threads versus processes
      2. Multithreading
      3. An example in Python
    3. An overview of the threading module
      1. The thread module in Python 2
      2. The threading module in Python 3
    4. Creating a new thread in Python
      1. Starting a thread with the thread module
      2. Starting a thread with the threading module
    5. Synchronizing threads
      1. The concept of thread synchronization
      2. The threading.Lock class
      3. An example in Python
    6. Multithreaded priority queue
      1. A connection between real-life and programmatic queues
      2. The queue module
      3. Queuing in concurrent programming
      4. Multithreaded priority queue
    7. Summary
    8. Questions
    9. Further reading
  16. Using the with Statement in Threads
    1. Technical requirements
    2. Context management
      1. Starting from managing files
      2. The with statement as a context manager
      3. The syntax of the with statement
    3. The with statement in concurrent programming
      1. Example of deadlock handling
    4. Summary
    5. Questions
    6. Further reading
  17. Concurrent Web Requests
    1. Technical requirements
    2. The basics of web requests
      1. HTML
      2. HTTP requests
      3. HTTP status code
    3. The requests module
      1. Making a request in Python
      2. Running a ping test
    4. Concurrent web requests
      1. Spawning multiple threads
      2. Refactoring request logic
    5. The problem of timeout
      1. Support from httpstat.us and simulation in Python
      2. Timeout specifications
    6. Good practices in making web requests
      1. Consider the terms of service and data-collecting policies
      2. Error handling
      3. Update your program regularly
      4. Avoid making a large number of requests
    7. Summary
    8. Questions
    9. Further reading
  18. Working with Processes in Python
    1. Technical requirements
    2. The concept of a process
      1. Processes versus threads
      2. Multiprocessing
      3. Introductory example in Python
    3. An overview of the multiprocessing module
      1. The process class
      2. The Pool class
      3. Determining the current process, waiting, and terminating processes
        1. Determining the current process
        2. Waiting for processes
        3. Terminating processes
    4. Interprocess communication
      1. Message passing for a single worker
      2. Message passing between several workers
    5. Summary
    6. Questions
    7. Further reading
  19. Reduction Operators in Processes
    1. Technical requirements
    2. The concept of reduction operators
      1. Properties of a reduction operator
      2. Examples and non-examples
    3. Example implementation in Python
    4. Real-life applications of concurrent reduction operators
    5. Summary
    6. Questions
    7. Further reading
  20. Concurrent Image Processing
    1. Technical requirements
    2. Image processing fundamentals
      1. Python as an image processing tool
        1. Installing OpenCV and NumPy
      2. Computer image basics
        1. RGB values
        2. Pixels and image files
        3. Coordinates inside an image
      3. OpenCV API
      4. Image processing techniques
        1. Grayscaling
        2. Thresholding
    3. Applying concurrency to image processing
    4. Good concurrent image processing practices
      1. Choosing the correct way (out of many)
      2. Spawning an appropriate number of processes
      3. Processing input/output concurrently
    5. Summary
    6. Questions
    7. Further reading
  21. Introduction to Asynchronous Programming
    1. Technical requirements
    2. A quick analogy
    3. Asynchronous versus other programming models
      1. Asynchronous versus synchronous programming
      2. Asynchronous versus threading and multiprocessing
    4. An example in Python
    5. Summary
    6. Questions
    7. Further reading
  22. Implementing Asynchronous Programming in Python
    1. Technical requirements
    2. The asyncio module
      1. Coroutines, event loops, and futures
      2. Asyncio API
    3. The asyncio framework in action
      1. Asynchronously counting down
      2. A note about blocking functions
      3. Asynchronous prime-checking
      4. Improvements from Python 3.7
      5. Inherently blocking tasks
    4. concurrent.futures as a solution for blocking tasks
      1. Changes in the framework
      2. Examples in Python
    5. Summary
    6. Questions
    7. Further reading
  23. Building Communication Channels with asyncio
    1. Technical requirements
    2. The ecosystem of communication channels
      1. Communication protocol layers
      2. Asynchronous programming for communication channels
      3. Transports and protocols in asyncio
      4. The big picture of asyncio's server client
    3. Python example
      1. Starting a server
      2. Installing Telnet
      3. Simulating a connection channel
      4. Sending messages back to clients
      5. Closing the transports
    4. Client-side communication with aiohttp
      1. Installing aiohttp and aiofiles
      2. Fetching a website's HTML code
      3. Writing files asynchronously
    5. Summary
    6. Questions
    7. Further reading
  24. Deadlocks
    1. Technical requirements
    2. The concept of deadlock
      1. The Dining Philosophers problem
      2. Deadlock in a concurrent system
      3. Python simulation
    3. Approaches to deadlock situations
      1. Implementing ranking among resources
      2. Ignoring locks and sharing resources
        1. An additional note about locks
        2. Concluding note on deadlock solutions
    4. The concept of livelock
    5. Summary
    6. Questions
    7. Further reading
  25. Starvation
    1. Technical requirements
    2. The concept of starvation
      1. What is starvation?
      2. Scheduling
      3. Causes of starvation
      4. Starvation's relationship to deadlock
    3. The readers-writers problem
      1. Problem statement
      2. The first readers-writers problem
      3. The second readers-writers problem
      4. The third readers-writers problem
    4. Solutions to starvation
    5. Summary
    6. Questions
    7. Further reading
  26. Race Conditions
    1. Technical requirements
    2. The concept of race conditions
      1. Critical sections
      2. How race conditions occur
    3. Simulating race conditions in Python
    4. Locks as a solution to race conditions
      1. The effectiveness of locks
      2. Implementation in Python
      3. The downside of locks
        1. Turning a concurrent program sequential
        2. Locks do not lock anything
    5. Race conditions in real life
      1. Security
      2. Operating systems
      3. Networking
    6. Summary
    7. Questions
    8. Further reading
  27. The Global Interpreter Lock
    1. Technical requirements
    2. An introduction to the Global Interpreter Lock
      1. An analysis of memory management in Python
      2. The problem that the GIL addresses
      3. Problems raised by the GIL
    3. The potential removal of the GIL from Python
    4. How to work with the GIL
      1. Implementing multiprocessing, rather than multithreading 
      2. Getting around the GIL with native extensions
      3. Utilizing a different Python interpreter
    5. Summary
    6. Questions
    7. Further reading
  28. The Factory Pattern
    1. The factory method
      1. Real-world examples 
      2. Use cases
      3. Implementing the factory method
    2. The abstract factory
      1. Real-world examples
      2. Use cases
      3. Implementing the abstract factory pattern
    3. Summary
  29. The Builder Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  30. Other Creational Patterns
    1. The prototype pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    2. Singleton
      1. Real-world examples
      2. Use cases
      3. Implementation
    3. Summary
  31. The Adapter Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  32. The Decorator Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  33. The Bridge Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  34. The Facade Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  35. Other Structural Patterns
    1. The flyweight pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    2. The model-view-controller pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    3. The proxy pattern
      1. Real-world examples
      2. Use cases
      3. Implementation
    4. Summary
  36. The Chain of Responsibility Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  37. The Command Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  38. The Observer Pattern
    1. Real-world examples
    2. Use cases
    3. Implementation
    4. Summary
  39. Appendix
    1. Chapter 8
    2. Chapter 9
    3. Chapter 10
    4. Chapter 11
    5. Chapter 12
    6. Chapter 13
    7. Chapter 14
    8. Chapter 15
    9. Chapter 16
    10. Chapter 17
    11. Chapter 18
    12. Chapter 19
    13. Chapter 20
    14. Chapter 21
    15. Chapter 22
  40. Other Books You May Enjoy
    1. Leave a Review - Let Other Readers Know What You Think

Product information

  • Title: Advanced Python Programming
  • Author(s): Dr. Gabriele Lanaro, Quan Nguyen, Sakis Kasampalis
  • Release date: February 2019
  • Publisher(s): Packt Publishing
  • ISBN: 9781838551216