High Performance Python, 3rd Edition

Chapter 8. The multiprocessing Module

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 9th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Questions You’ll Be Able to Answer After This Chapter

• What does the multiprocessing module offer?

• What’s the difference between processes and threads?

• How do I choose the right size for a process pool?

• How do I use nonpersistent queues for work processing?

• What are the costs and benefits of interprocess communication?

• How can I process numpy data with many CPUs?

• How would I use Joblib to simplify parallelized and cached scientific work?

• Why do I need locking to avoid data loss?

CPython doesn’t use multiple CPUs by default. This is partly because Python was designed back in a single-core era, and partly because parallelizing can actually be quite difficult to do efficiently. Python gives us the tools to do it but leaves us to make our own choices. It is painful to see your multicore machine using just one CPU on a long-running process, though, so in this chapter we’ll review ...