Python Cookbook, 2nd Edition

Table of contents

1. A Note Regarding Supplemental Files
2. Preface
   1. The Design of the Book
   2. The Implementation of the Book
   3. Using the Code from This Book
   4. Audience
   5. Organization
   6. Further Reading
   7. Conventions Used in This Book
   8. How to Contact Us
   9. Safari® Enabled
   10. Acknowledgments
3. 1. Text
   1. Introduction
   2. 1.1. Processing a String One Character at a Time
   3. 1.2. Converting Between Characters and Numeric Codes
   4. 1.3. Testing Whether an Object Is String-like
   5. 1.4. Aligning Strings
   6. 1.5. Trimming Space from the Ends of a String
   7. 1.6. Combining Strings
   8. 1.7. Reversing a String by Words or Characters
   9. 1.8. Checking Whether a String Contains a Set of Characters
   10. 1.9. Simplifying Usage of Strings’ translate Method
   11. 1.10. Filtering a String for a Set of Characters
   12. 1.11. Checking Whether a String Is Text or Binary
   13. 1.12. Controlling Case
   14. 1.13. Accessing Substrings
   15. 1.14. Changing the Indentation of a Multiline String
   16. 1.15. Expanding and Compressing Tabs
   17. 1.16. Interpolating Variables in a String
   18. 1.17. Interpolating Variables in a Stringin Python 2.4
   19. 1.18. Replacing Multiple Patterns in a Single Pass
   20. 1.19. Checking a String for Any of Multiple Endings
   21. 1.20. Handling International Text with Unicode
   22. 1.21. Converting Between Unicode and Plain Strings
   23. 1.22. Printing Unicode Charactersto Standard Output
   24. 1.23. Encoding Unicode Data for XML and HTML
   25. 1.24. Making Some Strings Case-Insensitive
   26. 1.25. Converting HTML Documents to Texton a Unix Terminal
4. 2. Files
   1. Introduction
   2. 2.1. Reading from a File
   3. 2.2. Writing to a File
   4. 2.3. Searching and Replacing Text in a File
   5. 2.4. Reading a Specific Line from a File
   6. 2.5. Counting Lines in a File
   7. 2.6. Processing Every Word in a File
   8. 2.7. Using Random-Access Input/Output
   9. 2.8. Updating a Random-Access File
   10. 2.9. Reading Data from zip Files
   11. 2.10. Handling a zip File Inside a String
   12. 2.11. Archiving a Tree of Files into a Compressed tar File
   13. 2.12. Sending Binary Data to Standard Output Under Windows
   14. 2.13. Using a C++-like iostream Syntax
   15. 2.14. Rewinding an Input File to the Beginning
   16. 2.15. Adapting a File-like Object to a True File Object
   17. 2.16. Walking Directory Trees
   18. 2.17. Swapping One File Extension for Another Throughout a Directory Tree
   19. 2.18. Finding a File Given a Search Path
   20. 2.19. Finding Files Given a Search Path and a Pattern
   21. 2.20. Finding a File on the Python Search Path
   22. 2.21. Dynamically Changing the PythonSearch Path
   23. 2.22. Computing the Relative Path from One Directory to Another
   24. 2.23. Reading an Unbuffered Character in a Cross-Platform Way
   25. 2.24. Counting Pages of PDF Documents on Mac OS X
   26. 2.25. Changing File Attributes on Windows
   27. 2.26. Extracting Text from OpenOffice.org Documents
   28. 2.27. Extracting Text from Microsoft Word Documents
   29. 2.28. File Locking Using a Cross-Platform API
   30. 2.29. Versioning Filenames
   31. 2.30. Calculating CRC-64 Cyclic Redundancy Checks
5. 3. Time and Money
   1. Introduction
   2. 3.1. Calculating Yesterday and Tomorrow
   3. 3.2. Finding Last Friday
   4. 3.3. Calculating Time Periods in a Date Range
   5. 3.4. Summing Durations of Songs
   6. 3.5. Calculating the Number of Weekdays Between Two Dates
   7. 3.6. Looking up Holidays Automatically
   8. 3.7. Fuzzy Parsing of Dates
   9. 3.8. Checking Whether Daylight Saving Time Is Currently in Effect
   10. 3.9. Converting Time Zones
   11. 3.10. Running a Command Repeatedly
   12. 3.11. Scheduling Commands
   13. 3.12. Doing Decimal Arithmetic
   14. 3.13. Formatting Decimals as Currency
   15. 3.14. Using Python as a Simple Adding Machine
   16. 3.15. Checking a Credit Card Checksum
   17. 3.16. Watching Foreign Exchange Rates
6. 4. Python Shortcuts
   1. Introduction
   2. 4.1. Copying an Object
   3. 4.2. Constructing Lists with List Comprehensions
   4. 4.3. Returning an Element of a List If It Exists
   5. 4.4. Looping over Items and Their Indices in a Sequence
   6. 4.5. Creating Lists of Lists Without Sharing References
   7. 4.6. Flattening a Nested Sequence
   8. 4.7. Removing or Reordering Columnsin a List of Rows
   9. 4.8. Transposing Two-Dimensional Arrays
   10. 4.9. Getting a Value from a Dictionary
   11. 4.10. Adding an Entry to a Dictionary
   12. 4.11. Building a Dictionary Without Excessive Quoting
   13. 4.12. Building a Dict from a List of Alternating Keys and Values
   14. 4.13. Extracting a Subset of a Dictionary
   15. 4.14. Inverting a Dictionary
   16. 4.15. Associating Multiple Values with Each Key in a Dictionary
   17. 4.16. Using a Dictionary to Dispatch Methods or Functions
   18. 4.17. Finding Unions and Intersections of Dictionaries
   19. 4.18. Collecting a Bunch of Named Items
   20. 4.19. Assigning and Testing with One Statement
   21. 4.20. Using printf in Python
   22. 4.21. Randomly Picking Items with Given Probabilities
   23. 4.22. Handling Exceptions Within an Expression
   24. 4.23. Ensuring a Name Is Defined in a Given Module
7. 5. Searching and Sorting
   1. Introduction
   2. 5.1. Sorting a Dictionary
   3. 5.2. Sorting a List of Strings Case-Insensitively
   4. 5.3. Sorting a List of Objects by an Attribute of the Objects
   5. 5.4. Sorting Keys or Indices Basedon the Corresponding Values
   6. 5.5. Sorting Strings with Embedded Numbers
   7. 5.6. Processing All of a List’s Items in Random Order
   8. 5.7. Keeping a Sequence Ordered as Items Are Added
   9. 5.8. Getting the First Few Smallest Items of a Sequence
   10. 5.9. Looking for Items in a Sorted Sequence
   11. 5.10. Selecting the nth Smallest Element of a Sequence
   12. 5.11. Showing off quicksort in Three Lines
   13. 5.12. Performing Frequent Membership Tests on a Sequence
   14. 5.13. Finding Subsequences
   15. 5.14. Enriching the Dictionary Type with Ratings Functionality
   16. 5.15. Sorting Names and Separating Them by Initials
8. 6. Object-Oriented Programming
   1. Introduction
   2. 6.1. Converting Among Temperature Scales
   3. 6.2. Defining Constants
   4. 6.3. Restricting Attribute Setting
   5. 6.4. Chaining Dictionary Lookups
   6. 6.5. Delegating Automatically as an Alternative to Inheritance
   7. 6.6. Delegating Special Methods in Proxies
   8. 6.7. Implementing Tuples with Named Items
   9. 6.8. Avoiding Boilerplate Accessors for Properties
   10. 6.9. Making a Fast Copy of an Object
   11. 6.10. Keeping References to Bound Methods Without Inhibiting Garbage Collection
   12. 6.11. Implementing a Ring Buffer
   13. 6.12. Checking an Instance for Any State Changes
   14. 6.13. Checking Whether an Object Has Necessary Attributes
   15. 6.14. Implementing the State Design Pattern
   16. 6.15. Implementing the “Singleton” Design Pattern
   17. 6.16. Avoiding the “Singleton” Design Pattern with the Borg Idiom
   18. 6.17. Implementing the Null Object Design Pattern
   19. 6.18. Automatically Initializing Instance Variables from _ _init_ _ Arguments
   20. 6.19. Calling a Superclass _ _init_ _ Method If It Exists
   21. 6.20. Using Cooperative Supercalls Concisely and Safely
9. 7. Persistence and Databases
   1. Introduction
   2. 7.1. Serializing Data Using the marshal Module
   3. 7.2. Serializing Data Using the pickle and cPickle Modules
   4. 7.3. Using Compression with Pickling
   5. 7.4. Using the cPickle Module on Classes and Instances
   6. 7.5. Holding Bound Methods in a Picklable Way
   7. 7.6. Pickling Code Objects
   8. 7.7. Mutating Objects with shelve
   9. 7.8. Using the Berkeley DB Database
   10. 7.9. Accessing a MySQL Database
   11. 7.10. Storing a BLOB in a MySQL Database
   12. 7.11. Storing a BLOB in a PostgreSQL Database
   13. 7.12. Storing a BLOB in a SQLite Database
   14. 7.13. Generating a Dictionary Mapping Field Names to Column Numbers
   15. 7.14. Using dtuple for Flexible Accessto Query Results
   16. 7.15. Pretty-Printing the Contents of Database Cursors
   17. 7.16. Using a Single Parameter-Passing Style Across Various DB API Modules
   18. 7.17. Using Microsoft Jet via ADO
   19. 7.18. Accessing a JDBC Database from a Jython Servlet
   20. 7.19. Using ODBC to Get Excel Data with Jython
10. 8. Debugging and Testing
    1. Introduction
    2. 8.1. Disabling Execution of Some Conditionals and Loops
    3. 8.2. Measuring Memory Usage on Linux
    4. 8.3. Debugging the Garbage-Collection Process
    5. 8.4. Trapping and Recording Exceptions
    6. 8.5. Tracing Expressions and Comments in Debug Mode
    7. 8.6. Getting More Information from Tracebacks
    8. 8.7. Starting the Debugger Automatically After an Uncaught Exception
    9. 8.8. Running Unit Tests Most Simply
    10. 8.9. Running Unit Tests Automatically
    11. 8.10. Using doctest with unittest in Python 2.4
    12. 8.11. Checking Values Against Intervals in Unit Testing
11. 9. Processes, Threads, and Synchronization
    1. Introduction
    2. 9.1. Synchronizing All Methods in an Object
    3. 9.2. Terminating a Thread
    4. 9.3. Using a Queue.Queue as a Priority Queue
    5. 9.4. Working with a Thread Pool
    6. 9.5. Executing a Function in Parallel on Multiple Argument Sets
    7. 9.6. Coordinating Threads by Simple Message Passing
    8. 9.7. Storing Per-Thread Information
    9. 9.8. Multitasking Cooperatively Without Threads
    10. 9.9. Determining Whether Another Instanceof a Script Is Already Running in Windows
    11. 9.10. Processing Windows Messages Using MsgWaitForMultipleObjects
    12. 9.11. Driving an External Process with popen
    13. 9.12. Capturing the Output and Error Streams from a Unix Shell Command
    14. 9.13. Forking a Daemon Process on Unix
12. 10. System Administration
    1. Introduction
    2. 10.1. Generating Random Passwords
    3. 10.2. Generating Easily Remembered Somewhat-Random Passwords
    4. 10.3. Authenticating Users by Means of a POP Server
    5. 10.4. Calculating Apache Hits per IP Address
    6. 10.5. Calculating the Rate of Client Cache Hits on Apache
    7. 10.6. Spawning an Editor from a Script
    8. 10.7. Backing Up Files
    9. 10.8. Selectively Copying a Mailbox File
    10. 10.9. Building a Whitelist of Email Addresses From a Mailbox
    11. 10.10. Blocking Duplicate Mails
    12. 10.11. Checking Your Windows Sound System
    13. 10.12. Registering or Unregistering a DLL on Windows
    14. 10.13. Checking and Modifying the Set of Tasks Windows Automatically Runs at Login
    15. 10.14. Creating a Share on Windows
    16. 10.15. Connecting to an Already Running Instance of Internet Explorer
    17. 10.16. Reading Microsoft Outlook Contacts
    18. 10.17. Gathering Detailed System Informationon Mac OS X
13. 11. User Interfaces
    1. Introduction
    2. 11.1. Showing a Progress Indicator on a Text Console
    3. 11.2. Avoiding lambda in Writing Callback Functions
    4. 11.3. Using Default Values and Bounds with tkSimpleDialog Functions
    5. 11.4. Adding Drag and Drop Reordering to a Tkinter Listbox
    6. 11.5. Entering Accented Characters in Tkinter Widgets
    7. 11.6. Embedding Inline GIFs Using Tkinter
    8. 11.7. Converting Among Image Formats
    9. 11.8. Implementing a Stopwatch in Tkinter
    10. 11.9. Combining GUIs and Asynchronous I/Owith Threads
    11. 11.10. Using IDLE’s Tree Widget in Tkinter
    12. 11.11. Supporting Multiple Values per Row in a Tkinter Listbox
    13. 11.12. Copying Geometry Methods and Options Between Tkinter Widgets
    14. 11.13. Implementing a Tabbed Notebook for Tkinter
    15. 11.14. Using a wxPython Notebook with Panels
    16. 11.15. Implementing an ImageJ Plug-in in Jython
    17. 11.16. Viewing an Image from a URL with Swing and Jython
    18. 11.17. Getting User Input on Mac OS
    19. 11.18. Building a Python Cocoa GUI Programmatically
    20. 11.19. Implementing Fade-in Windows with IronPython
14. 12. Processing XML
    1. Introduction
    2. 12.1. Checking XML Well-Formedness
    3. 12.2. Counting Tags in a Document
    4. 12.3. Extracting Text from an XML Document
    5. 12.4. Autodetecting XML Encoding
    6. 12.5. Converting an XML Document into a Tree of Python Objects
    7. 12.6. Removing Whitespace-only Text Nodes from an XML DOM Node’s Subtree
    8. 12.7. Parsing Microsoft Excel’s XML
    9. 12.8. Validating XML Documents
    10. 12.9. Filtering Elements and Attributes Belonging to a Given Namespace
    11. 12.10. Merging Continuous Text Events with a SAX Filter
    12. 12.11. Using MSHTML to Parse XML or HTML
15. 13. Network Programming
    1. Introduction
    2. 13.1. Passing Messages with Socket Datagrams
    3. 13.2. Grabbing a Document from the Web
    4. 13.3. Filtering a List of FTP Sites
    5. 13.4. Getting Time from a Server via the SNTP Protocol
    6. 13.5. Sending HTML Mail
    7. 13.6. Bundling Files in a MIME Message
    8. 13.7. Unpacking a Multipart MIME Message
    9. 13.8. Removing Attachments from an Email Message
    10. 13.9. Fixing Messages Parsed by Python 2.4 email.FeedParser
    11. 13.10. Inspecting a POP3 Mailbox Interactively
    12. 13.11. Detecting Inactive Computers
    13. 13.12. Monitoring a Network with HTTP
    14. 13.13. Forwarding and Redirecting Network Ports
    15. 13.14. Tunneling SSL Through a Proxy
    16. 13.15. Implementing the Dynamic IP Protocol
    17. 13.16. Connecting to IRC and Logging Messages to Disk
    18. 13.17. Accessing LDAP Servers
16. 14. Web Programming
    1. Introduction
    2. 14.1. Testing Whether CGI Is Working
    3. 14.2. Handling URLs Within a CGI Script
    4. 14.3. Uploading Files with CGI
    5. 14.4. Checking for a Web Page’s Existence
    6. 14.5. Checking Content Type via HTTP
    7. 14.6. Resuming the HTTP Download of a File
    8. 14.7. Handling Cookies While Fetching Web Pages
    9. 14.8. Authenticating with a Proxy for HTTPS Navigation
    10. 14.9. Running a Servlet with Jython
    11. 14.10. Finding an Internet Explorer Cookie
    12. 14.11. Generating OPML Files
    13. 14.12. Aggregating RSS Feeds
    14. 14.13. Turning Data into Web Pages Through Templates
    15. 14.14. Rendering Arbitrary Objects with Nevow
17. 15. Distributed Programming
    1. Introduction
    2. 15.1. Making an XML-RPC Method Call
    3. 15.2. Serving XML-RPC Requests
    4. 15.3. Using XML-RPC with Medusa
    5. 15.4. Enabling an XML-RPC Server to Be Terminated Remotely
    6. 15.5. Implementing SimpleXMLRPCServer Niceties
    7. 15.6. Giving an XML-RPC Server a wxPython GUI
    8. 15.7. Using Twisted Perspective Broker
    9. 15.8. Implementing a CORBA Server and Client
    10. 15.9. Performing Remote Logins Using telnetlib
    11. 15.10. Performing Remote Logins with SSH
    12. 15.11. Authenticating an SSL Client over HTTPS
18. 16. Programs About Programs
    1. Introduction
    2. 16.1. Verifying Whether a String Represents a Valid Number
    3. 16.2. Importing a Dynamically Generated Module
    4. 16.3. Importing from a Module Whose Name Is Determined at Runtime
    5. 16.4. Associating Parameters with a Function (Currying)
    6. 16.5. Composing Functions
    7. 16.6. Colorizing Python Source Using the Built-in Tokenizer
    8. 16.7. Merging and Splitting Tokens
    9. 16.8. Checking Whether a String Has Balanced Parentheses
    10. 16.9. Simulating Enumerations in Python
    11. 16.10. Referring to a List Comprehension While Building It
    12. 16.11. Automating the py2exe Compilation of Scripts into Windows Executables
    13. 16.12. Binding Main Script and Modules into One Executable on Unix
19. 17. Extending and Embedding
    1. Introduction
    2. 17.1. Implementing a Simple Extension Type
    3. 17.2. Implementing a Simple Extension Type with Pyrex
    4. 17.3. Exposing a C++ Library to Python
    5. 17.4. Calling Functions from a Windows DLL
    6. 17.5. Using SWIG-Generated Modules in a Multithreaded Environment
    7. 17.6. Translating a Python Sequence into a C Array with the PySequence_Fast Protocol
    8. 17.7. Accessing a Python Sequence Item-by-Item with the Iterator Protocol
    9. 17.8. Returning None from a Python-Callable C Function
    10. 17.9. Debugging Dynamically Loaded C Extensions with gdb
    11. 17.10. Debugging Memory Problems
20. 18. Algorithms
    1. Introduction
    2. 18.1. Removing Duplicates from a Sequence
    3. 18.2. Removing Duplicates from a Sequence While Maintaining Sequence Order
    4. 18.3. Generating Random Samples with Replacement
    5. 18.4. Generating Random Samples Without Replacement
    6. 18.5. Memoizing (Caching) the Return Values of Functions
    7. 18.6. Implementing a FIFO Container
    8. 18.7. Caching Objects with a FIFO Pruning Strategy
    9. 18.8. Implementing a Bag (Multiset) Collection Type
    10. 18.9. Simulating the Ternary Operator in Python
    11. 18.10. Computing Prime Numbers
    12. 18.11. Formatting Integers as Binary Strings
    13. 18.12. Formatting Integers as Strings in Arbitrary Bases
    14. 18.13. Converting Numbers to Rationals via Farey Fractions
    15. 18.14. Doing Arithmetic with Error Propagation
    16. 18.15. Summing Numbers with Maximal Accuracy
    17. 18.16. Simulating Floating Point
    18. 18.17. Computing the Convex Hulls and Diameters of 2D Point Sets
21. 19. Iterators and Generators
    1. Introduction
    2. 19.1. Writing a range-like Function with Float Increments
    3. 19.2. Building a List from Any Iterable
    4. 19.3. Generating the Fibonacci Sequence
    5. 19.4. Unpacking a Few Items in a Multiple Assignment
    6. 19.5. Automatically Unpacking the Needed Number of Items
    7. 19.6. Dividing an Iterable into n Slices of Stride n
    8. 19.7. Looping on a Sequence by Overlapping Windows
    9. 19.8. Looping Through Multiple Iterables in Parallel
    10. 19.9. Looping Through the Cross-Product of Multiple Iterables
    11. 19.10. Reading a Text File by Paragraphs
    12. 19.11. Reading Lines with Continuation Characters
    13. 19.12. Iterating on a Stream of Data Blocks as a Stream of Lines
    14. 19.13. Fetching Large Record Sets from a Database with a Generator
    15. 19.14. Merging Sorted Sequences
    16. 19.15. Generating Permutations, Combinations, and Selections
    17. 19.16. Generating the Partitions of an Integer
    18. 19.17. Duplicating an Iterator
    19. 19.18. Looking Ahead into an Iterator
    20. 19.19. Simplifying Queue-Consumer Threads
    21. 19.20. Running an Iterator in Another Thread
    22. 19.21. Computing a Summary Report with itertools.groupby
22. 20. Descriptors, Decorators,and Metaclasses
    1. Introduction
    2. 20.1. Getting Fresh Default Values at Each Function Call
    3. 20.2. Coding Properties as Nested Functions
    4. 20.3. Aliasing Attribute Values
    5. 20.4. Caching Attribute Values
    6. 20.5. Using One Method as Accessorfor Multiple Attributes
    7. 20.6. Adding Functionality to a Class by Wrapping a Method
    8. 20.7. Adding Functionality to a Class by Enriching All Methods
    9. 20.8. Adding a Method to a Class Instance at Runtime
    10. 20.9. Checking Whether Interfaces Are Implemented
    11. 20.10. Using _ _new_ _ and _ _init_ _ Appropriately in Custom Metaclasses
    12. 20.11. Allowing Chaining of Mutating List Methods
    13. 20.12. Using Cooperative Super calls with Terser Syntax
    14. 20.13. Initializing Instance Attributes Without Using _ _init_ _
    15. 20.14. Automatic Initialization of Instance Attributes
    16. 20.15. Upgrading Class Instances Automatically on reload
    17. 20.16. Binding Constants at Compile Time
    18. 20.17. Solving Metaclass Conflicts
23. Index
24. About the Authors
25. Colophon
26. Copyright