Python Cookbook

Chapter 13. Distributed Programming

Introduction

Credit: Jeremy Hylton, PythonLabs

The recipes in this chapter describe some simple techniques for using Python in distributed systems. Programming distributed systems is hard, and recipes alone won’t even come close to solving all your problems. What the recipes do is help you get programs on different computers talking to each other so you can start writing applications.

Remote procedure call (RPC) is an attractive approach to structuring a distributed system. The details of network communication are exposed through an interface that looks like normal procedure calls. When you call a function on a remote server, the RPC system is responsible for all the communication details. It encodes the arguments so they can be passed over the network to the server, which might use different internal representations for the data. It invokes the right function on the remote machine and waits for a response.

The recipes here use three different systems that provide RPC interfaces: CORBA, SOAP, and XML-RPC. These systems are attractive because they make it easy to connect programs, whether they are running on different computers or are written in different languages.

You can find Fredrik Lundh’s XML-RPC library for Python in the standard library, starting with the 2.2 release. For earlier versions of Python, and for CORBA and SOAP, you’ll need to install more software before you can get started. The recipes include pointers to the software you need.

The Python standard library also provides a good set of modules for doing the lower-level work of network programming: socket, select, asyncore, and asynchat. It also has modules for marshaling data and sending it across sockets: struct, pickle, and xdrlib. These modules, in turn, provide the plumbing for many other modules. Jeff Bauer offers a recipe using the telnetlib module to send commands to remote machines.

Four of the recipes focus on XML-RPC, a new protocol that uses XML and HTTP to exchange simple data structures between programs. Rael Dornfest demonstrates how to write an XML-RPC client program that retrieves data from O’Reilly’s Meerkat service. It’s a three-line recipe, including the import statement, which is its chief appeal.

Brian Quinlan and Jeff Bauer contribute two recipes for constructing XML-RPC servers. Quinlan shows how to use the SimpleXMLRPCServer module from the Python 2.2 standard library to handle incoming requests in Recipe 13.3. Bauer’s Recipe 13.4 uses Medusa, a framework for writing asynchronous network servers. In both cases, the libraries do most of the work. Other than a few lines of initialization and registration, the server looks like normal Python code.

SOAP is an XML-based protocol that shares its origins with XML-RPC. Graham Dumpleton explains how to create a server that can talk to clients with either protocol in Recipe 13.5, one of three recipes that use his OSE framework. The two protocols are similar enough that a single HTTP server and service implementation can support both protocols. There are gotchas, of course. Dumpleton mentions several. For starters, XML-RPC does not support None, and SOAP does not support empty dictionaries.

An alternative to the XML-based protocols is CORBA, an object-based RPC mechanism that uses its own protocol, IIOP. Compared to XML-RPC and SOAP, CORBA is a mature technology; it was introduced in 1991. The Python language binding was officially approved in February 2000, and several ORBs (roughly, CORBA servers) support Python. Duncan Grisby lays out the basics of getting a CORBA client and server running in Recipe 13.6, which uses omniORB, a free ORB, and the Python binding he wrote for it.

CORBA has a reputation for complexity, but Grisby’s recipe makes it look straightforward. There are more steps involved in the CORBA client example than in the XML-RPC client example, but they aren’t hard to follow. To connect an XML-RPC client to a server, you just need a URL. To connect a CORBA client to a server, you need a special corbaloc URL, and you need to know the server’s interface. Of course, you need to know the interface regardless of protocol, but CORBA uses it explicitly. Generally, CORBA offers more features—such as interfaces, type checking, passing references to objects, and more (and it supports both None and empty dictionaries).

Regardless of the protocols or systems you choose, the recipes here can help get you started. Interprogram communication is an important part of building a distributed system, but it’s just one part. Once you have a client and server working, you’ll find you have to deal with other interesting, hard problems, such as error detection, concurrency, and security, to name a few. The recipes here won’t solve these problems, but they will prevent you from getting caught up in unimportant details of the communication protocols.