Twisted Network Programming Essentials

Chapter 1. Getting Started

Before you can start developing applications using Twisted, you’ll need to download and install Twisted and its related packages. This chapter walks you through the installation process on various operating systems. It also shows you how to add the Twisted utilities to your path, familiarize yourself with the Twisted documentation, and get answers to your questions from the Twisted community.

Installing Twisted

First things first: you need to get Twisted installed on your computer. Downloads and instructions for installing Twisted on various operating systems can be found at http://twistedmatrix.com/projects/core/. To enable additional functionality in Twisted, you’ll have to install a couple of optional packages as well.

How Do I Do That?

Begin by downloading the latest Twisted release from http://www.twistedmatrix.com. Then install PyOpenSSL (a Python wrapper of the popular OpenSSL library), which Twisted uses to make encrypted Secure Socket Layer (SSL) connections. Finally, install PyCrypto , a package containing Python implementations of encryption algorithms used by the Secure SHell (SSH). Locations for these downloads are provided in each of the following platform-specific sections.

Tip

You don’t need to install PyOpenSSL or PyCrypto in order to use Twisted. Without these packages installed, you won’t be able to use Twisted’s SSL and SSH features, but everything else will still work.

Windows

Go to http://twistedmatrix.com/projects/core/. Download the Twisted Windows “Sumo” installer for Python 2.4 (or your Python version). The Sumo binary includes the Twisted core, as well as a number of extra modules to support specific groups of protocols like mail and web. You’ll need the full Sumo version of Twisted installed to run most of the examples in this book. Then go to http://twistedmatrix.com/products/download and find the section labeled “Twisted Dependencies for Windows.” There you’ll find links to installers for the latest versions of PyOpenSSL and PyCrypto.

Tip

It’s possible that some of these packages might move to different pages as the Twisted web site grows and is restructured in the future. If one if these links doesn’t work for you, try starting from the Twisted home page at http://twistedmatrix.com.

Once you’ve downloaded all three installers, run the Twisted Setup program. It’s a simple installer, and it will automatically detect your existing Python installation. Next, run the PyOpenSSL and and PyCrypto installers. These are even simpler than the Twisted installer and take only a few seconds to complete.

Tip

When byte-compiling the Twisted modules, the Twisted Setup program will throw up a tiny, floating progress bar that sits in a very tiny window of its own. The unpacking may take several minutes.

Verify that the installation worked by importing the twisted package from an interactive Python prompt:

    $ python
    Python 2.3 (#1, Sep 13 2003, 00:49:11)
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import twisted
    >>>

If the import twisted statement runs with no errors, you have a working Twisted install.

Mac OS X, Linux, and BSD

Many Linux distributions and BSD variants, including Debian, Ubuntu, Gentoo Linux, Red Hat Linux, and FreeBSD, have prepackaged versions of Twisted, PyOpenSSL, and PyCrypto available. Check the Twisted download page to see if it mentions your operating system, or search your favorite package repository. If there are packages available, make sure they are for Twisted 2.0 or higher, not the older 1.3 release.

If you can’t find a prepackaged version of Twisted for your operating system or platform, refer to the following instructions on installing from source.

Installing from Source Files

If you’re on an operating system for which no Twisted binary packages are available, you’ll need to install from source. Don’t worry, though; as source installs go, Python packages are among the easiest you’ll find.

How Do I Do That?

First, download the full “Sumo” source package for Twisted (choosing the version with documentation) from http://twistedmatrix.com/projects/core/. The Sumo package is the core of Twisted, plus a number of bundled modules from other projects developed under the Twisted umbrella; you’ll need the modules in the Sumo package to run most of the examples in this book. Once you’ve downloaded the package, extract it to a working directory:

    $ tar -xjvf ~/downloads/TwistedSumo-2005-03-22.tar.bz2
    TwistedSumo-2005-03-22/
    TwistedSumo-2005-03-22/bin/
    ...
    TwistedSumo-2005-03-22/README
    TwistedSumo-2005-03-22/LICENSE
    TwistedSumo-2005-03-22/setup.py
    TwistedSumo-2005-03-22/ZopeInterface-3.0.1.tgz

Next, enter the TwistedSumo-version directory. Twisted depends on the zope.interface package, which is bundled in the Twisted Sumo distribution. Unzip the ZopeInterface tarball:

    $ tar -xzvf ZopeInterface-3.0.1.tgz
    ZopeInterface-3.0.1/
    ZopeInterface-3.0.1/Support/
    ZopeInterface-3.0.1/Support/zpkgsetup/
    ZopeInterface-3.0.1/Support/zpkgsetup/publication.py
    ...
    ZopeInterface-3.0.1/setup.py
    ZopeInterface-3.0.1/setup.cfg
    ZopeInterface-3.0.1/MANIFEST

Enter the ZopeInterface-<version> directory, and run the command python setup.py install. This command will build and install the zope.interface package in your python installation’s lib/site-packages/twisted directory. You’ll need to have administrative/root permissions to do this, so use su or sudo to increase your permission level if necessary:

    $ cd ZopeInterface-3.0.1
    $ python setup.py install
    running install
    running build
    running build_py
    running build_ext
    building 'zope.interface._zope_interface_coptimizations' extension
    ...
    running install_lib
    copying build/lib.linux-i686-2.4/zope/interface/_zope_interface_coptimizations.so ->
    /usr/lib/python2.4/site-packages/zope/interface
    writing byte-compilation script '/tmp/tmpdY9dA9.py'
    /usr/bin/python -O /tmp/tmpdY9dA9.py
    removing /tmp/tmpdY9dA9.py

Once zope.interface is installed, you’re ready to install Twisted. In this TwistedSumo-<version> directory, run the command python setup.py install. The command will compile the Twisted C modules and install Twisted:

    $ cd TwistedSumo-2005-03-22
    $ python setup.py install
    Password:
    running install
    running build
    running build_py
    ...
    running install_data

Tip

If you have more than one version of Python installed, keep in mind that Twisted will be installed for only the version of Python you’re using when you run setup.py. The examples in this book require Python 2.3 or higher. To check your Python version, run python -V.

Congratulations—you’ve installed Twisted! You can make sure the installation worked by importing the Twisted package from an interactive Python prompt:

    $ python
    Python 2.3 (#1, Sep 13 2003, 00:49:11)
    [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import twisted
    >>>

If the import twisted statement runs with no errors, you have a working Twisted install.

Next, download the latest release of PyOpenSSL from http://pyopenssl.sourceforge.net. PyOpenSSL works on top of the OpenSSL library, so you’ll need to make sure you have OpenSSL installed first. Mac OS X comes with OpenSSL installed, along with its header files, and all Linux and BSD distributions should have a package available (if not installed by default).

Tip

If you’re using a really obscure operating system that doesn’t include OpenSSL, you can download and compile the source package from http://www.openssl.org.

To install PyOpenSSL, follow the same steps you did when installing Twisted. First, extract the contents of the downloaded file:

    $ tar -zxvf pyOpenSSL-0.6.tar.gz
    pyOpenSSL-0.6/
    pyOpenSSL-0.6/doc/
    pyOpenSSL-0.6/doc/html/
    ...
    pyOpenSSL-0.6/_ _init_ _.py
    pyOpenSSL-0.6/ChangeLog
    pyOpenSSL-0.6/COPYING
    pyOpenSSL-0.6/version.py

Next, switch to the PyOpenSSL directory and run python setup.py install as root or an administrative user:

    $ cd pyOpenSSL-0.6
    $ python setup.py install
    running install
    running build
    running build_py
    creating build
    ...
    byte-compiling
    /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/
    OpenSSL/__init__.py to _ _init_ _.pyc
    byte-compiling
    /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/
    OpenSSL/version.py to version.pyc

When the installation is complete, test to confirm that the OpenSSL package is now available, and that Twisted is making use of it in its internet.ssl module:

    $ python
    Python 2.3 (#1, Sep 13 2003, 00:49:11)
    [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import OpenSSL
    >>> import twisted.internet.ssl
    >>> twisted.internet.ssl.SSL
    <module 'OpenSSL.SSL' from
    '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-
    packages/OpenSSL/SSL.so'>

If you don’t see any errors, you’ve successfully added SSL support to your Twisted installation.

The final package to install is PyCrypto. PyCrypto, the Python Cryptography Toolkit, is a package developed by A. M. Kuchling that contains implementations of many cryptographic functions. Twisted uses PyCrypto to support SSH connections.

Start by downloading PyCrypto from http://www.amk.ca/python/code/crypto.html. Then extract the package:

    $ tar -xzvf pycrypto-2.0.tar.gz
    pycrypto-2.0/
    pycrypto-2.0/_ _init_ _.py
    pycrypto-2.0/ACKS
    pycrypto-2.0/ChangeLog
    ...
    pycrypto-2.0/Util/test/prime_speed.py
    pycrypto-2.0/Util/test.py

Run the now-familiar python setup.py install (as root or an administrative user) in the PyCrypto directory:

    $ cd pycrypto-2.0
    $ python setup.py install
    running install
    running build
    running build_py
    creating build
    ...
    byte-compiling
    /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/
    Crypto/Util/RFC1751.py to RFC1751.pyc
    byte-compiling
    /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/
    Crypto/Util/test.py to test.pyc

To verify that the package installed correctly, import it from an interactive Python prompt. You can also make sure Twisted’s twisted.conch.ssh.transport module is now using PyCrypto’s RSA implementation:

    $ python
    Python 2.3 (#1, Sep 13 2003, 00:49:11)
    [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import Crypto
    >>> import twisted.conch.ssh.transport
    >>> twisted.conch.ssh.transport.RSA
    <module 'Crypto.PublicKey.RSA' from
    '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-
    packages/Crypto/PublicKey/RSA.pyc'>

And that’s it! You’ve installed PyCrypto from source. At this point you have a complete, working Twisted installation, including support for SSL and SSH connections.

Adding Twisted Utilities to Your Path

Twisted includes a number of scripts and utilities that you’ll need to use. For convenience, you should make sure these are available in your path.

How Do I Do That?

Typically, all you have to do is add Twisted’s utility directory to your operating system’s command search path. Follow the specific instructions given in the following section for your platform.

Windows

Twisted’s utilities will be installed in the Python scripts directory (typically in a location such as c:\Python23\scripts). Twisted includes a helpful Programs menu entry that launches a Windows command prompt with the Python scripts directory added to %PATH%. It’s located under Programs → Twisted (Python [version]) → Twisted Command Prompt. Use this menu entry to launch your command prompt when you need to run the Twisted utilities, or edit your %PATH% to include the scripts directory.

Linux

Twisted’s utilities will be installed in the same directory as your python binary (probably /usr/bin or /usr/local/bin), so you shouldn’t need to make any changes to your $PATH.

Mac OS X

If you’re using the version of Python included with Mac OS X 2.3 “Jaguar” or later, Twisted’s utilities will be installed under /System/Library/Frameworks/Python.framework/Versions/Current/bin. Add this directory to your $PATH :

    $ set PATH=$PATH:/System/Library/Frameworks/Python.framework/Versions/Current/bin

Using the Twisted Documentation

Twisted includes a few different types of documentation: extensive API documentation, HOWTOs, a tutorial, and sample code. It’s a good idea to familiarize yourself with this documentation now, so that you’ll be able to refer to it during the development process.

How Do I Do That?

Documentation for Twisted is available online on the Twisted web site. A complete API reference can be found at http://twistedmatrix.com/documents/current/api. You’ll probably refer to this documentation many times to see which classes a module contains or to see the list of arguments for a specific function. The pages in the API documentation are automatically generated from the source code using lore, a custom documentation tool developed as part of Twisted.

Twisted is developed as a set of subprojects, and each project has additional documentation in its section of the Twisted site. For example, documentation on the core modules is at http://twistedmatrix.com/projects/core/documentation, and documentation on web modules is at http://twistedmatrix.com/projects/web/documentation. There are links to the full list of projects and documentation on the home page.

Within each project’s documentation, you’ll find the following types of information:

HOWTOs

These documents describe specific features of Twisted and how to use them. The HOWTOs don’t cover every part of Twisted, but they can provide a helpful starting point for certain tasks. Included in the HOWTOs is a tutorial called “Twisted From Scratch,” which shows how an application can be developed in Twisted, extended to take advantage of some advanced features, and then fully integrated with the Twisted utilities.

Examples

These are examples of short and specific Twisted programs. Like the HOWTOs, these aren’t comprehensive but can be an excellent resource when you need a working example of a certain feature.

Manual pages

These are HTML versions of the manpages describing how to use the Twisted utilities.

What About...

...viewing the API documentation without using a browser? If you’d prefer, you can view the API documentation using the standard pydoc utility. pydoc doesn’t understand all of the conventions used in the Twisted internal documentation, but it’s still quite usable. Figure 1-1 shows output from the command pydoc twisted.web.http.

Also, when you really want to understand how a part of Twisted works, you can always read the source code. With the exception of a few modules that are written in C for better performance, you don’t even need the source distribution—just go to your Python lib directory, look under site-packages/twisted, and open the appropriate .py file in your favorite editor.

Finding Answers to Your Questions

Even with this book, and the Twisted documentation, you’ll eventually run into a question whose answer you can’t figure out on your own. When that happens, it’s time to get help from the Twisted community.

Figure 1-1. Using pydoc to view API documentation

How Do I Do That?

There are a few excellent community resources you can look to for help. First, there are the mailing lists . The twisted-python list is for general discussion of Twisted. The twisted-web list is dedicated to discussion of web applications. It’s good etiquette to use the proper list; if you ask web-related questions on the twisted-python list, you’ll probably be asked to move the discussion to twisted-web. You can sign up for the twisted-python and twisted-web mailing lists at http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python and http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web.

Second, you can talk with Twisted users and developers in the #twisted and #twisted.web IRC channels on the freenode network (see http://freenode.net for a list of servers). These channels feature lively, and often funny, discussion, and give you the unique opportunity to ask questions directly to members of the Twisted development team. Keep in mind, though, that such real-time support is a privilege, not a right. Be polite, and understand that developers might not always have time to answer your questions right at that moment. If you don’t get an immediate answer on IRC, try sending a message to the appropriate mailing list. This approach will give the question to a wider audience, and let people answer when they have more time.

A final resource available to the Twisted community is Planet Twisted . Located at http://planet.twistedmatrix.com, this web site aggregates weblog posts made by members of the Twisted development team. It’s an excellent way to keep track of what’s going on with Twisted development, as well as to get to know the personalities of the Twisted team. Planet Twisted also provides an RSS feed at http://planet.twistedmatrix.com/rss.xml.