Python Cookbook

Chapter 16. Extending and Embedding

Introduction

Credit: David Beazley, University of Chicago

One of Python’s most powerful features is its ability to hook to libraries and programs written in compiled languages, such as C, C++, and Fortran. In fact, a large number of Python’s built-in library modules are written as extension modules in C, so that operating-system services, networking functions, databases, and other features can be easily accessed from the interpreter. In addition, a number of application programmers are writing extensions, which can use Python as a framework for controlling large software packages written in compiled languages.

The gory details of how Python interfaces with other languages can be found in various Python programming books and at http://www.python.org. However, the general approach revolves around the creation of special wrapper functions that hook into the interpreter. For example, say you have a C function such as this:

int gcd(int x, int y) {
    int g = y;
    while (x > 0) {
        g = x;
        x = y % x;
        y = g;
    }
    return g;
}

If you want to access it from Python in a spam module, you’d have to write a special wrapper code like this:

#include "Python.h"
extern int gcd(int, int);

PyObject *wrap_gcd(PyObject *self, PyObject *args) {
    int x,y,g;
    if(!PyArg_ParseTuple(args, "ii", &x, &y))
       return NULL;
    g = gcd(x, y);
    return Py_BuildValue("i", g);
}

/* List of all functions in the module */
static PyMethodDef spammethods[] = {
   {"gcd", wrap_gcd, METH_VARARGS },
   { NULL, NULL }
};

/* Module initialization function */
void initspam(void) {
    Py_InitModule("spam", spammethods);
}

Once this code is compiled into an extension module, the gcd function is used as you would expect. For example:

>>> import spam
>>> spam.gcd(63,56)
7
>>> spam.gcd(71,89)
1

This short example extends in a natural way to larger programming libraries—each function that you want to access from Python simply gets its own wrapper.

Although writing simple extension functions is fairly straightforward, the process of writing wrappers quickly becomes tedious and prone to error if you are building anything of reasonable complexity. Therefore, a lot programmers rely on automatic module-building tools to simplify the process. Python is fortunate to have a variety of such tools:

bgen

A module-building tool found in the Tools directory of a standard Python distribution. Maintained by Jack Jansen, it is used to generate many of the extension modules available in the Macintosh version of Python.

pyfort

A tool developed by Paul Dubois that can be used to build extension modules for Fortran code. Details are available at http://pyfortran.sourceforge.net.

CXX

Also developed by Paul Dubois, CXX is a library that provides a C++ friendly API for writing Python extensions. An interesting feature of CXX is that it allows Python objects such as lists and tuples to be used naturally with algorithms in the STL. The library also provides support for converting C++ exceptions into Python exceptions. Information about CXX is available at http://cxx.sourceforge.net.

f2py

A wrapper generator for creating extensions in Fortran 90/95 developed by Pearu Peterson. Details are available at http://cens.ioc.ee/projects/f2py2e/.

SIP

A C++ module builder developed by Phil Thompson that creates wrappers for C++ classes. The system has most notably been used to create the PyQt and PyKDE extension modules. More information can be found at http://www.thekompany.com/projects/pykde.

WrapPy

Another C++ module builder that produces extension modules by reading C++ header files. It was developed by Greg Couch and is available at http://www.cgl.ucsf.edu/home/gregc/wrappy/index.html.

Boost Python Library

Developed by David Abrahams, the Boost Python Library provides one of the more unusual C++ wrapping techniques. Classes are automatically wrapped into Python extensions by simply writing a few additional C++ classes that specify information about the extension module. More information is available at http://www.boost.org/libs/python/doc/.

SWIG

An automatic extension-building tool that reads annotated C and C++ header files and produces extension modules for Python, Tcl, Perl, and a variety of other scripting languages. SWIG can wrap a large subset of C++ language features into an Python extension module. However, since I developed SWIG, I may be a little biased. In any event, further details are available at http://www.swig.org.

Regardless of the approach used to build Python extension modules, certain topics remain somewhat mysterious to many extension programmers. Therefore, the recipes in this chapter describe some of the common problems and extension-building tricks that are rarely covered in the standard documentation or other Python books. Topics include interacting with threads, returning NULL values, defining classes from C, implementing C/C++ functions in Python, creating extension types, and debugging.