Programming Amazon Web Services

Distributed Application Services with BOTO

The full value of Amazon’s SQS messaging service is most evident when it is used to connect the components of a distributed application, bringing together a set of computing resources to cooperate on a task. In this application, we will apply everything we have learned about the Amazon’s Web Services (AWS) so far to create a distributed application that can run arbitrary Linux applications to process data files any way you wish. This application will use S3 for storage, EC2 instances for computer processing power, and SQS messaging to tie the whole system together.

In this application, we will use the open source boto library (version 0.9d) as a starting point. This Python library includes support for the three Amazon services we will use in this example, but most importantly it also includes a framework to create and run service components that process SQS messages. The boto library will allow us to concentrate on the functionality of our application without having to build the underlying plumbing from scratch.

The Boto Services Framework

Mitch Garnaat, the creator of boto, has written an excellent article explaining how his library can be used as the basis for a distributed application that converts video files from one format to another. In the article “Monster Muck Mashup: Mass Video Conversion Using AWS,” he describes the architecture, message format, and processing steps used in the application, as well as offering a prepackaged public ...