Cover | Table of Contents | Colophon

Table of Contents

Chapter 1: The Programmable Web and Its Inhabitants

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

When you write a computer program, you’re not limited to the algorithms you can think up. Your language’s standard library gives you some algorithms. You can get more from books, or in third-party libraries you find online. Only if you’re on the very cutting edge should you have to come up with your own algorithms.

If you’re lucky, the same is true for data. Some applications are driven entirely by the data the users type in. Sometimes data just comes to you naturally: if you’re analyzing spam, you should have no problem getting all you need. You can download a few public data sets—word lists, geographical data, lists of prime numbers, public domain texts—as though they were third-party libraries. But if you need some other kind of data, it doesn’t look good. Where’s the data going to come from? More and more often, it’s coming from the programmable web.

When you—a human being—want to find a book on a certain topic, you probably point your web browser to the URI of an online library or bookstore: say, http://www.amazon.com/.

The common term for the address of something on the Web is “URL.” I say “URI” throughout this book because that’s what the HTTP standard says. Every URI on the Web is also a URL, so you can substitute “URL” wherever I say “URI” with no loss of meaning.

You’re served a web page, a document in HTML format that your browser renders graphically. You visually scan the page for a search form, type your topic (say, “web services”) into a text box, and submit the form. At this point your web browser makes a second HTTP request, to a URI that incorporates your topic. To continue the Amazon example, the second URI your browser requests would be something like http://amazon.com/s?url=search-alias%3Dstripbooks&field-keywords=web+services.

The web server at amazon.com responds by serving a second document in HTML format. This document contains a description of your search results, links to additional search options, and miscellaneous commercial enticements (see ). Again, your browser renders the document in graphical form, and you look at it and decide what to do from there.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Kinds of Things on the Programmable Web

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The programmable web is based on HTTP and XML. Some parts of it serve HTML, JavaScript Object Notation (JSON), plain text, or binary documents, but most parts use XML. And it’s all based on HTTP: if you don’t use HTTP, you’re not on the web.Beyond that small island of agreement there is little but controversy. The isn’t set, and different people use common terms (like “REST,” the topic of this book) in ways that combine into a vague and confusing mess. What’s missing is a coherent way of classifying the programmable web. With that in place, the meanings of individual terms will become clear.

Imagine the programmable web as an ecosystem, like the ocean, containing many kinds of strange creatures. Ancient scientists and sailors classified sea creatures by their superficial appearance: whales were lumped in with the fish. Modern scientists classify animals according to their position in the evolutionary tree of all life: whales are now grouped with the other mammals. There are two analogous ways of classifying the services that inhabit the programmable web: by the technologies they use (URIs, SOAP, XML-RPC, and so on), or by the underlying architectures and design philosophies.

Usually the two systems for classifying sea creatures get along. You don’t need to do DNA tests to know that a tuna is more like a grouper than a sea anenome. But if you really want to understand why whales can’t breathe underwater, you need to stop classifying them as fish (by superficial appearance) and start classifying them as mammals (by underlying architecture).

When it comes to classifying the programmable web, most of today’s terminology sorts services by their superficial appearances: the technologies they use. These classifications work in most cases, but they’re conceptually lacking and they lead to whale-fish mistakes. I’m going to present a taxonomy based on architecture, which shows how technology choices follow from underlying design principles. I’m exposing divisions I’ll come back to throughout the book, but my main purpose is to zoom in on the parts of the programmable web that can reasonably be associated with the term “REST.”

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

HTTP: Documents in Envelopes

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

If I was classifying marine animals I’d start by talking about the things they have in common: DNA, cellular structure, the laws of embryonic development. Then I’d show how animals distinguish themselves from each other by specializing away from the common ground. To classify the programmable web, I’d like to start off with an overview of HTTP, the protocol that all web services have in common.

HTTP is a document-based protocol, in which the client puts a document in an envelope and sends it to the server. The server returns the favor by putting a response document in an envelope and sending it to the client. HTTP has strict standards for what the envelopes should look like, but it doesn’t much care what goes inside. shows a sample envelope: the HTTP request my web browser sends when I visit the homepage of oreilly.com. I’ve truncated two lines to make the text fit on the printed page.

Example . An HTTP GET request for http://www.oreilly.com/index.html

GET /index.html HTTP/1.1
Host: www.oreilly.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12)...
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,...
Accept-Language: us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-15,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

In case you’re not familiar with HTTP, now is a good time to point out the major parts of the HTTP request. I use these terms throughout the book.

The HTTP method

In this request, the method is “GET.” In other discussions of REST you may see this called the “HTTP verb” or “HTTP action.”

The name of the HTTP method is like a method name in a programming language: it indicates how the client expects the server to process this envelope. In this case, the client (my web browser) is trying to GET some information from the server (www.oreilly.com).

The path

This is the portion of the URI to the right of the hostname: here, http://www.oreilly.com/index.html becomes “/index.html.” In terms of the envelope metaphor, the path is the address on the envelope. In this book I sometimes refer to the “URI” as shorthand for just the path.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Method Information

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

HTTP is the one thing that all “animals” on the programmable web have in common. Now I’ll show you how web services distinguish themselves from each other. There are two big questions that today’s web services answer differently. If you know how a web service answers these questions, you’ll have a good idea of how well it works with the Web.

The first question is how the client can convey its intentions to the server. How does the server know a certain request is a request to retrieve some data, instead of a request to delete that same data or to overwrite it with different data? Why should the server do this instead of doing that?

I call the information about what to do with the data the method information. One way to convey method information in a web service is to put it in the HTTP method. Since this is how RESTful web services do it, I’ll have a lot more to say about this later. For now, note that the five most common HTTP methods are GET, HEAD, PUT, DELETE, and POST. This is enough to distinguish between “retrieve some data” (GET), “delete that same data” (DELETE), and “overwrite it with different data” (PUT).

The great advantage of HTTP method names is that they’re standardized. Of course, the space of HTTP method names is much more limited than the space of method names in a programming language. Some web services prefer to look for application-specific method names elsewhere in the HTTP request: usually in the URI path or the request document.

is a client for a web service that keeps its method information in the path: the web service for Flickr, Yahoo!’s online photo-sharing application. This sample application searches Flickr for photos. To run this program, you’ll need to create a Flickr account and apply for an API key.

Example . Searching Flickr for pictures

#!/usr/bin/ruby -w
# flickr-photo-search.rb
require 'open-uri'
require 'rexml/document'

# Returns the URI to a small version of a Flickr photo.
def small_photo_uri(photo)
  server = photo.attribute('server')
  id = photo.attribute('id')
  secret = photo.attribute('secret')
  return "http://static.flickr.com/#{server}/#{id}_#{secret}_m.jpg"
end

# Searches Flickr for photos matching a certain tag, and prints a URI
# for each search result.
def print_each_photo(api_key, tag)
  # Build the URI
  uri = "http://www.flickr.com/services/rest?method=flickr.photos.search" +
    "&api_key=#{api_key}&tags=#{tag}"

  # Make the HTTP request and get the entity-body.
  response = open(uri).read

  # Parse the entity-body as an XML document.
  doc = REXML::Document.new(response)

  # For each photo found...
  REXML::XPath.each(doc, '//photo') do |photo| 
    # ...generate and print its URI
    puts small_photo_uri(photo) if photo
  end
end

# Main program
#
if ARGV.size < 2
  puts "Usage: #{$0} [Flickr API key] [search term]"
  exit
end

api_key, tag = ARGV
print_each_photo(api_key, tag)

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Scoping Information

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The other big question web services answer differently is how the client tells the server which part of the data set to operate on. Given that the server understands that the client wants to (say) delete some data, how can it know which data the client wants to delete? Why should the server operate on this data instead of that data?

I call this information the scoping information. One obvious place to put it is in the URI path. That’s what most web sites do. Think once again about a search engine URI like http://www.google.com/search?q=REST. There, the method information is “GET,” and the scoping information is “/search?q=REST.” The client is trying to GET a list of search results about REST, as opposed to trying to GET something else: say, a list of search results about jellyfish (the scoping information for that would be “/search?q=jellyfish”), or the Google home page (that would be “/”).

Many web services put scoping information in the path. Flickr’s is one: most of the query variables in a Flickr API URI are scoping information. tags=penguin scopes the flickr.photos.search method so it only searches for photos tagged with “penguin.” In a service where the method information defines a method in the programming language sense, the scoping information can be seen as a set of arguments to that method. You could reasonably expect to see flickr.photos.search(tags=penguin) as a line of code in some programming language.

The alternative is to put the scoping information into the entity-body. A typical SOAP web service does it this way. contains a q tag whose contents are the string “REST.” That’s the scoping information, nestled conveniently inside the doGoogleSearch tag that provides the method information.

The service design determines what information is method information and what’s scoping information. This is most obvious in cases like Flickr and Google, where the web site and the web service do the same thing but have different designs. These two URIs contain the same information:

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

The Competing Architectures

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Now that I’ve identified the two main questions that web services answer differently, I can group web services by their answers to the questions. In my studies I’ve identified three common web service architectures: RESTful resource-oriented, RPC-style, and REST-RPC hybrid. I’ll cover each in turn.

The main topic of this book is the web service architectures which can be considered RESTful: those which get a good score when judged on the criteria set forth in Roy Fielding’s dissertation. Now, lots of architectures are technically RESTful,but I want to focus on the architectures that are best for web services. So when I talk about RESTful web services, I mean services that look like the Web. I’m calling this kind of service resource-oriented. In I’ll introduce the basic concepts of resource-oriented REST, in the context of a real web service: Amazon’s Simple Storage Service. Starting in , I’ll talk you through the defining characteristics of REST, and define a good architecture for RESTful web services: the Resource-Oriented Architecture.

In RESTful architectures, the method information goes into the HTTP method. In Resource-Oriented Architectures, the scoping information goes into the URI. The combination is powerful. Given the first line of an HTTP request to a resource-oriented RESTful web service (“GET /reports/open-bugs HTTP/1.1”), you should understand basically what the client wants to do. The rest of the request is just details; indeed, you can make many requests using only one line of HTTP. If the HTTP method doesn’t match the method information, the service isn’t RESTful. If the scoping information isn’t in the URI, the service isn’t resource-oriented. These aren’t the only requirements, but they’re good rules of thumb.

A few well-known examples of RESTful, resource-oriented web services include:

• Services that expose the Atom Publishing Protocol and its variants such as GData
• Amazon’s Simple Storage Service (S3)

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Technologies on the Programmable Web

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I’ve classified web services by their underlying architectures, distinguishing the fish from the whales. Now I can examine the technologies they use, without confusing technology and architecture.

All web services use HTTP, but they use it in different ways. A request to a RESTful web services puts the method information in the HTTP method and the scoping information in the URI. RPC-style web services tend to ignore the HTTP method, looking for method and scoping information in the URI, HTTP headers, or entity-body. Some RPC-style web services use HTTP as an envelope containing a document, and others only use it as an unlabelled envelope containing another envelope.

Again, all web services use URIs, but in different ways. What I’m about to say is a generalization, but a fairly accurate one. A RESTful, resource-oriented service exposes a URI for every piece of data the client might want to operate on. A REST-RPC hybrid exposes a URI for every operation the client might perform: one URI to fetch a piece of data, a different URI to delete that same data. An RPC-style service exposes one URI for every processes capable of handling Remote Procedure Calls (RPC). There’s usually only one such URI: the service “endpoint.”

A few, mostly legacy, web services use XML-RPC on top of HTTP. XML-RPC is a data structure format for representing function calls and their return values. As the name implies, it’s explicitly designed to use an RPC style.

Lots of web services use SOAP on top of HTTP. SOAP is an envelope format, like HTTP, but it’s an XML-based envelope format.

Now I’m going to say something controversial. To a first approximation, every current web service that uses SOAP also has an RPC architecture. This is controversial because many SOAP programmers think the RPC architecture is déclassé and prefer to call their services “message-oriented” or “document-oriented” services.

Well, all web services are message-oriented, because HTTP itself is message-oriented. An HTTP request is just a message: an envelope with a document inside. The question is what that document says. SOAP-based services ask the client to stick a second envelope (a SOAP document) inside the HTTP envelope. Again, the real question is what it says inside the envelope. A SOAP envelope can contain any XML data, just as an HTTP envelope can contain any data in its entity-body. But in every

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Leftover Terminology

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Believe it not, there are some common terms used in discussions of REST that I haven’t mentioned yet. I haven’t mentioned them because I think they’re inaccurate or entirely outside the scope of this book. But I owe you explanations of why I think this, so you can decide whether or not you agree. Feel free to skip this section if you haven’t heard these terms.

Service-Oriented Architecture

This is a big industry buzzword. I’m not going to dwell on it for two reasons. First, the term is not very well defined. Second, to the extent that it is defined, it means something like: “a software architecture based on the production and consumption of web services.” In this book I talk about the design of individual services. A book on service-oriented architecture should work on a slightly higher level, showing how to use services as software components, how to integrate them into a coherent whole. I don’t cover that sort of thing in this book.

SOAP as a competitor to REST

If you get involved with web service debates you’ll hear this one a lot. You won’t hear it here because it gives the wrong impression. The primary competitors to RESTful architectures are RPC architectures, not specific technologies like SOAP. It is true that basically every SOAP service that now exists has an RPC architecture, but SOAP is just a way of putting a document in an envelope with stickers on it, like HTTP. SOAP is tied to the RPC architecture mainly by historical contingency and the current generation of automated tools.

There is a real tension here, but it’s not one I’ll cover much in this book. Roughly speaking, it’s the tension between services that put their documents in a SOAP envelope and then an HTTP envelope; and services that only use the HTTP envelope.

HTTP+POX

Stands for HTTP plus Plain Old XML. This term covers roughly those services I call REST-RPC hybrid services. They overlap with RESTful designs, especially when it comes to retrieving data, but their basic architecture is RPC-oriented.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 2: Writing Web Service Clients

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In I showed some quick examples of clients for existing, public web services. Some of the services had resource-oriented RESTful architectures, some had RPC-style architectures, and some were hybrids. Most of the time, I accessed these services through wrapper libraries instead of making the HTTP requests myself.

You can’t always rely on the existence of a convenient wrapper library for your favorite web service, especially if you wrote the web service yourself. Fortunately, it’s easy to write programs that work directly with HTTP requests and responses. In this chapter I show how to write clients for RESTful and hybrid architecture services, in a variety of programming languages.

is a bare HTTP client for a RESTful web service: Yahoo!’s web search. You might compare it to , the client from the previous chapter that runs against the RPC-style SOAP interface to Google’s web search.

Example . Searching the Web with Yahoo!’s web service

#!/usr/bin/ruby
# yahoo-web-search.rb
require 'open-uri'
require 'rexml/document'
require 'cgi'

BASE_URI = 'http://api.search.yahoo.com/WebSearchService/V1/webSearch'

def print_page_titles(term)
  # Fetch a resource: an XML document full of search results.
  term = CGI::escape(term)
  xml = open(BASE_URI + "?appid=restbook&query=#{term}").read
  
  # Parse the XML document into a data structure.
  document = REXML::Document.new(xml)
  
  # Use XPath to find the interesting parts of the data structure.
  REXML::XPath.each(document, '/ResultSet/Result/Title/[]') do |title|
    puts title
  end        
end

(puts "Usage: #{$0} [search term]"; exit) if ARGV.empty?
print_page_titles(ARGV.join(' '))

This “web service” code looks just like generic HTTP client code. It uses Ruby’s standard open-uri library to make an HTTP request and Ruby’s standard REXML library to parse the output. I’d use the same tools to fetch and process a web page. These two URIs:

• http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=restbook&query=jellyfish
• http://search.yahoo.com/search?p=jellyfish

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Web Services Are Web Sites

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In I showed some quick examples of clients for existing, public web services. Some of the services had resource-oriented RESTful architectures, some had RPC-style architectures, and some were hybrids. Most of the time, I accessed these services through wrapper libraries instead of making the HTTP requests myself.

You can’t always rely on the existence of a convenient wrapper library for your favorite web service, especially if you wrote the web service yourself. Fortunately, it’s easy to write programs that work directly with HTTP requests and responses. In this chapter I show how to write clients for RESTful and hybrid architecture services, in a variety of programming languages.

is a bare HTTP client for a RESTful web service: Yahoo!’s web search. You might compare it to , the client from the previous chapter that runs against the RPC-style SOAP interface to Google’s web search.

Example . Searching the Web with Yahoo!’s web service

#!/usr/bin/ruby
# yahoo-web-search.rb
require 'open-uri'
require 'rexml/document'
require 'cgi'

BASE_URI = 'http://api.search.yahoo.com/WebSearchService/V1/webSearch'

def print_page_titles(term)
  # Fetch a resource: an XML document full of search results.
  term = CGI::escape(term)
  xml = open(BASE_URI + "?appid=restbook&query=#{term}").read
  
  # Parse the XML document into a data structure.
  document = REXML::Document.new(xml)
  
  # Use XPath to find the interesting parts of the data structure.
  REXML::XPath.each(document, '/ResultSet/Result/Title/[]') do |title|
    puts title
  end        
end

(puts "Usage: #{$0} [search term]"; exit) if ARGV.empty?
print_page_titles(ARGV.join(' '))

This “web service” code looks just like generic HTTP client code. It uses Ruby’s standard open-uri library to make an HTTP request and Ruby’s standard REXML library to parse the output. I’d use the same tools to fetch and process a web page. These two URIs:

• http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=restbook&query=jellyfish
• http://search.yahoo.com/search?p=jellyfish

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

del.icio.us: The Sample Application

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In this chapter I walk through the life cycle of a web service request from the client’s point of view. Though most of this book’s code examples are written in Ruby, in this chapter I show code written in a variety of programming languages. My example throughout this chapter is the web service provided by the social bookmarking web site del.icio.us. You can read a prose description of this web service at http://del.icio.us/help/api/.

If you’re not familiar with del.icio.us, here’s a brief digressionary introduction. del.icio.us is a web site that works like your web browser’s bookmark feature, but it’s public and better-organized (see ). When you save a link to del.icio.us, it’s associated with your account so you can find it later. You can also share your bookmarks with others.

You can associate short strings, called tags, with a URI. Tags are versatile little suckers. They make it easy for you to find a URI later, they make it possible to group URIs together, and when multiple people tag the same URI, they create a machine-readable vocabulary for that URI.

Figure : del.icio.us screenshot

The del.icio.us web service gives you programmatic access to your bookmarks. You can write programs that bookmark URIs, convert your browser bookmarks to del.icio.us bookmarks, or fetch the URIs you’ve bookmarked in the past. The best way to visualize the del.icio.us web service is to use the human-oriented web site for a while. There’s no fundamental difference between the del.icio.us web site and the del.icio.us web service, but there are variations:

• The web site is rooted at http://del.icio.us/ and the web service is rooted at https://api.del.icio.us/v1/. The web site communicates with clients through HTTP, the web service uses secure HTTPS.
• The web site and the web service expose different URI structures. To get your recent bookmarks from the web site, you fetch https://del.icio.us/{your-username}. To get your recent bookmarks from the web service, you fetch

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Making the Request: HTTP Libraries

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Every modern programming language has one or more libraries for making HTTP requests. Not all of these libraries are equally useful, though. To build a fully general web service client you need an HTTP library with these features:

• It must support HTTPS and SSL certificate validation. Web services, like web sites, use HTTPS to secure communication with their clients. Many web services (del.icio.us is one example) won’t accept plain HTTP requests at all. A library’s HTTPS support often depends on the presense of an external SSL library written in C.
• It must support at least the five main HTTP methods: GET, HEAD, POST, PUT, and DELETE. Some libraries support only GET and POST. Others are designed for simplicity and support only GET.
  You can get pretty far with a client that only supports GET and POST: HTML forms support only those two methods, so the entire human web is open to you. You can even do all right with just GET, because many web services (among them del.icio.us and Flickr) use GET even where they shouldn’t. But if you’re choosing a library for all your web service clients, or writing a general client like a WADL client, you need a library that supports all five methods. Additional methods like OPTIONS and TRACE, and WebDAV extensions like MOVE, are a bonus.
• It must allow the programmer to customize the data sent as the entity-body of a PUT or POST request.
• It must allow the programmer to customize a request’s HTTP headers.
• It must give the programmer access to the response code and headers of an HTTP response; not just access to the entity-body.
• It must be able to communicate through an HTTP proxy. The average programmer may not think about this, but many HTTP clients in corporate environments can only work through a proxy. Intermediaries like HTTP proxies are also a standard part of the REST meta-architecture, though not one I’ll be covering in much detail.

There are also some features of an HTTP library that make life easier as you write clients for RESTful and hybrid services. These features mostly boil down to knowledge about HTTP headers, so they’re technically optional. You can implement them yourself so long as your library gives you access to request and response HTTP headers. The advantage of library support is that you don’t have to worry about the details.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Processing the Response: XML Parsers

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The entity-body is usually the most important part of an HTTP response. Where web services are concerned, the entity-body is usually an XML document, and the client gets most of the information it needs by running this document through an XML parser.

Now, there are many HTTP client libraries, but they all have exactly the same task. Given a URI, a set of headers, and a body document, the client’s job is to construct an HTTP request and send it to a certain server. Some libraries have more features than others: cookies, authentication, caching, and the other ones I mentioned. But all these extra features are implemented within the HTTP request, usually as extra headers. A library might offer an object-oriented interface (like Net::HTTP) or a file-like interface (like open-uri), but both interfaces do the same thing. There’s only one kind of HTTP client library.

But there are three kinds of XML parsers. It’s not just that some XML parsers have features that others lack, or that one interface is more natural than another. There are two basic XML parsing strategies: the document-based strategy of DOM and other tree-style parsers, and the event-based strategy of SAX and “pull” parsers. You can get a tree-style or a SAX parser for any programming language, and a pull parser for almost any language.

The document-based, tree-style strategy is the simplest of the three models. A tree-style parser models an XML document as a nested data structure. Once you’ve got this data structure, you can search and process it with XPath queries, CSS selectors, or custom navigation functions: whatever your parser supports. A DOM parser is a tree-style parser that implements a specific interface defined by the W3C.

The tree-style strategy is easy to use, and it’s the one I use the most. With a tree-style parser, the document is just an object like the other objects in your program. The big shortcoming is that you have to deal with the document as a whole. You can’t start working on the document until you’ve processed the whole thing into a tree, and you can’t avoid loading the whole document into memory. For documents that are simple but very large, this is inefficient. It would be a lot better to handle tags as they’re parsed.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

JSON Parsers: Handling Serialized Data

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Most web services return XML documents, but a growing number return simple data structures (numbers, arrays, hashes, and so on), serialized as JSON-formatted strings. JSON is usually produced by services that expect to be consumed by the client half of an Ajax application. The idea is that it’s a lot easier for a browser to get a JavaScript data structure from a JSON data structure than from an XML document. Every web browser offers a slightly different JavaScript interface to its XML parser, but a JSON string is nothing but a tightly constrained JavaScript program, so it works the same way in every browser.

Of course, JSON is not tied to JavaScript, any more than JavaScript is to Java. JSON makes a lightweight alternative to XML-based approaches to data serialization, like XML Schema. The JSON web site links to implementations in many languages, and I refer you to that site rather than mentioning a JSON library for every language.

JSON is a simple and language-independent way of formatting programming language data structures (numbers, arrays, hashes, and so on) as strings. is a JSON representation of a simple data structure: a mixed-type array.

Example . A mixed-type array in JSON format

[3, "three"]

By comparison, is one possible XML representation of the same data.

Example . A mixed-type array in XML-RPC format

<value>
 <array>
  <data>
   <value><i4>3</i4></value>
   <value><string>three</string></value>
  </data>
 </array>
</value>

Since a JSON string is nothing but a tightly constrained JavaScript program, you can “parse” JSON simply by calling eval on the string. This is very fast, but you shouldn’t do it unless you control the web service that served your JSON. An untested or untrusted web service can send the client buggy or malicious JavaScript programs instead of real JSON structures. For the JavaScript examples in , I use a JSON parser written in JavaScript and available from json.org (see ).

Example . A JSON demo in JavaScript

<!-- json-demo.html -->
<!-- In a real application, you would save json.js locally
     instead of fetching it from json.org every time. -->
<script type="text/javascript" src="http://www.json.org/json.js">
</script>

<script type="text/javascript">
 array = [3, "three"]
 alert("Converted array into JSON string: '" + array.toJSONString() + "'")
  json = "[4, \"four\"]"
 alert("Converted JSON '" + json + "' into array:")
 array2 = json.parseJSON()
 for (i=0; i < array2.length; i++) 
 {
   alert("Element #" + i + " is " + array2[i])
 }
</script>

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Clients Made Easy with WADL

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

So far I’ve presented code in a variety of languages, but it always follows the same three-step pattern. To call a web service I build up the elements of an HTTP request (method, URI, headers, and entity-body). I use an HTTP library to turn that data into a real HTTP request, and the library sends the request to the appropriate server. Then I use an XML parser to parse the response into a data structure or a series of events. Once I make the request, I’m free to use the response data however I like. In this regard all RESTful web services, and most hybrid services, are the same. What’s more, as I’ll show in the chapters to come, all RESTful web services use HTTP the same way: HTTP has what’s called a uniform interface.

Can I take advantage of this similarity? Abstract this pattern out into a generic “REST library” that can access any web service that supports the uniform interface? There’s precedent for this. The Web Service Description Language (WSDL) describes the differences between RPC-style web services in enough detail that a generic library can access any RPC-style SOAP service, given an appropriate WSDL file.

For RESTful and hybrid services, I recommend using the Web Application Description Language. A WADL file describes the HTTP requests you can legitimately make of a service: which URIs you can visit, what data those URIs expect you to send, and what data they serve in return. A WADL library can parse this file and model the space of possible service requests as a native language API.

I describe WADL in more detail in , but here’s a taste. The del.icio.us client shown in is equivalent to the Ruby client in , but it uses Ruby’s WADL library and a bootleg WADL file I created for del.icio.us. (I’ll show you the WADL file in .)

Example . A Ruby/WADL client for del.icious

#!/usr/bin/ruby
# delicious-wadl-ruby.rb
require 'wadl'

if ARGV.size != 2
  puts "Usage: #{$0} [username] [password]"
  exit
end
username, password = ARGV

# Load an application from the WADL file
delicious = WADL::Application.from_wadl(open("delicious.wadl"))

# Give authentication information to the application
service = delicious.v1.with_basic_auth(username, password)

begin
  # Find the "recent posts" functionality
  recent_posts = service.posts.recent

  # For every recent post...
  recent_posts.get.representation.each_by_param('post') do |post|
    # Print its description and URI.
    puts "#{post.attributes['description']}: #{post.attributes['href']}"
  end
rescue WADL::Faults::AuthorizationRequired
  puts "Invalid authentication information!"
end

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 3: What Makes RESTful Services Different?

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I pulled a kind of bait-and-switch on you earlier, and it’s time to make things right. Though this is a book about RESTful web services, most of the real services I’ve shown you are REST-RPC hybrids like the del.icio.us API: services that don’t quite work like the rest of the Web. This is because right now, there just aren’t many well-known RESTful services that work like the Web. In previous chapters I wanted to show you clients for real services you might have heard of, so I had to take what I could get.

The del.icio.us and Flickr APIs are good examples of hybrid services. They work like the Web when you’re fetching data, but they’re RPC-style services when it comes time to modify the data. The various Yahoo! search services are very RESTful, but they’re so simple that they don’t make good examples. The Amazon E-Commerce Service (seen in ) is also quite simple, and defects the RPC style on a few obscure but important points.

These services are all useful. I think the RPC style is the wrong one for web services, but that never prevents me from writing an RPC-style client if there’s interesting data on the other side. I can’t use Flickr or the del.icio.us API as examples of how to design RESTful web services, though. That’s why I covered them early in the book, when the only thing I was trying to show was what’s on the programmable web and how to write HTTP clients. Now that we’re approaching a heavy design chapter, I need to show you what a service looks like when it’s RESTful and resource-oriented.

Two popular web services can answer this call: the Atom Publishing Protocol (APP), and Amazon’s Simple Storage Service (S3). ( lists some publicly deployed RESTful web services, many of which you may not have heard of.) The APP is less an actual service than a set of instructions for building a service, so I’m going to start with S3, which actually exists at a specific place on the Web. In I discuss the APP, Atom, and related topics like Google’s GData. For much of the rest of this chapter, I’ll explore S3.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Introducing the Simple Storage Service

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Two popular web services can answer this call: the Atom Publishing Protocol (APP), and Amazon’s Simple Storage Service (S3). ( lists some publicly deployed RESTful web services, many of which you may not have heard of.) The APP is less an actual service than a set of instructions for building a service, so I’m going to start with S3, which actually exists at a specific place on the Web. In I discuss the APP, Atom, and related topics like Google’s GData. For much of the rest of this chapter, I’ll explore S3.

S3 is a way of storing any data you like, structured however you like. You can keep your data private, or make it accessible by anyone with a web browser or BitTorrent client. Amazon hosts the storage and the bandwidth, and charges you by the gigabyte for both. To use the example S3 code in this chapter, you’ll need to sign up for the S3 service by going to http://aws.amazon.com/s3. The S3 technical documentation is at http://docs.amazonwebservices.com/AmazonS3/2006-03-01/.

There are two main uses for S3, as a:

Backup server

You store your data through S3 and don’t give anyone else access to it. Rather than buying your own backup disks, you’re renting disk space from Amazon.

Data host

You store your data on S3 and give others access to it. Amazon serves your data through HTTP or BitTorrent. Rather than paying an ISP for bandwidth, you’re paying Amazon. Depending on your existing bandwidth costs this can save you a lot of money. Many of today’s web startups use S3 to serve data files.

Unlike the services I’ve shown so far, S3 is not inspired by any existing web site. The del.icio.us API is based on the del.icio.us web site, and the Yahoo! search services are based on corresponding web sites, but there’s no web page on amazon.com where you fill out HTML forms to upload your files to S3. S3 is intended only for programmatic use. (Of course, if you use S3 as a data host, people will use it through their web browsers, without even knowing they’re making a web service call. It’ll act like a normal web site.)

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Object-Oriented Design of S3

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

S3 is based on two concepts: S3 “buckets” and S3 “objects.” An object is a named piece of data with some accompanying metadata. A bucket is a named container for objects. A bucket is analogous to the filesystem on your hard drive, and an object to one of the files on that filesystem. It’s tempting to compare a bucket to a directory on a filesystem, but filesystem directories can be nested and buckets can’t. If you want a directory structure inside your bucket, you need to simulate one by giving your objects names like “directory/subdirectory/file-object.”

A bucket has one piece of information associated with it: the name. A bucket name can only contain the characters A through Z, a through z, 0 through 9, underscore, period, and dash. I recommend staying away from uppercase letters in bucket names.

As I mentioned above, buckets cannot contain other buckets: only objects. Each S3 user is limited to 100 buckets, and your bucket name cannot conflict with anyone else’s. I recommend you either keep everything in one bucket, or name each bucket after one of your projects or domain names.

An object has four parts to it:

• A reference to the parent bucket.
• The data stored in that object (S3 calls this the “value”).
• A name (S3 calls it the “key”).
• A set of metadata key-value pairs associated with the object. This is mostly custom metadata, but it may also include values for the standard HTTP headers Content-Type and Content-Disposition.

If I wanted to host the O’Reilly web site on S3, I’d create a bucket called “oreilly.com,” and fill it with objects whose keys were “” (the empty string), “catalog,” “catalog/9780596529260,” and so on. These objects correspond to the URIs http://oreilly.com/, http://oreilly.com/catalog, and so on. The object’s values would be the HTML contents of O’Reilly’s web pages. These S3 objects would have their Content-Type metadata value set to text/html, so that people browsing the site would be served these objects as HTML documents, as opposed to XML or plain text.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Resources

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Amazon exposes S3 as two different web services: a RESTful service based on plain HTTP envelopes, and an RPC-style service based on SOAP envelopes. The RPC-style service exposes functions much like the methods in ’s hypothetical Ruby library: ListAllMyBuckets, CreateBucket, and so on. Indeed, many RPC-style web services are automatically generated from their implementation methods, and expose the same interfaces as the programming-language code they call behind the scenes. This works because most modern programming (including object-oriented programming) is procedural.

The RESTful S3 service exposes all the functionality of the RPC-style service, but instead of doing it with custom-named functions, it exposes standard HTTP objects called resources. Instead of responding to custom method names like getObjects, a resource responds to one or more of the six standard HTTP methods: GET, HEAD, POST, PUT, DELETE, and OPTIONS.

The RESTful S3 service provides three types of resources. Here they are, with sample URIs for each:

• The list of your buckets (https://s3.amazonaws.com/). There’s only one resource of this type.
• A particular bucket (https://s3.amazonaws.com/{name-of-bucket}/). There can be up to 100 resources of this type.
• A particular S3 object inside a bucket (https://s3.amazonaws.com/{name-of-bucket}/{name-of-object}). There can be infinitely many resources of this type.

Each method from my hypothetical object-oriented S3 library corresponds to one of the six standard methods on one of these three types of resources. The getter method S3Object#name corresponds to a GET request on an “S3 object” resource, and the setter method S3Object#value= corresponds to a PUT request on the same resource. Factory methods like S3Bucket.getBuckets and relational methods like S3Bucket#getObjects correspond to GET methods on the “bucket list” and “bucket” resources.

Every resource exposes the same interface and works the same way. To get an object’s value you send a GET request to that object’s URI. To get only the metadata for an object you send a HEAD request to the same URI. To create a bucket, you send a PUT request to a URI that incorporates the name of the bucket. To add an object to a bucket, you send PUT to a URI that incorporates the bucket name and object name. To delete a bucket or an object, you send a DELETE request to its URI.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

HTTP Response Codes

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Another defining feature of a RESTful architecture is its use of HTTP response codes. If you send a request to S3, and S3 handles it with no problem, you’ll probably get back an HTTP response code of 200 (“OK”), just like when you successfully fetch a web page in your browser. If something goes wrong, the response code will be in the 3xx, 4xx, or 5xx range: for instance, 500 (“Internal Server Error”). An error response code is a signal to the client that the metadata and entity-body should not be interpreted as a response to the request. It’s not what the client asked for: it’s the server’s attempt to tell the client about a problem. Since the response code isn’t part of the document or the metadata, the client can see whether or not an error occurred just by looking at the first three bytes of the response.

shows a sample error response. I made an HTTP request for an object that didn’t exist (https://s3.amazonaws.com/crummy.com/nonexistent/object). The response code is 404 (“Not Found”).

Example . A sample error response from S3

404 Not Found
Content-Type: application/xml
Date: Fri, 10 Nov 2006 20:04:45 GMT
Server: AmazonS3
Transfer-Encoding: chunked
X-amz-id-2: /sBIPQxHJCsyRXJwGWNzxuL5P+K96/Wvx4FhvVACbjRfNbhbDyBH5RC511sIz0w0
X-amz-request-id: ED2168503ABB7BF4

<?xml version="1.0" encoding="UTF-8"?>
<Error>
 <Code>NoSuchKey</Code>
 <Message>The specified key does not exist.</Message>
 <Key>nonexistent/object</Key>
 <RequestId>ED2168503ABB7BF4</RequestId>
 <HostId>/sBIPQxHJCsyRXJwGWNzxuL5P+K96/Wvx4FhvVACbjRfNbhbDyBH5RC511sIz0w0</HostId>
</Error>

HTTP response codes are underused on the human web. Your browser doesn’t show you the HTTP response code when you request a page, because who wants to look at a numeric code when you can just look at the document to see whether something went wrong? When an error occurs in a web application, most web applications send 200 (“OK”) along with a human-readable document that talks about the error. There’s very little chance a human will mistake the error document for the document they requested.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

An S3 Client

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The Amazon sample libraries, and the third-party contributions like AWS::S3, eliminate much of the need for custom S3 client libraries. But I’m not telling you about S3 just so you’ll know about a useful web service. I want to use it to illustrate the theory behind REST. So I’m going to write a Ruby S3 client of my own, and dissect it for you as I go along.

Just to show it can be done, my library will implement an object-oriented interface, like the one from , on top of the S3 service. The result will look like ActiveRecord or some other object-relational mapper. Instead of making SQL calls under the covers to store data in a database, though, it’ll make HTTP requests under the covers to store data on the S3 service. Rather than give my methods resource-specific names like getBuckets and getObjects, I’ll try to use names that reflect the underlying RESTful interface: get, put, and so on.

The first thing I need is an interface to Amazon’s rather unusual web service authorization mechanism. But that’s not as interesting as seeing the web service in action, so I’m going to skip it for now. I’m going to create a very small Ruby module called S3::Authorized, just so my other S3 classes can include it. I’ll come back to it at the end, and fill in the details.

shows a bit of throat-clearing code.

Example . S3 Ruby client: Initial code

#!/usr/bin/ruby -w
# S3lib.rb

# Libraries neccessary for making HTTP requests and parsing responses.
require 'rubygems'
require 'rest-open-uri'
require 'rexml/document'

# Libraries neccessary for request signing
require 'openssl'
require 'digest/sha1'
require 'base64'
require 'uri'

module S3 # This is the beginning of a big, all-encompassing module.

module Authorized
  # Enter your public key (Amazon calls it an "Access Key ID") and
  # your private key (Amazon calls it a "Secret Access Key"). This is
  # so you can sign your S3 requests and Amazon will know who to
  # charge.
  @@public_key = ''
  @@private_key = ''

  if @@public_key.empty? or @@private_key.empty?	
    raise "You need to set your S3 keys."
  end

  # You shouldn't need to change this unless you're using an S3 clone like
  # Park Place.
  HOST = 'https://s3.amazonaws.com/'
end

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Request Signing and Access Control

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I’ve put it off as long as I can, and now it’s time to deal with S3 authentication. If your main interest is in RESTful services in general, feel free to skip ahead to the section on using the S3 library in clients. But if the inner workings of S3 have piqued your interest, read on.

The code I’ve shown you so far makes HTTP requests all right, but S3 rejects them, because they don’t contain the all-important Authorization header. S3 has no proof that you’re the owner of your own buckets. Remember, Amazon charges you for the data stored on their servers and the bandwidth used in transferring that data. If S3 accepted requests to your buckets with no authorization, anyone could store data in your buckets and you’d get charged for it.

Most web services that require authentication use a standard HTTP mechanism to make sure you are who you claim to be. But S3’s needs are more complicated. With most web services you never want anyone else using your data. But one of the uses of S3 is as a hosting service. You might want to host a big movie file on S3, let anyone download it with their BitTorrent client, and have Amazon send you the bill.

Or you might be selling access to movie files stored on S3. Your e-commerce site takes payment from a customer and gives them an S3 URI they can use to download the movie. You’re delegating to someone else the right to make a particular web service call (a GET request) as you, and have it charged to your account.

The standard mechanisms for HTTP authentication can’t provide security for that kind of application. Normally, the person who’s sending the HTTP request needs to know the actual password. You can prevent someone from spying on your password, but you can’t say to someone else: “here’s my password, but you must promise only to use it to request this one URI.”

This is a job for public-key cryptography. Every time you make an S3 request, you use your “private” key (remember, not truly private: Amazon knows it too) to sign the important parts of the request. That’d be the URI, the HTTP method you’re using, and a few of the HTTP headers. Only someone with the “private” key can create these signatures for your requests, which is how Amazon knows it’s okay to charge you for the request. But once you’ve signed a request, you can send the signature to a third party without revealing your “private” key. The third party is then free to send an identical HTTP request to the one you signed, and have Amazon charge you for it. In short: someone else can make a specific request as you, for a limited time, without having to know your “private” key.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Using the S3 Client Library

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I’ve now shown you a Ruby client library that can access just about the full capabilities of Amazon’s S3 service. Of course, a library is useless without clients that use it. In the previous section I showed you a couple of small clients to demonstrate points about security, but now I’d like to show something a little more substantial.

is a simple command-line S3 client that can create a bucket and an object, then list the contents of the bucket. This client should give you a high-level picture of how S3’s resources work together. I’ve annotated the lines of code that trigger HTTP requests, by describing the HTTP requests in comments off to the right.

Example . A sample S3 client

#!/usr/bin/ruby -w
# s3-sample-client.rb
require 'S3lib'

# Gather command-line arguments
bucket_name, object_name, object_value = ARGV
unless bucket_name
  puts "Usage: #{$0} [bucket name] [object name] [object value]"
  exit
end

# Find or create the bucket.
buckets = S3::BucketList.new.get               # GET /
bucket = buckets.detect { |b| b.name == bucket_name }
if bucket
  puts "Found bucket #{bucket_name}."
else
  puts "Could not find bucket #{bucket_name}, creating it."
  bucket = S3::Bucket.new(bucket_name)
  bucket.put                                   # PUT /{bucket}
end

# Create the object.
object = S3::Object.new(bucket, object_name)
object.metadata['content-type'] = 'text/plain'
object.value = object_value
object.put                                     # PUT /{bucket}/{object}

# For each object in the bucket...
bucket.get[0].each do |o|                      # GET /{bucket}
  # ...print out information about the object.
  puts "Name: #{o.name}"
  puts "Value: #{o.value}"                     # GET /{bucket}/{object}
  puts "Metadata hash: #{o.metadata.inspect}"
  puts
end

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Clients Made Transparent with ActiveResource

Content preview·Buy PDF of this chapter|