RESTful Web Services

Chapter 2. Writing Web Service Clients

Web Services Are Web Sites

In Chapter 1 I showed some quick examples of clients for existing, public web services. Some of the services had resource-oriented RESTful architectures, some had RPC-style architectures, and some were hybrids. Most of the time, I accessed these services through wrapper libraries instead of making the HTTP requests myself.

You can’t always rely on the existence of a convenient wrapper library for your favorite web service, especially if you wrote the web service yourself. Fortunately, it’s easy to write programs that work directly with HTTP requests and responses. In this chapter I show how to write clients for RESTful and hybrid architecture services, in a variety of programming languages.

Example 2-1 is a bare HTTP client for a RESTful web service: Yahoo!’s web search. You might compare it to Example 1-8, the client from the previous chapter that runs against the RPC-style SOAP interface to Google’s web search.

Example 2-1. Searching the Web with Yahoo!’s web service

#!/usr/bin/ruby
# yahoo-web-search.rb
require 'open-uri'
require 'rexml/document'
require 'cgi'

BASE_URI = 'http://api.search.yahoo.com/WebSearchService/V1/webSearch'

def print_page_titles(term)
  # Fetch a resource: an XML document full of search results.
  term = CGI::escape(term)
  xml = open(BASE_URI + "?appid=restbook&query=#{term}").read
  
  # Parse the XML document into a data structure.
  document = REXML::Document.new(xml)
  
  # Use XPath to find the interesting parts of the data structure.
  REXML::XPath.each(document, '/ResultSet/Result/Title/[]') do |title|
    puts title
  end        
end

(puts "Usage: #{$0} [search term]"; exit) if ARGV.empty?
print_page_titles(ARGV.join(' '))

This “web service” code looks just like generic HTTP client code. It uses Ruby’s standard open-uri library to make an HTTP request and Ruby’s standard REXML library to parse the output. I’d use the same tools to fetch and process a web page. These two URIs:

http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=restbook&query=jellyfish
http://search.yahoo.com/search?p=jellyfish

point to different forms of the same thing: “a list of search results for the query ‘jellyfish.’” One URI serves HTML and is intended for use by web browsers; the other serves XML and is intended for use by automated clients.

Reading from right to left, the expression /ResultSet/Result/Title/[] means:

Find the direct children	[ ]
of every `Title` tag	Title/
that’s the direct child of a `Result` tag	Result/
that’s the direct child of the `ResultSet` tag	ResultSet/
at the root of the document.	/

If you look at the XML files served by the Yahoo! search service, you’ll see a ResultSet tag that contains Result tags, each of which contains a Title tag. The contents of those tags are what I’m after in Example 2-1.

There is no magic dust that makes an HTTP request a web service request. You can make requests to a RESTful or hybrid web service using nothing but your programming language’s HTTP client library. You can process the results with a standard XML parser. Every web service request involves the same three steps:

Come up with the data that will go into the HTTP request: the HTTP method, the URI, any HTTP headers, and (for requests using the PUT or POST method) any document that needs to go in the request’s entity-body.
Format the data as an HTTP request, and send it to the appropriate HTTP server.
Parse the response data—the response code, any headers, and any entity-body—into the data structures the rest of your program needs.

In this chapter I show how different programming languages and libraries implement this three-step process.

Wrappers, WADL, and ActiveResource

Although a web service request is just an HTTP request, any given web service has a logic and a structure that is missing from the World Wide Web as a whole. If you follow the three-step algorithm every time you make a web service request, your code will be a mess and you’ll never take advantage of that underlying structure.

Instead, as a smart programmer you’ll quickly notice the patterns underlying your requests to a given service, and write wrapper methods that abstract away the details of HTTP access. The print_page_titles method defined in Example 2-1 is a primitive wrapper. As a web service gets popular, its users release polished wrapper libraries in various languages. Some service providers offer official wrappers: Amazon gives away clients in five different languages for its RESTful S3 service. That hasn’t stopped outside programmers from writing their own S3 client libraries, like jbucket and s3sh.

Wrappers make service programming easy, because the API of a wrapper library is tailored to one particular service. You don’t have to think about HTTP at all. The downside is that each wrapper is slightly different: learning one wrapper doesn’t prepare you for the next one.

This is a little disappointing. After all, these services are just variations on the three-step algorithm for making HTTP requests. Shouldn’t there be some way of abstracting out the differences between services, some library that can act as a wrapper for the entire space of RESTful and hybrid services?

This is the problem of service description. We need a language with a vocabulary that can describe the variety of RESTful and hybrid services. A document written in this language could script a generic web service client, making it act like a custom-written wrapper. The SOAP RPC community has united around WSDL as its service description language. The REST community has yet to unite around a description language, so in this book I do my bit to promote WADL as a resource-oriented alternative to WSDL. I think it’s the simplest and most elegant solution that solves the whole problem. I show a simple WADL client in this chapter and it is covered in detail in the WADL” section.

There’s also a generic client called ActiveResource, still in development. ActiveResource makes it easy to write clients for many kinds of web services written with the Ruby on Rails framework. I cover ActiveResource at the end of Chapter 3.

del.icio.us: The Sample Application

In this chapter I walk through the life cycle of a web service request from the client’s point of view. Though most of this book’s code examples are written in Ruby, in this chapter I show code written in a variety of programming languages. My example throughout this chapter is the web service provided by the social bookmarking web site del.icio.us. You can read a prose description of this web service at http://del.icio.us/help/api/.

Tip

If you’re not familiar with del.icio.us, here’s a brief digressionary introduction. del.icio.us is a web site that works like your web browser’s bookmark feature, but it’s public and better-organized (see Figure 2-1). When you save a link to del.icio.us, it’s associated with your account so you can find it later. You can also share your bookmarks with others.

You can associate short strings, called tags, with a URI. Tags are versatile little suckers. They make it easy for you to find a URI later, they make it possible to group URIs together, and when multiple people tag the same URI, they create a machine-readable vocabulary for that URI.

Figure 2-1. del.icio.us screenshot

The del.icio.us web service gives you programmatic access to your bookmarks. You can write programs that bookmark URIs, convert your browser bookmarks to del.icio.us bookmarks, or fetch the URIs you’ve bookmarked in the past. The best way to visualize the del.icio.us web service is to use the human-oriented web site for a while. There’s no fundamental difference between the del.icio.us web site and the del.icio.us web service, but there are variations:

The web site is rooted at http://del.icio.us/ and the web service is rooted at https://api.del.icio.us/v1/. The web site communicates with clients through HTTP, the web service uses secure HTTPS.
The web site and the web service expose different URI structures. To get your recent bookmarks from the web site, you fetch https://del.icio.us/{your-username}. To get your recent bookmarks from the web service, you fetch https://api.del.icio.us/v1/posts/recent.
The web site serves HTML documents, and the web service serves XML documents. The formats are different, but they contain the same data.
The web site lets you see a lot of information without logging in or even having an account. The web service makes you authenticate for every request.
Both offer features for personal bookmark management, but the web site also has social features. On the web site, you can see lists of URIs other people have bookmarked, lists of people who have bookmarked a particular URI, lists of URIs tagged with a certain tag, and lists of popular bookmarks. The web service only lets you see your own bookmarks.

These variations are important but they don’t make the web service a different kind of thing from the web site. The web service is a stripped-down web site that uses HTTPS and serves funny-looking documents. (You can flip this around and look at the web site as a more functional web service, though the del.icio.us administrators discourage this viewpoint.) This is a theme I’m coming back to again and again: web services should work under the same rules as web sites.

Aside from its similarity to a web site, the del.icio.us web service does not have a very RESTful design. The programmers have laid out the service URIs in a way that suggests an RPC-style rather than a resource-oriented design. All requests to the del.icio.us web service use the HTTP GET method: the real method information goes into the URI and might conflict with “GET”. A couple sample URIs should illustrate this point: consider https://api.del.icio.us/v1/posts/add and https://api.del.icio.us/v1/tags/rename. Though there’s no explicit methodName variable, the del.icio.us API is just like the Flickr API I covered in Chapter 1. The method information (“add” and “rename”) is kept in the URIs, not in the HTTP method.

So why have I chosen del.icio.us for the sample clients in this chapter? Three reasons. First, del.icio.us is an easy application to understand, and its web service is popular and easy to use.

Second, I want to make it clear that what I say in the coming chapters is prescriptive, not descriptive. When you implement a web service, following the constraints of REST will give your clients a nice, usable web service that acts like the web. But when you implement a web service client, you have to work with the service as it is. The only alternatives are to lobby for a change or boycott the service. If a web service designer has never heard of REST, or thinks that hybrid services are “RESTful,” there’s little you can do about it. Most existing services are hybrids or full-blown RPC services. A snooty client that can feed only on the purest of REST services isn’t very useful, and won’t be for the forseeable future. Servers should be idealistic; clients must be pragmatic. This is a variant of Postel’s Law: “Be conservative in what you do; be liberal in which you accept from others.”

Third, in Chapter 7 I present a bookmark-tracking web service that’s similar to del.icio.us but designed on RESTful principles. I want to introduce the social bookmarking domain to you now, so you’ll be thinking about it as I introduce the principles of REST and my Resource-Oriented Architecture. In Chapter 7, when I design and implement a RESTful interface to del.icio.us-like functionality, you’ll see the difference.

What the Sample Clients Do

In the sections that follow, I show you simple del.icio.us clients in a variety of programming languages. All of these clients do exactly the same thing, and it’s worth spelling out what that is. First, they open up a TCP/IP socket connection to port 443 (the standard HTTPS port) on the server at api.del.icio.us. Then they send something like the HTTP request in Example 2-2. Like all HTTP responses, this one has three parts: a status code, a set of headers, and an entity-body. In this case, the entity-body is an XML document.

Example 2-2. A possible request to the del.icio.us web service

GET /v1/posts/recent HTTP/1.1
Host: api.del.icio.us
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

The del.icio.us web service sends back something like the HTTP response in Example 2-3, then closes the socket connection.

Example 2-3. A possible response from the del.icio.us web service

200 OK
Content-Type: text/xml
Date: Sun, 29 Oct 2006 15:09:36 GMT
Connection: close

<?xml version='1.0' standalone='yes'?>
<posts tag="" user="username">
  <post href="http://www.foo.com/" description="foo" extended="" 
   hash="14d59bdc067e3c1f8f792f51010ae5ac" tag="foo"
   time="2006-10-29T02:56:12Z" />
  <post href="http://amphibians.com/" description="Amphibian Mania" 
   extended="" hash="688b7b2f2241bc54a0b267b69f438805" tag="frogs toads" 
   time="2006-10-28T02:55:53Z" />
</posts>

The clients I write are only interested in the entity-body part. Specifically, they’re only interested in the href and description attributes of the post tags. They’ll parse the XML document into a data structure and use the XPath expression /posts/post to iterate over the post tags. They’ll print to standard output the href and description attribute of every del.icio.us bookmark:

foo: http://www.foo.com/
Amphibian Mania: http://amphibians.com/

To use any of these clients, you’ll need to create a del.icio.us account. Some of the clients hard-code the strings “username” and “password”; you’ll need to substitute your del.icio.us username and password.

Making the Request: HTTP Libraries

Every modern programming language has one or more libraries for making HTTP requests. Not all of these libraries are equally useful, though. To build a fully general web service client you need an HTTP library with these features:

It must support HTTPS and SSL certificate validation. Web services, like web sites, use HTTPS to secure communication with their clients. Many web services (del.icio.us is one example) won’t accept plain HTTP requests at all. A library’s HTTPS support often depends on the presense of an external SSL library written in C.
It must support at least the five main HTTP methods: GET, HEAD, POST, PUT, and DELETE. Some libraries support only GET and POST. Others are designed for simplicity and support only GET.
You can get pretty far with a client that only supports GET and POST: HTML forms support only those two methods, so the entire human web is open to you. You can even do all right with just GET, because many web services (among them del.icio.us and Flickr) use GET even where they shouldn’t. But if you’re choosing a library for all your web service clients, or writing a general client like a WADL client, you need a library that supports all five methods. Additional methods like OPTIONS and TRACE, and WebDAV extensions like MOVE, are a bonus.
It must allow the programmer to customize the data sent as the entity-body of a PUT or POST request.
It must allow the programmer to customize a request’s HTTP headers.
It must give the programmer access to the response code and headers of an HTTP response; not just access to the entity-body.
It must be able to communicate through an HTTP proxy. The average programmer may not think about this, but many HTTP clients in corporate environments can only work through a proxy. Intermediaries like HTTP proxies are also a standard part of the REST meta-architecture, though not one I’ll be covering in much detail.

Optional Features

There are also some features of an HTTP library that make life easier as you write clients for RESTful and hybrid services. These features mostly boil down to knowledge about HTTP headers, so they’re technically optional. You can implement them yourself so long as your library gives you access to request and response HTTP headers. The advantage of library support is that you don’t have to worry about the details.

An HTTP library should automatically request data in compressed form to save bandwidth, and transparently decompress the data it receives. The HTTP request header here is Accept-Encoding, and the response header is Encoding. I discuss these in more detail in Chapter 8.
It should automatically cache the responses to your requests. The second time you request a URI, it should return an item from the cache if the object on the server hasn’t changed. The HTTP headers here are ETag and If-Modified-Since for the request, and Etag and Last-Modified for the response. These, too, I discuss in Chapter 8.
It should transparently support the most common forms of HTTP authentication: Basic, Digest, and WSSE. It’s useful to support custom, company-specific authentication methods such as Amazon’s, or to have plug-ins that support them.
The request header is Authorization and the response header (the one that demands authentication) is WWW-Authenticate. I cover the standard HTTP authentication methods, plus WSSE, in Chapter 8. I cover Amazon’s custom authentication method in Chapter 3.
It should be able to transparently follow HTTP redirects, while avoiding infinite redirects and redirect loops. This should be an optional convenience for the user, rather than something that happens on every single redirect. A web service may reasonably send a status code of 303 (“See Other”) without implying that the client should go fetch that other URI right now!
It should be able to parse and create HTTP cookie strings, rather than forcing the programmer to manually set the Cookie header. This is not very important for RESTful services, which shun cookies, but it’s very important if you want to use the human web.

When you’re writing code against a specific service, you may be able to do without some or all of these features. Ruby’s standard open-uri library only supports GET requests. If you’re writing a client for del.icio.us, there’s no problem, since that web service expects only GET requests. But try to use open-uri with Amazon S3 (which uses GET, HEAD, PUT, and DELETE), and you’ll quickly run into a wall. In the next sections I recommend good HTTP client libraries for some popular programming languages.

Ruby: rest-open-uri and net/http

Ruby comes with two HTTP client libraries, open-uri and the lower-level net/http. Either can make HTTPS requests if you’ve got the net/https extension installed. Windows installations of Ruby should be able to make HTTPS requests out of the box. If you’re not on Windows, you may have to install net/https separately.^[8]

The open-uri library has a simple and elegant interface that lets you treat URIs as filenames. To read a web page, you simply open its URI and read data from the “filehandle.” You can pass in a hash to open containing custom HTTP headers and open-specific keyword arguments. This lets you set up a proxy, or specify authentication information.

Unfortunately, right now open-uri only supports one HTTP method: GET. That’s why I’ve made some minor modifications to open-uri and made the result available as the rest-open-uri Ruby gem.^[9] I’ve added two keyword arguments to open:method, which lets you customize the HTTP method, and :body, which lets you send data in the entity-body.

Example 2-4 is an implementation of the standard del.icio.us example using the open-uri library (rest-open-uri works the same way). This code parses the response document using the REXML::Document parser, which you’ve seen before.

Example 2-4. A Ruby client using open-uri

#!/usr/bin/ruby -w
# delicious-open-uri.rb


require 'open-uri'
require 'rexml/document'

# Fetches a del.icio.us user's recent bookmarks, and prints each one.
def print_my_recent_bookmarks(username, password)
  # Make the HTTPS request.
  response = open('https://api.del.icio.us/v1/posts/recent',
                  :http_basic_authentication => [username, password])

  # Read the response entity-body as an XML document.
  xml = response.read

  # Turn the document into a data structure.
  document = REXML::Document.new(xml)

  # For each bookmark...
  REXML::XPath.each(document, "/posts/post") do |e|
    # Print the bookmark's description and URI
    puts "#{e.attributes['description']}: #{e.attributes['href']}"
  end
end

# Main program
username, password = ARGV
unless username and password
  puts "Usage: #{$0} [username] [password]"
  exit
end
print_my_recent_bookmarks(username, password)

I mentioned earlier that Ruby’s stock open-uri can only make HTTP GET requests. For many purposes, GET is enough, but if you want to write a Ruby client for a fully RESTful service like Amazon’s S3, you’ll either need to use rest-open-uri, or turn to Ruby’s low-level HTTP library: net/http.

This built-in library provides the Net::HTTP class, which has several methods for making HTTP requests (see Table 2-1). You can build a complete HTTP client out of this class, using nothing more than the Ruby standard library. In fact, open-uri and rest-open-uri are based on Net::HTTP. Those libraries only exist because Net::HTTP provides no simple, easy-to-use interface that supports all the features a REST client needs (proxies, HTTPS, headers, and so on). That’s why I recommend you use rest-open-uri.

Table 2-1. HTTP feature matrix for Ruby HTTP client libraries

	open-uri	rest-open-uri	Net:HTTP
HTTPS^[a]	Yes	Yes	Yes
HTTP verbs	GET	All	All
Custom data	No	Yes	Yes
Custom headers	Yes	Yes	Yes
Proxies	Yes	Yes	Yes
Compression	No	No	No
Caching	No	No	No
Auth methods	Basic	Basic	Basic
Cookies	No	No	No
Redirects	Yes	Yes	No
^[a]Assuming the `net/https` library is installed

Python: httplib2

The Python standard library comes with two HTTP clients: urllib2, which has a file-like interface like Ruby’s open-uri; and httplib, which works more like Ruby’s Net::HTTP. Both offer transparent support for HTTPS, assuming your copy of Python was compiled with SSL support. There’s also an excellent third-party library, Joe Gregorio’s httplib2, which is the one I recommend in general. httplib2 is an excellent piece of software, supporting nearly every feature on my wish list—most notably, transparent caching. Table 2-2 lists the features available in each library.

Table 2-2. HTTP feature matrix for Python HTTP client libraries

	urllib2	httplib	httplib2
HTTPS^[a]	Yes	Yes	Yes
HTTP verbs	GET, POST	All	All
Custom data	Yes	Yes	Yes
Custom headers	Yes	Yes	Yes
Proxies	Yes	No	No
Compression	No	No	Yes
Caching	No	No	Yes
Auth methods	Basic, Digest	None	Basic, Digest, WSSE, Google
Cookies	Yes (Use `urllib2.build_opener(HTTPCookieProcessor)`)	No	No
Redirects	Yes	No	Yes
^[a]Assuming Python was compiled with SSL support

Example 2-5 is a del.icio.us client that uses httplib2. It uses the ElementTree library to parse the del.icio.us XML.

Example 2-5. A del.icio.us client in Python

#!/usr/bin/python2.5
# delicious-httplib2.py
import sys
from xml.etree import ElementTree
import httplib2

# Fetches a del.icio.us user's recent bookmarks, and prints each one.
def print_my_recent_bookmarks(username, password):
    client = httplib2.Http(".cache")
    client.add_credentials(username, password)

    # Make the HTTP request, and fetch the response and the entity-body.
    response, xml = client.request('https://api.del.icio.us/v1/posts/recent')

    # Turn the XML entity-body into a data structure.
    doc = ElementTree.fromstring(xml)

    # Print information about every bookmark.
    for post in doc.findall('post'):
        print "%s: %s" % (post.attrib['description'], post.attrib['href'])

# Main program
if len(sys.argv) != 3:    
    print "Usage: %s [username] [password]" % sys.argv[0]
    sys.exit()

username, password = sys.argv[1:]
print_my_recent_bookmarks(username, password)

Java: HttpClient

The Java standard library comes with an HTTP client, java.net.HttpURLConnection. You can get an instance by calling open on a java.net.URL object. Though it supports most of the basic features of HTTP, programming to its API is very difficult. The Apache Jakarta project has a competing client called HttpClient, which has a better design. There’s also Restlet. I cover Restlet as a server library in Chapter 12, but it’s also an HTTP client library. The class org.restlet.Client makes it easy to make simple HTTP requests, and the class org.restlet.data.Request hides the HttpURLConnection programming necessary to make more complex requests. Table 2-3 lists the features available in each library.

Table 2-3. HTTP feature matrix for Java HTTP client libraries.

	HttpURLConnection	HttpClient	Restlet
HTTPS	Yes	Yes	Yes
HTTP verbs	All	All	All
Custom data	Yes	Yes	Yes
Custom headers	Yes	Yes	Yes
Proxies	Yes	Yes	Yes
Compression	No	No	Yes
Caching	Yes	No	Yes
Auth methods	Basic, Digest, NTLM	Basic, Digest, NTLM	Basic, Amazon
Cookies	Yes	Yes	Yes
Redirects	Yes	Yes	Yes

Example 2-6 is a Java client for del.icio.us that uses HttpClient. It works in Java 1.5 and up, and it’ll work in previous versions if you install the Xerces parser (see Java: javax.xml, Xerces, or XMLPull” later in this chapter).

Example 2-6. A del.icio.us client in Java

// DeliciousApp.java
import java.io.*;

import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.GetMethod;

import org.w3c.dom.*;
import org.xml.sax.SAXException;
import javax.xml.parsers.*;
import javax.xml.xpath.*;

/**
 * A command-line application that fetches bookmarks from del.icio.us
 * and prints them to standard output.
 */
public class DeliciousApp
{
  public static void main(String[] args)
    throws HttpException, IOException, ParserConfigurationException,
           SAXException, XPathExpressionException
  {        
    if (args.length != 2)
    {
      System.out.println("Usage: java -classpath [CLASSPATH] "
                         + "DeliciousApp [USERNAME] [PASSWORD]");
      System.out.println("[CLASSPATH] - Must contain commons-codec, " +
                         "commons-logging, and commons-httpclient");
      System.out.println("[USERNAME]  - Your del.icio.us username");
      System.out.println("[PASSWORD]  - Your del.icio.us password");
      System.out.println();

      System.exit(-1);
    }

    // Set the authentication credentials.
    Credentials creds = new UsernamePasswordCredentials(args[0], args[1]);
    HttpClient client = new HttpClient();
    client.getState().setCredentials(AuthScope.ANY, creds);

    // Make the HTTP request.
    String url = "https://api.del.icio.us/v1/posts/recent";
    GetMethod method = new GetMethod(url);
    client.executeMethod(method);
    InputStream responseBody = method.getResponseBodyAsStream();

    // Turn the response entity-body into an XML document.
    DocumentBuilderFactory docBuilderFactory =
      DocumentBuilderFactory.newInstance();
    DocumentBuilder docBuilder = 
      docBuilderFactory.newDocumentBuilder();
    Document doc = docBuilder.parse(responseBody);
    method.releaseConnection();

    // Hit the XML document with an XPath expression to get the list
    // of bookmarks.
    XPath xpath = XPathFactory.newInstance().newXPath();        
    NodeList bookmarks = (NodeList)xpath.evaluate("/posts/post", doc,
                                                  XPathConstants.NODESET);

    // Iterate over the bookmarks and print out each one.
    for (int i = 0; i < bookmarks.getLength(); i++)
    {
       NamedNodeMap bookmark = bookmarks.item(i).getAttributes();
       String description = bookmark.getNamedItem("description")
           .getNodeValue();
       String uri = bookmark.getNamedItem("href").getNodeValue();
       System.out.println(description + ": " + uri);
    }

    System.exit(0);
  }
}

C#: System.Web.HTTPWebRequest

The .NET Common Language Runtime (CLR) defines HTTPWebRequest for making HTTP requests, and NetworkCredential for authenticating the client to the server. The HTTPWebRequest constructor takes a URI. The NetworkCredential constructor takes a username and password (see Example 2-7).

Example 2-7. A del.icio.us client in C#

using System;
using System.IO;
using System.Net;
using System.Xml.XPath;

public class DeliciousApp {
    static string user = "username";
    static string password = "password";
    static Uri uri = new Uri("https://api.del.icio.us/v1/posts/recent");

    static void Main(string[] args) {
        HttpWebRequest request = (HttpWebRequest) WebRequest.Create(uri);
        request.Credentials = new NetworkCredential(user, password);
        HttpWebResponse response = (HttpWebResponse) request.GetResponse();

        XPathDocument xml = new
	    XPathDocument(response.GetResponseStream());
        XPathNavigator navigator = xml.CreateNavigator();
        foreach (XPathNavigator node in navigator.Select("/posts/post")) {
          string description = node.GetAttribute("description","");
          string href = node.GetAttribute("href","");
          Console.WriteLine(description + ": " + href);
        }
    }
}

PHP: libcurl

PHP comes with a binding to the C library libcurl, which can do pretty much anything you might want to do with a URI (see Example 2-8).

Example 2-8. A del.icio.us client in PHP

<?php
  $user = "username";
  $password = "password";

  $request = curl_init();
  curl_setopt($request, CURLOPT_URL,
              'https://api.del.icio.us/v1/posts/recent');
  curl_setopt($request, CURLOPT_USERPWD, "$user:$password");
  curl_setopt($request, CURLOPT_RETURNTRANSFER, true);

  $response = curl_exec($request);
  $xml = simplexml_load_string($response);
  curl_close($request);

  foreach ($xml->post as $post) {
    print "$post[description]: $post[href]\n";
  }
?>

JavaScript: XMLHttpRequest

If you’re writing a web service client in JavaScript, you probably intend it to run inside a web browser as part of an Ajax application. All modern web browsers implement a HTTP client library for JavaScript called XMLHttpRequest.

Because Ajax clients are developed differently from standalone clients, I’ve devoted an entire chapter to them: Chapter 11. The first example in that chapter is a del.icio.us client, so you can skip there right now without losing the flow of the examples.

The Command Line: curl

This example is a bit different: it doesn’t use a programming language at all. A program called curl is a capable HTTP client that runs from the Unix or Windows command line. It supports most HTTP methods, custom headers, several authentication mechanisms, proxies, compression, and many other features. You can use curl to do quick one-off HTTP requests, or use it in conjunction with shell scripts. Here’s curl in action, grabbing a user’s del.icio.us bookmarks:

$ curl https://username:password@api.del.icio.us/v1/posts/recent
<?xml version='1.0' standalone='yes'?>
<posts tag="" user="username">
 ...
</posts>

Other Languages

I don’t have the space or the expertise to cover every popular programming language in depth with a del.icio.us client example. I can, however, give brief pointers to HTTP client libraries for some of the many languages I haven’t covered yet.

ActionScript

Flash applications, like JavaScript applications, generally run inside a web browser. This means that when you write an ActionScript web service client you’ll probably use the Ajax architecture described in Chapter 11, rather than the standalone architecture shown in this chapter.

ActionScript’s XML class gives functionality similar to JavaScript’s XmlHttpRequest. The XML.load method fetches a URI and parses the response document into an XML data structure. ActionScript also provides a class called LoadVars, which works on form-encoded key-value pairs instead of on XML documents.

C

The libwww library for C was the very first HTTP client library, but most C programmers today use libcurl, the basis for the curl command-line tool. Earlier I mentioned PHP’s bindings to libcurl, but there are also bindings for more than 30 other languages. If you don’t like my recommendations, or I don’t mention your favorite programming language in this chapter, you might look at using the libcurl bindings.

C++

Use libcurl, either directly or through an object-oriented wrapper called cURLpp.

Common Lisp

simple-http is easy to use, but doesn’t support anything but basic HTTP GET and POST. The AllegroServe web server library includes a complete HTTP client library.

Perl

The standard HTTP library for Perl is libwww-perl (also known as LWP), available from CPAN or most Unix packaging systems. libwww-perl has a long history and is one of the best-regarded Perl libraries. To get HTTPS support, you should also install the Crypt:SSLeay module (available from CPAN).

Processing the Response: XML Parsers

The entity-body is usually the most important part of an HTTP response. Where web services are concerned, the entity-body is usually an XML document, and the client gets most of the information it needs by running this document through an XML parser.

Now, there are many HTTP client libraries, but they all have exactly the same task. Given a URI, a set of headers, and a body document, the client’s job is to construct an HTTP request and send it to a certain server. Some libraries have more features than others: cookies, authentication, caching, and the other ones I mentioned. But all these extra features are implemented within the HTTP request, usually as extra headers. A library might offer an object-oriented interface (like Net::HTTP) or a file-like interface (like open-uri), but both interfaces do the same thing. There’s only one kind of HTTP client library.

But there are three kinds of XML parsers. It’s not just that some XML parsers have features that others lack, or that one interface is more natural than another. There are two basic XML parsing strategies: the document-based strategy of DOM and other tree-style parsers, and the event-based strategy of SAX and “pull” parsers. You can get a tree-style or a SAX parser for any programming language, and a pull parser for almost any language.

The document-based, tree-style strategy is the simplest of the three models. A tree-style parser models an XML document as a nested data structure. Once you’ve got this data structure, you can search and process it with XPath queries, CSS selectors, or custom navigation functions: whatever your parser supports. A DOM parser is a tree-style parser that implements a specific interface defined by the W3C.

The tree-style strategy is easy to use, and it’s the one I use the most. With a tree-style parser, the document is just an object like the other objects in your program. The big shortcoming is that you have to deal with the document as a whole. You can’t start working on the document until you’ve processed the whole thing into a tree, and you can’t avoid loading the whole document into memory. For documents that are simple but very large, this is inefficient. It would be a lot better to handle tags as they’re parsed.

Instead of a data structure, a SAX-style or pull parser turns a document into a stream of events. Starting and closing tags, XML comments, and entity declarations are all events.

A pull parser is useful when you need to handle almost every event. A pull parser lets you handle one event at a time, “pulling” the next one from the stream as needed. You can take action in response to individual events as they come in, or build up a data structure for later use—presumably a smaller data structure than the one a tree-style parser would build. You can stop parsing the document at any time and come back to it later by pulling the next event from the stream.

A SAX parser is more complex, but useful when you only care about a few of the many events that will be streaming in. You drive a SAX parser by registering callback methods with it. Once you’re done defining callbacks, you set the parser loose on a document. The parser turns the document into a series of events, and processes every event in the document without stopping. When an event comes along that matches one of your callbacks, the parser triggers that callback, and your custom code runs. Once the callback completes, the SAX parser goes back to processing events without stopping.

The advantage of the document-based approach is that it gives you random access to the document’s contents. With event-based parsers, once the events have fired, they’re gone. If you want to trigger them again you need to re-parse the document. What’s more, an event-based parser won’t notice that a malformed XML document is malformed until it tries to parse the bad spot, and crashes. Before passing a document into an event-based parser, you’ll need to make sure the document is well formed, or else accept that your callback methods can be triggered for a document that turns out not to be good.

Some programming languages come with a standard set of XML parsers. Others have a canonical third-party parser library. For the sake of performance, some languages also have bindings to fast parsers written in C. I’d like to go through the list of languages again now, and make recommendations for document- and event-based XML parsers. I’ll rate commonly available parsers on speed, the quality of their interface, how well they support XPath (for tree-style parsers), how strict they are, and whether or not they support schema-based validation. Depending on the application, a strict parser may be a good thing (because an XML document will be parsed the correct way or not at all) or a bad thing (because you want to use a service that generates bad XML).

In the sample del.icio.us clients given above, I showed not only how to use my favorite HTTP client library for a language, but how to use my favorite tree-style parser for that language. To show you how event-based parsers work, I’ll give two more examples of del.icio.us clients using Ruby’s built-in SAX and pull parsers.

Ruby: REXML, I Guess

Ruby comes with a standard XML parser library, REXML, that supports both DOM and SAX interfaces, and has good XPath support. Unfortunately, REXML’s internals put it in a strange middle ground: it’s too strict to be used to parse bad XML, but not strict enough to reject all bad XML.

I use REXML throughout this book because it’s the default choice, and because I only deal with well-formed XML. If you want to guarantee that you only deal with well-formed XML, you’ll need to install the Ruby bindings to the GNOME project’s libxml2 library (described in Other Languages” later in this chapter).

If you want to be able to handle bad markup, the best choice is hpricot, available as the hpricot gem. It’s fast (it uses a C extension), and it has an intuitive interface including support for common XPath expressions.

Example 2-9 is an implementation of the del.icio.us client using REXML’s SAX interface.

Example 2-9. A Ruby client using a SAX parser

#!/usr/bin/ruby -w
# delicious-sax.rb
require 'open-uri'
require 'rexml/parsers/sax2parser'

def print_my_recent_bookmarks(username, password)
  # Make an HTTPS request and read the entity-body as an XML document.
  xml = open('https://api.del.icio.us/v1/posts/recent',
             :http_basic_authentication => [username, password])

  # Create a SAX parser whose destiny is to parse the XML entity-body.
  parser = REXML::Parsers::SAX2Parser.new(xml)

  # When the SAX parser encounters a 'post' tag...
  parser.listen(:start_element, ["post"]) do |uri, tag, fqtag, attributes|
    # ...it should print out information about the tag.
    puts "#{attributes['description']}: #{attributes['href']}"
  end

  # Make the parser fulfil its destiny to parse the XML entity-body.
  parser.parse 
end

# Main program.
username, password = ARGV
unless username and password
  puts "Usage: #{$0} [USERNAME] [PASSWORD]"  
  exit
end
print_my_recent_bookmarks(username, password)

In this program, the data isn’t parsed (or even read from the HTTP connection) until the call to SAXParser#parse. Up to that point I’m free to call listen and set up pieces of code to run in response to parser events. In this case, the only event I’m interested in is the start of a post tag. My code block gets called every time the parser finds a post tag. This is the same as parsing the XML document with a tree-style parser, and running the XPath expression “//post” against the object tree. What does my code block do? The same thing my other example programs do when they find a post tag: print out the values of the description and href attributes.

This implementation is faster and much more memory-efficient than the equivalent tree-style implementation. However, complex SAX-based programs are much more difficult to write than equivalent tree-style programs. Pull parsers are a good compromise. Example 2-10 shows a client implementation that uses REXML’s pull parser interface.

Example 2-10. A del.icio.us client using REXML’s pull parser

#!/usr/bin/ruby -w
# delicious-pull.rb
require 'open-uri'
require 'rexml/parsers/pullparser'

def print_my_recent_bookmarks(username, password)
  # Make an HTTPS request and read the entity-body as an XML document.
  xml = open('https://api.del.icio.us/v1/posts/recent',
             :http_basic_authentication => [username, password])

  # Feed the XML entity-body into a pull parser
  parser = REXML::Parsers::PullParser.new(xml)

  # Until there are no more events to pull...
  while parser.has_next?
    # ...pull the next event.
    tag = parser.pull    
    # If it's a 'post' tag...
    if tag.start_element?
      if tag[0] == 'post'       
        # Print information about the bookmark.
        attrs = tag[1]
        puts "#{attrs['description']}: #{attrs['href']}"
      end
    end
  end
end

# Main program.
username, password = ARGV
unless username and password
  puts "Usage: #{$0} [USERNAME] [PASSWORD]"  
  exit
end
print_my_recent_bookmarks(username, password)

Python: ElementTree

The world is full of XML parsers for Python. There are seven different XML interfaces in the Python 2.5 standard library alone. For full details, see the Python library reference.

For tree-style parsing, the best library is ElementTree. It’s fast, it has a sensible interface, and as of Python 2.5 you don’t have to install anything because it’s in the standard library. On the downside, its support for XPath is limited to simple expressions—of course, nothing else in the standard library supports XPath at all. If you need full XPath support, try 4Suite.

Beautiful Soup is a slower tree-style parser that is very forgiving of invalid XML, and offers a programmatic interface to a document. It also handles most character set conversions automatically, letting you work with Unicode data.

For SAX-style parsing, the best choice is the xml.sax module in the standard library. The PyXML suite includes a pull parser.

Java: javax.xml, Xerces, or XMLPull

Java 1.5 includes the XML parser written by the Apache Xerces project. The core classes are found in the packages javax.xml.*, (for instance, javax.xml.xpath). The DOM interface lives in org.w3c.dom.*, and the SAX interface lives in org.xml.sax.*. If you’re using a previous version of Java, you can install Xerces yourself and take advantage of the same interface found in Java 1.5 (http://xerces.apache.org/xerces2-j/).

There are a variety of pull parsers for Java. Sun’s Web Services Developer Pack includes a pull parser in the javax.xml.stream package.

For parsing bad XML, you might try TagSoup.

C#: System.Xml.XmlReader

The .NET Common Language Runtime comes with a pull parser interface, in contrast to the more typical (and more complex) SAX-style interface. You can also create a full W3C DOM tree using XmlDocument. The XPathDocument class lets you iterate over nodes in the tree that match an XPath expression.

If you need to handle broken XML documents, check out Chris Lovett’s SgmlReader at http://www.gotdotnet.com/Community/UserSamples/.

PHP

You can create a SAX-style parser with the function xml_parser_create, and a pull parser with the XMLReader extension. The DOM PHP extension (included in PHP 5) provides a tree-style interface to the GNOME project’s libxml2 C library. You might have an easier time using SimpleXML, a tree-style parser that’s not an official DOM implementation. That’s what I used in Example 2-8.

There’s also a pure PHP DOM parser called DOMIT!.

JavaScript: responseXML

If you’re using XMLHttpRequest to write an Ajax client, you don’t have to worry about the XML parser at all. If you make a request and the response entity-body is in XML format, the web browser parses it with its own tree-style parser, and makes it available through the responseXML property of the XMLHttpRequest object. You manipulate this document with JavaScript DOM methods: the same ones you use to manipulate HTML documents displayed in the browser. Chapter 11 has more information on how to use responseXML—and how to handle non-XML documents with the responseData member.

There’s a third-party XML parser, XML for <SCRIPT>, which works independently of the parser built into the client’s web browser. “XML for <SCRIPT>” offers DOM and SAX interfaces, and supports XPath queries.

Other Languages

ActionScript: When you load a URI with XML.load, it’s automatically parsed into an XML object, which exposes a tree-style interface.
C: Expat is the most popular SAX-style parser. The GNOME project’s libxml2 contains DOM, pull, and SAX parsers.
C++: You can use either of the C parsers, or the object-oriented Xerces-C++ parser. Like the Java version of Xerces, Xerces-C++ exposes both DOM and SAX interfaces.
Common Lisp: Use SXML. It exposes a SAX-like interface, and can also turn an XML document into tree-like S-expressions or Lisp data structures.
Perl: As with Python, there are a variety of XML parsers for Perl. They’re all available on CPAN. XML::XPath has XPath support, and XML::Simple turns an XML document into standard Perl data structures. For SAX-style parsing, use XML::SAX::PurePerl. For pull parsing, use XML::LibXML::Reader. The Perl XML FAQ has an overview of the most popular Perl XML libraries.

JSON Parsers: Handling Serialized Data

Most web services return XML documents, but a growing number return simple data structures (numbers, arrays, hashes, and so on), serialized as JSON-formatted strings. JSON is usually produced by services that expect to be consumed by the client half of an Ajax application. The idea is that it’s a lot easier for a browser to get a JavaScript data structure from a JSON data structure than from an XML document. Every web browser offers a slightly different JavaScript interface to its XML parser, but a JSON string is nothing but a tightly constrained JavaScript program, so it works the same way in every browser.

Of course, JSON is not tied to JavaScript, any more than JavaScript is to Java. JSON makes a lightweight alternative to XML-based approaches to data serialization, like XML Schema. The JSON web site links to implementations in many languages, and I refer you to that site rather than mentioning a JSON library for every language.

JSON is a simple and language-independent way of formatting programming language data structures (numbers, arrays, hashes, and so on) as strings. Example 2-11 is a JSON representation of a simple data structure: a mixed-type array.

Example 2-11. A mixed-type array in JSON format

[3, "three"]

By comparison, Example 2-12 is one possible XML representation of the same data.

Example 2-12. A mixed-type array in XML-RPC format

<value>
 <array>
  <data>
   <value><i4>3</i4></value>
   <value><string>three</string></value>
  </data>
 </array>
</value>

Since a JSON string is nothing but a tightly constrained JavaScript program, you can “parse” JSON simply by calling eval on the string. This is very fast, but you shouldn’t do it unless you control the web service that served your JSON. An untested or untrusted web service can send the client buggy or malicious JavaScript programs instead of real JSON structures. For the JavaScript examples in Chapter 11, I use a JSON parser written in JavaScript and available from json.org (see Example 2-13).

Example 2-13. A JSON demo in JavaScript

<!-- json-demo.html -->
<!-- In a real application, you would save json.js locally
     instead of fetching it from json.org every time. -->
<script type="text/javascript" src="http://www.json.org/json.js">
</script>

<script type="text/javascript">
 array = [3, "three"]
 alert("Converted array into JSON string: '" + array.toJSONString() + "'")
 json = "[4, \"four\"]"
 alert("Converted JSON '" + json + "' into array:")
 array2 = json.parseJSON()
 for (i=0; i < array2.length; i++) 
 {
   alert("Element #" + i + " is " + array2[i])
 }
</script>

The Dojo JavaScript framework has a JSON library in the dojo.json package, so if you’re using Dojo you don’t have to install anything extra. A future version of the ECMAScript standard may define JSON serialization and deserialization methods as part of the JavaScript language, making third-party libraries obsolete.

In this book’s Ruby examples, I’ll use the JSON parser that comes from the json Ruby gem. The two most important methods are Object#to_json and JSON.parse. Try running the Ruby code in Example 2-14 through the irb interpreter.

Example 2-14. A JSON demo in Ruby

# json-demo.rb
require 'rubygems'
require 'json'

[3, "three"].to_json                 # => "[3,\"three\"]"
JSON.parse('[4, "four"]')            # => [4, "four"]

Right now, Yahoo! Web Services are the most popular public web services to serve JSON. Example 2-15 shows a command-line program, written in Ruby, that uses the Yahoo! News web service to get a JSON representation of current news stories.

Example 2-15. Searching the Web with Yahoo!’s web service (JSON edition)

#!/usr/bin/ruby
# yahoo-web-search-json.rb
require 'rubygems'
require 'json'
require 'open-uri'
$KCODE = 'UTF8'

# Search the web for a term, and print the titles of matching web pages.
def search(term)
  base_uri = 'http://api.search.yahoo.com/NewsSearchService/V1/newsSearch'

  # Make the HTTP request and read the response entity-body as a JSON
  # document.
  json = open(base_uri + "?appid=restbook&output=json&query=#{term}").read

  # Parse the JSON document into a Ruby data structure.
  json = JSON.parse(json)

  # Iterate over the data structure...
  json['ResultSet']['Result'].each do
    # ...and print the title of each web page.
    |r| puts r['Title']
  end
end

# Main program.
unless ARGV[0]
  puts "Usage: #{$0} [search term]"
  exit
end
search(ARGV[0])

Compare this to the program yahoo-web-search.rb in Example 2-1. That program has the same basic structure, but it works differently. It asks for search results formatted as XML, parses the XML, and uses an XPath query to extract the result titles. This program parses a JSON data structure into a native-language data structure (a hash), and traverses it with native-language operators instead of XPath.

If JSON is so simple, why not use it for everything? You could do that, but I don’t recommend it. JSON is good for representing data structures in general, and the Web mainly serves documents: irregular, self-describing data structures that link to each other. XML and HTML are specialized for representing documents. A JSON representation of a web page would be hard to read, just like the XML representation of an array in Example 2-12 was hard to read. JSON is useful when you need to describe a data structure that doesn’t fit easily into the document paradigm: a simple list, for instance, or a hash.

Clients Made Easy with WADL

So far I’ve presented code in a variety of languages, but it always follows the same three-step pattern. To call a web service I build up the elements of an HTTP request (method, URI, headers, and entity-body). I use an HTTP library to turn that data into a real HTTP request, and the library sends the request to the appropriate server. Then I use an XML parser to parse the response into a data structure or a series of events. Once I make the request, I’m free to use the response data however I like. In this regard all RESTful web services, and most hybrid services, are the same. What’s more, as I’ll show in the chapters to come, all RESTful web services use HTTP the same way: HTTP has what’s called a uniform interface.

Can I take advantage of this similarity? Abstract this pattern out into a generic “REST library” that can access any web service that supports the uniform interface? There’s precedent for this. The Web Service Description Language (WSDL) describes the differences between RPC-style web services in enough detail that a generic library can access any RPC-style SOAP service, given an appropriate WSDL file.

For RESTful and hybrid services, I recommend using the Web Application Description Language. A WADL file describes the HTTP requests you can legitimately make of a service: which URIs you can visit, what data those URIs expect you to send, and what data they serve in return. A WADL library can parse this file and model the space of possible service requests as a native language API.

I describe WADL in more detail in Chapter 9, but here’s a taste. The del.icio.us client shown in Example 2-16 is equivalent to the Ruby client in Example 2-4, but it uses Ruby’s WADL library and a bootleg WADL file I created for del.icio.us. (I’ll show you the WADL file in Chapter 8.)

Example 2-16. A Ruby/WADL client for del.icious

#!/usr/bin/ruby
# delicious-wadl-ruby.rb
require 'wadl'

if ARGV.size != 2
  puts "Usage: #{$0} [username] [password]"
  exit
end
username, password = ARGV

# Load an application from the WADL file
delicious = WADL::Application.from_wadl(open("delicious.wadl"))

# Give authentication information to the application
service = delicious.v1.with_basic_auth(username, password)

begin
  # Find the "recent posts" functionality
  recent_posts = service.posts.recent

  # For every recent post...
  recent_posts.get.representation.each_by_param('post') do |post|
    # Print its description and URI.
    puts "#{post.attributes['description']}: #{post.attributes['href']}"
  end
rescue WADL::Faults::AuthorizationRequired
  puts "Invalid authentication information!"
end

Behind the scenes, this code makes exactly the same HTTP request as the other del.icio.us clients seen in this chapter. The details are hidden in the WADL file delicious.wadl, which is interpreted by the WADL client library inside WADL::Application.from_WADL. This code is not immediately recognizable as a web service client. That’s a good thing: it means the library is doing its job. And yet, when we come back to this code in Chapter 9, you’ll see that it follows the principles of REST as much as the examples that made their own HTTP requests. WADL abstracts away the details of HTTP, but not the underlying RESTful interface.

As of the time of writing, WADL adoption is very poor. If you want to use a WADL client for a service, instead of writing a language-specific client, you’ll probably have to write the WADL file yourself. It’s not difficult to write a bootleg WADL file for someone else’s service: I’ve done it for del.icio.us and a few other services. You can even write a WADL file that lets you use a web application—designed for human use—as a web service. WADL is designed to describe RESTful web services, but it can describe almost anything that goes on the Web.

A Ruby library called ActiveResource takes a different strategy. It only works with certain kinds of web services, but it hides the details of RESTful HTTP access behind a simple object-oriented interface. I cover ActiveResource in the next chapter, after introducing some REST terminology.

^[8]On Debian GNU/Linux and Debian-derived systems like Ubuntu, the package name is libopenssl-ruby. If your packaging system doesn’t include net/https, you’ll have to download it from http://www.nongnu.org/rubypki/ and install it by hand.

^[9]For more information on Ruby gems, see http://rubygems.org/. Once you have the gem program installed, you can install rest-open-uri with the command gem install rest-open-uri. Hopefully my modifications to open-uri will one day make it into the core Ruby code, and the rest-open-uri gem will become redundant.

Get RESTful Web Services now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

RESTful Web Services by Leonard Richardson, Sam Ruby

Chapter 2. Writing Web Service Clients

Web Services Are Web Sites

Wrappers, WADL, and ActiveResource

del.icio.us: The Sample Application

Tip

What the Sample Clients Do

Making the Request: HTTP Libraries

Optional Features

Ruby: rest-open-uri and net/http

Python: httplib2

Java: HttpClient

C#: System.Web.HTTPWebRequest

PHP: libcurl

JavaScript: XMLHttpRequest

The Command Line: curl

Other Languages

Processing the Response: XML Parsers

Ruby: REXML, I Guess

Python: ElementTree

Java: javax.xml, Xerces, or XMLPull

C#: System.Xml.XmlReader

PHP

JavaScript: responseXML

Other Languages

JSON Parsers: Handling Serialized Data

Clients Made Easy with WADL

Don’t leave empty-handed

It’s yours, free.

Check it out now on O’Reilly