Chapter 1. The Programmable Web and Its Inhabitants
When you write a computer program, you’re not limited to the algorithms you can think up. Your language’s standard library gives you some algorithms. You can get more from books, or in third-party libraries you find online. Only if you’re on the very cutting edge should you have to come up with your own algorithms.
If you’re lucky, the same is true for data. Some applications are driven entirely by the data the users type in. Sometimes data just comes to you naturally: if you’re analyzing spam, you should have no problem getting all you need. You can download a few public data sets—word lists, geographical data, lists of prime numbers, public domain texts—as though they were third-party libraries. But if you need some other kind of data, it doesn’t look good. Where’s the data going to come from? More and more often, it’s coming from the programmable web.
When you—a human being—want to find a book on a certain topic, you probably point your web browser to the URI of an online library or bookstore: say, http://www.amazon.com/.
Tip
The common term for the address of something on the Web is “URL.” I say “URI” throughout this book because that’s what the HTTP standard says. Every HTTP URI on the Web is also a URL, so you can substitute “URL” wherever I say “URI” with no loss of meaning.
You’re served a web page, a document in HTML format that your browser renders graphically. You visually scan the page for a search form, type your topic (say, “web services”) into a text box, and submit the form. At this point your web browser makes a second HTTP request, to a URI that incorporates your topic. To continue the Amazon example, the second URI your browser requests would be something like http://amazon.com/s?url=search-alias%3Dstripbooks&field-keywords=web+services.
The web server at amazon.com
responds by serving a second document in HTML format. This document
contains a description of your search results, links to additional search
options, and miscellaneous commercial enticements (see Example 1-1). Again, your browser renders the
document in graphical form, and you look at it and decide what to do from
there.
...
<a href="http://www.amazon.com/Restful-Web-Services-Leonard-Richardson/dp/...>
<span class="srTitle">RESTful Web Services</span>
</a>
by Leonard Richardson and Sam Ruby
<span class="bindingBlock">
(<span class="binding">Paperback</span> - May 1, 2007)
</span>The Web you use is full of data: book information, opinions, prices, arrival times, messages, photographs, and miscellaneous junk. It’s full of services: search engines, online stores, weblogs, wikis, calculators, and games. Rather than installing all this data and all these programs on your own computer, you install one program—a web browser—and access the data and services through it.
The programmable web is just the same. The main difference is that instead of arranging its data in attractive HTML pages with banner ads and cute pastel logos, the programmable web usually serves stark, brutal XML documents. The programmable web is not necessarily for human consumption. Its data is intended as input to a software program that does something amazing.
Example 1-2 shows a Ruby script that uses the programmable web to do a traditional human web task: find the titles of books matching a keyword. It hides the web access under a programming language interface, using the Ruby/Amazon library.
#!/usr/bin/ruby -w
# amazon-book-search.rb
require 'amazon/search'
if ARGV.size != 2
puts "Usage: #{$0} [Amazon Web Services AccessKey ID] [text to search for]"
exit
end
access_key, search_request = ARGV
req = Amazon::Search::Request.new(access_key)
# For every book in the search results...
req.keyword_search(search_request, 'books', Amazon::Search::LIGHT) do |book|
# Print the book's name and the list of authors.
puts %{"#{book.product_name}" by #{book.authors.join(', ')}}
endTo run this program, you’ll need to sign up for an Amazon Web Services account and pass in the Access Key ID as a command-line argument. Here’s a sample run of the program:
$ruby ruby amazon-book-search.rb C1D4NQS41IMK2 "restful web services""RESTful Web Services" by Leonard Richardson, Sam Ruby"Hacking with Ruby: Ruby and Rails for the Real World" by Mark Watson
At its best, the programmable web works the same way as the human
web. When amazon-book-search.rb calls
the method Amazon::Search::Request#keyword_search, the
Ruby program starts acting like a web browser. It makes an HTTP request to
a URI: in this case, something like http://xml.amazon.com/onca/xml3?KeywordSearch=restful+web+services&mode=books&f=xml&type=lite&page=1.
The web server at xml.amazon.com
responds with an XML document. This document, shown in Example 1-3, describes the search results, just like
the HTML document you see in your web browser, but in a more structured
form.
... <ProductName>RESTful Web Services</ProductName> <Catalog>Book</Catalog> <Authors> <Author>Leonard Richardson</Author> <Author>Sam Ruby</Author> </Authors> <ReleaseDate>01 May, 2007</ReleaseDate> ...
Once a web browser has submitted its HTTP request, it has a fairly easy task. It needs to render the response in a way a human being can understand. It doesn’t need to figure out what the HTTP response means: that’s the human’s job. A web service client doesn’t have this luxury. It’s programmed in advance, so it has to be both the web browser that fetches the data, and the “human” who decides what the data means. Web service clients must automatically extract meaning from HTTP responses and make decisions based on that meaning.
In Example 1-2, the web service client parses the XML document, extracts some interesting information (book titles and authors), and prints that information to standard output. The program amazon-book-search.rb is effectively a small, special-purpose web browser, relaying data to a human reader. It could easily do something else with the Amazon book data, something that didn’t rely on human intervention at all: stick the book titles into a database, maybe, or use the author information to drive a recommendation engine.
And the data doesn’t have to always flow toward the client. Just as you can bend parts of the human web to your will (by posting on your weblog or buying a book), you can write clients that modify the programmable web. You can use it as a storage space or as another source of algorithms you don’t have to write yourself. It depends on what service you need, and whether you can find someone else to provide it.
Example 1-4 is an example of a web service client that modifies the programmable web: the s3sh command shell for Ruby. It’s one of many clients written against another of Amazon’s web services: S3, or the Simple Storage Service. In Chapter 3 I cover S3’s workings in detail, so if you’re interested in using s3sh for yourself, you can read up on S3 there.
To understand this s3sh transcript, all you need to know is that Amazon S3 lets its clients store labelled pieces of data (“objects”) in labelled containers (“buckets”). The s3sh program builds an interactive programming interface on top of S3. Other clients use S3 as a backup tool or a web host. It’s a very flexible service.
$s3sh>>Service.buckets.collect { |b| b.name }=> ["example.com"]>>my_bucket = Bucket.find("example.com")>>contents = open("disk_file.txt").read=> "This text is the contents of the file disk_file.txt">>S3Object.store("mydir/mydocument.txt", contents, my_bucket.name)>>my_bucket['directory/document.txt'].value=> "This text is the contents of the file disk_file.txt"
In this chapter I survey the current state of the programmable web. What technologies are being used, what architectures are they used to implement, and what design styles are the most popular? I show some real code and some real HTTP conversations, but my main goal in this chapter is to get you thinking about the World Wide Web as a way of connecting computer programs to each other, on the same terms as it connects human beings to each other.
Kinds of Things on the Programmable Web
The programmable web is based on HTTP and XML. Some parts of it serve HTML, JavaScript Object Notation (JSON), plain text, or binary documents, but most parts use XML. And it’s all based on HTTP: if you don’t use HTTP, you’re not on the web.[5]Beyond that small island of agreement there is little but controversy. The terminology isn’t set, and different people use common terms (like “REST,” the topic of this book) in ways that combine into a vague and confusing mess. What’s missing is a coherent way of classifying the programmable web. With that in place, the meanings of individual terms will become clear.
Imagine the programmable web as an ecosystem, like the ocean, containing many kinds of strange creatures. Ancient scientists and sailors classified sea creatures by their superficial appearance: whales were lumped in with the fish. Modern scientists classify animals according to their position in the evolutionary tree of all life: whales are now grouped with the other mammals. There are two analogous ways of classifying the services that inhabit the programmable web: by the technologies they use (URIs, SOAP, XML-RPC, and so on), or by the underlying architectures and design philosophies.
Usually the two systems for classifying sea creatures get along. You don’t need to do DNA tests to know that a tuna is more like a grouper than a sea anenome. But if you really want to understand why whales can’t breathe underwater, you need to stop classifying them as fish (by superficial appearance) and start classifying them as mammals (by underlying architecture).[6]
When it comes to classifying the programmable web, most of today’s terminology sorts services by their superficial appearances: the technologies they use. These classifications work in most cases, but they’re conceptually lacking and they lead to whale-fish mistakes. I’m going to present a taxonomy based on architecture, which shows how technology choices follow from underlying design principles. I’m exposing divisions I’ll come back to throughout the book, but my main purpose is to zoom in on the parts of the programmable web that can reasonably be associated with the term “REST.”
HTTP: Documents in Envelopes
If I was classifying marine animals I’d start by talking about the things they have in common: DNA, cellular structure, the laws of embryonic development. Then I’d show how animals distinguish themselves from each other by specializing away from the common ground. To classify the programmable web, I’d like to start off with an overview of HTTP, the protocol that all web services have in common.
HTTP is a document-based protocol, in which the client puts a
document in an envelope and sends it to the server. The server returns
the favor by putting a response document in an envelope and sending it
to the client. HTTP has strict standards for what the envelopes should
look like, but it doesn’t much care what goes inside. Example 1-5 shows a sample envelope: the HTTP request my web
browser sends when I visit the homepage of oreilly.com. I’ve truncated two lines to make
the text fit on the printed page.
GET /index.html HTTP/1.1 Host: www.oreilly.com User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12)... Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,... Accept-Language: us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-15,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive
In case you’re not familiar with HTTP, now is a good time to point out the major parts of the HTTP request. I use these terms throughout the book.
- The HTTP method
In this request, the method is “GET.” In other discussions of REST you may see this called the “HTTP verb” or “HTTP action.”
The name of the HTTP method is like a method name in a programming language: it indicates how the client expects the server to process this envelope. In this case, the client (my web browser) is trying to GET some information from the server (
www.oreilly.com).
- The path
This is the portion of the URI to the right of the hostname: here, http://www.oreilly.com/index.html becomes “/index.html.” In terms of the envelope metaphor, the path is the address on the envelope. In this book I sometimes refer to the “URI” as shorthand for just the path.
- The request headers
These are bits of metadata: key-value pairs that act like informational stickers slapped onto the envelope. This request has eight headers:
Host,User-Agent,Accept, and so on. There’s a standard list of HTTP headers (see Appendix C), and applications can define their own.
- The entity-body, also called the document or representation
This is the document that inside the envelope. This particular request has no entity-body, which means the envelope is empty! This is typical for a GET request, where all the information needed to complete the request is in the path and the headers.
The HTTP response is also a document in a envelope. It’s almost
identical in form to the HTTP request. Example 1-6 shows a trimmed version of what the
server at oreilly.com sends my web
browser when I make the request in Example 1-5.
HTTP/1.1 200 OK
Date: Fri, 17 Nov 2006 15:36:32 GMT
Server: Apache
Last-Modified: Fri, 17 Nov 2006 09:05:32 GMT
Etag: "7359b7-a7fa-455d8264
Accept-Ranges: bytes
Content-Length: 43302
Content-Type: text/html
X-Cache: MISS from www.oreilly.com
Keep-Alive: timeout=15, max=1000
Connection: Keep-Alive
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
...
<title>oreilly.com -- Welcome to O'Reilly Media, Inc.</title>
...The response can be divided into three parts:
- The HTTP response code
This numeric code tells the client whether its request went well or poorly, and how the client should regard this envelope and its contents. In this case the GET operation must have succeeded, since the response code is 200 (“OK”). I describe the HTTP response codes in Appendix B.
- The response headers
Just as with the request headers, these are informational stickers slapped onto the envelope. This response has 10 headers:
Date,Server, and so on.
- The entity-body or representation
Again, this is the document inside the envelope, and this time there actually is one! The entity-body is the fulfillment of my GET request. The rest of the response is just an envelope with stickers on it, telling the web browser how to deal with the document.
The most important of these stickers is worth mentioning separately. The response header
Content-Typegives the media type of the entity-body. In this case, the media type istext/html. This lets my web browser know it can render the entity-body as an HTML document: a web page.There’s a standard list of media types (http://www.iana.org/assignments/media-types/). The most common media types designate textual documents (
text/html), structured data documents (application/xml), and images (image/jpeg). In other discussions of REST or HTTP, you may see the media type called the “MIME type,” “content type,” or “data type.”
Method Information
HTTP is the one thing that all “animals” on the programmable web have in common. Now I’ll show you how web services distinguish themselves from each other. There are two big questions that today’s web services answer differently. If you know how a web service answers these questions, you’ll have a good idea of how well it works with the Web.
The first question is how the client can convey its intentions to the server. How does the server know a certain request is a request to retrieve some data, instead of a request to delete that same data or to overwrite it with different data? Why should the server do this instead of doing that?
I call the information about what to do with the data the method information. One way to convey method information in a web service is to put it in the HTTP method. Since this is how RESTful web services do it, I’ll have a lot more to say about this later. For now, note that the five most common HTTP methods are GET, HEAD, PUT, DELETE, and POST. This is enough to distinguish between “retrieve some data” (GET), “delete that same data” (DELETE), and “overwrite it with different data” (PUT).
The great advantage of HTTP method names is that they’re standardized. Of course, the space of HTTP method names is much more limited than the space of method names in a programming language. Some web services prefer to look for application-specific method names elsewhere in the HTTP request: usually in the URI path or the request document.
Example 1-7 is a client for a web service that keeps its method information in the path: the web service for Flickr, Yahoo!’s online photo-sharing application. This sample application searches Flickr for photos. To run this program, you’ll need to create a Flickr account and apply for an API key.
#!/usr/bin/ruby -w
# flickr-photo-search.rb
require 'open-uri'
require 'rexml/document'
# Returns the URI to a small version of a Flickr photo.
def small_photo_uri(photo)
server = photo.attribute('server')
id = photo.attribute('id')
secret = photo.attribute('secret')
return "http://static.flickr.com/#{server}/#{id}_#{secret}_m.jpg"
end
# Searches Flickr for photos matching a certain tag, and prints a URI
# for each search result.
def print_each_photo(api_key, tag)
# Build the URI
uri = "http://www.flickr.com/services/rest?method=flickr.photos.search" +
"&api_key=#{api_key}&tags=#{tag}"
# Make the HTTP request and get the entity-body.
response = open(uri).read
# Parse the entity-body as an XML document.
doc = REXML::Document.new(response)
# For each photo found...
REXML::XPath.each(doc, '//photo') do |photo|
# ...generate and print its URI
puts small_photo_uri(photo) if photo
end
end
# Main program
#
if ARGV.size < 2
puts "Usage: #{$0} [Flickr API key] [search term]"
exit
end
api_key, tag = ARGV
print_each_photo(api_key, tag)This program makes HTTP requests to URIs like http://www.flickr.com/services/rest?method=flickr.photos.search&api_key=xxx&tag=penguins.
How does the server know what the client is trying to do? Well, the
method name is pretty clearly flickr.photos.search. Except: the HTTP method
is GET, and I am getting information, so it might
be that the method thing is a red
herring. Maybe the method information really goes in the HTTP
action.
This hypothesis doesn’t last for very long, because the Flickr API
supports many methods, not just “get”-type methods such as flickr.photos.search and flickr.people.findByEmail, but also methods
like flickr.photos.addTags,
flickr.photos.comments.deleteComment, and so
on. All of them are invoked with an HTTP GET request, regardless of
whether or not they “get” any data. It’s pretty clear that Flickr is
sticking the method information in the method query variable, and expecting the
client to ignore what the HTTP method says.
By contrast, a typical SOAP service keeps its method information in the entity-body and in a HTTP header. Example 1-8 is a Ruby script that searches the Web using Google’s SOAP-based API.
#!/usr/bin/ruby -w
# google-search.rb
require 'soap/wsdlDriver'
# Do a Google search and print out the title of each search result
def print_page_titles(license_key, query)
wsdl_uri = 'http://api.google.com/GoogleSearch.wsdl'
driver = SOAP::WSDLDriverFactory.new(wsdl_uri).create_rpc_driver
result_set = driver.doGoogleSearch(license_key, query, 0, 10, true, ' ',
false, ' ', ' ', ' ')
result_set.resultElements.each { |result| puts result.title }
end
# Main program.
if ARGV.size < 2
puts "Usage: #{$0} [Google license key] [query]"
exit
end
license_key, query = ARGV
print_page_titles(license_key, query)Tip
While I was writing this book, Google announced that it was deprecating its SOAP search service in favor of a RESTful, resource-oriented service (which, unfortunately, is encumbered by legal restrictions on use in a way the SOAP service isn’t). I haven’t changed the example because Google’s SOAP service still makes the best example I know of, and because I don’t expect you to actually run this program. I just want you to look at the code, and the SOAP and WSDL documents the code relies on.
OK, that probably wasn’t very informative, because the WSDL library hides most of the details. Here’s what
happens. When you call the doGoogleSearch method, the WSDL library makes
a POST request to the “endpoint” of the Google SOAP service, located at
the URI http://api.google.com/search/beta2. This
single URI is the destination for every API call, and only POST requests
are ever made to it. All of these details are in the WSDL file found at
http://api.google.com/GoogleSearch.wsdl, which
contains details like the definition of doGoogleSearch (Example 1-9).
<operation name="doGoogleSearch"> <input message="typens:doGoogleSearch"/> <output message="typens:doGoogleSearchResponse"/> </operation>
Since the URI and the HTTP method never vary, the method information—that “doGoogleSearch”—can’t go in either place. Instead, it goes into the entity-body of the POST request. Example 1-10 shows what HTTP request you might make to do a search for REST.
POST search/beta2 HTTP/1.1 Host: api.google.com Content-Type: application/soap+xml SOAPAction: urn:GoogleSearchAction <?xml version="1.0" encoding="UTF-8"?> <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Body> <gs:doGoogleSearch xmlns:gs="urn:GoogleSearch"> <q>REST</q> ... </gs:doGoogleSearch> </soap:Body> </soap:Envelope>
The method information is “doGoogleSearch.” That’s the name of the
XML tag inside the SOAP Envelope,
it’s the name of the operation in the
WSDL file, and it’s the name of the Ruby method in Example 1-8.
Let’s bring things full circle by considering not the Google SOAP search API, but the Google search engine itself. To use your web browser to search Google’s data set for REST, you’d send a GET request to http://www.google.com/search?q=REST and get an HTML response back. The method information is kept in the HTTP method: you’re GETting a list of search results.
Scoping Information
The other big question web services answer differently is how the client tells the server which part of the data set to operate on. Given that the server understands that the client wants to (say) delete some data, how can it know which data the client wants to delete? Why should the server operate on this data instead of that data?
I call this information the scoping information. One obvious place to put it is in the URI path. That’s what most web sites do. Think once again about a search engine URI like http://www.google.com/search?q=REST. There, the method information is “GET,” and the scoping information is “/search?q=REST.” The client is trying to GET a list of search results about REST, as opposed to trying to GET something else: say, a list of search results about jellyfish (the scoping information for that would be “/search?q=jellyfish”), or the Google home page (that would be “/”).
Many web services put scoping information in the path. Flickr’s is
one: most of the query variables in a Flickr API URI are scoping
information. tags=penguin scopes the
flickr.photos.search method so it
only searches for photos tagged with “penguin.” In a service where the
method information defines a method in the programming language sense,
the scoping information can be seen as a set of arguments to that
method. You could reasonably expect to see flickr.photos.search(tags=penguin) as a line
of code in some programming language.
The alternative is to put the scoping information into
the entity-body. A typical SOAP web service does it this way.
Example 1-10 contains a q tag whose contents are the string “REST.”
That’s the scoping information, nestled conveniently inside the doGoogleSearch tag that
provides the method information.
The service design determines what information is method information and what’s scoping information. This is most obvious in cases like Flickr and Google, where the web site and the web service do the same thing but have different designs. These two URIs contain the same information:
In the first URI, the method information is “GET” and the scoping
information is “photos tagged ‘penguin.’” In the second URI, the method
information is “do a photo search” and the scoping information is
“penguin.” From a technical standpoint, there’s no difference between
the two: both of them use HTTP GET. The differences only become apparent
at the level of architecture, when you take a step back and notice
values for methodname like flickr.photos.delete, which take HTTP’s GET
method into places it wasn’t meant to go.
Another example: in the Google SOAP API, the fact that you’re
doing a search is method information (doGoogleSearch). The search query is scoping
information (q). On the Google web
site, both “search” and the value for “q” are scoping information. The
method information is HTTP’s standard GET. (If the Google SOAP API
offered a method called doGoogleSearchForREST, it would be defining
the method information so expansively that you’d need no scoping
information to do a search for REST.)
The Competing Architectures
Now that I’ve identified the two main questions that web services answer differently, I can group web services by their answers to the questions. In my studies I’ve identified three common web service architectures: RESTful resource-oriented, RPC-style, and REST-RPC hybrid. I’ll cover each in turn.
RESTful, Resource-Oriented Architectures
The main topic of this book is the web service architectures which can be considered RESTful: those which get a good score when judged on the criteria set forth in Roy Fielding’s dissertation. Now, lots of architectures are technically RESTful,[7]but I want to focus on the architectures that are best for web services. So when I talk about RESTful web services, I mean services that look like the Web. I’m calling this kind of service resource-oriented. In Chapter 3 I’ll introduce the basic concepts of resource-oriented REST, in the context of a real web service: Amazon’s Simple Storage Service. Starting in Chapter 4, I’ll talk you through the defining characteristics of REST, and define a good architecture for RESTful web services: the Resource-Oriented Architecture.
In RESTful architectures, the method information goes into the HTTP method. In resource-oriented architectures, the scoping information goes into the URI. The combination is powerful. Given the first line of an HTTP request to a resource-oriented RESTful web service (“GET /reports/open-bugs HTTP/1.1”), you should understand basically what the client wants to do. The rest of the request is just details; indeed, you can make many requests using only one line of HTTP. If the HTTP method doesn’t match the method information, the service isn’t RESTful. If the scoping information isn’t in the URI, the service isn’t resource-oriented. These aren’t the only requirements, but they’re good rules of thumb.
A few well-known examples of RESTful, resource-oriented web services include:
Services that expose the Atom Publishing Protocol and its variants such as GData
Most of Yahoo!’s web services
Most other read-only web services that don’t use SOAP
Static web sites
Many web applications, especially read-only ones like search engines
Whenever I cover unRESTful architectures, as well as architectures that aren’t resource-oriented, I do it with some ulterior motive. In this chapter, I want to put RESTful web services into perspective, against the larger backdrop of the programmable web. In Chapter 2, I’m widening the book’s coverage of real web services, and showing that you can use the same client tools whether or not a service exactly fits my preferred architecture. In Chapter 10, I’m making an argument in a long-running debate about what the programmable web should look like.
RPC-Style Architectures
An RPC-style web service accepts an envelope full of data from its client, and sends a similar envelope back. The method and the scoping information are kept inside the envelope, or on stickers applied to the envelope. What kind of envelope is not important to my classification, but HTTP is a popular envelope format, since any web service worthy of the name must use HTTP anyway. SOAP is another popular envelope format (transmitting a SOAP document over HTTP puts the SOAP envelope inside an HTTP envelope). Every RPC-style service defines a brand new vocabulary. Computer programs work this way as well: every time you write a program, you define functions with different names. By contrast, all RESTful web services share a standard vocabulary of HTTP methods. Every object in a RESTful service responds to the same basic interface.
The XML-RPC protocol for web services is the most obvious example of the RPC architecture. XML-RPC is mostly a legacy protocol these days, but I’m going to start off with it because it’s relatively simple and easy to explain. Example 1-11 shows a Ruby client for an XML-RPC service that lets you look up anything with a Universal Product Code.
#!/usr/bin/ruby -w
# xmlrpc-upc.rb
require 'xmlrpc/client'
def find_product(upc)
server = XMLRPC::Client.new2('http://www.upcdatabase.com/rpc')
begin
response = server.call('lookupUPC', upc)
rescue XMLRPC::FaultException => e
puts "Error: "
puts e.faultCode
puts e.faultString
end
end
puts find_product("001441000055")['description']
# "Trader Joe's Thai Rice Noodles"An XML-RPC service models a programming language like C. You
call a function (lookupUPC) with
some arguments (“001441000055”) and get a return value back. The
method data (the function name) and the scoping data (the arguments)
are put inside an XML document. Example 1-12
gives a sample document.
<?xml version="1.0" ?> <methodCall> <methodName>lookupUPC</methodName> <params> <param><value><string>001441000055</string></value></param> </params> </methodCall>
This XML document is put into an envelope for transfer to the server. The envelope is an HTTP request with a method, URI, and headers (see Example 1-13). The XML document becomes the entity-body inside the HTTP envelope.
POST /rpc HTTP/1.1 Host: www.upcdatabase.com User-Agent: XMLRPC::Client (Ruby 1.8.4) Content-Type: text/xml; charset=utf-8 Content-Length: 158 Connection: keep-alive <?xml version="1.0" ?> <methodCall> <methodName>lookupUPC</methodName> ... </methodCall>
The XML document changes depending on which method you’re calling, but the HTTP envelope is always the same. No matter what you do with the UPC database service, the URI is always http://www.upcdatabase.com/rpc and the HTTP method is always POST. Simply put, an XML-RPC service ignores most features of HTTP. It exposes only one URI (the “endpoint”), and supports only one method on that URI (POST).
Where a RESTful service would expose different URIs for different values of the scoping information, an RPC-style service typically exposes a URI for each “document processor”: something that can open the envelopes and transform them into software commands. For purposes of comparison, Example 1-14 shows what that code might look like if the UPC database were a RESTful web service.
require 'open-uri'
upc_data = open('http://www.upcdatabase.com/upc/00598491').read()
...Here, the method information is contained in the HTTP method.
The default HTTP method is GET, which is equivalent in this scenario
to lookupUPC. The scoping
information is contained in the URI. The hypothetical service exposes
an enormous number of URIs: one for every possible UPC. By contrast,
the HTTP envelope is empty: an HTTP GET request contains no
entity-body at all.
For another example of a client for an RPC-style service, look back at Example 1-8. Google’s SOAP search API is an RPC-style service that uses SOAP as its envelope format.
A service that uses HTTP POST heavily or exclusively is probably an RPC-style service. Again, this isn’t a sure sign, but it’s a tip-off that the service isn’t very interested in putting its method information in the HTTP method. An otherwise RESTful service that uses HTTP POST a lot tends to move toward a REST-RPC hybrid architecture.
A few well-known examples of RPC-style web services:
All services that use XML-RPC
Just about every SOAP service (see the Technologies on the Programmable Web” section later in this chapter for a defense of this controversial statement)
A few web applications (generally poorly designed ones)
REST-RPC Hybrid Architectures
This is a term I made up for describing web services that fit somewhere in between the RESTful web services and the purely RPC-style services. These services are often created by programmers who know a lot about real-world web applications, but not much about the theory of REST.
Take another look at this URI used by the Flickr web service: http://www.flickr.com/services/rest?api_key=xxx&method=flickr.photos.search&tags=penguin. Despite the “rest” in the URI, this was clearly designed as an RPC-style service, one that uses HTTP as its envelope format. It’s got the scoping information (“photos tagged ‘penguin’”) in the URI, just like a RESTful resource-oriented service. But the method information (“search for photos”) also goes in the URI. In a RESTful service, the method information would go into the HTTP method (GET), and whatever was leftover would become scoping information. As it is, this service is simply using HTTP as an envelope format, sticking the method and scoping information wherever it pleases. This is an RPC-style service. Case closed.
Except…look at Example 1-15.
GET services/rest?api_key=xxx&method=flickr.photos.search&tags=penguin HTTP/1.1 Host: www.flickr.com
That’s the HTTP request a client makes when remotely calling this procedure. Now it looks like the method information is in the HTTP method. I’m sending a GET request to get something. What am I getting? A list of search results for photos tagged “penguin.” What used to look like method information (“photoSearch()”) now looks like scoping information (“photos/tag/penguin”). Now the web service looks RESTful.
This optical illusion happens when an RPC-style service uses plain old HTTP as its envelope format, and when both the method and the scoping information happen to live in the URI portion of the HTTP request. If the HTTP method is GET, and the point of the web service request is to “get” information, it’s hard to tell whether the method information is in the HTTP method or in the URI. Look at the HTTP requests that go across the wire and you see the requests you’d see for a RESTful web service. They may contain elements like “method=flickr.photos.search” but that could be interpreted as scoping information, the way “photos/” and “search/” are scoping information. These RPC-style services have elements of RESTful web services, more or less by accident. They’re only using HTTP as a convenient envelope format, but they’re using it in a way that overlaps with what a RESTful service might do.
Many read-only web services qualify as entirely RESTful and resource-oriented, even though they were designed in the RPC style! But if the service allows clients to write to the data set, there will be times when the client uses an HTTP method that doesn’t match up with the true method information. This keeps the service from being as RESTful as it could be. Services like these are the ones I consider to be REST-RPC hybrids.
Here’s one example. The Flickr web API asks clients to use HTTP
GET even when they want to modify the data set. To delete a photo you
make a GET request to a URI that includes method=flickr.photos.delete. That’s just not
what GET is for, as I’ll show in Chapter 5. The
Flickr web API is a REST-RPC hybrid: RESTful when the client is
retrieving data through GET, RPC-style when the client is modifying
the data set.
A few well-known examples of REST-RPC hybrid services include:
The del.icio.us API
The “REST” Flickr web API
Many other allegedly RESTful web services
Most web applications
From a design standpoint, I don’t think anybody sets out to to design a service as a REST-RPC hybrid. Because of the way HTTP works, any RPC-style service that uses plain HTTP and exposes multiple URIs tends to end up either RESTful or hybrid. Many programmers design web services exactly as they’d design web applications, and end up with hybrid services.
The existence of hybrid architectures has caused a lot of confusion. The style comes naturally to people who’ve designed web applications, and it’s often claimed that hybrid architectures are RESTful: after all, they work “the same way” as the human web. A lot of time has been spent trying to distinguish RESTful web services from these mysterious others. My classification of the “others” as REST-RPC hybrids is just the latest in a long line of neologisms. I think this particular neologism is the most accurate and useful way to look at these common but baffling services. If you’ve encountered other ways of describing them (“HTTP+POX” is the most popular at the time of writing), you might want read on, where I explain those other phrases in terms of what I’m saying in this book.
The Human Web Is on the Programmable Web
In the previous sections I claimed that all static web sites are RESTful. I claimed that web applications fall into one of the three categories, the majority being REST-RPC hybrids. Since the human web is made entirely of static web sites and web applications, this means that the entire human web is also on the programmable web! By now this should not be surprising to you. A web browser is a software program that makes HTTP requests and processes the responses somehow (by showing them to a human). That’s exactly what a web service client is. If it’s on the Web, it’s a web service.
My goal in this book is not to make the programmable web bigger. That’s almost impossible: the programmable web already encompasses nearly everything with an HTTP interface. My goal is to help make the programmable web better: more uniform, better-structured, and using the features of HTTP to greatest advantage.
Technologies on the Programmable Web
I’ve classified web services by their underlying architectures, distinguishing the fish from the whales. Now I can examine the technologies they use, without confusing technology and architecture.
HTTP
All web services use HTTP, but they use it in different ways. A request to a RESTful web service puts the method information in the HTTP method and the scoping information in the URI. RPC-style web services tend to ignore the HTTP method, looking for method and scoping information in the URI, HTTP headers, or entity-body. Some RPC-style web services use HTTP as an envelope containing a document, and others only use it as an unlabelled envelope containing another envelope.
URI
Again, all web services use URIs, but in different ways. What I’m about to say is a generalization, but a fairly accurate one. A RESTful, resource-oriented service exposes a URI for every piece of data the client might want to operate on. A REST-RPC hybrid exposes a URI for every operation the client might perform: one URI to fetch a piece of data, a different URI to delete that same data. An RPC-style service exposes one URI for every processes capable of handling Remote Procedure Calls (RPC). There’s usually only one such URI: the service “endpoint.”
XML-RPC
A few, mostly legacy, web services use XML-RPC on top of HTTP. XML-RPC is a data structure format for representing function calls and their return values. As the name implies, it’s explicitly designed to use an RPC style.
SOAP
Lots of web services use SOAP on top of HTTP. SOAP is an envelope format, like HTTP, but it’s an XML-based envelope format.
Now I’m going to say something controversial. To a first approximation, every current web service that uses SOAP also has an RPC architecture. This is controversial because many SOAP programmers think the RPC architecture is déclassé and prefer to call their services “message-oriented” or “document-oriented” services.
Well, all web services are message-oriented, because HTTP itself is message-oriented. An HTTP request is just a message: an envelope with a document inside. The question is what that document says. SOAP-based services ask the client to stick a second envelope (a SOAP document) inside the HTTP envelope. Again, the real question is what it says inside the envelope. A SOAP envelope can contain any XML data, just as an HTTP envelope can contain any data in its entity-body. But in every existing SOAP service, the SOAP envelope contains a description of an RPC call in a format similar to that of XML-RPC.
There are various ways of shuffling this RPC description around and giving it different labels—“document/literal” or “wrapped/literal”—but any way you slice it, you have a service with a large vocabulary of method information, a service that looks for scoping information inside the document rather than on the envelope. These are defining features of the RPC architecture.
I emphasize that this is not a fact about SOAP, just a fact about how it’s currently used. SOAP, like HTTP, is just a way of putting data in an envelope. Right now, though, the only data that ever gets put in that envelope is XML-RPC-esque data about how to call a remote function, or what’s the return value from such a function. I argue this point in more detail in Chapter 10.
WS-*
These standards define special XML “stickers” for the SOAP envelope. The stickers are analagous to HTTP headers.
WSDL
The Web Service Description Language (WSDL) is an XML vocabulary used to describe SOAP-based web services. A client can load a WSDL file and know exactly which RPC-style methods it can call, what arguments those methods expect, and which data types they return. Nearly every SOAP service in existence exposes a WSDL file, and most SOAP services would be very difficult to use without their WSDL files serving as guides. As I discuss in Chapter 10, WSDL bears more responsiblity than any other technology for maintaining SOAP’s association with the RPC style.
WADL
The Web Application Description Language (WADL) is an XML vocabulary used to describe RESTful web services. As with WSDL, a generic client can load a WADL file and be immediately equipped to access the full functionality of the corresponding web service. I discuss WADL in Chapter 9.
Since RESTful services have simpler interfaces, WADL is not nearly as necessary to these services as WSDL is to RPC-style SOAP services. This is a good thing, since as of the time of writing there are few real web services providing official WADL files. Yahoo!’s web search service is one that does.
Leftover Terminology
Believe it not, there are some common terms used in discussions of REST that I haven’t mentioned yet. I haven’t mentioned them because I think they’re inaccurate or entirely outside the scope of this book. But I owe you explanations of why I think this, so you can decide whether or not you agree. Feel free to skip this section if you haven’t heard these terms.
- Service-Oriented Architecture
This is a big industry buzzword. I’m not going to dwell on it for two reasons. First, the term is not very well defined. Second, to the extent that it is defined, it means something like: “a software architecture based on the production and consumption of web services.” In this book I talk about the design of individual services. A book on service-oriented architecture should work on a slightly higher level, showing how to use services as software components, how to integrate them into a coherent whole. I don’t cover that sort of thing in this book.
- SOAP as a competitor to REST
If you get involved with web service debates you’ll hear this one a lot. You won’t hear it here because it gives the wrong impression. The primary competitors to RESTful architectures are RPC architectures, not specific technologies like SOAP. It is true that basically every SOAP service that now exists has an RPC architecture, but SOAP is just a way of putting a document in an envelope with stickers on it, like HTTP. SOAP is tied to the RPC architecture mainly by historical contingency and the current generation of automated tools.
There is a real tension here, but it’s not one I’ll cover much in this book. Roughly speaking, it’s the tension between services that put their documents in a SOAP envelope and then an HTTP envelope; and services that only use the HTTP envelope.
- HTTP+POX
Stands for HTTP plus Plain Old XML. This term covers roughly those services I call REST-RPC hybrid services. They overlap with RESTful designs, especially when it comes to retrieving data, but their basic architecture is RPC-oriented.
I don’t like this term because Plain Old XML is inaccurate. The interesting thing about these services is not that they produce plain old XML documents (as opposed to XML documents wrapped in SOAP envelopes). Some of these services don’t serve XML at all: they serve JSON, plain text, or binary files. No, the interesting thing about these services is their RPC architecture. That’s what puts them in opposition to REST.
- STREST
Means Service-Trampled REST. This is another term for REST-RPC hybrid architectures. It’s more accurate than HTTP+POX since it conveys the notion of a RESTful architecture taken over by something else: in this case, the RPC style.
This is a cute acronym but I don’t like it, because it buys into a myth that the only true web services are RPC-style services. After all, the service that trampled your REST was an RPC service. If you think that REST services are real services, it doesn’t make sense to cry “Help! I had some REST but then this Service got into it!” RPC-Trampled REST would be more accurate, but that’s a lousy acronym.
- High and low REST
Yet another way of distinguishing between truly RESTful services and the ones I call REST-RPC hybrids. High REST services are just those that adhere closely to the Fielding dissertation. Among other things, they put method information in the HTTP method and scoping information in the URI. Low REST services are presumed to have deviated. Since low REST services tend to deviate from orthodoxy in a particular direction (toward the RPC style), I prefer a more specific terminology.
[5] Thanks to Big Web Services’ WS-Addressing standard, it’s now possible to create a web service that’s not on the Web: one that uses email or TCP as its transport protocol instead of HTTP. I don’t think absolutely everything has to be on the Web, but it does seem like you should have to call this bizarre spectacle something other than a web service. This point isn’t really important, since in practice nearly everyone uses HTTP. Thus the footnote. The only exceptions I know of are eBay’s web services, which can send you SOAP documents over email as well as HTTP.
[6] Melville, in Moby-Dick, spends much of Chapter 22 (“Cetology”) arguing that the whale is a fish. This sounds silly but he’s not denying that whales have lungs and give milk; he’s arguing for a definition of “fish” based on appearance, as opposed to Linnaeus’s definition “from the law of nature” (ex lege naturae).
[7] More than you’d think. The Google SOAP API for web search technically has a RESTful architecture. So do many other read-only SOAP and XML-RPC services. But these are bad architectures for web services, because they look nothing like the Web.
Get RESTful Web Services now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
