Preface
A complex system that works is invariably found to have evolved from a simple system that worked.
We wrote this book to tell you about an amazing new technology. It’s here, it’s hot, and it promises to radically change the way we write distributed systems. We’re talking about the World Wide Web.
Okay, it’s not a new technology. It’s not as hot as it used to be, and from a technical standpoint it’s not incredibly amazing. But everything else is true. In 10 years the Web has changed the way we live, but it’s got more change left to give. The Web is a simple, ubiquitous, yet overlooked platform for distributed programming. The goal of this book is to pull out that change and send it off into the world.
It may seem strange to claim that the Web’s potential for distributed programming has been overlooked. After all, this book competes for shelf space with any number of other books about web services. The problem is, most of today’s “web services” have nothing to do with the Web. In opposition to the Web’s simplicity, they espouse a heavyweight architecture for distributed object access, similar to COM or CORBA. Today’s “web service” architectures reinvent or ignore every feature that makes the Web successful.
It doesn’t have to be that way. We know the technologies behind the Web can drive useful remote services, because those services exist and we use them every day. We know such services can scale to enormous size, because they already do. Consider the Google search engine. What is it but a remote service for querying a massive database and getting back a formatted response? We don’t normally think of web sites as “services,” because that’s programming talk and a web site’s ultimate client is a human, but services are what they are.
Every web application—every web site—is a service. You can harness this power for programmable applications if you work with the Web instead of against it, if you don’t bury its unique power under layers of abstraction. It’s time to put the “web” back into “web services.”
The features that make a web site easy for a web surfer to use also make a web service API easy for a programmer to use. To find the principles underlying the design of these services, we can just translate the principles for human-readable web sites into terms that make sense when the surfers are computer programs.
That’s what we do in this book. Our goal throughout is to show the power (and, where appropriate, the limitations) of the basic web technologies: the HTTP application protocol, the URI naming standard, and the XML markup language. Our topic is the set of principles underlying the Web: Representational State Transfer, or REST. For the first time, we set down best practices for “RESTful” web services. We cut through the confusion and guesswork, replacing folklore and implicit knowledge with concrete advice.
We introduce the Resource-Oriented Architecture (ROA), a commonsense set of rules for designing RESTful web services. We also show you the view from the client side: how you can write programs to consume RESTful services. Our examples include real-world RESTful services like Amazon’s Simple Storage Service (S3), the various incarnations of the Atom Publishing Protocol, and Google Maps. We also take popular services that fall short of RESTfulness, like the del.icio.us social bookmarking API, and rehabilitate them.
The Web Is Simple
Why are we so obsessed with the Web that we think it can do everything? Perhaps we are delusional, the victims of hype. The web is certainly the most-hyped part of the Internet, despite the fact that HTTP is not the most popular Internet protocol. Depending on who’s measuring, the bulk of the world’s Internet traffic comes from email (thanks to spam) or BitTorrent (thanks to copyright infringement). If the Internet were to disappear tomorrow, email is the application people would miss the most. So why the Web? What makes HTTP, a protocol designed to schlep project notes around a physics lab, also suited for distributed Internet applications?
Actually, to say that HTTP was designed for anything is to pay it a pretty big compliment. HTTP and HTML have been called “the Whoopee Cushion and Joy Buzzer of Internet protocols, only comprehensible as elaborate practical jokes”—and that’s by someone who likes them.[1]The first version of HTTP sure looked like a joke. Here’s a sample interaction between client and server:
| Client request | Server response |
GET /hello.txt | Hello, world! |
That’s it. You connected to the server, gave it the path to a document, and then the server sent you the contents of that document. You could do little else with HTTP 0.9. It looked like a featureless rip-off of more sophisticated file transfer protocols like FTP.
This is, surprisingly, a big part of the answer. With tongue only slightly in cheek we can say that HTTP is uniquely well suited to distributed Internet applications because it has no features to speak of. You tell it what you want, and it gives it to you. In a twist straight out of a kung-fu movie,[2]HTTP’s weakness is its strength, its simplicity its power.
In that first version of HTTP, cleverly disguised as a lack of features, we can see addressability and statelessness: the two basic design decisions that made HTTP an improvement on its rivals, and that keep it scalable up to today’s mega-sites. Many of the features lacking in HTTP 0.9 have since turned out to be unnecessary or counterproductive. Adding them back actually cripples the Web. Most of the rest were implemented in the 1.0 and 1.1 revisions of the protocol. The other two technologies essential to the success of the Web, URIs and HTML (and, later, XML), are also simple in important senses.
Obviously, these “simple” technologies are powerful enough to give us the Web and the applications we use on it. In this book we go further, and claim that the World Wide Web is a simple and flexible environment for distributed programming. We also claim to know the reason for this: that there is no essential difference between the human web designed for our own use, and the “programmable web” designed for consumption by software programs. We say: if the Web is good enough for humans, it’s good enough for robots. We just need to make some allowances. Computer programs are good at building and parsing complex data structures, but they’re not as flexible as humans when it comes to interpreting documents.
Big Web Services Are Not Simple
There are a number of protocols and standards, mostly built on top of HTTP, designed for building Web Services (note the capitalization). These standards are collectively called the WS-* stack. They include WS-Notification, WS-Security, WSDL, and SOAP. Throughout this book we give the name “Big Web Services” to this collection of technologies as a fairly gentle term of disparagement.
This book does not cover these standards in any great detail. We believe you can implement web services without implementing Big Web Services: that the Web should be all the service you need. We believe the Web’s basic technologies are good enough to be considered the default platform for distributed services.
Some of the WS-* standards (such as SOAP) can be used in ways compatible with REST and our Resource-Oriented Architecture. In practice, though, they’re used to implement Remote Procedure Call applications over HTTP. Sometimes an RPC style is appropriate, and sometimes other needs take precedence over the virtues of the Web. This is fine.
What we don’t like is needless complexity. Too often a programmer or a company brings in Big Web Services for a job that plain old HTTP could handle just fine. The effect is that HTTP is reduced to a transport protocol for an enormous XML payload that explains what’s “really” going on. The resulting service is far too complex, impossible to debug, and won’t work unless your clients have the exact same setup as you do.
Big Web Services do have one advantage: modern tools can create a web service from your code with a single click, especially if you’re developing in Java or C#. If you’re using these tools to generate RPC-style web services with the WS-* stack, it probably doesn’t matter to you that a RESTful web service would be much simpler. The tools hide all the complexity, so who cares? Bandwidth and CPU are cheap.
This attitude works when you’re working in a homogeneous group, providing services behind a firewall for other groups like yours. If your group has enough political clout, you may be able to get people to play your way outside the firewall. But if you want your service to grow to Internet scale, you’ll have to handle clients you never planned for, using custom-built software stacks to do things to your service you never imagined were possible. Your users will want to integrate your service with other services you’ve never heard of. Sound difficult? This already happens on the Web every day.
Abstractions are never perfect. Every new layer creates failure points, interoperability hassles, and scalability problems. New tools can hide complexity, but they can’t justify it—and they always add it. Getting a service to work with the Web as a whole means paying attention to adaptability, scalability, and maintainability. Simplicity—that despised virtue of HTTP 0.9—is a prerequisite for all three. The more complex the system, the more difficult it is to fix when something goes wrong.
If you provide RESTful web services, you can spend your complexity on additional features, or on making multiple services interact. Success in providing services also means being part of the Web instead of just “on” the Web: making your information available under the same rules that govern well-designed web sites. The closer you are to the basic web protocols, the easier this is.
The Story of the REST
REST is simple, but it’s well defined and not an excuse for implementing web services as half-assed web sites because “they’re the same.” Unfortunately, until now the main REST reference was chapter five of Roy Fielding’s 2000 Ph.D. dissertation, which is a good read for a Ph.D. dissertation, but leaves most of the real-world questions unanswered. [3]That’s because it presents REST not as an architecture but as a way of judging architectures. The term “RESTful” is like the term “object-oriented.” A language, a framework, or an application may be designed in an object-oriented way, but that doesn’t make its architecture the object-oriented architecture.
Even in object-oriented languages like C++ and Ruby, it’s possible to write programs that are not truly object-oriented. HTTP in the abstract does very well on the criteria of REST. (It ought to, since Fielding co-wrote the HTTP standard and wrote his dissertation to describe the architecture of the Web.) But real web sites, web applications, and web services often betray the principles of REST. How can you be sure you’re correctly applying the principles to the problem of designing a specific web service?
Most other sources of information on REST are informal: mailing lists, wikis, and weblogs (I list some of the best in Appendix A). Up to now, REST’s best practices have been a matter of folklore. What’s needed is a concrete architecture based on the REST meta-architecture: a set of simple guidelines for implementing typical services that fulfill the potential of the Web. We present one such architecture in this book as the Resource-Oriented Architecture (see Chapter 4). It’s certainly not the only possible high-level RESTful architecture, but we think it’s a good one for designing web services that are easy for clients to use.
We wrote the ROA to bring the best practices of web service design out of the realm of folklore. What we’ve written is a suggested baseline. If you’ve tried to figure out REST in the past, we hope our architecture gives you confidence that what you’re doing is “really” REST. We also hope the ROA will help the community as a whole make faster progress in coming up with and codifying best practices. We want to make it easy for programmers to create distributed web applications that are elegant, that do the job they’re designed for, and that participate in the Web instead of merely living on top of it.
We know, however, that it’s not enough to have all these technical facts at your disposal. We’ve both worked in organizations where major architectural decisions didn’t go our way. You can’t succeed with a RESTful architecture if you never get a chance to use it. In addition to the technical know-how, we must give you the vocabulary to argue for RESTful solutions. We’ve positioned the ROA as a simple alternative to the RPC-style architecture used by today’s SOAP+WSDL services. The RPC architecture exposes internal algorithms through a complex programming-language-like interface that’s different for every service. The ROA exposes internal data through a simple document-processing interface that’s always the same. In Chapter 10, we compare the two architectures and show how to argue for the ROA.
Reuniting the Webs
Programmers have been using web sites as web services for years—unofficially, of course.[4]It’s difficult for a computer to understand web pages designed for human consumption, but that’s never stopped hackers from fetching pages with automated clients and screen-scraping the interesting bits. Over time, this drive was sublimated into programmer-friendly technologies for exposing a web site’s functionality in officially sanctioned ways—RSS, XML-RPC, and SOAP. These technologies formed a programmable web, one that extended the human web for the convenience of software programs.
Our ultimate goal in this book is to reunite the programmable web with the human web. We envision a single interconnected network: a World Wide Web that runs on one set of servers, uses one set of protocols, and obeys one set of design principles. A network that you can use whether you’re serving data to human beings or computer programs.
The Internet and the Web did not have to exist. They come to us courtesy of misallocated defense money, skunkworks engineering projects, worse-is-better engineering practices, big science, naive liberal idealism, cranky libertarian politics, techno-fetishism, and the sweat and capital of programmers and investors who thought they’d found an easy way to strike it rich.
The result is, amazingly, a simple, open (for now), almost universal platform for networked applications. This platform contains much of human knowledge and supports most fields of human endeavor. We think it’s time to seriously start applying its rules to distributed programming, to open up that information and those processes to automatic clients. If you agree, this book will show you how to do it.
What’s in This Book?
In this book we focus on practical issues: how to design and implement RESTful web services, and clients for those services. Our secondary focus is on theory: what it means to be RESTful, and why web services should be more RESTful instead of less. We don’t cover everything, but we try to hit today’s big topics, and because this is the first book of its kind, we return to the core issue—how to design a RESTful service—over and over again.
The first three chapters introduce web services from the client’s perspective and show what’s special about RESTful services.
- Chapter 1, The Programmable Web and Its Inhabitants
In this chapter we introduce web services in general: programs that go over the Web and ask a foreign server to provide data or run an algorithm. We demonstrate the three common web service architectures: RESTful, RPC-style, and REST-RPC hybrid. We show sample HTTP requests and responses for each architecture, along with typical client code.
- Chapter 2, Writing Web Service Clients
In this chapter we show you how to write clients for existing web services, using an HTTP library and an XML parser. We introduce a popular REST-RPC service (the web service for the social bookmarking site del.icio.us) and demonstrate clients written in Ruby, Python, Java, C#, and PHP. We also give technology recommendations for several other languages, without actually showing code. JavaScript and Ajax are covered separately in Chapter 11.
The next six chapters form the core of the book. They focus on designing and implementing your own RESTful services.
- Chapter 4, The Resource-Oriented Architecture
A formal introduction to REST, not in its abstract form but in the context of a specific architecture for web services. Our architecture is based on four important REST concepts: resources, their names, their representations, and the links between them. Its services should be judged by four RESTful properties: addressability, statelessness, connectedness, and the uniform interface.
- Chapter 5, Designing Read-Only Resource-Oriented Services
We present a procedure for turning an idea or a set of requirements into a set of RESTful resources. These resources are read-only: clients can get data from your service but they can’t send any data of their own. We illustrate the procedure by designing a web service for serving navigable maps, inspired by the Google Maps web application.
- Chapter 6, Designing Read/Write Resource-Oriented Services
We extend the procedure from the previous chapter so that clients can create, modify, and delete resources. We demonstrate by adding two new kinds of resource to the map service: user accounts and user-defined places.
- Chapter 8, REST and ROA Best Practices
In this chapter we collect our earlier suggestions for service design into one place, and add new suggestions. We show how standard features of HTTP can help you with common problems and optimizations. We also give resource-oriented designs for tough features like transactions, which you may have thought were impossible to do in RESTful web services.
- Chapter 9, The Building Blocks of Services
Here we describe extra technologies that work on top of REST’s big three of HTTP, URI, and XML. Some of these technologies are file formats for conveying state, like XHTML and its microformats. Some are hypermedia formats for showing clients the levers of state, like WADL. Some are sets of rules for building RESTful web services, like the Atom Publishing Protocol.
The last three chapters cover specialized topics, each of which could make for a book in its own right:
- Chapter 10, The Resource-Oriented Architecture Versus Big Web Services
We compare our architecture, and REST in general, to another leading brand. We think that RESTful web services are simpler, more scalable, easier to use, better attuned to the philosophy of the Web, and better able to handle a wide variety of clients than are services based on SOAP, WSDL, and the WS-* stack.
- Chapter 11, Ajax Applications as REST Clients
Here we explain the Ajax architecture for web applications in terms of web services: an Ajax application is just a web service client that runs inside your web browser. That makes this chapter an extension of Chapter 2. We show how to write clients for RESTful web services using
XMLHttpRequestand the standard JavaScript library.
- Chapter 12, Frameworks for RESTful Services
In the final chapter we cover three popular frameworks that make it easy to implement RESTful web services: Ruby on Rails, Restlet (for Java), and Django (for Python).
We also have three appendixes we hope you find useful:
- Appendix A, Some Resources for REST and Some RESTful Resources
The first part lists interesting standards, tutorials, and communities related to RESTful web services. The second part lists some existing, public RESTful web services that you can use and learn from.
- Appendix B, The HTTP Response Code Top 42
Describes every standard HTTP response code (plus one extension), and explains when you’d use each one in a RESTful web service.
- Appendix C, The HTTP Header Top Infinity
Does the same thing for HTTP headers. It covers every standard HTTP header, and a few extension headers that are useful for web services.
Administrative Notes
This book has two authors (Leonard and Sam), but for the rest of the book we’ll be merging our identities into a single authorial “I.” In the final chapter (Chapter 12), the authorial “I” gets a little bit more crowded, as Django and Restlet developers join in to show how their frameworks let you build RESTful services.
We assume that you’re a competent programmer, but not that you have any experience with web programming in particular. What we say in this book is not tied to any programming language, and we include sample code for RESTful clients and services in a variety of languages. But whenever we’re not demonstrating a specific framework or language, we use Ruby as our implementation language.
We chose Ruby because it’s concise and easy to read, even for programmers who don’t know the language. (And because it’s nice and confusing in conjunction with Sam’s last name.) Ruby’s standard web framework, Ruby on Rails, is also one of the leading implementation platforms for RESTful web services. If you don’t know Ruby, don’t worry: we include lots of comments explaining Ruby-specific idioms.
The sample programs in this book are available for download from this book’s official web site. This includes the entire Rails application from Chapter 7, and the corresponding Restlet and Django applications from Chapter 12. It also includes Java implementations of many of the clients that only show up in the book as Ruby implementations. These client programs use the Restlet library, and were written by Restlet developers Jerome Louvel and Dave Pawson. If you’re more familiar with Java than with Ruby, these implementations may help you grasp the concepts behind the code. Most notably, there’s a full Java implementation of the Amazon S3 client from Chapter 3 in there.
Conventions Used in This Book
The following typographical conventions are used in this book:
- Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant widthUsed for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
Constant width boldShows commands or other text that should be typed literally by the user.
Constant width italicShows text that should be replaced with user-supplied values or by values determined by context.
Tip
This icon signifies a tip, suggestion, or general note.
Caution
This icon indicates a warning or caution.
Using Code Examples
This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “RESTful Web Services by Leonard Richardson and Sam Ruby. Copyright 2007 O’Reilly Media, Inc., 978-0-596-52926-0.”
If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.
Safari® Enabled
Note
When you see a Safari® Enabled icon on the cover of your favorite technology book, that means the book is available online through the O’Reilly Network Safari Bookshelf.
Safari offers a solution that’s better than e-books. It’s a virtual library that lets you easily search thousands of top tech books, cut and paste code samples, download chapters, and find quick answers when you need the most accurate, current information. Try it for free at http://safari.oreilly.com.
How to Contact Us
Please address comments and questions concerning this book to the publisher:
| O’Reilly Media, Inc. |
| 1005 Gravenstein Highway North |
| Sebastopol, CA 95472 |
| 800-998-9938 (in the United States or Canada) |
| 707-829-0515 (international or local) |
| 707 829-0104 (fax) |
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at:
| http://www.oreilly.com/catalog/9780596529260 |
To comment or ask technical questions about this book, send email to:
| bookquestions@oreilly.com |
For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see our web site at:
| http://www.oreilly.com |
Acknowledgments
We’re ultimately indebted to the people whose work made us see that we could program directly with HTTP. For Sam, it was Rael Dornfest with his Blosxom blogging application. Leonard’s experience stems from building screen-scraping applications in the mid-90s. His thanks go to those whose web design made their sites usable as web services: notably, the pseudonymous author of the online comic “Pokey the Penguin.”
Once we had this insight, Roy Fielding was there to flesh it out. His thesis named and defined something that was for us only a feeling. Roy’s theoretical foundation is what we’ve tried to build on.
In writing this book we had an enormous amount of help from the REST community. We’re grateful for the feedback we got from Benjamin Carlyle, David Gourley, Joe Gregorio, Marc Hadley, Chuck Hinson, Pete Lacey, Larry Liberto, Benjamin Pollack, Aron Roberts, Richard Walker, and Yohei Yamamoto. Others helped us unknowingly, through their writings: Mark Baker, Tim Berners-Lee, Alex Bunardzic, Duncan Cragg, David Heinemeier Hansson, Ian Hickson, Mark Nottingham, Koranteng Ofosu-Amaah, Uche Ogbuji, Mark Pilgrim, Paul Prescod, Clay Shirky, Brian Totty, and Jon Udell. Of course, all opinions in this book, and any errors and omissions, are our own.
Our editor Michael Loukides was helpful and knowledgeable throughout the process of developing this book. We’d also like to thank Laurel Ruma and everyone else at O’Reilly for their production work.
Finally, Jerome Louvel, Dave Pawson, and Jacob Kaplan-Moss deserve special thanks. Their knowledge of Restlet and Django made Chapter 12 possible.
[1] Clay Shirky, In Praise of Evolvable Systems
[2] Legend of The Drunken Protocol (1991)
[3] Fielding, Roy Thomas. Architectural Styles and the Design of Network-Based Software Architectures, Doctoral dissertation, University of California, Irvine, 2000 (http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm)
[4] For an early example, see Jon Udell’s 1996 Byte article “On-Line Componentware”. Note: “A powerful capability for ad hoc distributed computing arises naturally from the architecture of the Web.” That’s from 1996, folks.
Get RESTful Web Services now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
