Learning HTTP/2

Chapter 1. The Evolution of HTTP

In the 1930s, Vannevar Bush, an electrical engineer from the United States then at MIT’s School of Engineering, had a concern with the volume of information people were producing relative to society’s ability to consume that information. In his essay published in the Atlantic Monthly in 1945 entitled, “As We May Think,” he said:

Professionally our methods of transmitting and reviewing the results of research are generations old and by now are totally inadequate for their purpose. If the aggregate time spent in writing scholarly works and in reading them could be evaluated, the ratio between these amounts of time might well be startling.

He envisioned a system where our aggregate knowledge was stored on microfilm and could be “consulted with exceeding speed and flexibility.” He further stated that this information should have contextual associations with related topics, much in the way the human mind links data together. His memex system was never built, but the ideas influenced those that followed.

The term Hypertext that we take for granted today was coined around 1963 and first published in 1965 by Ted Nelson, a software designer and visionary. He proposed the concept of hypertext to mean:

…a body of written or pictorial material interconnected in such a complex way that it could not conveniently be presented or represented on paper. It may contain summaries, or maps of its contents and their interrelations; it may contain annotations, additions and footnotes from scholars who have examined it. ¹

Nelson wanted to create a “docuverse” where information was interlinked and never deleted and easily available to all. He built on Bush’s ideas and in the 1970s created a prototype implementation of a hypertext system with his project Xanadu. It was unfortunately never completed, but provided the shoulders to stand on for those to come.

HTTP enters the picture in 1989. While at CERN, Tim Berners-Lee proposed² a new system for helping keep track of the information created by “the accelerators” (referencing the yet-to-be-built Large Hadron Collider) and experiments at the institution. He embraces two concepts from Nelson: Hypertext, or “Human-readable information linked together in an unconstrained way,” and Hypermedia, a term to “indicate that one is not bound to text.” In the proposal he discussed the creation of a server and browsers on many machines that could provide a “universal system.”

HTTP/0.9 and 1.0

HTTP/0.9 was a wonderfully simple, if limited, protocol. It had a single method (GET), there were no headers, and it was designed to only fetch HTML (meaning no images—just text).

Over the next few years, use of HTTP grew. By 1995 there were over 18,000 servers handling HTTP traffic on port 80 across the world. The protocol had evolved well past its 0.9 roots, and in 1996 RFC 1945³ codified HTTP/1.0.

Version 1.0 brought a massive amount of change to the little protocol that started it all. Whereas the 0.9 spec was about a page long, the 1.0 RFC came in at 60 pages. You could say it had grown from a toy into a tool. It brought in ideas that are very familiar to us today:

• Headers

• Response codes

• Redirects

• Errors

• Conditional requests

• Content encoding (compression)

• More request methods

and more. HTTP/1.0, though a large leap from 0.9, still had a number of known flaws that had to be addressed—most notably, the inability to keep a connection open between requests, the lack of a mandatory Host header, and bare bones options for caching. These three items had consequences on how the web could scale and needed to be addressed.

HTTP/1.1

Right on the heels of 1.0 came 1.1, the protocol that has lived on for over 20 years. It fixed a number of the aforementioned 1.0 problems. By making the Host header mandatory, it was now possible to perform virtual hosting, or serving multiple web properties on a singe IP address. When the new connection directives are used, a web server is not required to close a connection after a response. This was a boon for performance and efficiency since the browser no longer needed to reestablish the TCP connection on every request.

Additional changes included:

• An extension of cacheability headers

• An OPTIONS method

• The Upgrade header

• Range requests

• Compression with transfer-encoding

• Pipelining

Note

Pipelining is a feature that allows a client to send all of its requests at once. There were a couple of problems with pipelining that prevented its popularity. Servers still had to respond to the requests in order. This meant if one request takes a long time, this head of line blocking will get in the way of the other requests. Additionally, pipelining implementations in servers and proxies on the internet tended to range from nonexistent (bad) to broken (worse).

HTTP/1.1 was the result of HTTP/1.0’s success and the experience gained running the older protocol for a few years.

RFCs for HTTP/1.1

The Internet Engineering Task Force (IETF) publishes protocol specifications in committee-created drafts called Request for Comments (RFC). These committees are open to anyone with the time and inclination to participate. HTTP/1.1 was first defined in RFC 2068, then later replaced by RFC 2616, and finally revised in RFCs 7230 through 7235.

Beyond 1.1

Since 1999, RFC 2616, which specified HTTP/1.1, has defined the standard that the modern web is built on. Written in stone, it did not evolve or change. The web, however, and the way we used it continued to change in ways likely unimagined by its originators. The interactivity and utility of your average commerce site go well beyond the vision of an interwoven docuverse and fundamentally change the way we participate in our world. That evolution came despite the limitation of the protocol that we see today.

The most tangible change we can point to is in the makeup of web pages. The HTTP Archives only goes back to 2010, but in even that relatively short time the change has been dramatic. Every added object adds complexity and strains a protocol designed to request one object at a time.

SPDY

In 2009, Mike Belshe and Roberto Peon of Google proposed an alternative to HTTP, which they called SPDY⁴ (pronounced “speedy”). SPDY was not the first proposal to replace HTTP, but it was the most important as it moved the perceived performance mountain. Before SPDY, it was thought that there was not enough will in the industry to make breaking, or incompatible, changes to HTTP/1.1. The effort that would be required to coordinate the changes between browsers, servers, proxies, and various middle boxes was seen to be too great.

SPDY, however, changed everything. It quickly proved that there was a desire for something more efficient and a willingness to change. SPDY laid the groundwork for HTTP/2 and was responsible for proving out some of its key features such as multiplexing, framing, and header compression, among others. It was integrated quickly, even in internet time, into Chrome and Firefox, and eventually would be adopted by almost every major browser. Similarly, the necessary support in servers and proxies came along at about the same pace.

HTTP/2

In early 2012, the HTTP working group (the IETF group responsible for the HTTP specifications) was rechartered to work on the next version of HTTP. A key portion of its charter⁵ laid out their expectations for this new protocol:

It is expected that HTTP/2.0 will:

• Substantially and measurably improve end-user perceived latency in most cases, over HTTP/1.1 using TCP.

• Address the “head of line blocking” problem in HTTP.

• Not require multiple connections to a server to enable parallelism, thus improving its use of TCP, especially regarding congestion control.

• Retain the semantics of HTTP/1.1, leveraging existing documentation (see above), including (but not limited to) HTTP methods, status codes, URIs, and header fields.

• Clearly define how HTTP/2.0 interacts with HTTP/1.x, especially in intermediaries (both 2->1 and 1->2).

• Clearly identify any new extensibility points and policy for their appropriate use.

A call for proposals was sent out and it was decided to use SDPY as a starting point for HTTP/2.0. Finally, on May 14, 2015, RFC 7540 was published and HTTP/2 was official.

The remainder of this book lays out the rest of the story.

¹ T. H. Nelson. “Complex information processing: a file structure for the complex, the changing and the indeterminate.” ACM ’65 Proceedings of the 1965 20th national conference.

² https://www.w3.org/History/1989/proposal.html

³ https://tools.ietf.org/html/rfc1945

⁴ http://bit.ly/2oi9ZS9

⁵ http://bit.ly/2oid6cP