Web application development is different (and better)

The Web became the most ubiquitous distributed application system because it didn’t have to think of itself as a programming environment. Almost every day I see comments or complaints from programmers (even brilliant programmers) muttering about how many strange and inferior parts they have to deal with, how they’d like to fix a historical accident by ripping out HTML completely and replacing it with Canvas, and how separation of concerns is an inconvenience. Everything should be JavaScript.

(Apologies to Tom Dale, who tweeted a perfect series of counterpoints just as I was writing. He has visions of rebuilding the rendering stack in JavaScript, but those tweets are not unusual opinions.)

The Web is different, and I can see why programmers might have little tolerance for the paths it chose, but this time the programmers are wrong. It’s not that the Web is perfect – it certainly has glitches. It’s not that success means something is better. Many terrible things have found broad audiences, and there are infinite levels to the Worse is Better conversations. And of course, the Web doesn’t solve every programming need. Many problems just don’t fit, and that’s fine.

So why is the Web better?

The Web made it possible to address the project scalability problem, making some key choices that allowed both human and technical distribution of responsibilities. Those choices are compromises, but the balance they achieved lets developers at any level contribute to the Web.

The Web had great timing. It had a few years at the beginning to settle and develop a story about what it was. Next came a few years of insane rampaging growth in which developers large and small could make mistakes, energy to fix those mistakes, and then a quieter period for those fixes to set in. Even in the quiet period, the Web maintained mammoth scale, and web projects continued growing in scale and scope.

It’s not just that the Web is a great way to build distributed applications. It’s not just that it showed up at a time when a lot of us wanted to do that. It’s that the front end, the back end, and the connections between them are built with a unique set of features and constraints that make it much easier to build great (and maintainable!) applications.

The Front End

In the Beginning, there was HTML. HTML, to be precise, with hypertext links serving as GOTO-like glue. Content, structure, formatting, behavior, and links were all stuck together in text documents, creating Ted Nelson’s worst technical nightmare.

As the web (and web sites) got bigger, search and replace hit their limits. The base layer, HTML, still contains a mix of content, structure, and hyperlink behavior, but other separations of concerns made it much easier to maintain and distribute web content. Eventually that spawned the web application context we have today.

HyperText Markup Language (HTML) remains approachable to a large number of people. Not everyone, but many more people than “programmers” or “designers”. With basic knowledge of how markup works and a few key tags, people can create content that others can use in many different contexts, from basic pages to complex applications. I still do a double-take when I see “HTML code,” but that usage is correct – HTML is not a programming language per se, but certainly encodes content and structure.

Sketching out an application in HTML is often easier than it is on paper or in Photoshop. Text is easy to store and manipulate (though encoding headaches exist). Google relies on this, to take just one key search engine example. The markup structure also defines a tree. The Document Object Model (DOM), or one of the many wrappers that seek to improve the DOM experience, give developers access to that tree.

Cascading Style Sheets (CSS) spares people writing HTML the headache of figuring out what exactly it should look like. Despite continuing concerns about the challenges of layout within a web browser, and efforts to add more logic to what is becoming a complex world, CSS hit a sweet spot. Its declarative approach made it easy for humans to create it and browsers to apply it efficiently. The ‘cascade’, though sometimes mysterious in its details, usually only sees use in situations complex enough to require that level of solution. Most amazing to me, CSS has also helped integrate hardware acceleration for animation and transformation in ways that would be much more difficult in a procedural environment.

CSS plays a key role in web architecture, though, beyond helping make things beautiful. CSS makes it much much easier to manage libraries of HTML documents, including application components, reducing redundancy and errors. The simple fact that style information applies to multiple documents has many benefits:

• From a programming perspective, the “separation of concerns” between HTML and CSS fits neatly with the “Don’t Repeat Yourself” (DRY) approach to programming. Style sheets may have repetition internally, but compared to every document repeating presentation information, it’s an easy win.
• From a network performance perspective, separate reused stylesheets makes caching work more efficiently.
• From a design perspective, it makes it much easier to have designers manage design with much simpler guidelines for people creating content.

Yes, having presentation information in separate documents is sometimes inconvenient. (Using the style attribute doesn’t win points for elegance.) That inconvenience, though, disappears rapidly as soon a site or application includes more than one document!

CSS had two cultural problems, one from the design side and one from the programming side. While HTML documents were (mostly) naturally responsive, reflowing to fit whatever size browser container they needed, the additional control CSS gave designers encouraged them to create beautiful but brittle layouts. New features for layout and media queries are making Responsive Web Design workable once again. On the programming side, well, CSS isn’t programming, as this excessively plausible satire of that attitude makes clear.

JavaScript provides behavior. Long criticized as a scrambled mess of a not quite object-oriented, but not quite functional programming language, JavaScript has gone from toy language to must-learn language. The cursing and criticism of JavaScript hasn’t stopped, and JavaScript programmers come in many varieties, but JavaScript’s strengths match distributed application development very well.

Over the past few years, browser vendors have created application programming interfaces (APIs) that give developers more opportunities to interact with both content and devices. Once JavaScript moved beyond (now banned) status bar tricks, the Document Object Model (DOM) was the first step. DOM rarely makes the “best API design” list, but it has improved over time and developers have wrapped the DOM to make it more usable. More recent APIs have provided access to local storage and device capabilities, breaking browsers out of their sandbox without giving them access to everything on a user’s computer.

Ajax, a development pattern built on an API, appeared on the scene in 2005 when Jesse James Garrett gave that name to a pattern of development he’d noticed a few times, combining these Web technologies with the XMLHttpRequest object that had appeared in Internet Explorer in 1999. It took time for the idea to spread to more browsers, but suddenly a huge group of programmers was building front-end applications. (Despite the object name, they often use JSON, a data format pulled from JavaScript, instead of XML.)

The Web keeps doing more with these tools, of course.

JavaScript, now further strengthened by Web Components, lets developers extend the browser’s behavior. CSS lets developers tell a browser how to present content. Between those options and additional APIs, it’s possible to extend HTML beyond what Alex Russell complained was a “3rd-grade vocabulary”. It’s now possible to lay down the “all HTML must be standard” mantra because the other parts of the web front-end stack are themselves standardized.

What does that mean? It means that:

• Developers can wrap APIs to tune their interfaces.
• Developers can fill gaps in browsers that are missing APIs or other functionality.
• Developers can extend the HTML vocabulary, using Web Components. You want a calendar, or a tab bar, or a tooltip? Just add one with brick or create one with Polymer. Or roll your own. HTML’s existing vocabulary becomes a foundation wall rather than a boundary wall.

The browser itself, not just the markup language, is now extensible, and these tools work across browser environments. Some of those extensions may be as simple as using new elements and providing CSS styles for them. Programmers hold the keys to more substantial extensions, adding behavior, but in this model they can share their extensions with non-programmers who just include them in their markup. You don’t have to be a programmer to use the new features, and the familiar toolset still works. (Complex polyfills can create questions around accessibility, but WAI-ARIA provides a foundation for many cases and the W3C is working on maintaining accessibility in the harder cases.)

Polyfills, at least polyfills using JavaScript, still have one small flaw: they require that JavaScript be on to do their work. The failures aren’t as dramatic as most of those shown, for example, at Sigh, JavaScript, but subtle failures may actually create more difficult problems.

Most of the failures at Sigh, JavaScript come from a corner of web application development that has turned its back on the HTML/CSS/JavaScript stack. That approach considers web applications primarily as JavaScript applications. The HTML is just a shell loading JavaScript which then sets up everything through the DOM (or in more extreme cases, Canvas). While generated markup is visible through the debugger, this model expects the programmer to be in control of everything. CSS may still be separate, but always has to be tested against the application, not against more easily iterated documents.

Single-Page Apps, the most prominent of those approaches, is a great match for the expectations of developers coming from other desktop or mobile application models, or even Flash. The Web can accommodate this model (on both the front and back ends), and browser performance has improved to the point where it can compete in other ways with native applications. However, it discards much of what made Web culture scalable. Programmers may be kings and queens of their applications, but there aren’t enough kings and queens to go around.

Single-Page Apps also bear a stigma created by their insistence on using JavaScript to populate their underlying markup and content: they are effectively invisible to Google. For some apps, that’s fine: they don’t want Google or other scrapers looking inside their customers’ data. When it matters, you can, of course, run server farms to generate the markup when Google comes looking, and years of experience in shifting code between client and server may pay off in multiple contexts. Isomorphic JavaScript, a more hybrid architecture, seems to be finding its footing.

The Glue

Almost at the Beginning, in 1994, there was the URL. A uniform resource locator or web address, the URL not only connects the front and back ends of web development, but makes hyperlinking easy. Despite a variety of attempts (from the CueCat to QR codes) to replace text-based URLs with computer-readable graphics, the URL remains both popular and the foundation for its attempted successors.

On the front end, URLs were the heart of navigation. They provided starting points (think home page), hyperlink targets, bookmarkable content, and a simple model for the back and forward buttons. When Ajax arrived, and pages suddenly started loading new content without changing their base URL, the back button became unpredictable at best. A solution, the History API, lets JavaScript code manage its contribution to browser history manually. It helps, but also creates demos bearing disclaimers like “Note: since these urls aren’t real, refreshing the page will land on an invalid url.”.

Fragment identifiers, the optional part on the right end of a URL that starts with #, let you go further, reaching to a specific point in a document. Unfortunately, they quickly become complicated once you leave behind the basic expectations of either the HTML vocabulary or multi-page browsing models. Re-creating fragment identifiers in a single-page app was not my favorite part of the project a few years ago. Not everyone is as excited as I am about fragment identifiers of course (even proposing syntax to extend their pointing abilities – more here).

On the back end, the way that URLs, at least the parts to the right of the domain name, actually work has changed over the years. They used to map to files on the server, and still sometimes do, but many web frameworks now centralize control through routing approaches. PHP has been a good standard-bearer for distributing application logic across a large set of files, while Rails made routing a central component of its development story. Despite the different models, back-end developers have largely accepted the notion that URLs can work as pointers to different aspects of the same application.

Stop for a second, though, and think about your expectations of other application contexts. When I open an application on the desktop or a mobile device, my options are pretty much that it opens where I was last time or it opens on a new screen. It’s not so easy to have an application open up the way I want it, especially if I have multiple “the way I want it” options. With the Web, this is generally trivial – URLs let you link to specific points in an application, and halfway-smart development models let you log in as needed and get on with your work. Maybe there’s room to extend this capability further, but URLs definitely create capabilities for web apps that other apps have barely begun to consider.

(URLs have, in a sense, been punished for their success. Tim Berners-Lee grew less excited about them as he shifted focus toward visions of the Semantic Web. Instead of a mostly clear story about URLs, the W3C now spends much of its time working around the theology of the URI. If you have a dark sense of humor, a fondness for tech conversation, and lots of time on your hands, you could waste a year on this. Just seek out httpRange-14, the closed but never actually ending tear in the URI space-time continuum, or its predecessor conversation.)

The Back End

In the Beginning, there was HTTP. All it offered clients was the ability to GET content from a URL, but over time the number of methods – verbs – expanded. GET and POST remain dominant, but PUT and DELETE have also become central for developers managing information in web applications.

HTTP’s original charm was its simplicity. It didn’t actually do much, and its ubiquity and apparent harmlessness combined to make firewall vendors largely willing to let it pass through. Because interest in consuming Web content grew faster than bandwidth, HTTP’s maintainers placed high priority on caching and proxies. After a few iterations that led to a protocol that was still relatively simple, but with a strong interest in performance. (As is all too often the case, many of the advantages of that simplicity are fading with the need to add security, but the architectural legacy remains.)

Statelessness was another feature of HTTP, forcing developers to consider how they would maintain conversations across multiple requests. Most of us settled for cookies – I now repent having written a book on them – that made the challenge less visible to users and developers, but also, because they offered persistence, opened the door to endless user tracking strategies.

Around the turn of the millennium, “Web Services” lurched forward, using HTTP POST requests for transportation and as an easy way to cut through firewalls. Like Single-Page Apps, they used web infrastructure in pursuit of an architecture that didn’t resemble web architecture or take much advantage of its features. Instead, it took much of its inspiration from CORBA, and enterprise architectures of the time.

At the time, I worried mostly that Web Services were destroying XML, turning it toward programming tasks for which it wasn’t well-suited. Fortunately HTTP at least was looking out for itself, and struck back in the unusual form of a doctoral dissertation chapter that found traction.

REST recognized the strengths of HTTP, in particular its providing a uniform (and minimal) interface to resources. HTTP’s constraints were in fact its virtues, though the religious-sounding nature of that claim has led to pretty much infinite debate about what is and is not truly RESTful. Early discussions of CREATE-READ-UPDATE-DELETE (CRUD) as parallels to the HTTP verbs led to of applications and frameworks (notably Rails) that used those methods blindly.

The most remarkable part of REST’s growth, to me, is that proponents of Web Services insisted that to reach both the public world and the internal enterprise world, Web Services had to base themselves on the expectations of enterprise developers. To a substantial extent, REST has demonstrated that enterprise developers can lean a lot from the architecture and practices of a larger public Web world.

More recently, it’s not just REST, but hypermedia interfaces pushing against traditional expectations of how you structure a program. Despite some programmers’ complaints that documents and hypertext as the basis of web programming is just an unfortunate quirk, I can’t help but see document-based approaches to distributed development as a feature, not a bug.

HTTP does have some key limitations – it’s a protocol that expects clients and servers and intermediaries to be transferring relatively large chunks of content in bursts. It doesn’t do peer-to-peer, handle tiny content efficiently, or do a great job of dealing with always-open connections. Those cases teach different lessons and need different technologies, notably WebRTC for the peer-to-peer and WebSockets for the small and always-on.

However, like its Web peers, HTTP has triumphed in a wide range of situations, not just despite its limitations, but because of them. HTTP even has a descendant, Constrained Application Protocol, which supports the RESTful HTTP verbs in a lighter-weight context!

The Web is (Mostly) Better

In Worse is Better, Richard Gabriel argued that “Unix and C are the ultimate computer viruses.” They were. Today, the Web has proven even more viral.

Structured document exchange was not a field anyone forecast would change the way we write and share programs, but it has turned out to be a remarkably good fit. It doesn’t solve every problem – please don’t try to rewrite Photoshop or video-processing software to run in a browser – but it hits an 80/20 point and then some, with signs of going further.

Why has this worked so well? Distributed application development has always been difficult, and the Web has been distributed almost from the start. Some of it, I think, has to do with the peculiar stresses of the early Web: the need to scale projects with limited resources, and the need to scale content delivery with limited bandwidth. Those best practices grew out of challenging situations we have largely forgotten. The other key piece, though, is the URL, which provided glue and a basic set of expectations.

I was delighted to see the Extensible Web Manifesto last year, which seemed to me to pursue a vision of building on this existing web infrastructure to create new possibilities. “Small mutations” sound good, but are they small? Lately I’m getting a stronger sense that others read it as an opportunity to tear things down and rebuild them in programmers’ own image, chucking or at the very least destabilizing the very layers that make the Web scalable and accessible.

Let’s extend the Web and help it do more – but let’s do that by valuing the many strengths it already brings.

If you find this insanity interesting or want to explore its roots more deeply, you might explore The Web Platform: Building a Solid Stack of HTML, CSS, and JavaScript, a free collection of related writing, or come to O’Reilly’s Fluent Conference and ask me questions.