WTF?: What's the Future and Why It's Up to Us

2

TOWARD A GLOBAL BRAIN

MY FOCUS ON THE INTERNET RATHER THAN ON LINUX EVENTUALLY led me in very different directions from other open source advocates. They wanted to argue about the best open source licenses. I thought that licenses didn’t matter as much as everyone else thought they did.

I was fascinated by the massive next-generation infrastructure and business processes Google was building. Others were interested in these things too, but they thought very few companies would need Google’s kind of infrastructure, or to use its techniques. They were wrong.

This is my next lesson. If the future is here, but just not evenly distributed yet, find seeds of that future, study them, and ask yourself how things will be different when they are the new normal. What happens if this trend keeps going?

Over the next few years, I refined my argument, eventually developing a talk called “The Open Source Paradigm Shift,” which I delivered hundreds of times to business and technical audiences. I always started the talk with a question: “How many of you use Linux?” Sometimes only a few hands would go up; sometimes many. But when I asked my next question, “How many of you use Google?” almost every hand in the room would go up. “You just told me,” I said, “that you still think of the software you use as what is running on your local computer. Google is built on top of Linux. You’re all using it.”

The way you view the world limits what you can see.

Microsoft had come to define a paradigm in which competitive advantage and user control came through proprietary software running on a desktop computer. Most free and open source advocates had accepted that map of the world, and were looking to Linux to compete with Microsoft Windows as an operating system for desktop and laptop computers. Instead, I argued, open source was becoming the “Intel Inside” of the next generation of computer applications. I was thinking about what was different in how that next generation of software was going to work, and how that would upset the power dynamics of the computer industry far more deeply than any competition by Linux on the desktop.

As open source developers gave away their software for free, many could see only the devaluation of something that was once a locus of enormous value. Thus Red Hat founder Bob Young told me, “My goal is to shrink the size of the operating system market.” (Red Hat, however, aimed to own a large part of that smaller market.) Defenders of the status quo, such as Microsoft VP Jim Allchin, claimed that “open source is an intellectual property destroyer,” and painted a bleak picture in which a great industry is destroyed, with nothing to take its place.

The commoditization of operating systems, databases, web servers and browsers, and related software was indeed threatening to Microsoft’s core business. But that destruction created the opportunity for the killer applications of the Internet era. It is worth remembering this history when contemplating the effect of on-demand services like Uber, self-driving cars, and artificial intelligence.

I found that Clayton Christensen, the author of The Innovator’s Dilemma and The Innovator’s Solution, had developed a framework that explained what I was observing. In a 2004 article in Harvard Business Review, he articulated “the law of conservation of attractive profits” as follows:

“When attractive profits disappear at one stage in the value chain because a product becomes modular and commoditized, the opportunity to earn attractive profits with proprietary products will usually emerge at an adjacent stage.”

I saw Christensen’s law of conservation of attractive profits at work in the paradigm shifts required by open source software. Just as IBM’s commoditization of the basic design of the personal computer led to opportunities for attractive profits “up the stack” in software, new fortunes were being made up the stack from the commodity open source software that underlies the Internet, in a new class of proprietary applications.

Google and Amazon provided a serious challenge to the traditional understanding of free and open source software. Here were applications built on top of Linux, but they were fiercely proprietary. What’s more, even when using and modifying software distributed under the most restrictive of free software licenses, the GPL (GNU Public License), these sites were not constrained by any of its provisions, all of which were framed in terms of the old paradigm. The GPL’s protections were triggered by the act of software distribution, yet web-based applications don’t distribute any software: It is simply performed on the Internet’s global stage, delivered as a service rather than as a packaged software application.

But even more important, even if these sites gave out their source code, users would not easily be able to create a full copy of the running application. I told free software advocates like Richard Stallman that even if they had all of the software that Amazon or Google had built on top of Linux, they wouldn’t have Amazon or Google. These sites didn’t just consist of a set of software programs. They consisted of massive amounts of data and the people and business processes used to gather, manage, and build ongoing services using that data.

As I had been exploring this line of argument, the tectonic processes of technology were adding new continents that had to be reflected in the map. In June 1999, Internet file-sharing site Napster turned the industry on its head by allowing users to share music files with each other free of charge across the net. What was most interesting from the technical point of view was that Napster, and soon other file-sharing networks like FreeNet and Gnutella (and a bit later, BitTorrent), didn’t keep all the files in one place like existing online music sites. Instead they stored them on the hard drives of millions of users across the Internet. Andy Oram, one of the editors at my publishing company, made the point to me that the architectural implications of these programs were more important than their business implications. (This is a history that has repeated itself fifteen years later with bitcoin and the blockchain.)

This was a kind of decentralization beyond even the World Wide Web. It was becoming clear that the future demanded even more extreme rethinking of what the Internet could become as a platform for next-generation software applications and content.

Nor was this future limited to file sharing. The SETI@home project, started in mid-1999, was signing up Internet users to contribute unused computing power on their home PCs to analyze radio telescope signals for signs of extraterrestrial intelligence. Computation, not just files and data, could be split across thousands of computers. And developers were increasingly beginning to understand that the powerful applications of the web could be treated as components callable by other programs—what we now call “web services.” An API was no longer just a way for an operating system vendor like Microsoft to provide developers with access to its system services, but a sort of door that an Internet site could leave open for others to come and securely retrieve data.

Jon Udell, a prescient observer of technology, had actually given a talk on this topic at the first Perl Conference in 1997. He noted that when a website called a back-end database to retrieve information, it encoded the information that it wanted into the URL (the web’s Uniform Resource Locator format), and that this URL could be constructed by a program, essentially turning any website into a program-callable component.

Programmers had been using these kinds of hidden clues to remotely control websites since the early days. “Web spidering,” using a program to visit and copy millions of websites in succession, was an essential part of search engines, for instance, but people were now starting to think about how to generalize the process of calling remote websites to allow more specific functions.

All of this was adding up to a completely new paradigm in computing. The Internet was replacing the personal computer as the platform for a new generation of applications. The World Wide Web was the most powerful face of this platform, but peer-to-peer file sharing, distributed computation, and Internet messaging systems like ICQ demonstrated that an even bigger story was afoot.

So in September 2000, I once again pulled together a set of people who I thought ought to be in the same room to find out what they had in common.

Early the next year, based on the insights drawn from that “peer-to-peer” summit, we launched the O’Reilly Peer-to-Peer and Web Services Conference. By 2002 we had renamed it the O’Reilly Emerging Technology Conference and reframed its theme as “Building the Internet Operating System.”

I still remember the perplexity with which some people responded to my choice of keynotes for the event: One was on Napster and Internet file sharing, one was on distributed computation, and the third was on web services. “What do these things have to do with each other?” people asked. It was clear to me that they were all aspects of the evolution of the Internet into a generalized platform for new kinds of applications.

Remember, putting the right pieces of the puzzle on the table is the first step toward assembling them into a coherent picture.

At that first Peer-to-Peer and Web Services Conference in 2001, Clay Shirky memorably summarized the shift to network computing by telling an apocryphal story about Thomas Watson Sr., the head of IBM during the birth of the mainframe computer. Watson was said to have remarked that he saw no need for more than five computers worldwide. Clay noted, “We now know that Thomas Watson was wrong.” We all laughed as we thought of the hundreds of millions of personal computers that had been sold. But then Clay socked us with the punch line: “He overstated the number by four.”

Clay Shirky was right: For all practical purposes, there is now only one computer. Google is now running on well over a million servers, using services distributed across those servers to deliver instant access to documents and services available from nearly a hundred million other independent web servers—to users running on billions of smartphones and PCs. It is all woven into one seamless whole. John Gage, the chief scientist at Sun Microsystems, had first uttered this prescient insight in 1985 when he coined Sun’s slogan: “The Network is the Computer.”

WEB 2.0

The last piece of the puzzle arrived in 2003, and much as had happened with open source, it was a term, Web 2.0, coined by someone else.

Dale Dougherty, one of my earliest employees, who had played a key role in transforming O’Reilly & Associates (later renamed O’Reilly Media) from a technical writing consulting company into a technology book publishing company in the late 1980s, and whom I’d come to consider a cofounder, had gone on to explore online publishing. He created our first ebook project in 1987, and in trying to develop a platform for ebook publishing that would be open and available to all publishers, had discovered the nascent World Wide Web.

Dale had brought the web to my attention, introducing me to Tim Berners-Lee in the summer of 1992. We quickly became convinced that the web was a truly important technology that we had to cover in our forthcoming book about the Internet, which was just then opening up for commercial use. Ed Krol, the author, didn’t yet know much about the web, so Mike Loukides, his editor at O’Reilly, wrote the chapter and we added it to the book just before its publication in October 1992.

Ed’s book, The Whole Internet User’s Guide & Catalog, went on to sell over a million copies and be named by the New York Public Library as one of the most significant books of the twentieth century. It was that book that introduced the World Wide Web to the world. There were only about 200 websites worldwide when we published it in the fall of 1992. Within a few years, there were millions.

Dale had gone on to create the Global Network Navigator (GNN), O’Reilly’s online magazine about the people and trends behind the web and a catalog of the most interesting websites. It was the first web portal (launched a year before Yahoo!) and the first website to carry advertising. Realizing that the web was growing faster than we could keep up with as a private company, and not wanting to lose control of O’Reilly by taking in venture capital, we instead sold GNN to AOL in 1995, in the first content transaction of what was to become the dot-com boom.

Even after that boom went bust in 2000, while venture capitalists were downcast and the markets swooned, we remained convinced that it was still just the early days of the web. In 2003, at an O’Reilly Media company management retreat, we identified our key strategic goal as “reigniting enthusiasm in the computer industry.” Dale was the one who found out how to do that. While brainstorming with Craig Cline, an executive for a conference company called MediaLive International, which had long wanted to partner with O’Reilly on conferences, Dale had come up with the name Web 2.0 to describe the second coming of the World Wide Web after the bust. I gave Dale the go-ahead to partner with MediaLive on the new event, which we launched a year later as the Web 2.0 Conference, with John Battelle, the author and media entrepreneur, as the host and third partner.

As with my work on open source, we began to build a new map by trying to put projects like Google’s pay-per-click ad model, Wikipedia, file-sharing systems like Napster and BitTorrent, web services, and syndicated content systems like blogging into the same frame. And as had happened with open source, the introduction of the new term caught the zeitgeist perfectly and was quickly embraced. Companies rebranded themselves as “Web 2.0 companies,” distancing themselves from the old “dot-com” moniker whether they actually were doing something new or not. Consultants came out of the woodwork promising to help companies adopt the new paradigm.

By 2005, I realized that I needed to put a bit more substance behind the term, and wrote an essay that summarized everything I’d learned since “The Open Source Paradigm Shift.” The essay was called “What is Web 2.0?” It was that essay, more than anything else, that gave me a reputation as a futurist, because I identified so many of the key trends that came together in this next generation of computing.

I didn’t predict the future. I drew a map of the present that identified the forces shaping the technology and business landscape.

THE INTERNET AS PLATFORM

The first principle of Web 2.0 was that the Internet was replacing Windows as the dominant platform on which the next generation of applications was being built. Today this is so obvious as to make you wonder how anyone could have missed it. Yet Netscape, the great challenger to Microsoft’s dominance in the late 1990s, had failed because they had accepted the rules of the game as it was being played by Microsoft. They were using the old map. The defining company of Web 2.0, Google, was using the new map.

Netscape had also used the term the web as platform, but they had framed it in terms of the old software paradigm: Their flagship product was the web browser, a desktop application, and their strategy was to use their dominance in the browser market to establish a market for high-priced server products. Much as the “horseless carriage” framed the automobile as an extension of the familiar, Netscape promoted a “webtop” to replace the desktop, and planned to populate that webtop with information updates and applets pushed to the webtop by information providers who would purchase Netscape servers. Control over standards for displaying content and applications in the browser would, in theory, give Netscape the kind of market power enjoyed by Microsoft in the PC market.

In the end, both web browsers and web servers turned out to be commodities, and value moved up the stack to services delivered over the true web platform.

Google, by contrast, began its life as a native web application, never sold or packaged, but delivered as a service, with customers paying, directly or indirectly, for the use of that service. None of the trappings of the old software industry were present. No scheduled software releases, just continuous improvement. No licensing or sale, just usage. No porting to different platforms so that customers can run the software on their own equipment, just a massively scalable collection of commodity PCs running open source operating systems plus homegrown applications and utilities that no one outside the company ever gets to see.

“Google’s service is not a server,” I wrote, “though it is delivered by a massive collection of Internet servers—nor a browser—though it is experienced by the user within the browser. Nor does its flagship search service even host the content that it enables users to find. Much like a phone call, which happens not just on the phones at either end of the call, but on the network in between, Google happens in the space between browser and search engine and destination content server, as an enabler or middleman between the user and his or her online experience.”

While both Netscape and Google could be described as software companies, Netscape belonged to the same software world as Lotus, Microsoft, Oracle, SAP, and other companies that got their start in the 1980s software revolution, while Google’s fellows were other Internet applications like eBay, Amazon, Napster, DoubleClick, and Akamai.

As we moved from the Web 2.0 era into the “mobile-social” era and now into the “Internet of Things,” the same principle continues to hold true. Applications live on the Internet itself—in the space between the device and remote servers—not just on the device in the user’s hands. This idea was expressed by another of the principles I laid out in the paper, which I called “Software Above the Level of a Single Device,” using a phrase first introduced by Microsoft open source lead David Stutz in his open letter to the company when he left in 2003.

The implications of this principle continue to unfold. When I first wrote about the idea of software above the level of a single device, I wasn’t just thinking about web applications like Google but also hybrid applications like iTunes, which used three tiers of software—a cloud-based music store, a personal PC-based application, and a handheld device (at the time, the iPod). Today’s applications are even more complex. Consider Uber. The system (it’s hard to call it an “application” anymore) simultaneously spans code running in Uber’s data centers, on GPS satellites and real-time traffic feeds, and apps on the smartphones of hundreds of thousands of drivers and of millions of passengers, in a complex choreography of data and devices.

HARNESSING COLLECTIVE INTELLIGENCE

Another key to what distinguished the web applications that survived the dot-com bust from those that died was that the survivors all, in one way or another, worked to harness the collective intelligence of their users. Google is an aggregator of hundreds of millions of websites built by people all over the world, and uses hidden signals from its own users and from the people who create those websites to rank and organize them. Amazon not only aggregates products from a worldwide network of suppliers, but allows its customers to annotate its product database with reviews and ratings, using the power of the crowd to ferret out the best products.

I’d originally seen this pattern in the way that the Internet turbocharged the global collaboration around open source projects. And as the future continued to unfold, once again, the pattern held true. The iPhone leapt to dominance in the early mobile era not just because of its touch-screen interface and sleek, innovative design but because the App Store enabled a worldwide developer community to add features in the form of apps. Social media platforms like YouTube, Facebook, Twitter, Instagram, and Snapchat all gain their power by aggregating the contributions of billions of users.

When people asked me what came after Web 2.0, I was quick to answer “collective intelligence applications driven by data from sensors rather than from people typing on keyboards.” Sure enough, advances in areas like speech recognition and image recognition, real-time traffic and self-driving cars, all depend on massive amounts of data harvested from sensors on connected devices.

The current race in autonomous vehicles is a race not just to develop new algorithms, but to collect larger and larger amounts of data from human drivers about road conditions, and ever-more-detailed maps of the world created by millions of unwitting contributors. It’s easy to forget that in 2007, when Stanford won the DARPA Grand Challenge for self-driving vehicles, they did so by completing a seven-mile course in seven hours. Yet by 2011, Google had managed more than a million miles on ordinary highways. One of their secret weapons: Google Street View cars, driven by human drivers, using cameras, GPS, and LIDAR (LIght Detection And Ranging) to collect data. As Peter Norvig, a director of research at Google, once said to me, “It is a hard AI problem to pick a traffic light out of a video image. It’s much easier to tell whether it’s green or red when you already know it’s there.” (In the years since Peter said that, the first problem has gotten easier too, but you get the idea.)

Today, companies like Tesla and Uber have a shot at leadership in self-driving cars because they have large fleets of instrumented vehicles, vehicles whose sensors are used not just for the task at hand, but as input to the algorithmic systems of the future. But remember: Those vehicles are driven by humans. The data they capture is the next stage in harnessing the collective intelligence of billions of instrumented humans going about their daily lives.

DATA AS THE NEXT INTEL INSIDE

Contribution of user data for collective intelligence sounds like kumbaya, and in the first years of the new century many of the people celebrating user-contributed sites like Wikipedia or new media networks like blogging saw only the utopian possibilities. I argued that data would turn out to be the key to market dominance for companies like Google and Amazon. As I put it in one talk at the time, “‘Harnessing collective intelligence’ is how the Web 2.0 revolution begins; ‘Data is the Intel Inside’ is how it ends.”

Intel, of course, was the company that, along with Microsoft, had captured a monopoly position in the personal computer market, such that every PC bore the sticker INTEL INSIDE. Intel had done this by becoming the sole source for the processor, the brain of the PC. Microsoft had done it by controlling access to its software operating system.

Open source software and the open communications protocols of the Internet had changed the game for Microsoft and Intel. But my map told me that the game didn’t end there. Per Clayton Christensen’s Law of Conservation of Attractive Profits, I knew that something else was going to become valuable. In a word: data. In particular, I thought that building a critical mass of user-contributed data led to self-reinforcing network effects.

The term network effect generally refers to systems that gain in utility the more people use them. A single telephone is not very useful, but once enough people have them, it is very hard not to join the network. So too, the competition in social networks has been to assemble massive user bases, because the lock-in is not via software but through the number of other people using the same service.

The network effects that I observed in data were more indirect, and had to do with the way that companies were learning to harvest value from the users of their systems. Barnes & Noble had all the same products as Amazon, but Amazon had vastly more user reviews and comments. People came not just for the products but for the intelligence added by other users. So too, in addition to Google’s superior algorithms and commitment to constantly improving the product, Google Search kept getting better because more people were using it, which meant that Google could accumulate more data and therefore learn faster than competitors, keeping them perpetually ahead.

Returning to the question of who will win in self-driving cars, one has to ask not just who will have the best software, but who will have the most data.

In a 2016 conversation with Uber executives, they argued that their hundreds of millions of miles of data collected from driver and passenger apps will give them the edge. However, it’s hard to believe that the data from smartphone apps alone will match the level of detail that Google has been collecting with its specially equipped vehicles. That’s why Uber believes it is so urgent to get self-driving vehicles offered as part of their service, even if these remain crewed by drivers for many years to come. Tesla too has detailed telemetry from every vehicle, and in the case of the second-generation vehicles with self-driving features, that does include detailed camera and radar data. The big question for automakers without this edge is whether the sensors used for accident avoidance or automated parking will be sufficient for them to collect enough data to compete.

A lot depends not just on how much data you have, of course, but how able you are to make sense of it. There Google, Tesla, and Uber have a big edge on traditional auto companies.

THE END OF THE SOFTWARE RELEASE CYCLE

In the PC era, we were accustomed to thinking of software as an artifact. Companies had to start thinking of software as a service. This meant we’d see a whole new approach to software development. While I didn’t develop this idea as fully as the previous three, it was clear even in 2005 that what we now call “iterative, user-centered, data-driven development” would be the new normal. Software built in what we now call “the cloud” is constantly updated.

But it’s not just updated many times faster than PC-era software. Today’s software is developed by watching what users do in real time—with A/B testing of features on subsets of users, measurement of what works and what doesn’t work informing development on an ongoing basis. In this way, the collaborative model of open source software development—“given enough eyeballs, all bugs become shallow”—has been taken to its logical conclusion, and completely divorced from the original licensing model of free and open source software.

In the end, I was able to see the future more clearly because my map was more useful than one based on a battle between proprietary software and free software licensing models. Having the right orientation matters. But even then, it had taken years to explore the landscape sufficiently to fill in all the blank spaces on the map.

THINKING IN VECTORS

We all know that the world is changing, but too often we take refuge in the familiar, and fail to stretch our thinking to look at current trends and ask ourselves, “What happens if this goes on?” We also fail to take into account that some trends are potentially much more powerful than others, developing at a faster rate, or taking things in a radically different direction rather than as a simple continuation of the familiar.

The path I traveled from noticing these trends to predicting the future began with the fact that the free software narrative had left out the software behind the Internet. Putting that observation together with my knowledge of the early history of the PC and the rise of Microsoft, and thinking about the long arc of Internet-enabled collaboration, is an example of what I call “thinking in vectors.”

A vector is defined in mathematics as a quantity that can only be fully described by both a magnitude and a direction. You have to take both into account. Some of the most famous “laws” that have been cited in the computer industry are essentially descriptions of vectors.

Moore’s Law, originally formulated by Intel cofounder Gordon Moore in 1965, noted that the number of transistors on an integrated circuit had roughly doubled every year, and looked to continue for the foreseeable future. In 1975, Moore revised his prediction to predict a doubling of the transistor count every two years. Intel executive David House proposed that the actual performance increase would be closer to a doubling every eighteen months, due to an increase in processor speed as well as the increase in chip density, and it is that version that largely held true for many decades.

One of my favorite popular definitions of Moore’s Law came in a conversation I had with Reid Hoffman, the founder and chairman of LinkedIn, and Senator Sheldon Whitehouse (D-RI) over dinner in San Francisco seven or eight years ago. “We need to start seeing Moore’s Law apply to healthcare,” I said. “What’s Moore’s Law?” the senator asked. “You have to understand, Senator,” Reid interjected, “that in Washington, you assume that every year things cost more and do less. In Silicon Valley, everyone expects our products to cost less every year but do more.”

Whether through Moore’s Law proper, or through related advances, like the speed and density of memory storage, hard disk density, networking interconnections, display pixels per dollar, and many other systematic advances, that broader “Hoffman’s Law,” as I now dub it, that every year technology products cost less and do more, has generally held true for a very long time.

In the case of Hoffman’s Law and some of the other fundamental drivers of progress in the computer industry, the vector is clear. We don’t necessarily know where the next increment will come from, but the line has been drawn through enough data points that there is a reasonable expectation that it will continue.

You must always be alert, though, for an inflection point where the old gives way to something profoundly new. For example, we know that Moore’s Law proper cannot continue forever, because of the physical limits of transistor density. Without some breakthrough like quantum computing, which uses subatomic particles for computation, transistor density is limited by the size of the atom, which we will approach in only a few more generations of Moore’s Law. That being said, as Moore’s Law is slowing down, multi-core processors have provided the industry with a temporary workaround, so even though we hit limits in transistors and clock speed, we are still increasing throughput.

Vectors are not only a productive way to think about well-defined trends like Moore’s Law, but also a way to make sense of virtually everything that changes. The future is the outcome of millions of intersecting vectors, which add up in unexpected ways. The art is to pick out important vectors and weave a net from them in which to catch a view of the future.

At O’Reilly Media, when we first take note of a new trend but don’t have the quantification yet to fully characterize it as a vector, with both a magnitude and a direction, we still begin to plot a line, extending it as each new data point comes in. This needn’t be entirely conscious. Instead, it requires an attitude of receptivity, in which new information is always coming in, in which multiple scenarios, multiple futures, are unfolding, all still possible, but gradually collapsing into the present. Lawrence Wilkinson, one of the cofounders of Global Business Network, the company that pioneered a technique called scenario planning, whom I met in 2005, introduced me to a wonderful phrase that captured how my mind works: “news from the future.”

So, for example, consider how the “Harnessing Collective Intelligence” vector became clear to us:

      1.   In the late 1980s and early 1990s, we were exposed to the “barn raising” style of collaborative software development of the early Unix community—what we later came to call open source software.

      2.   In developing our first books, we practiced a version of this kind of crowdsourcing ourselves. In 1987, I wrote a book called Managing UUCP and Usenet, which described how to use a program called the Unix-to-Unix Copy Program (UUCP) to connect to Usenet, a distributed dial-up precursor to today’s social web. It was on Usenet that the world’s software developers conversed about their work, shared tips and advice, and, increasingly, talked about everything from sex to politics. At first the book was based on my own experience connecting systems to Usenet, but that experience was limited. Readers sent me information about how to use additional equipment that I didn’t have access to and the fine points of geekery (“Here’s the ‘chat script’ for calling in through a Develcon switch,” or “Here are the pins you need to connect in an RS-232 cable” for some particular brand of modem.)

                We reprinted the book every six months or so, and every time we reprinted, it grew by another thirty or forty pages, almost entirely composed of contributions from readers. Over its first three years, it went from about 80 pages to over 200. You might say it was an early, printed-on-paper wiki.

                In 1992, trying to create a print book that emulated the link style of the World Wide Web, I designed and coauthored a book called Unix Power Tools, which wove together tips and tricks harvested from hundreds of Internet contributors into a hyperlinked web of short articles, each of which could be read independently because it also contained links to additional articles providing tutorial and background information that my coauthors Jerry Peek and Mike Loukides and I felt was needed to make sense of the crowdsourced lessons.

      3.   In 1992 and 1993, as we turned “the Whole Internet Catalog” into GNN, the Global Network Navigator, every day we sought out the best of the new sites joining the World Wide Web, curating them into a rich catalog of experiences created, as if by magic, by a distributed network of people pursuing their own passions.

      4.   We watched the early search engines, starting with Web-crawler in 1994, automatically collect links not just to the best websites, but to every website. And in 1998, when Google launched, with far better results, it became clear that they had found hidden intelligence in web links. A link wasn’t just a pointer to a page that might previously have gone unnoticed by the crawler, it was a way of discovering new content on the web. The number of links was also a vote about the value of that site. And the site making the link also had links pointing to it; the nature and quality of those links could tell the search engine something about the value of the page making the connection. How long had that site been on the net? How many people pointed to it? How valuable did people find the links that it made? Not only that, but there was further human intent signaled by the “anchor text”—the words in the source document that hyperlinked to another one. Google found a gold mine of data, and never looked back.

                I still remember a blog post by Robert Scoble in which he gleefully demonstrated how human contribution was central to search engines. “I just discovered a new restaurant in Seattle. Its website isn’t in Google. But it will be tomorrow, because I just linked to it!”

      5.   In 1995, we saw how eBay and Craigslist brought crowdsourcing to products and services, and began to realize that the magical aggregation of millions of people into new kinds of services wasn’t limited to “content,” but could also be used in the physical world.

      6.   We watched how Amazon ran rings around Barnes & Noble and Borders in online bookselling by applying the same principles that Google used to make a better search engine to more effective e-commerce. While Barnes & Noble followed its retail store practices of letting publishers buy search placement, so that a search for, say, a computer book on JavaScript or Perl would turn up the book of whichever publisher had paid them the most to feature it, Amazon used multiple signals to choose the book that had the most “relevance,” as defined by a mix of sales, positive reviews, inbound links from “Associates,” and other factors based on collective intelligence. We were always pleased to find our books at the top of Amazon searches, because we knew that that meant they were seen as the best by tens of thousands of readers.

As a result of all these prior data points, in 2004, when I sought to define “Web 2.0,” and thought about what distinguished the companies that had survived the dot-com bust from those that had failed, it was clear that all of the survivors were, in one way or another, harnessing the power of their users to create their product.

And in 2009, when I wrote “Web Squared: Web 2.0 Five Years On,” it was straightforward to see what was coming next. “The smartphone revolution has moved the Web from our desks to our pockets,” I wrote. “Collective intelligence applications are no longer being driven solely by humans typing on keyboards but, increasingly, by sensors. Our phones and cameras are being turned into eyes and ears for applications; motion and location sensors tell where we are, what we’re looking at, and how fast we’re moving. Data is being collected, presented, and acted upon in real time. The scale of participation has increased by orders of magnitude.

“The Web is no longer a collection of static pages of HTML that describe something in the world,” I continued. “Increasingly, the Web is the world—everything and everyone in the world casts an ‘information shadow,’ an aura of data which, when captured and processed intelligently, offers extraordinary opportunity and mind bending implications.”

What’s important to note, though, is that even when you’ve spotted a vector, it doesn’t mean that you understand all of its implications. Yes, I was able to identify in 2009 that sensors would be the key to the next generation of applications, but that didn’t lead me to “predict” Google’s breakthrough with self-driving cars, or that Uber was about to realize the potential of the sensors in the phone to revolutionize on-demand transportation.

I also often didn’t take the time to act on my own insights. Technology journalist John Dvorak once reminded me that very early in the history of the web, I had confidently predicted to him that there would be a market for buying and selling domain names. They would be extremely valuable. Yet I never bothered to go out and buy any myself.

Once you’ve identified a trend, though, it’s easier to recognize early which new developments are important, because they are the next step in the continued acceleration along the vector, as entrepreneurs and inventors continue, in Wallace Stevens’s magnificent phrase, to “search a possible for its possibleness.” In other words, the news from the future mindset helps you to pay attention to the right things, and learn from them.

TWITTER MAKES IT REAL FOR NONPROGRAMMERS

The notion of the Internet as an operating system for the next generation of applications had taken me a long way. By 2010, the idea had taken hold in the industry. Developers were routinely writing applications that relied on data from Internet services—about location, search results, social networks, music, products, and so much more. Startups were no longer building local applications in their own data centers but rather in what was now called the cloud. I didn’t need to keep preaching that gospel.

And frankly, I was ready to move on. As T. S. Eliot so memorably put it:

 . . . one has only learnt to get the better of words

For the thing one no longer has to say, or the way in which

One is no longer disposed to say it. And so each venture

Is a new beginning, a raid on the inarticulate

I was tired of talking about Web 2.0. And there was more to what was happening than just a cloud-based platform for computer applications. Social media was showing how the Internet connects people on a global scale, and I began to see the power of a different metaphor. A metaphor too is a kind of map; it may be all you have when you are first encountering a new territory shrouded in mist.

Increasingly, I’d been watching a kind of Cambrian explosion in applications for collective intelligence that were qualitatively different from those of the desktop web. Smartphones had put a camera in everyone’s hand, and Twitter had created a real-time platform from which those photos and text updates could be instantly disseminated to the world. Billions of connected humans and devices were being woven into a global brain. That brain was all of us, augmented and connected.

Twitter was an especially fertile ground for reinvention. Three features we now take for granted were all created by users and only later adopted by the platform. The @ symbol to reply to another user first appeared in November 2006; it was formally adopted by the platform in May 2007, turning Twitter into a place for conversations as well as status updates. The first “retweet” of someone else’s tweet happened in April 2007, though it wasn’t formally adopted as a feature until 2009.

In August 2007, Chris Messina proposed the use of the # symbol as a way to label events or groups of tweets on Twitter. It became clear just how powerful an amplifier this was of collective knowledge and sentiment during the San Diego wildfires a few months later. Before long, hashtags, as they came to be called, were everywhere. Many of them didn’t stick, but if enough people adopted one, it became the real world equivalent of Obi-Wan Kenobi’s words in Star Wars: “I felt a great disturbance in the Force . . . as if millions of voices suddenly cried out.”

And the voices cried out: #iranelection #haitiearthquake #occupywallstreet.

Beginning in July 2009, Twitter responded to the outside-in innovation and began hyperlinking hashtags, so users could search on them. The app had already begun showing “trending topics” (using algorithms to detect common events even if they do not have the same hashtag), but hashtags added fuel to the fire.

When photos were added to Twitter (again by an outside developer providing features that the platform developer itself hadn’t imagined), Twitter’s power to reveal the real-time pulse of the world increased even further. On January 15, 2009, four minutes after Captain “Sully” Sullenberger ditched US Airways Flight 1549 in the Hudson after multiple bird strikes had disabled the engines, Jim Hanrahan posted the first tweet. Janis Krums snapped an iPhone photo of passengers standing on the wing of the downed plane a few minutes later and shared it on Twitter via a third-party app called TwitPic, and it went worldwide long before the story appeared on the television news.

Facebook also began to have an effect on global affairs. In 2010, an Egyptian Google employee named Wael Ghonim created a Facebook page called “We Are All Khaled Said,” commemorating a young Egyptian who’d been tortured to death by police. The page became a focus for activism that led to antigovernment protests culminating in the revolution of January 25, 2011.

Wikipedia too had become a fulcrum for real-time collective intelligence about the world. After the 2011 Tohuku earthquake and tsunami in Japan, which led to the meltdown of the Fukushima nuclear plant, I had watched in awe as the Wikipedia page grew from a single line in fractured, misspelled English to a full-featured encyclopedia entry. The first entry appeared a mere thirty-two minutes after the earthquake, before the tsunami had struck. Over a short period, hundreds, then thousands of contributors made more than 5,000 edits, creating a comprehensive and authoritative account of the disaster. I still show animations of the transformation in some of my talks. It is a WTF? moment for anyone who sees it.

The debates behind the scenes on Wikipedia “talk” pages, about controversial elements on the published pages, are also eye-opening. In Reinventing Discovery, his wonderful book about lessons from the consumer Internet for the practice of science, Michael Nielsen writes, “Wikipedia is not an encyclopedia. It is a virtual city, a city whose main export to the world is its encyclopedia articles, but with an internal life of its own.”

In response to the speed of blogging and social media, Google sped up its web crawl, and Google’s search results too became increasingly real-time. This led to a qualitative difference in how quickly information was transmitted, and magnified its impact. Now news, ideas, and images propagate across the global brain in seconds rather than weeks or months.

In one sense, this is nothing new. There has always been a global brain. As recounted by Jeff Bezos in a 2005 talk at my Emerging Technology Conference, computer scientist Danny Hillis once said to him that “global consciousness is that thing that decided that decaffeinated coffeepots should be orange.” The idea that “orange means decaffeinated” originated during World War II, when Sanka promoted its decaffeinated coffee brand by giving away orange-rimmed coffeepots to restaurants across America. The idea took hold—not universally, to be sure, but sufficiently that the pattern propagates. At some point, it no longer belonged to Sanka but to the world.

The association of “orange” with “decaffeinated” is an example of what Richard Dawkins called a “meme”—a self-replicating idea. Today people often think of memes as images and slogans shared on social media, but any great idea that takes hold is a meme. In 1880, “Darwin’s Bulldog” Thomas Henry Huxley wrote, “The struggle for existence holds as much in the intellectual as in the physical world. A theory is a species of thinking, and its right to exist is coextensive with its power of resisting extinction by its rivals.”

Knowledge spread from mind to mind even before the advent of writing. But the printed word made it possible for ideas and news to reach people in distant lands, first at the speed of walking, then riding, and eventually of steamships and rail. The first electronic transmissions by telephone and telegraph cut a delay of weeks or months to minutes. With radio and television, transmission became almost instantaneous, but creation and vetting of what was to be transmitted was still slow, done in offices and boardrooms, because the channels for dissemination of instantaneous media were so limited. The Internet, and in particular the combination of the Internet and the smartphone, changed all that. Anyone could share anything at any time; others could pick it up and pass it on even more quickly.

It isn’t just ideas and sensations (news of current events) that spread across the network. We talk of information as “going viral,” but there are malicious programs designed to do exactly that, to reproduce themselves whether we wish it or not. But perhaps more important than hostile viruses are those with which we willingly cooperate.

In his magnificent history of the origins of modern computing, Turing’s Cathedral, George Dyson notes that some of the earliest thinkers about digital computing realized that the spread of “codes”—that is, programs—from computer to computer is akin to the spread of viruses, and perhaps of more complex living organisms, that take over a host and put its machinery to work reproducing that program.

“Numerical organisms were replicated, nourished, and rewarded according to their ability to go out and do things: they performed arithmetic, processed words, designed nuclear weapons, and accounted for money in all its forms. They made their creators fabulously wealthy,” Dyson writes. “They . . . then influenced the computational atmosphere as pervasively as the oxygen released by early microbes influenced the subsequent course of life. They coalesced into operating systems amounting to millions of lines of code—allowing us to more efficiently operate computers while allowing computers to more efficiently operate us. They learned how to divide into packets, traverse the network, correct any errors suffered along the way and reassemble themselves at the other end. By representing music, imagery, voice, knowledge, friendship, status, money, and sex—the things people value most—they secured unlimited resources, forming complex metazoan organisms running on a multitude of individual processors the way a genome runs on a multitude of cells.”

When people join the web, or download a new mobile app, they reproduce its code onto their local machine; they interact with the program, and it changes their behavior. This is true of all programs, but in the network age there are a set of programs whose explicit goal is to get their users to share them more widely. Thus the global brain is actively building new capacity.

The kinds of “thoughts” that a global brain has are different from those of an individual, or of a less connected society. At their best, these thoughts allow for coordinated memory on a scale never seen before, and sometimes even for unforeseen ingenuity and new forms of cooperation; at their worst, they allow for the adoption of misinformation as truth, for corrosive attacks on the fabric of society as one portion of the network seeks advantage at the expense of others (think of spam and fraud, or of the behavior of financial markets in recent decades, or of the rash of fake news sites during the 2016 US presidential election).

But perhaps the most riveting thing to realize is that, bit by bit, the global brain is getting a body. It has eyes and ears (billions of connected cameras and microphones), a sense of position and motion (GPS and motion sensors) that is far more precise and powerful than that of humans, and with specialized sensors, data-gathering capabilities that far outstrip our own.

Now it starts to move. Self-driving cars are a manifestation of the global brain; their memory is the memory of roads traveled under the tutelage of human drivers but recorded with their uncanny senses. But not unsurprisingly, the most powerful manifestation of the global brain’s ability to touch the physical world relies not on robots but on the power of networked applications to direct human activity.

There is usually a paradigmatic company or group of companies that best exemplifies the next wave of technology. “Unpacking” the lessons of that company can help you draw your map of the future.

From 1998 to 2005, I’d built my map of the future by thinking about what we could learn from Amazon and Google. Today two of the companies that teach us the most about the trends shaping the future are Uber and its rival Lyft.

Many readers may bristle at the notion that Uber is a positive model for the technology-driven economy of the future. After all, the company has been embroiled in controversy almost from the beginning. Critics question whether it truly provides economic opportunity for drivers or traps them with deceptive promises of income that it can’t deliver. Cities fume at its brazen confrontations with regulators and use of technology to deflect their investigations. Rivals sue over claims of stolen technology. Former employees make accusations of a toxic workplace culture that tolerates sexual harassment.

It is easy to forget that many of the people who invent the future do so by crashing through barriers, crushing competitors, and dominating a new industry by force of will as well as intellect. Sometimes dirty tricks come into play. Thomas Edison and John D. Rockefeller, Bill Gates and Larry Ellison, were all justifiably reviled at various points in their careers. When I began my work in computing, Microsoft was routinely referred to as “the Evil Empire.”

Whatever you may think of Uber, it is hard to deny its impact on the economy. If we want to understand the future, we have to understand Uber. Like it or not, it is the poster child for many of the ways that technology is changing the world of work.

Lyft, Uber’s smaller rival, is a more idealistic, worker-friendly company that, in practice, has the same business model. Each of the companies has introduced key innovations that were copied by the other. In many ways, they are co-inventing the future of urban transportation. We will consider them together throughout the book.