The Strange Case of the Disappearing Open Source Vendors

"Is there any point to which you would wish to draw my attention?"
"To the curious incident of the dog in the night-time."
"The dog did nothing in the night-time."
"That was the curious incident," remarked Sherlock Holmes."

--Sir Arthur Conan Doyle, Silver Blaze, 1894

Each year, in preparation for the Open Source Convention, I try to wrap my head around the current state of open source. What's the big story?

Back in 1998 when the term "open source" was coined, the story was simple: "Just because there's no vendor behind software, it doesn't mean it isn't important. The Internet runs on open source software (BIND, Sendmail, Apache, Perl), so any enterprise that depends on the Internet also depends on open source." As in the Sherlock Holmes story, Silver Blaze, the "curious incident" was the absence of the expected. The press, the analysts, and the computer industry as a whole ignored open source software because no one was paying for it. An industry whose measurements were focused on vendor market share had no way to come to grips with a market segment in which software could be adopted by end users without money changing hands.

But like the misdirection that often characterizes mystery stories, the first and most obvious conclusion may lead the investigation astray. Many people rushed to the conclusion that the answer was to monetize this market segment! Over the next few years, open source boomed along with the Internet, and we saw a huge influx of venture capital (especially into Linux) and several multibillion-dollar IPOs.

In the summer of 2002, though, the story is much more sober. The dot-com boom has ended, the VCs and the stock market are in retreat, and of all the much-hyped open source companies, only a few are left. Red Hat is still flourishing, but VA Linux Systems has taken "Linux" out of its name; Caldera, SuSe, Turbolinux, and Connectiva are joining forces; Eazel, Great Bridge, and Lutris are out of business, among many others. (Note: In addition to Red Hat, there are some quiet successes: Zope, ActiveState, and CollabNet all have continued solid growth. Disclaimer ((or should it be bragging rights?)): O'Reilly is an investor in all three.)

At the same time that many open source vendors have stumbled, though, the acceptance of open source has gone through the roof. And it is here that we can see the true significance of the Sherlock Holmes story with which I began this piece: the dog did nothing because it was the owner of the dog who came in during the night.

Open source is ultimately about empowering users, not vendors. Eric Raymond made this point convincingly in the third of his essays on open source, The Magic Cauldron (now collected as The Cathedral and the Bazaar), pointing out that far more software is written for use than for sale. In a section entitled "The Manufacturing Delusion," he notes:

"We need to begin by noticing that computer programs, like all other kinds of tools or capital goods, have two distinct kinds of economic value. They have use value and sale value.

The use value of a program is its economic value as a tool, a productivity multiplier. The sale value of a program is its value as a salable commodity.

When most people try to reason about software-production economics, they...have a strong tendency to assume that software has the value characteristics of a typical manufactured good. [This assumption is] demonstrably false.

Scanning the employment section of your local newspaper is an enlightening experiment that I urge the reader to perform for him- or herself. Examine the jobs listings under programming, data processing, and software engineering for positions that involve the development of software. Categorize each such job according to whether the software is being developed for use or for sale.

It will quickly become clear that, even given the most inclusive definition of "for sale", at least 19 in 20 of the salaries offered are being funded strictly by use value (that is, value as an intermediate good)."

It is the users of Apache and Perl and Sendmail, from dot-com sites like Yahoo, Amazon, and E*Trade, to ISPs and other Internet infrastructure providers such as UUnet, Rackspace, and Akamai, to thousands of small Web design firms and other service businesses, not to mention large end users like Wal-Mart*, that are making (or saving) money with open source.

(*Yes, I know that Wal-Mart shows up on Netcraft as running Microsoft IIS, but curiously, the operating system is Linux. So, it appears to be a case of the fairly common Apache hack, in which the Apache source is modified to output IIS as the server string. Mike Prettejohn of Netcraft assures me that the method used to find out the underlying operating system is less susceptible to modification in this way than the Web server signature.)

Bob Young, the former chairman of Red Hat, once remarked that what Red Hat actually sells is control, the power of end users to extend, modify, and service their own software, rather than depending on the whims of a vendor. He repeated this idea in a recent interview with ZDNet UK:

"...for the first time the customer was given control over the technology that you're asking them to invest in.... The beauty of open source software is that when you run into a bug you can get it fixed. You get source code and you get a licence that allows you to modify it. It's like buying a car with a hood that you can open, as opposed to the traditional model in the software industry where the hood is locked shut. If you can open the hood it means you can fix your car, but it also means you have access to 10,000 car repair shops across the U.K. Whereas if the hood's locked shut, and if your vendor denies that it's a bug, which is known to happen, you're completely stuck."

In many of its recent attacks, Microsoft has argued that open source is bad for business, but you have to ask, "Whose business? Theirs, or yours?" The answer to that question is very different if you're an end user rather than a software vendor.

If you look at the sessions at the upcoming O'Reilly Open Source Convention in San Diego, you won't see lots of software-vendor product pitches. Instead, you'll see open source being put to work by its owners. Lincoln Stein, Ewan Birney, and Jim Kent are deciphering the human genome. Robert Spier of Walt Disney Feature Animation and Milton Ngan of Weta Digital (the animation company that worked on the movie version of Lord of the Rings) are using Perl and Linux to make better movies. Jason Asbahr is using Python to develop the logic for games on the Sony Playstation and the Nintendo GameCube. Jeremy Zawodney of Yahoo talks about MySQL Optimization not because Yahoo sells MySQL but because it uses it to deliver its service. And of course, it's not just the speakers but the attendees at the conference who are using open source to improve productivity, cut costs, and drive innovation in the products and services that they do sell.

Some of the industries and well-known companies represented by attendees at the conference include aerospace (Boeing, Lockheed Martin, General Dynamics, Raytheon, NASA); computers and semiconductors (Agilent, Apple, Fujitsu, HP, Intel, IBM, Philips, Intuit Macromedia, SAIC, Sun, Texas Instruments, Veritas); telecom (ATT Wireless, Nokia, Qualcomm, Verizon Wireless); finance, insurance, and accounting (Barclays Global Investors, Morgan Stanley, Federal Reserve Bank, PriceWaterhouseCoopers, Prudential); media (AOL Time Warner, BBC, Disney, LexisNexis, Reuters, USA Today, Yahoo); and pharmaceuticals (GlaxoSmithKline, McKesson, Merck, Novartis, Pfizer).

(If this list (derived from a few minutes looking over the conference program and attendee list) isn't enough to convince you how widely open source is used in business, Chris DiBona, Jeff Bates, Karim Lakhani, and Robert G. Wolf will be doing a session on Open Source Demographics: Who They Are and Why They Do It. And Paul Pangaro and Elaine Coleman of Sun Microsystems talk about the changing relationship between business and developers.)

It may turn out that the observation that first triggered the wave of open source activism ("Just because there's no vendor behind software, it doesn't mean it isn't important") remains the touchstone of the movement. The "strange case of the disappearing open source vendors" doesn't reflect the weakness of open source but its strength. And the fact that open source may reduce the revenues of some software vendors does not mean that it reduces economic activity or economic success, but instead that it correctly allocates the profits to the developers of that software, its users.

Open Source in Government

And then there's the biggest IT end user of them all: government. Governments around the world are increasingly showing an interest in the cost savings and increased control by end users that open source provides.

At the conference, Todd Ogasawara tells how shifting budget priorities pushed the state of Hawaii to move away from commercial software solutions to Linux, Apache, MySQL, Zope, Python, and PHP for its Intranet portal. Lisa Nyman and Rachael LaPorte Taylor explain how the FedStats.gov site was built with Perl, Apache, and MySQL. Kim Brand and Matthew Hartmann talk about open source opportunities in schools. (Their work is with a small group of Catholic schools in Indianapolis, but as Matthew Szulik, CEO of Red Hat, points out: "The average technology expenditure per student is $105 per year--money that should be stretched as far as possible." It's in the public interest for public schools to use publicly available software whenever possible.)

And of course, as with the commercial sector, conference attendees span a wide range of national, state, and local government agencies, from the Department of Defense, the Census Bureau, and the EPA all the way down to Medina County in Ohio.

Ever prescient about marketplace threats, Microsoft was the first to realize that government was perhaps the key battleground between open source and proprietary software. It began its public campaign against Linux and open source in early 2001 with much publicized remarks by Jim Allchin ("Open source is an intellectual-property destroyer") and Steve Ballmer ("Linux is a cancer"). These were not attacks on deep-pocketed Linux backers such as IBM and Hewlett-Packard, but an attempt to undermine support for Linux by U.S. government agencies. The focus on Linux in government has continued and intensified in recent months, with salvos against the Pentagon and the government of Peru.

Not content with direct attacks, the Microsoft-funded Alexis de Tocqueville Institution released a "study" claiming that open source might undermine our national security:

In a paper to be released next week, the Alexis de Tocqueville Institution outlines how open source might facilitate efforts to disrupt or sabotage electronic commerce, air-traffic control, or even sensitive surveillance systems.

Unlike proprietary software, open source software does not make the underlying code of a software confidential.

"Computer systems are the backbone of U.S. national security", says Fossedal, chairman of the Alexis de Tocqueville Institution and its Committee for the Common Defense, which will release the study. "Before the Pentagon and other federal agencies make uninformed decisions to alter the very foundation of computer security, they should study the potential consequences carefully."

This is FUD of the highest order, because, of course, it is Microsoft's software, not open source software, that has been the focus of security breakdowns so common as to become routine. Most security experts believe that open source software is likely to be more secure than proprietary software, because it doesn't rely on obscurity for security, but on effective, secure systems design.

The de Tocqueville paper consists of little more than unsupported assertions about the danger of open source, and has been refuted at length elsewhere. What is germane to my argument here, of course, is that Microsoft has correctly identified government as one of the key axes of open source adoption, and the fierceness of their attacks indicate just how concerned they are about it.

The willingness to make scurrilous accusations ("open source might facilitate efforts to disrupt or sabotage electronic commerce, air-traffic control or even sensitive surveillance systems") is symptomatic of the disregard for the truth afflicting corporate America these days. The willingness to harness misinformation as a tool of corporate strategy springs from the same "me first at all costs" mentality that led us to the Enron debacle. Just as Enron thought it was appropriate business practice to manipulate the California energy markets to raise its profits, Microsoft seeks to influence public policy to raise the costs of software and prohibit government support for a low-cost alternative.

The national security threat promised by the de Tocqueville press release turned out to be nothing more than a few inflammatory paragraphs buried in a much larger document repeating and expanding on Microsoft's attacks on the "business-unfriendly" nature of the GPL.

The chief argument appears to be that code covered by the GPL can't be combined with proprietary code because doing so requires the resulting product also to be released under the GPL. Of course, this argument trades on ambiguity. It implies that, as Steve Ballmer said last year in his "Linux is a cancer" interview, GPL'd software "attaches itself in an intellectual property sense to everything it touches", when in fact, the "viral" effect of the GPL applies only if the code from a GPL'd program is actually incorporated into another program. And of course, Microsoft code can't be incorporated into other products either, under any terms at all (or at least without paying substantial sums to Microsoft), while GPL'd code is useful for any public or private sector projects that care to play by its rules.

As I've long made clear, I prefer BSD- or Apache-style licenses, or dual-licensing schemes such as the SleepyCat license, because I believe they allow for a richer ecology of open source and proprietary commercial development. Of course, this preference is within the context of support for all licenses, from the most proprietary to the most free, since my primary belief is in the right of developers to set the terms for the reuse of their code in an way that best serves their objectives. And that includes Microsoft's right to set the terms under which they license their own software--but only so long as the user is free to accept or reject those terms in a fair marketplace.

While it may not seem so from this article, I'm actually a big fan of Microsoft. They had a vision early on of cheap, ubiquitous personal computing, and the fact that computers are so widespread today is in part a tribute to their singleminded pursuit of that vision. And now, with .NET, they have a new vision, of an Internet-scale operating system, which is driving the industry forward into uncharted territory. Despite the many knocks on them from the open source community, they are an innovative company. But they are also an arrogant company, and one that has fallen into the trap of assuming that they somehow have a right to profit levels that they've enjoyed in the past, and that any method to maintain that profitability (even when growth slows) is legitimate.

I part ways with Microsoft when they seek to restrict the choice of their users, whether by unfairly using their monopoly position to extinguish competitors, or by undermining industry standards with non-interoperable extensions, or by locking in users, or by seeking to influence public policy to shut out open source software.

The vehemence of Microsoft's attacks on the GPL have led me to believe that GPL advocates may be right that it is the public's best defense against companies that want to take from the commons without giving anything back, or worse, block the commons so that people will have to buy their products. As a result, I've invited Richard Stallman to keynote at this year's Open Source Convention. Richard's uncompromising vision that software should belong to its users so that they can repair and extend it as needed is a refreshing corrective to the excesses of an industry that pits itself against its customers in order to extract maximum profits from them.

The concept that the GPL is "business unfriendly" is based entirely on the concept that only software vendors make money with software. If in fact, the great majority of software is developed by users of the software, and not by vendors, then open source is very good for business, and a policy that encourages lower-cost alternatives to high-cost proprietary software is a good public policy. It is no accident that inside Microsoft, decisions that lead to software lock-in are referred to as "the strategy tax." This tax is imposed not just on Microsoft's own developers, but on corporations and government institutions.

Customer lock-in is the real enemy of business, not the GPL.

In a recent piece on the difficulty of extracting his mail and contacts from Outlook 2000, Dale Dougherty wrote a paragraph that ought to be studied hard by decisionmakers at Microsoft:

Nike is running a series of bold, new commercials featuring Tiger Woods, who says his contract with Nike doesn't require him to use its equipment unless he finds it to be the best in the market. He says with amusement that it puts the pressure on Nike to be the best or else. If Microsoft is the best at what it does, then it shouldn't have to resort to this kind of lock-in of its contract with users. Let us choose the best.

This is the very same point made by Peruvian Congressman Edgar Villanueva Nuñez, in his letter to Microsoft responding to their protest against his bill to have the Peruvian government mandate use of free software:

"To guarantee the free access of citizens to public information, it is indispensable that the encoding of data is not tied to a single provider. The use of standard and open formats gives a guarantee of this free access, if necessary through the creation of compatible free software.

To guarantee the permanence of public data, it is necessary that the usability and maintenance of the software does not depend on the goodwill of the suppliers, or on the monopoly conditions imposed by them. For this reason the State needs systems the development of which can be guaranteed due to the availability of the source code."

My concern to air the issues around open source in government was triggered not just by the de Tocqueville piece, but because I've heard tales that intense Microsoft lobbying is making some open source advocates in government keep their heads down. This pressure has hit several of our planned speakers at the Open Source Convention. Below are excerpts from two recent mail messages I received from one of the conference planners. Names and other identifying information are deleted to protect the privacy of the individuals in question, who already appear to be in trouble:

I have been talking with name deleted. Long story short, other speaker name deleted called this morning and said he can't do the talk. I got the impression he was in deep doo doo with his bosses. He asked me to jerk him off the web site quickly, along with any reference in the description to name of government agency deleted. He said the talk can in no way appear to represent agency.

and

name deleted is the speaker for session title deleted I called him earlier in the week to ask how things were going. He said that he just ... walked into a beehive. Said he was thrown into [his boss's] office and told that he may not speak about anything having to do with name of agency deleted. So, he said he can speak on a different topic if you like, or cancel. He is planning to come to the conference regardless of the outcome and will pay if he must.

Now, I can't say that these two about-faces from speakers who were originally eager to proclaim their agency's use of open source have any relationship to the fact that Microsoft is pouring money into lobbying against open source as anti-competitive, anti-American, and a potential aid to terrorism. But the recent FUD sure can't help open source advocates in government. (It's also true that government officials are under strict regulations about what they can endorse, in order to keep a level playing field.) Nonetheless, it seems to me that there's a real need to understand what's going on. Government procurement and standards-setting systems are byzantine and impenetrable, so it's hard to follow the money or find out who makes decisions about software, and on what they base those decisions.

I'd like to see more serious debate about public policy regarding software. Like most members of the industry, I believe that less government intervention is better, but once the government does get involved, and companies with full time lobbyists (like Microsoft) are working the issues, it behooves us all to make sure that our voices are heard and the issues thoroughly explored.

If you have information (either positive or negative) about the use of open source in government, or about the pressures that are placed on its advocates, please post the information in the talkback at the end of this article, or if you need to keep some or all of the details private, send me mail (tim at oreilly.com).

And of course, government agencies that use open source software to reduce costs or serve their customers, as well as companies, universities, research labs, and individual users, should stick the course, choosing the software that best meets their needs, rejecting software that unfairly limits their choice or seeks to extract too high a price for the value it delivers, and continuing to share with others, in a cooperative marketplace, tools that can make the pie bigger for everyone.

Copyright © 2009 O'Reilly Media, Inc.