The Benefits of Open Source and the Risks of Open Core

Chapter 1. What Is Open Source and Why Is It Popular?

Effectively, open source is a model for software development that allows everyone to use software without restrictions, adapt it to their needs, and share their changes with other people. The concept of open source also involves a constellation of practices:

• Communities of programmers and users, often spanning the globe, who collaborate to improve the software

• Transparent communications, fully documented in open archives

• Software licenses that guarantee the rights to use, change, and share the software

• Public, inclusive review processes

• Living repositories that preserve the entire history of a software project’s development, including discussions about choices made

Strictly speaking, most of the items in this list are not needed to call a project open source. Although modern open source projects focus on processes—building community, adhering to healthy governance practices, achieving stable funding, etc.—technically, any time a developer releases code under an open license, it’s open source. The Open Source Initiative (OSI), a nonprofit set up by leaders throughout the worldwide open source community, has approved many such licenses. Most of those licenses also pass muster with another key organization that was one of the earliest and most influential leaders in this movement, the Free Software Foundation (FSF).

Some open source licenses are short and simple, whereas others are longer and strive to prevent many possible problems that have been seen over the decades. But most don’t come close to the complexity of the licenses that normally accompany closed software, which often incorporate surprising and cumbersome restrictions on users.

Still, the previous list of practices applies to most important open source projects today. We will see in the course of this report that the first two items in the list, communities and communications, are central to the success of open source and tend to be problematic in open core.

How Open Source Changed the Software Industry

Open source historically reflects an intentional return to an earlier ethos in the technology industry. During the initial rise of software, most practitioners were either scientists or hobbyists who freely shared information and innovation as the norm. Academics, engineers in the field, and software hobbyists took for granted the exchange of source code through floppy disks, magnetic tape, and eventually electronic networks. These programmers realized that it took the combined work of all practitioners to debug and co-invent their ideas.

By the time the concept of open source arrived, however, sharing and collaboration in computing had been overshadowed by the tremendous commercial value of software, engendering fierce competition over prices and features. Software and hardware manufacturers had discovered the power of vendor lock-in, a clever and ever-evolving set of practices that make it difficult for customers to switch vendors. Examples of vendor lock-in include physical keys (dongles) or encryption that make data available only to the vendor’s program, opaque file formats that may change arbitrarily in new versions of the product and features deliberately designed to differ slightly from the features on competing products.

The vendors still strived for actual customer value, but it suffered in the wake of artificial monopolies created by vendor lock-in. Customers frequently had to wait months to years for bug fixes, even critical security patches, because the vendor would choose to invest its limited programmer staff in other areas such as new features. It was a common practice for companies to make small changes to file formats in order to wring more money from the installed base, by forcing paid upgrades by customers who merely wanted to retain access to previously created documents.

At the same time, as the World Wide Web was driving online commerce, internet servers relied mostly on a proprietary implementation of Unix running on bespoke server hardware; this solution was expensive and a real barrier to new companies getting off the ground. Something had to give.

In 1999, the groundbreaking Netscape web browser had become an open source project called Mozilla, whose best-known product currently is the popular Firefox browser. This release set the stage for open source to disrupt the status quo in the tech industry. Investments from leading companies such as IBM, Hewlett-Packard, and Sun Microsystems helped launch many of the foundational open source project communities, starting with a historic IBM announcement in 2000 that they planned to spend $1 billion on GNU/Linux. Once a powerhouse like IBM was willing to throw their brand behind Linux, other businesses could take Linux seriously.

At the beginning of the open source movement, most established vendors were notably skeptical and removed. They were afraid that the obligations of strict licenses, such as the General Public License (GPL) from the GNU Project, would legally impel them to publish trade secrets from their proprietary code if mixed too closely with open source code. They thought the ultimate success of open source to be unlikely.

For instance, in 1999, the legal department at a major hardware vendor in the server space at that time, Sun Microsystems, published strict guidelines around shipping open source modules. They had to be on a separate DVD from proprietary product executables, with a separate installation to be completed by the end user (not automatically included as part of any one-click installer for setting up a product). Open source components were not to be tightly coupled in Sun’s proprietary products (e.g., the software had to work acceptably without any open source add-ons, and such add-ons could cover only nonessential functionality). And above all, Sun would not initiate any project under the GPL family of licenses.

Within less than two years, Sun would release the code for Open­Office.org under the Lesser GPL (LGPL) because part of the strategy for that project was to entice GNU/Linux desktop distributions to bundle it, and because it needed to work well inside the GNOME desktop interface, which used the GPL license. But although this policy shift happened fairly quickly for a strategic reason, Sun was not so quick to make similar shifts for their flagship software products: the Java programming language and Solaris operating system. It took a full decade from the first proposals to actual open source releases of both products, because even in a progressive culture, change has a very long tail.

Another tipping point in the maturation of open source in the enterprise came when the stranglehold of proprietary Unix, networking, and web technologies broke up under a combination of open source technologies known as the LAMP stack. LAMP includes Linux as the operating system, Apache as the web server, MySQL as the database, and Perl or PHP as the programming language for backend servers.

The financial services sector here was the vanguard for a customer revolt. The CTO at Morgan Stanley realized that Linux could be used to drive a farm of relatively cheap commercial x86 servers to provide acceptable compute power that could drive their online business at a fraction of the cost of bespoke hardware and software. This bold move came just as the first dot-com bubble burst in 2000, and companies became desperate overnight to survive by trimming costs. Within a year, all the major fintech companies were switching.

That same LAMP stack fueled the rise of a new type of company, one that offered services on the web. Ecommerce was one such service, but there were many others. Searching the web itself became a service, with Yahoo!, AltaVista, and later Google offering very high quality, zero-cost search engines through a web interface. The “product” that actually drove the search engine companies’ success was the capture, analysis, and sale of the data generated by those searches, with market trends and intelligence available for a fee. Experimental business models like this couldn’t be brought to life without increasingly cheap hardware and open source software without license fees.

Open source became a democratizing force in the tech industry, enabling new companies to thrive by driving down costs while enabling offerings of better services. Even very proprietary companies such as Apple started using open source. Indeed, an open source project was the basis for the pioneering version 10 of Mac OS (now macOS), driven by the return of Steve Jobs to Apple in the late 1990s. Apple took a version of the historic Berkeley Software Distribution (BSD) and released their core software under the name Darwin. Apple was also choosing to release some of their own products’ source code (notably WebKit, an important layout engine used in many web browsers) under open source licenses, because doing that could drive adoption and set up de facto standards.

Sun Microsystems was by now using open source to push their Java agenda. Sun donated the source code for their Java Servlet API to the Apache Software Foundation to create the Apache Tomcat Project, driving mass adoption of Java over Microsoft’s competing Active Server Pages technology.

Where Is Open Source Dominant?

If the average computer user—not a computer programmer or dedicated free software user—runs down the list of installed applications, they will probably not see many open source applications. Perhaps the user has Firefox, a leading web browser mentioned in the previous section. The user may also have found it useful to install LibreOffice (descendant of the Open­Office.org suite mentioned earlier) for authoring documents, slide presentations, and spreadsheets, GIMP for photo editing, or Thunderbird to manage email. Most layout engines—the software that is central to web browsers—are open source, the most important of these being the previously mentioned WebKit as well as Gecko and Blink.

Applications do not tell the real story of open source and free software. If a user could dig down into their operating system, and further into the networking infrastructure underlying the web, they would discover a world of open source. If they decided to try out professional activities such as creating their own software or managing a collection of servers, they would establish an even stronger bond with open source.

In short, open source programmers are particularly good at creating tools for their own use. Proprietary companies charge a lot of money for such tools, and over time it has become clear that the users of these tools—that is, programmers, professional computer or network administrators, AI developers, etc.—do a better job of meeting their own needs. While some companies still offer proprietary solutions in this area, a growing number simply offer open source tools packaged up for convenient use.

Most open source software is hidden from an end user who is creating a spreadsheet or doing ecommerce, but open source is pervasive and makes the end-user activities possible.

Governments have also jumped on the open source bandwagon. In the early 2000s, many governments adopted laws or regulations favoring the use of open source software in government, where feasible. Notable examples include the United States, the United Kingdom, and Peru, although in general the follow-through has not matched the enthusiasm of the initial public flourish.

The triumph of open source can be traced in the evolution of the world’s most famous software company, Microsoft. As late as the ’00s, Microsoft managers were calling open source a “cancer” and a danger to the software industry. But Microsoft is now one of the top contributors to the Linux kernel and collaborates with open source communities to create tools that work with Microsoft products.

Aren’t Web Services Taking Over?

Web services, which you use every time you go online to visit a retailer, bank, or professional service such as SAP or Salesforce, are almost always proprietary. The topic of web services, also sometimes called software as a service (SaaS), lies beyond the scope of this report, but we’ll say a bit about it because of its widespread use.

Most web service companies want to fix bugs and add features quickly. They use a variety of processes, such as Agile programming, DevOps, and continuous integration (CI), to allow the quick deployment of new versions of their applications. In other words, when you log in to Facebook (or some other popular service) this morning, you’re probably using a different version from when you logged in last night.

This pace is incompatible with the deliberative, collaborative process used by open source communities. A company that adopts such continuous change will quickly lose touch with the open source version of the software and is really using a totally different, proprietary product. Thus, web services are not generally relevant to this report. But a few popular web services—such as the Drupal and WordPress content management systems and the Instaclustr cloud service—provide their services as open source software and work well with communities to update it.

Other reasons for SaaS companies to maintain a firm grip on their software will be explored in Chapter 2.

We’re not saying any particular practice is wrong. Continuous updates, handled responsibly, can fix security problems and improve the visitor experience in websites. The CI process can save a lot of time and effort on open source as well as proprietary projects.

Finally, many companies that offer proprietary web services also contribute a lot of developer time and money to open source software, which the companies recognize is superior for their infrastructure. Later chapters will look at this practice.

Why Users Ask for Open Source

The impressive strength of the open source movement, and the popularity of its products, can lead one to ask what makes it so appealing. The benefits of the open source and free software models, listed in this section, help to explain why companies try to ride the powerful wave of open source. People with considerable experience in using and procuring software, having seen the negative impact of proprietary software, tend to look for free and open source alternatives for the reasons cited here.

Avoiding Lock-In

This is almost certainly the most compelling reason users adopt open source. Anyone who has used software for a few years has encountered horror stories of lock-in: companies that precipitously raise prices, remove product features that users consider crucial, go out of business without providing an upgrade path to the companies’ abandoned customer bases, etc. An open source project depends on contributions, but all users know that the product will be there and will be open to all users so long as enough of them care enough about the product to maintain it.

If a company bases its product on an open standard, the clients might be able to find alternatives (proprietary or open source) without having to change their code or behavior. However, few proprietary products are based entirely on open standards—and if the vendor claims to follow a standard such as SQL, clients will still end up having trouble porting their move code because (as the old joke goes) SQL offers “so many standards to choose from.”

Accelerated Ramp-Up

With proprietary software, businesses have to contact the vendor, negotiate licenses, and manage the software’s installation and payments. All these steps are eliminated or streamlined with open source. A few individuals can try out the software as long as they need, without the end of a “free trial” looming over them. They can then spread the software quickly as far throughout the organization as its managers want.

Staff Availability

This advantage of open source is often underestimated by its advocates but is acutely noticed by adopters. Learning open source software is cheaper and easier than studying proprietary software (which might force learners to take courses that are expensive and geographically remote). In an industry grievously short of staff, open source makes it easier to find programmers and administrators who know the product, an advantage particularly strong among the most popular open source projects. Job-takers don’t have to be trained in an unknown proprietary offering but can work productively from day one with the familiar open source tools.

Freedom from Licensing Tangles

Proprietary software is usually licensed on some scale. The simplest cost model is “per seat,” but modern corporate environments make the calculations more calculated, and SaaS adds even more variables. Such fuss is totally out of place for modern container-based services that scale up and down over a period of minutes. It is also inappropriate for interpreted languages such as Python: a company can’t license a compiler on a “per seat” basis because there is no compiler, but no one would want to pay a fee just to run a Python program.

Proprietary licenses often require each employee (or an administrator acting on the employee’s behalf) to go through cumbersome registration processes that involve typing in long random numbers. In any case, the counting and monitoring required for proprietary licenses is intrusive.

Adaptability

If you encounter a bug in proprietary software, you have to ask the vendor to fix it, and they will do it on their schedule. They may have other priorities. They may require you to upgrade to a more expensive or complex version of the product to get the bug fix.

With open source, you can hire someone to fix the bug and contribute it back to the core project so all can benefit.

Having a Say

Bug-fixing is one obvious advantage of open source, but it only scratches the surface of open source’s benefits to involved users. Instead of being at the mercy of a business plan cooked up by a vendor, users in healthy open source projects set the agendas and roadmaps.

If the bulk of the community chooses to go in a different direction from your needs, you can create your own branch of a project and implement the changes you want. Usually, there is a way to accommodate multiple pieces of functionality through modules and options for building and configuring the software. If you need to create your own version (known as forking), you can do that.

Compatibility

Open source developers are used to making their tools work with others. Commercial vendors also want to be compatible with outside tools but are more selective about what they support or how much compatibility they offer. If you use an open source tool and you need it to work with a hardware device, operating system, database or other component that’s not yet supported, you can go ahead and add that support.

The reasons cited in this section should demonstrate why so many people insist on using open source for their software needs.

Sustainability and Health in Open Source Projects

To compare open source with open core constructively, it’s important to understand the traits and practices that sustain a robust open source project. Leaders of the open source movement have long been concerned that choosing an appropriate license isn’t enough to ensure a robust open source project. Some important properties of open source development aren’t mentioned in the OSI or FSF definitions, leaving wiggle room for interpretation and intentional or unintentional abuse.

For example, open source projects work best when barriers to entry and open collaboration are removed so that any practitioner can rise through the ranks, collaborate with anyone else, gain reputation, and evolve to participate at any level. Several high-profile, corporate-backed projects have failed because they never managed to achieve this leveling, preferring their own employees’ contributions over those of nonemployees.

Overt and implicit exclusionary practices make it hard for women and marginalized communities to participate in open source, just as in other parts of society. During the past decade, most open source projects that develop communities institute codes of conduct to enforce fair and respectful behavior. To realistically attract communities that haven’t historically participated much, truly committed projects foster educational projects and other forms of outreach to make the pool of available contributors grow.

Although Apple announced interest in fostering a community around the development of their open source macOS core, Darwin, they failed to follow through with the practices that companies use successfully to create a community.

A similar problem ruined the prospects for VistA, a highly regarded electronic health record software from the US Department of Veterans Affairs. They “threw their software over the fence” from time to time, but arrived too late at attempts to create a community, which would have been critical to render the software useful outside of its original setting in the VA. Several open source teams and companies tried to pull together useful distributions but failed to work together and never managed to achieve critical mass.

In addition to the attributes just described, what are the other considerations that lead to a healthy open source project? Red Hat, a leading provider of GNU/Linux distributions and open cloud solutions, cites these things to consider for a healthy open source project:

Project life cycle

This measure examines things such as how mature the project is and whether it’s attracting new contributors. You can decide from this research how you can best participate and whether the project will last.

Target audience

A project should state how it expects to be used, and by whom. From this information, you can judge whether the project is meeting the right needs and whether it’s right for you.

Governance

A term with many definitions, governance in this context describes how decisions are being made and how contributors are managed and organized.

Project leaders

You should be able to identify easily who the current leaders are—and how to contact them. A healthy project has leaders who know its history and uphold its culture and vision.

Release manager and process

You’re seeking here a process for releases that is well thought out, well documented, and managed competently.

Infrastructure

We’ve mentioned that the coordination of a community is supported by its tools. Numerous tools support version control, issue requests, builds and releases, communications, and so on. Determine whether the tools on the project are meeting its needs.

Goals and roadmap

These attributes deal with the long-term direction of the project. The project should define goals and a roadmap and communicate them clearly.

Onboarding processes

There are many ways to contribute to a project: coding, advocacy, fundraising, documentation, and more. New contributors in all these areas should be supported by documentation and mentors.

Internal communication

People shouldn’t be going off in all directions doing whatever is right in their own eyes. See whether they are communicating, whether decisions are being preserved, and whether communication channels are adequate.

Outreach

Outreach tries to ensure that users, funders, and others with a potential interest in a project know what it’s doing. This task can be hard for many software projects, which can easily become ingrown and focus only on people currently showing interest.

Awareness

How well do potential users and stakeholders know the project? This attribute is related both to outreach and to target audience.

Ecosystem

As we’ve explained, open source projects work with many others. The project should maintain good relations with the projects that it uses or serves and should be a responsible member of this ecosystem.

These guidelines have been compiled from decades of experience at Red Hat and other participants working with open source projects. The guidelines are very helpful to an organization looking to commit to a specific project, whether that organization is new to the open source space or longtime allies of open source. We will later refer to some of these guidelines when looking at open core.