Chapter 1. Introduction
Why are we really excited about introducing you to Erlang? What do we feel is really special about the language? Its lightweight concurrency model with massive process scalability independent of the underlying operating system is second to none. With its approach that avoids shared data, Erlang is the perfect fit for multicore processors, in effect solving many of the synchronization problems and bottlenecks that arise with many conventional programming languages. Its declarative nature makes Erlang programs short and compact, and its built-in features make it ideal for fault-tolerant, soft real-time systems. Erlang also comes with very strong integration capabilities, so Erlang systems can be seamlessly incorporated into larger systems. This means that gradually bringing Erlang into a system and displacing less-capable conventional languages is not at all unusual.
Although Erlang might have been around for some time, the language itself, the virtual machine, and its libraries have been keeping pace with the rapidly changing requirements of the software industry. They are constantly being improved by a competent, enthusiastic, and dedicated team, aided by computer science researchers from universities around the world.
This introduction gives a high-level overview of the characteristics and features that have made Erlang so successful, providing insight into the context in which the language was designed, and how this influenced its current shape. Using case studies from commercial, research, and open source projects, we talk about how Erlang is used for real, comparing it with other languages and highlighting its strengths. We conclude by explaining the approaches that have worked best for us when running Erlang projects.
Why Should I Use Erlang?
What makes Erlang the best choice for your project? It depends on what you are looking to build. If you are looking into writing a number-crunching application, a graphics-intensive system, or client software running on a mobile handset, then sorry, you bought the wrong book. But if your target system is a high-level, concurrent, robust, soft real-time system that will scale in line with demand, make full use of multicore processors, and integrate with components written in other languages, Erlang should be your choice. As Tim Bray, director of Web Technologies at Sun Microsystems, expressed in his keynote at OSCON in July 2008:
If somebody came to me and wanted to pay me a lot of money to build a large scale message handling system that really had to be up all the time, could never afford to go down for years at a time, I would unhesitatingly choose Erlang to build it in.
Many companies are using Erlang in their production systems:
Amazon uses Erlang to implement SimpleDB, providing database services as a part of the Amazon Elastic Compute Cloud (EC2).
Yahoo! uses it in its social bookmarking service, Delicious, which has more than 5 million users and 150 million bookmarked URLs.
Facebook uses Erlang to power the backend of its chat service, handling more than 100 million active users.
Motorola is using Erlang in call processing products in the public-safety industry.
Ericsson uses Erlang in its support nodes, used in GPRS and 3G mobile networks worldwide.
The most popular open source Erlang applications include the following:
The 3D subdivision modeler Wings 3D, used to model and texture polygon meshes.
The Ejabberd system, which provides an Extensible Messaging and Presence Protocol (XMPP) based instant messaging (IM) application server.
The CouchDB “schema-less” document-oriented database, providing scalability across multicore and multiserver clusters.
The MochiWeb library that provides support for building lightweight HTTP servers. It is used to power services such as MochiBot and MochiAds, which serve dynamically generated content to millions of viewers daily.
RabbitMQ, an AMQP messaging protocol implementation. AMQP is an emerging standard for high-performance enterprise messaging.
Although Uppsala University has for many years led the way with research on Erlang through the High Performance Erlang Project (HiPE), many other universities around the world are not far behind. They include the University of Kent in the United Kingdom and Eötvös Loránd University in Hungary, which are both working on refactoring tools. The Universidad Politécnica de Madrid of Spain together with Chalmers University of Technology and the IT University (both in Sweden) are working on Erlang property-based testing tools that are changing the way people verify Erlang programs.
With these companies, open source projects, and universities, we have just scratched the surface of what has today become a vibrant international community spread across six continents. Blogs, user groups, mailing lists, and dedicated sites are now helping to take the community to its next level.
The suitability of Erlang for server-side software has its roots in the history of the language, as it was originally developed to solve problems in a subset of this particular space, namely the telecom sector, and so it’s worth looking back to the invention of Erlang in the 1980s.
The History of Erlang
In the mid-1980s, Ericsson’s Computer Science Laboratory was given the task of investigating programming languages suitable for programming the next generation of telecom products. Joe Armstrong, Robert Virding, and Mike Williams—under the supervision of Bjarne Däcker—spent two years prototyping telecom applications with all of the available programming languages of the time. Their conclusion was that although many of the languages had interesting and relevant features, no single language encompassed them all. As a result, they decided to invent their own. Erlang was influenced by functional languages such as ML and Miranda, concurrent languages such as ADA, Modula, and Chill, as well as the Prolog logic programming language. The software upgrade properties of Smalltalk played a role, as did the Ericsson proprietary languages EriPascal and PLEX.
With a Prolog-based Erlang virtual machine (VM), the lab spent four years prototyping telecom applications with an evolving language that through trial and error became the Erlang we know today. In 1991, Mike Williams wrote the first C-based virtual machine, and a year later, the first commercial project with a small team of developers was launched. The project was a mobility server, allowing DECT cordless phone users to roam across private office networks. The product was successfully launched in 1994, providing valuable feedback on improvements and missing features that got integrated into the 1995 Erlang release.
Only then was the language deemed mature enough to use in major projects with hundreds of developers, including Ericsson’s broadband, GPRS, and ATM switching solutions. In conjunction with these projects, the OTP framework was developed and released in 1996. OTP provides a framework to structure Erlang systems, offering robustness and fault tolerance together with a set of tools and libraries.
The history of Erlang is important in understanding its philosophy. Although many languages were developed before finding their niche, Erlang was developed to solve the “time-to-market” requirements of distributed, fault-tolerant, massively concurrent, soft real-time systems. The fact that web services, retail and commercial banking, computer telephony, messaging systems, and enterprise integration, to mention but a few, happen to share the same requirements as telecom systems explains why Erlang is gaining headway in these sectors.
Ericsson made the decision to release Erlang as open source in December 1998 using the EPL license, a derivative of the Mozilla Public License. This was done with no budget or press releases, nor with the help of the corporate marketing department. In January 1999, the erlang.org site had about 36,000 page impressions. Ten years later, this number had risen to 2.8 million. This rise is a reflection of an ever-growing community resulting from a combination of successful commercial, research, and open source projects, viral marketing, blogging, and books, all driven by the need to solve hard software problems in the domain for which Erlang had originally been created.
Erlang’s Characteristics
Although Erlang on its own is an attractive programming language, its real strength becomes apparent when you put it together with the virtual machine (VM) and the OTP middleware and libraries. Each of them contributes to making software development in Erlang special. So, what are the features of Erlang that differentiate it from many of its peers?
High-Level Constructs
Erlang is a declarative language. Declarative languages work on the principle of trying to describe what should be computed, rather than saying how this value is calculated. A function definition—particularly one that uses pattern matching to select among different cases, and to extract components from complex data structures—will read like a set of equations:
area({square, Side}) -> Side * Side ;
area({circle, Radius}) -> math:pi() * Radius * Radius.This definition takes a shape—here a square or a circle—and depending on which kind of shape it receives, it matches the correct function clause and returns the corresponding area.
In Erlang, you can pattern-match not only on high-level data but also on bit sequences, allowing a startlingly high-level description of protocol manipulation functions. Here is the start of a function to decode TCP segments:
decode(<< SourcePort:16, DestinationPort:16,
SequenceNumber:32,
AckNumber:32,
DataOffset:4, _Reserved:4, Flags:8, WindowSize:16,
Checksum:16, UrgentPointer:16,
Payload/binary>>) when DataOffset>4 ...In the preceding code, each numeric length, such as 4 in DataOffset:4, gives the number of
bits to be matched to that variable. By comparison,
think of how you would achieve the same effect in C or Java.
Another aspect of Erlang is that functions (or closures) are first-class data. They can be bound to a variable and can be treated just like any other data item: stored in a list, returned by a function, or communicated between processes.
List comprehensions, also taken from the functional programming paradigm, combine list generators and filters, returning a list containing the elements of the list generators after the filters have been applied. The following example of list comprehensions, which we explain fully in Chapter 9, is an implementation of the quicksort algorithm in a couple of lines of code:
qsort([]) -> [];
qsort([X|Xs]) ->
qsort([Y || Y<-Xs, Y =< X]) ++ [X] ++ qsort([Y || Y<-Xs, Y > X]).Concurrent Processes and Message Passing
Concurrency in Erlang is fundamental to its success. Rather than providing threads that share memory, each Erlang process executes in its own memory space and owns its own heap and stack. Processes can’t interfere with each other inadvertently, as is all too easy in threading models, leading to deadlocks and other horrors.
Processes communicate with each other via message passing, where the message can be any Erlang data value at all. Message passing is asynchronous, so once a message is sent, the process can continue processing. Messages are retrieved from the process mailbox selectively, so it is not necessary to process messages in the order they are received. This makes the concurrency more robust, particularly when processes are distributed across different computers and the order in which messages are received will depend on ambient network conditions. Figure 1-1 shows an example, where an “area server” process calculates areas of shapes for a client, as we did earlier in High-Level Constructs.
Scalable, Safe, and Efficient Concurrency
Erlang concurrency is fast and scalable. Its processes are lightweight in that the Erlang virtual machine does not create an OS thread for every created process. They are created, scheduled, and handled in the VM, independent of the underlying operating system. As a result, process creation time is of the order of microseconds and independent of the number of concurrently existing processes. Compare this with Java and C#, where for every process an underlying OS thread is created: you will get some very competitive comparisons, with Erlang greatly outperforming both languages.
Erlang processes communicate with each other through message passing. Regardless of the number of concurrent processes in your system, exchanging messages within the system takes microseconds. All that is involved in message passing is the copying of data from the memory space of one process to the memory space of the other, all within the same virtual machine. This differs from Java and C#, which work with shared memory, semaphores, and OS threads. Even here, benchmarks show that Erlang manages to outperform these languages for the same reasons it outperforms them in process creation times.
You might think that comparing Erlang to C# and Java is unfair to these two languages, as we are comparing apples and oranges. Well, you are right. Our point is that if you want to build massively concurrent systems, you should be using the tool that is best for the job, regardless of the underlying concurrency mechanism. As a result, the concurrency model of an Erlang program would differ from that of languages where process creation and message passing times are not as small. We describe the Erlang way of dealing with concurrency in Chapters 4 through 6, and Chapter 12.
Soft Real-Time Properties
Even though Erlang is a high-level language, you can use it for tasks with soft real-time constraints. Storage management in Erlang is automated, with garbage collection implemented on a per-process basis. This gives system response times on the order of milliseconds even in the presence of garbage-collected memory. Because of this, Erlang can handle high loads with no degradation in throughput, even during sustained peaks.
Robustness
How do you build a robust system? Although Erlang might not solve all your problems, it will greatly facilitate your task at a fraction of the effort of other programming languages. Thanks to a set of simple but powerful error-handling mechanisms and exception monitoring constructs, very general library modules have been built, with robustness designed into their core. By programming for the correct case and letting these libraries handle the errors, not only are programs shorter and easier to understand, but they will usually contain fewer bugs.
The libraries are collectively known as the OTP middleware. What exception monitoring and error-handling mechanisms do they contain, and what libraries are built on top of them?
Erlang processes can be linked together so that if one crashes, the other will be informed, and then can either handle the crash or choose to crash itself.
OTP provides a number of generic behaviors, such as servers, finite state machines, and event handlers. These worker processes have built-in robustness, since they handle all the general (and therefore difficult) concurrent parts of these patterns; all the user needs to do is to program the specific behavior of the particular server, which is much more straightforward to program than the general behavior.
These generic behaviors are linked to a supervisor behavior whose only task is to monitor and handle process termination. OTP puts the idea of links into a framework whereby a process supervises other workers and supervisors, and may itself be supervised by yet another process, all in a hierarchical structure. Figure 1-2 illustrates a typical supervision tree.
Using this supervision and linking, Erlang programmers can concentrate on programming for the correct case, and can let the process fail in any other circumstances. This avoidance of defensive programming makes a programmer’s task much easier, as well as making it more straightforward to understand how a program behaves.
Although in this book we concentrate on Erlang and its error-handling mechanisms and exception monitoring properties, we also provide an introduction to the OTP design patterns in Chapter 12.
Distributed Computation
Erlang has distribution incorporated into the language’s syntax and semantics, allowing systems to be built with location transparency in mind. The default distribution mode is based on TCP/IP, allowing a node (or Erlang runtime system) on a heterogeneous network to connect to any other node running on any operating system. As long as these nodes are connected through a TCP/IP network and the firewall has been correctly configured, the result is a fully meshed network of nodes, where all the nodes can communicate with each other.
As Erlang clusters were designed to execute behind firewalls, security is based on secret cookies with very few restrictions on access rights. You have the ability to create more disparate networks of distributed Erlang nodes using gateways, and if necessary, make them communicate using secure Internet protocols such as SSL.
Erlang programs consist of processes that communicate via message passing. When you start programming in Erlang, these will all be on one node, but as the syntax of sending a message within the node is the same as sending it to a remote node, you can easily distribute your processes across a cluster of computers. With distribution built into the language, operations such as clustering, load balancing, the addition of hardware and nodes, communication, and reliability come with very little overhead and correspondingly little code.
Integration and Openness
You want to use the right tool for the right job. Erlang is an open language allowing you to integrate legacy code or new code where programming languages other than Erlang are more suitable for the job. As a result, there are mechanisms for interworking with C, Java, Ruby, and other programming languages, including Python, Perl, and Lisp.
High-level libraries allow Erlang nodes to communicate with nodes executing Java or C, making them appear and behave like distributed Erlang nodes. Other external languages can be tied in more tightly using drivers that are linked into the Erlang runtime system itself, as a device driver would be, and sockets can also be used for communication between Erlang nodes and systems written in other languages using popular protocols such as HTTP, SNMP, and IIOP.
The fact that distribution is built into Erlang means that integrating it with other systems is more natural than in other languages. The facilities for handling network data formats are an important part of the language and its libraries, rather than a bolted-on afterthought. The tracing and logging facilities also give you a clear picture of how the integration is working, enabling you to debug and tune systems much more effectively.
Erlang and Multicore
The shift to multicore is inevitable. Parallelizing legacy C and Java code is very hard, and debugging parallelized C and Java is even harder...but what alternative is there?
The Erlang model for concurrency—separate processes with no shared memory communicating via message passing—naturally transfers to multicore processors in a way that is largely transparent to the programmer, so that you can run your Erlang programs on more powerful hardware without having to redesign them.
Symmetric multiprocessing (SMP) support in Erlang was first developed experimentally in the late 1990s, and is now an integral part of the standard release. The ethos of the Erlang/OTP development team at Ericsson is to make SMP work, measure its performance, find the bottlenecks, and optimize. Since releasing the first SMP-enabled version of Erlang, this has been their approach. Over recent releases, the virtual machine model has evolved from a single monolithic run queue—possibly with processes running on different processors—to a run queue for each processor, ensuring that the run queue is no longer a bottleneck for the system, as illustrated in Figure 1-3. As more complex processors emerge, the runtime system will be able to evolve with them.
The goal with Erlang’s SMP is to hide the problems and awareness of SMP from the programmer. Programmers should develop and structure their code as they have always done, optimally using concurrency and without having to worry about the underlying operating system and hardware. As a result, Erlang programs should run perfectly well on any system, regardless of the number of cores or processors.
Case Studies
Let’s start looking at how the features we just described have contributed to some of Erlang’s successes. Ericsson’s first major Erlang product was the AXD301 ATM switch; more recently, Erlang has been the key to implementing the CouchDB schema-free, document-oriented database. Finally, we report on a Motorola-based research project comparing the productivity of Erlang and C++ head on.
The AXD301 ATM Switch
The AXD301, a telephony-class 10–160 Gbps ATM switch, was designed and implemented from scratch in less than three years. At the heart of the AXD301 are more than 1.5 million lines of Erlang code, handling all the complex control logic, and overseeing operations and maintenance. This integrates with about half a million lines of C/C++ implementing low-level protocol and device drivers, much of it coming from third-party sources.
This ATM switch has been installed in networks all over the world, but the installation that shot to prominence was used by British Telecom to build what was at the time the largest “Voice over ATM” backbone in the world. According to an Ericsson press release issued at the end of the trial period, “Since cut-over of the first nodes in BT’s network in January 2002 only one minor fault has occurred, resulting in 99.9999999% availability.” The director of Ericsson’s Next Generation Systems program, Bernt Nilsson, confirmed that “the network performance has been so reliable that there is almost a risk that our field engineers do not learn maintenance skills.”
Experiences with the AXD301 suggest that “five nines” availability, downtime for software upgrades included, is a more realistic assessment. For nonstop operations, you need multiple computers, redundant power supplies, multiple network interfaces and reliable networks, cooling systems that never fail, and cables that system administrators cannot trip over, not to mention engineers who are well practiced in their maintenance skills. Considering that this target has been achieved at a fraction of the effort that would have been needed in a conventional programming language, it is still something to be very proud of.
How did Erlang contribute to the success of the AXD301? It supports incremental development, with the absence of side effects, making it easier to add or modify single components. Support for robustness and concurrency is built into the language and available from the start.
Erlang was very popular with the programming teams that found they were building much more compact code, thus dramatically improving their productivity. Experience from the project, although not scientifically documented, suggests that the Erlang code was 4 to 10 times shorter than similar systems written in C/C++, Java, and PLEX,[1] while the fault rate per thousand lines of code was the same.
Ericsson has gone on to use Erlang on other projects across the company, including a SIP telephony stack, control software for wireless base stations, telephony gateway controllers, media gateways, broadband solutions, and in GPRS and 3G data transmission. And these are just a few of the many we are allowed to talk about.
CouchDB
When Damien Katz decided to implement CouchDB, he wanted to be the one developing “cool stuff.” He wanted to see whether he was good enough to develop something from scratch, pushing the code base to new levels. CouchDB is an open source database that provides a schema-less replicated document store, storing objects in JSON format and accessed through a RESTful interface.
He wrote the first version of CouchDB in C++. His system consisted of three components: a storage engine, a view engine, and a query language. The complexity of his components increased, and when he started hitting concurrency issues, he felt like he had hit a wall. He stumbled upon Erlang, downloaded it, and quickly came to the realization that it would solve his problems.
From the world Damien was coming from, Erlang initially sounded very complicated, and he believed it would be hard to learn. But when he got down to the details, what instead struck him was the simplicity of the language. Getting something to work with Erlang took extra effort compared to Java, as there were fewer tools and IDEs available, but to get something working reliably ended up taking much less talent and time than any of the other languages he knew.
Erlang gave Damien the features he needed for CouchDB at a fraction of the effort of using conventional languages. When migrating CouchDB to Erlang, he focused on the concurrency aspects and integrating it with his existing C++ components. He ended up replacing the entire C++ code base, as Erlang had all of the qualities he was looking for in a database application. They included support for intensive I/O, high reliability, and facilities for dealing with failure gracefully. The first benchmarks on the code, even before it was profiled, allowed in excess of 20,000 simultaneous connections. This compared pretty favorably with the 500 he expected to get on the C++ version!
Once it was released, CouchDB started getting lots of attention in the open source community. Damien made the decision to release the code under the Apache license, giving him the freedom he needed to continue development. Today, CouchDB is one of the best known Erlang open source applications currently being used in production systems worldwide.
What happened to Damien? He got a job with IBM, allowing him to continue developing CouchDB as an open source project. In Damien’s words, now he is indeed “the guy who gets paid to work on cool stuff.” You can read out more about CouchDB at http://www.couchdb.org.
Comparing Erlang to C++
Most experienced Erlang programmers will confirm that the Erlang programs they have written are substantially shorter than their counterparts in other mainstream programming languages used by the industry. Indeed, this was an urban legend among Ericsson programmers long before Erlang was released as open source. But until recently, there was very little scientific evidence to back up these claims. Quicksort (Chapter 9) using list comprehensions or remote procedure call server examples (Chapter 11), both of which we cover in this book, were used to argue the case. When comparing programming languages, however, you must benchmark whole systems in the application domain for which those languages were designed, not code snippets or simple functions.
Heriot-Watt University in the United Kingdom received an EPSRC[2] grant to study the impact of distributed functional programming languages in the telecom sector. When we first heard about this grant, our reaction was, why not speak with Ericsson and get it over with? We quickly changed our minds when we realized the research project was being done in cooperation with Motorola Labs, one of Ericsson’s competitors. Although Heriot-Watt might have taken Ericsson’s word that Erlang was suitable for programming telecom applications, Motorola wasn’t having any of it.
The focus of the study consisted of two C++-based systems referred to as the Data Mobility (DM) component and the Dispatch Call Controller (DCC). These systems handled digital communication streams for pocket radio systems as used by emergency services. The DM been written with fault tolerance and reliability in mind. The implementation was done by good C++ programmers who based their development work on proprietary Motorola libraries. The DCC was an internal research prototype intended for evaluating the use of C++ and CORBA to gain scalability.
The Erlang rewrites were implemented by Jan Henry Nyström, an experienced Erlang programmer with an academic background. Two Erlang rewrites were done of the DM and only one of the DCC. The first DM implementation interfaced with Motorola’s libraries, and the second was a pure Erlang implementation. The DCC was a pure Erlang implementation. Comparisons were made of the performance, robustness, productivity, and impact of the programming language constructs.
The interesting conclusions of this research came with the pure Erlang implementations. In the DM, there was an 85% reduction in code. This was explained by the fact that 27% of the C++ code consisted of defensive programming, 11% of memory management, and 23% of high-level communication, all features which in Erlang are part of the semantics of the language or are implemented in the OTP libraries. The DCC’s code base was more in line with the folklore and urban legends, namely that it was about 70% smaller than its C++ counterpart.
The Erlang DM resulted in a 100% performance increase when compared to the C++ version, which crashed when severely overloaded. Although the throughput might sound surprising at first, it was a result of Erlang and its lightweight concurrency model being the right tool for the task. The mobility application in question had lots of concurrency, short messages, and little in terms of heavy processing and number crunching. The C++ implementation was never implemented to handle the loads it was subjected to, so as a result, the conclusion was that these load results might not be relevant and are certainly unfair to the C++ implementation. They do, however, demonstrate an important property of Erlang-based systems, which are stable under heavy loads and recover automatically when the load drops.
Although Erlang pioneers argued their case of shorter and more compact code based on experience, this study has finally provided empirical data to support the claims. A full report that fully confirms the “Erlang advantage” is available in “High-level distribution for the rapid production of robust telecoms software: comparing C++ and Erlang.”[3]
How Should I Use Erlang?
The philosophy used to develop Erlang fits equally well with the development of Erlang-based systems. Quoting Mike Williams, one of the three inventors of Erlang:
Find the right methods—Design by Prototyping.
It is not good enough to have ideas, you must also be able to implement them and know they work.
Make mistakes on a small scale, not in a production project.
In line with these quotes, all successful Erlang projects should start with a prototype, and Erlang has all the support you need to get a prototype up and running quickly. Working prototypes usually cover a subset of the functionality and allow end-to-end tests of the system. If, for example, you were building an IM server (a recurring theme throughout this book), valid functionality to test could include the ability to sign on and send messages to a remote server without worrying about issues such as redundancy, persistency, and security.
Software development in Erlang is best achieved using an agile approach, incrementally delivering systems of increasing functionality over a short cycle period. Teams should be small in size, and, where possible, tests should be automated. The tools available with Erlang, discussed in the body of this book and in the section Integration and Openness, give excellent software development support. Testing is aided by EUnit for unit testing, and Common Test for system testing. Other tools include cover, providing coverage analysis, and Dialyzer, a static analysis tool that identifies software discrepancies such as type errors, dead code, and unsafe code.
If you’re going to introduce Erlang to your organization, it can be a good strategy to start small, bringing in Erlang for a small project (or subsystem) where you can play to Erlang’s strengths. This works particularly well for Erlang because it has distribution and integration designed in from the start, as we described in the section Integration and Openness, and virtually all production Erlang systems interwork with other languages and systems. Once you have achieved success on a small scale, you can start to think bigger!
The website for this book and the Appendix A contain links about where you can go to learn more about Erlang itself, the tools that support program development, and the Erlang community. But now, it’s time to get to work....
[1] PLEX is a proprietary language developed by Ericsson and used extensively in the AXE-10 switches. Just like Erlang, many of its features were ahead of its time. It was never released to the public.
[2] The Engineering and Physical Sciences Research Council provides U.K. government support for science research in universities.
[3] Nyström, J.H., P.W. Trinder, and D.J. King. Concurrency and Computation: Practice & Experience, 20(8), 2008.
Get Erlang Programming now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
