Introduction to designing for scalability with Erlang/OTP

Defining the Problem

As we show throughout this book, Erlang/OTP is unique among programming languages and
frameworks in the breadth, depth, and consistency of the features it
provides for scalable, fault-tolerant systems with requirements for high
availability. Designing, implementing, operating, and maintaining these
systems is challenging. Teams that succeed in building and running them do
so by continuously iterating through those four phases, constantly using
feedback from production metrics and monitoring to help find areas they
can improve not only in their code, but also in their development and
operating processes. Successful teams also learn how to improve
scalability through other means, such as testing, experimentation, and
benchmarking, and they keep up on research and development relevant to
their system characteristics. Nontechnical issues such as organizational
values and culture can also play a significant part in determining whether
teams can meet or exceed their system requirements.

We used the terms distributed,
fault-tolerant, scalable,
soft real-time, and highly
available to describe the systems we plan on building with OTP.
But what do these words actually mean?

Scalable refers to how well a computing system can adapt to changes in load or
available resources. Scalable websites, for example, are able to smoothly
handle traffic spikes without dropping any client requests, even when
hardware fails. A scalable chat system might be able to accommodate
thousands of new users per day without disruption of the service it
provides to its current users.

Distributed refers to how systems are clustered together and interact with each
other. Clusters can be architected to scale horizontally by adding
commodity (or regular) hardware, or on a single machine, where additional
instances of standalone nodes are deployed to better utilize the available
cores. Single machines can also be virtualized, so that instances of an
operating system run on other operating systems or share the bare-metal
resources. Adding more processing power to a database cluster could enable
it to scale in terms of the amount of data it can store or how many
requests per second it can handle. Scaling downward is often equally as
important; for example, a web application built on cloud services might
want to deploy extra capacity at peak times and release unused computing
instances as soon as usage drops.

Systems that are fault tolerant
continue to operate predictably when things in their environment are
failing. Fault tolerance has to be designed into a system from the start;
don’t even consider adding it as an afterthought. What if there is a bug
in your code or your state gets corrupted? Or what if you experience a
network outage or hardware failure? If a user sending a message causes a
process to crash, the user is notified of whether the message was
delivered or not and can be assured that the notification received is
correct.

By soft real-time, we mean the predictability of response and latency, handling a
constant throughput, and guaranteeing a response within an acceptable time
frame. This throughput has to remain constant regardless of traffic spikes
and number of concurrent requests. No matter how many simultaneous
requests are going through the system, throughput must not degrade under
heavy loads. Response time, also known as latency, has to be relative to
the number of simultaneous requests, avoiding large variances in requests
caused by “stop the world” garbage collectors or other sequential
bottlenecks. If your system throughput is a million messages per second
and a million simultaneous requests happen to be processed, it should take
1 second to process and deliver a request to its recipient. But if during
a spike, two million requests are sent, there should be no degradation in
the throughput; not some, but all of the requests should be handled within
2 seconds.

High availability minimizes or
completely eliminates downtime as a result of bugs, outages,
upgrades, or other operational activities. What if a process crashes? What
if the power supply to your data center is cut off? Do you have a
redundant supply or battery backup that gives you enough time to migrate
your cluster and cleanly shut down the affected servers? Or network and
hardware redundancy? Have you dimensioned your system ensuring that, even
after losing part of your cluster, the remaining hardware has enough CPU
capacity to handle peak loads? It does not matter if you lose part of your
infrastructure, if your cloud provider is experiencing an embarrassing
outage, or if you are doing maintenance work; a user sending a chat
message wants to be reassured that it reaches its intended recipient. The
system’s users expect it to just work. This is in contrast to fault tolerance, where the user is told it did
not work, but the system itself is unaffected and continues to run.
Erlang’s ability to do software upgrades during runtime helps. But if you
start thinking of what is involved when dealing with database schema
changes, or upgrades to non–backward-compatible protocols in potentially
distributed environments handling requests during the upgrade, simplicity
fades very quickly. When doing your online banking on weekends or at
night, you want to be sure you will not be met with an embarrassing
“closed for routine maintenance” sign posted on the website.

Erlang indeed facilitates solving many of these problems. But at the
end of the day, it is still just a programming language. For the complex
systems you are going to implement, you need ready-built applications and
libraries you can use out of the box. You also need design principles and
patterns that inform the architecture of your system with an aim to create
distributed, reliable clusters. You need guidelines on how to design your
system, together with tools to implement, deploy, monitor, operate, and
maintain it. In this book we cover libraries and tools that allow you to
isolate failure on a node level, and create and distribute multiple nodes
for scalability and availability.

You need to think hard about your requirements and properties,
making certain you pick the right libraries and design patterns that
ensure the final system behaves the way you want it to and does what you
originally intended. In your quest, you will have to make tradeoffs that
are mutually dependent—tradeoffs on time, resources, and features and
tradeoffs on availability, scalability, and reliability. No ready-made
library can help you if you do not know what you want to get out of your
system. In this book, we guide you through the steps in understanding
these requirements, and walk you through the steps involved in making
design choices and the tradeoffs needed to achieve them.

OTP

OTP is a domain-independent set of frameworks, principles, and
patterns that guide and support the structure, design, implementation, and
deployment of Erlang systems. Using OTP in your projects will help you
avoid accidental complexity: things that are difficult because you picked
inadequate tools. But other problems remain difficult, irrespective of the
programming tools and middleware you choose.

Ericsson realized this very early on. In 1993, alongside the development of the first Erlang
product, Ericsson started a project to tackle tools, middleware, and
design principles. The developers wanted to avoid accidental difficulties
that had already been solved, and instead focus their energy on the hard
problems. The result was BOS, the Basic Operating System. In 1995, BOS merged with
the development of Erlang, bringing everything under one roof to form
Erlang/OTP as we know it today. You might have heard the
dream team that supports Erlang being referred to as the OTP team. This
group was a spinoff of this merge, when Erlang was moved out of a research
organization and a product group was formed to further develop and
maintain it.

Spreading knowledge of OTP can promote Erlang adoption in more
“tried and true” corporate IT environments. Just knowing there is a stable
and mature platform available for application development helps
technologists sell Erlang to management, a crucial step in making its
industrial adoption more widespread. Startups, on the other hand, just get
on with it, with Erlang/OTP allowing them to achieve speed to market and
reduce their development and operations costs.

OTP is said to consist of three building blocks (Figure 1-1) that, when used together, provide a solid
approach to designing and developing systems in the problem domain we’ve
just described. They are Erlang itself, tools and libraries, and a set of
design principles. We’ll look at each in turn.

System Design Principles

The third building block of OTP consists of a set of abstract principles,
design rules, and generic behaviors. The abstract principles describe
the software architecture of an Erlang system, using processes in the
form of generic behaviors as basic ingredients. Design rules keep the
tools you use compatible with the system you are developing. Using this
approach provides a standard way of solving problems, making code easier
to understand and maintain, as well as providing a common language and
vocabulary among the teams.

OTP generic behaviors can be seen as formalizations of concurrent
design patterns. Behaviors are packaged into library modules containing
generic code that solves a common problem. They have built-in support
for debugging, software upgrade, generic error handling, and built-in
functionality for upgrades.

Behaviors can be worker processes, which
do all of the hard work, and supervisor
processes, whose only tasks are to start, stop, and monitor
workers or other supervisors. Because supervisors can monitor other
supervisors, the functionality within an application can be chained so
that it can be more easily developed in a modular fashion. The processes
monitored by a supervisor are called its children.

OTP provides predefined libraries for workers and supervisors,
allowing you to focus on the business logic of the system. We structure
processes into hierarchical supervision trees,
yielding fault-tolerant structures that isolate failure and facilitate
recovery. OTP allows you to package a supervision tree into an
application, as seen in Figure 1-2, where circles
with double rings are supervisors and the other processes are
workers.

Generic behaviors that come as part of the OTP middleware include:

• Generic servers, providing a client-server design pattern

• Generic finite state machines, allowing you to implement
  FSMs

• Event handlers and managers, allowing you to generically
  deal with event streams

• Supervisors, monitoring other worker and supervision processes

• Applications, allowing you to package resources, including
  supervision trees

We cover them all in detail in this book, as well as explaining
how to implement your own. We use behaviors to create supervision trees,
which are packaged into applications. We then group applications
together to form a release. A release
describes what runs in a node.

Erlang Nodes

An Erlang node consists of several loosely coupled applications, which
might be comprised of some of the applications described in Tools and Libraries combined with other third-party
applications and applications you write specifically for the system you
are trying to implement. These applications could be independent of each
other or rely on the services and APIs of other applications. Figure 1-3 illustrates a typical release of an Erlang node with
the virtual machine (VM) dependent on the hardware and operating system,
and Erlang applications running on top of the VM interfacing with
non-Erlang components that are OS and hardware dependent.

Group together a cluster of Erlang nodes—potentially pairing them
up with nodes written in other programming languages—and you have a
distributed system. You can now scale your system by adding nodes until
you hit certain physical limits. These may be dictated by how you shared
your data, by hardware or network constraints, or by external
dependencies that act as bottlenecks.

Distribution, Infrastructure, and Multicore

Fault tolerance—one of Erlang’s fundamental requirements from its telecom roots—has
distribution as its mainspring. Without distribution, the reliability and
availability of an application running on just a single host would depend
heavily on the reliability of the hardware and software comprising that
host. Any problems with the host’s CPU, memory, persistent storage,
peripherals, power supply, or backplane could easily take down the entire
machine and the application along with it. Similarly, problems in the
host’s operating system or support libraries could bring down the
application or otherwise render it unavailable. Achieving fault tolerance
requires multiple computers with some degree of coordination between them,
and distribution provides the avenue for that coordination.

For decades, the computing industry has explored how programming
languages can support distribution. Designing general-purpose languages is
difficult enough; designing them to support distribution significantly
adds to that difficulty. Because of this, a common approach is to add
distribution support to nondistributed programming languages through
optional libraries. This approach has the benefit of allowing distribution
support to evolve separately from the language itself, but it often
suffers from an impedance mismatch with the language, feeling to
developers as if it were “bolted on.” Since most languages use function
calls as the primary means of transferring control and data from one part
of an application to another, add-on distribution libraries often model
exchanges between distributed parts of an application as function calls as
well. While convenient, this approach is fundamentally broken because the
semantics of local and remote function calls, especially their failure
modes, are markedly different.

In Erlang, processes communicate via asynchronous message passing. This works even if a process
is on a remote node because the Erlang virtual machine supports passing
messages from one node to another. When one node joins another, it also
becomes aware of any nodes already known to the other. In this manner, all
the nodes in a cluster form a mesh, enabling any process to send a message
to another process on any other node in the cluster. Each node in the
cluster also automatically tracks liveness of other nodes in order to
become aware of nonresponsive nodes. The advantages of asynchronous
message passing in systems running on a node is extended to systems
running in clusters, as replies can be received alongside errors and
timeouts.

Erlang’s message passing and clustering primitives can serve as the
basis for a wide variety of distributed system architectures. For example,
service-oriented architecture (SOA), especially in its more modern variant, microservices, is a
natural fit for Erlang given the ease of developing and deploying
server-like processes. Clients treat such processes as services,
communicating with them by exchanging messages. As another example,
consider that Erlang clusters do not require master or leader nodes, which
means that using them for peer-to-peer systems of replicas works well.
Clients can send service request messages to any peer node in the cluster,
and the peer can either handle the request itself or route it to another
peer. The concept of standalone clusters, known as groups that communicate with each other through
gateway nodes that can go up and down or lose connectivity exists in a
framework called SD Erlang. Another
popular distributed framework, inspired by the Amazon Dynamo paper published in 2007, is
Riak Core, offering consistent hashing to schedule jobs, recovery from
partitioned networks and failed nodes through consistent hashing, eventual
consistency, and virtual nodes dividing state and the data into small,
manageable entities that can be replicated and moved across nodes.

With distributed systems, you can also achieve scalability. In fact, availability, consistency, and
scalability go hand in hand, each affecting the others. It starts with the
concurrency model and the concept of message passing within the node,
which we extend across the network to use for clustering nodes. Erlang’s
virtual machine takes advantage of today’s multicore systems by allowing processes to execute with true
concurrency, running simultaneously on different cores. Because of the
symmetric multiprocessing (SMP) capabilities of the Erlang virtual machine, Erlang is
already prepared to help applications scale vertically as the number of
cores per CPU continues to increase. And because adding new nodes to a
cluster is easy—all it takes is to have that node contact just one other
node to join the mesh—horizontal scaling is also well within easy reach.
This, in turn, allows you to focus on the real challenge when dealing with
distributed systems: namely, distributing your data and state across hosts
and networks that are unreliable.

Summing Up

To make design, implementation, operation, and maintainability
easier and more robust, your programming language and middleware have to
be compact, their behavior in runtime predictable, and the resulting code
base maintainable. We keep talking about fault-tolerant, scalable, soft
real-time systems with requirements for high availability. The problems
you have to solve do not have to be complicated in order to benefit from
the advantages Erlang/OTP brings to the table. Advantages will be evident
if you are developing solutions targeted for embedded hardware platforms
such as the Parallela board, the BeagleBoard, or the Raspberry Pi. You
will find Erlang/OTP ideal for the orchestration code in embedded devices,
for server-side development where concurrency comes in naturally, and all
the way up to scalable and distributed multicore architectures and
supercomputers. It eases the development of the harder software problems
while making simpler programs even easier to implement.

What You’ll Learn in This Book

This book is divided into two sections. The first part, from Chapter 3 to Chapter 10,
deals with the design and implementation of a single node. You should read
these chapters sequentially, because their examples and explanations build
on prior ones. The second half of the book, from Chapter 11 to Chapter 16,
focuses on tools, techniques, and architectures used for deployment,
monitoring, and operations, while explaining the theoretical approaches
needed to tackle issues such as reliability, scalability, and high
availability. The second half builds in part on the examples covered in
the first half of the book, but can be read independently of it.

We begin with an overview of Erlang in Chapter 2, intended not to teach you the language but
rather as a refresher course. If you do not yet know Erlang, we recommend
that you first consult one or more of the excellent books designed to help
you learn the language, such as Simon St. Laurent’s Introducing
Erlang, Erlang Programming
by Francesco Cesarini and Simon Thompson, or any of the other books we
mention in Chapter 2. Our overview touches on the
major elements of the language, such as lists, functions, processes and
messages, and the Erlang shell, as well as those features that make Erlang
unique among languages, such as process linking and monitoring, live
upgrades, and distribution.

Following the Erlang overview, Chapter 3
dives into process structures. Erlang processes can handle a wide variety
of tasks, yet regardless of the particular tasks or their problem domains,
similar code structures and process lifecycles surface, akin to the common
design patterns that have been observed and documented for popular
object-oriented languages like Java and C++. OTP captures and formalizes
these common process-oriented structures and lifecycles into behaviors, which serve as the base elements of
OTP’s reusable frameworks.

In Chapter 4 we explore in detail our first
worker process. It is the most popular and frequently used OTP behavior,
the gen_server. As its name implies, it supports generic client-server structures,
with the server governing particular computing resources—perhaps just a
simple Erlang Term Storage (ETS) instance, or a pool of network
connections to a remote non-Erlang server—and granting clients access to
them. Clients communicate with generic servers synchronously in a
call-response fashion, asynchronously via a one-way message called a
cast, or via regular Erlang messaging
primitives. Full consideration of these modes of communication requires us
to scrutinize various aspects of the processes involved, such as what
happens if the client or server dies in the middle of a message exchange,
how timeouts apply, and what might happen if a server receives a message
it does not understand. By addressing these and other common issues, the
gen_server handles a lot of details independently of the
problem domain, allowing developers to focus more of their time and energy
on their applications. The gen_server behavior is so useful
that it not only appears in most nontrivial Erlang applications but is
used throughout OTP itself as well.

Prior to examining more OTP behaviors, we follow our discussion of
gen_server with a look at some of the control and observation
points the OTP behaviors provide (Chapter 5). These features reflect
another aspect of Erlang/OTP that sets it apart from other languages and
frameworks: built-in observability. If you want to know what your
gen_server process is doing, you can simply enable debug
tracing for that process, either at compile time or at runtime from an
Erlang shell. Enabling traces causes it to emit information that indicates
what messages it is receiving and what actions it is taking to handle
them. Erlang/OTP also provides functions for peering into running
processes to see their backtraces, process dictionaries, parent processes,
linked processes, and other details. There are also OTP functions for
examining status and internal state specifically for behaviors and other
system processes. Because of these debug-oriented features, Erlang
programmers often forego the use of traditional debuggers and instead rely
on tracing to help them diagnose errant programs, as it is typically both
faster to set up and more informative.

We then examine another OTP behavior, gen_fsm (Chapter 6), which supports a generic FSM pattern. As you may already know, an FSM is a system that has a finite
number of states, and incoming messages can advance the system from one
state to another, with side effects potentially occurring as part of the
transitions. For example, you might consider your television set-top box
as being an FSM where the current state represents the selected channel
and whether any on-screen display is shown. Pressing buttons on your
remote causes the set-top box to change state, perhaps selecting a
different channel, or changing its on-screen display to show the channel
guide or list any on-demand shows that might be available for purchase.
FSMs are applicable to a wide variety of problem domains because they
allow developers to more easily reason about and implement the potential
states and state transitions of their applications. Knowing when and how
to use gen_fsm can save you from trying to implement your own
ad hoc state machines, which often quickly devolve into spaghetti code
that is hard to maintain and extend.

Logging and monitoring are critical parts of any scalability success
story, since they allow you to glean important information about your
running systems that can help pinpoint bottlenecks and problematic areas
that require further investigation. The Erlang/OTP gen_event behavior (Chapter 7) provides support for subsystems that
emit and manage event streams reflecting changes in system state that can
impact operational characteristics, such as sustained increases in CPU
load, queues that appear to grow without bound, or the inability of one
node in a distributed cluster to reach another. These streams do not have
to stop with your system events. They could handle your
application-specific events originating from user interaction, sensor
networks, or third-party applications. In addition to exploring the
gen_event behavior, we also take a look at the OTP system
architecture support libraries (SASL) error-logging event handlers, which provide flexibility for
managing supervisor reports, crash reports, and progress reports.

Event handlers and error handlers are staples of numerous
programming languages, and they are incredibly useful in Erlang/OTP as
well, but do not let their presence here fool you: dealing with errors in
Erlang/OTP is strikingly different from the approaches to which most
programmers are accustomed.

After gen_event, the next behavior we study is
the supervisor (Chapter 8),
which manages worker processes. In Erlang/OTP, supervisor processes start
workers and then keep an eye on them while they carry out application
tasks. Should one or more workers die unexpectedly, the supervisor can
deal with the problem in one of several ways that we explain later in the
book. This form of handling errors, known as “let it crash,” differs
significantly from the defensive programming tactics that most programmers
employ. “Let it crash” and supervision, together a critical cornerstone of
Erlang/OTP, are highly effective in practice.

We then look into the final fundamental OTP behavior, the application (Chapter 9),
which serves as the primary point of integration between the Erlang/OTP
runtime and your code. OTP applications have configuration files that
specify their names, versions, modules, the applications upon which they
depend, and other details. When started by the Erlang/OTP runtime, your
application instance in turn starts a top-level supervisor that brings up
the rest of the application. Structuring modules of code into applications
also lets you perform code upgrades on live systems. A release of an
Erlang/OTP package typically comprises a number of applications, some of
which are part of the Erlang/OTP open source distribution and others that
you provide.

Having examined the standard behaviors, we next turn our attention
to explaining how to write your own behaviors and special processes (Chapter 10). Special processes are processes that follow certain design rules, allowing them to
be added to OTP supervision trees. Knowing these design rules can not only
help you understand implementation details of the standard behaviors, but
also inform you of their tradeoffs and allow you to better decide when to
use them and when to write your own instead.

Chapter 11 describes how OTP applications
in a single node are coupled together and started as a whole. You will
have to create your own release files, referred to in the Erlang world as
rel files. The rel file lists the
versions of the applications and the runtime system that are used by the
systools module to bundle up the software into a standalone
release directory that includes the virtual machine. This release
directory, once configured and packaged, is ready to be deployed and run
on target hosts. We cover the community-contributed tools rebar3 and relx, the best way to build your code and your
releases.

The Erlang virtual machine has configurable system limits and
settings you need to be aware of when deploying your systems. There are
many, ranging from limits regulating the maximum number of ETS tables or
processes to included code search paths and modes used for loading
modules. Modules in Erlang can be loaded at startup, or when they are
first called. In systems with strict revision control, you will have to
run them in embedded mode, loading modules at startup and crashing if modules do not
exist, or in interactive mode, where
if a module is not available, an attempt to load it is made
before terminating the process. An external monitoring heart process
monitors the Erlang virtual machine by sending heartbeats and invoking a
script that allows you to react when these heartbeats are not
acknowledged. You implement the script yourself, allowing you to decide
whether restarting the node is enough or whether—based on a history of
previous restarts—you want to escalate the crash and terminate the virtual
instance or reboot the whole machine.

Although Erlang’s dynamic typing allows you to upgrade your module
at runtime while retaining the process state, it does not coordinate
dependencies among modules, changes in process state, or
non–backward-compatible protocols. OTP has the tools to support system
upgrades on a system level, including not only the applications, but also
the runtime system. The principles and supporting libraries are presented
in Chapter 12, from defining your own
application-upgrade scripts to writing scripts that support release
upgrades. Approaches and strategies for handling changes to your database
schema are provided, as are guidelines for upgrades in distributed
environments and non–backward-compatible protocols. For major upgrades in
distributed environments where bugs are fixed, protocols improved, and
database schema changed, runtime upgrades are not for the faint of heart.
But they are incredibly powerful, allowing automated upgrades and nonstop
operations. Finding your online banking is unavailable because of
maintenance should now be a thing of the past. If it isn’t, send a copy of
this book to your bank’s IT department.

Operating and maintaining any system requires visibility into what
is going on. Scaling clusters require strategies for how you share your
data and state. And fault tolerance requires an approach to how you
replicate and persist it. In doing so, you have to deal with unreliable
networks, failure, and recovery strategies. While each of these subjects
merits a book of its own, the final chapters of this book will provide you
with the theoretical background needed when distributing your systems and
making them reliable and scalable. We provide this theory by describing
the steps needed to design a scalable, highly available architecture in
Erlang/OTP.

Chapter 13 will give you an overview of
the approaches needed when designing your distributed architecture,
breaking up your functionality into standalone nodes. In doing so, each
standalone node type will be assigned a
specific purpose, such as acting as a client gateway managing TCP/IP
connection pools or providing a service such as authentication or
payments. For each node type, we define an approach to specifying
interfaces and defining the state and data each node needs. We conclude
the chapter by describing the most common distributed architectural
patterns and the different network protocols that can be used to connect
them.

When you have your distributed architecture in place, you need to
make design choices that will impact fault tolerance, resilience,
reliability, and availability. You know what data and state you need in
your node types, but how are you going to distribute it and keep it
consistent? Are you going for the share-everything, share-something, or
share-nothing approach, and what are the tradeoffs you need to make when
choosing strong, causal, or eventual consistency? In Chapter 14, we describe the different approaches
you can take, introducing the retry strategies you need to be aware of in
case a request times out as the result of process, node, or network
failure or the mere fact that the network or your servers are running over
capacity.

It is easy to say that you are going to add hardware to make your
system scale horizontally, but alas, the design choices introduced in
Chapter 14 will have an impact on your
system’s scalability. In Chapter 15, we describe the impacts resulting from
your data-sharing strategy, consistency model, and retry strategy. We
cover capacity planning, including the load, peak, and stress tests you
need to subject your system to to guarantee it behaves in a predictable
way under heavy load even when the hardware, software, and infrastructure
around it are failing.

Once you’ve designed your scalability and availability strategies, you need to tackle monitoring. If
you want to achieve five-nines uptime, you need to not only know what is
going on, but also be able to quickly determine what happened, and why. We
conclude the book with Chapter 16, looking at
how monitoring is used for preemptive support and postmortem
debugging.

Monitoring focuses on metrics, alarms, and logs. This chapter
discusses the importance of system and business metrics. Examples of system metrics
include the amount of memory your node is using, process message queue
length, and hard-disk utilization. Combining these with business metrics,
such as the number of failed and successful login attempts, message
throughput per second, and session duration, yields full visibility of how
your business logic is affecting your system resources.

Complementing metrics is alarming, where you detect and report anomalies, allowing the system to take
action to try to resolve them or to alert an operator when human
intervention is required. Alarms could include a system running out of
disk space (resulting in the automatic invocation of scripts for
compressing or deleting logs) or a large number of failed message
submissions (requiring human intervention to troubleshoot connectivity
problems). Preemptive support at its best, detecting and resolving issues
before they escalate, is a must when dealing with high availability. If
you do not have a real-time view of what is going on, resolving issues
before they escalate becomes extremely difficult and cumbersome.

And finally, logging of
major events in the system helps you troubleshoot your
system after a crash where you lost its state, so you can retrieve the
call flow of a particular request among millions of others to handle a
customer services query, or just provide data records for billing
purposes.

With your monitoring in place, you will be ready to architect
systems that are not only scalable, but also resilient and highly
available. Happy reading! We hope you enjoy the book as much as we enjoyed
writing it.