Why Reactive?

In Search of the Optimal Utilization Level

In the previous section, we learned that using asynchronous APIs and programming techniques helps to increase utilization of your hardware. This sounds good, and indeed we do want to use the hardware that we’re paying for to its fullest. However, the other side of the coin is that pushing utilization beyond a certain point will yield diminishing (or even negative if pushed further) returns. This observation has been formalized by Neil J. Gunther in 1993 and is called the Universal Scalability Law (USL).⁹

The relation between the USL, Amdahl’s law, and queueing theory is material worth an entire paper by itself, so I’ll only give some brief intuitions in this report. If after reading this section you feel intrigued and would like to learn more, please check out the white paper “Practical Scalability Analysis with the Universal Scalability Law” by Baron Schwartz (O’Reilly).

The USL can be seen as a more practical model than the more widely known Amdahl’s law, first defined by Gene Amdahl in 1967, which only talks about the theoretical speedup of an algorithm depending on how much of it can be executed in parallel. USL on the other hand takes the analysis a step further by introducing the cost of communication, the cost of keeping data in sync—coherency—as variable in the quotation, and suggests that pushing a system beyond its utilization sweet spot will not only not yield any more speedup, but will actually have a negative impact on the system’s overall throughput, since all kind of coordination is happening in the background. This coordination might be on the hardware level (e.g., memory bandwidth saturation, which clearly does not scale with the number of processors) or network level (e.g., bandwidth saturation or incast and retransmission problems).

One should note that we can compete for various resources and that the over-utilization problem applies not only to CPU, but—in a similar vein—to network resources. For example, with some of the high-throughput messaging libraries, it is possible to max out the 1 Gbps networks which are the most commonly found in various cloud provider setups (unless specific network/node configurations are available and provisioned, such as 10 Gbps network interfaces available for specific high-end instances on Amazon EC2). So while the USL applies both to local and distributed settings, for now let’s focus on the application-level implications of it.

Using Back-Pressure to Maintain Optimal Utilization Levels

When using synchronous APIs, the system is “automatically” back-pressured by the blocking operations. Since we won’t do anything else until the blocking operation has completed, we’re wasting a lot of resources by waiting. But with asynchronous APIs, we’re able to max out on performing our logic more intensely, although we run the risk of overwhelming some other (slower) downstream system or other part of the application. This is where back-pressure (or flow-control) mechanisms come into play.

Similar to the Reactive Manifesto, the Reactive Streams initiative emerged from a collaboration between industry-leading companies building concurrent and distributed applications that wanted to standardize an interop protocol around bounded-memory stream processing. This initial collaboration included Lightbend, Netflix, and Pivotal, but eventually grew to encompass developers from RedHat and Oracle.¹⁰ The specification is aimed to be a low-level interop protocol between various streaming libraries, and it requires and enables applying back-pressure transparently to users of these streaming libraries. As the result of over a year of iterating on the specification, its TCK, and semantic details of Reactive Streams, they have been incorporated in the OpenJDK, as part of the JEP-266 “More Concurrency Updates” proposal.¹¹ With these interfaces and a few helper methods that have become part of the Java ecosystem directly inside the JDK, it is safe to bet on libraries that implement the Reactive Streams interfaces to be able to move on to the ones included in the JDK, and be compatible even in the future—with the release of JDK9.

It is important to keep in mind that back-pressure, Reactive Streams, or any other part of the puzzle is not quite enough to make a system resilient, scalable, and responsive. It is the combination of the techniques described here which yields a fully reactive system. With the use of asynchronous and back-pressured APIs, we’re able to push our systems to their limits, but not beyond them. Answering the question of how much utilization is in fact optimal is tricky, as it’s always a balance between being able to cope with a sudden spike in traffic, and wasting resources. It also is very dependent on the task that the system is performing. A simple rule of thumb to get started with (and from there on, optimize according to your requirements) is to keep system utilization below 80%. An interesting discussion about battles fought for the sake of optimizing utilization, among other things, can be read in the excellent Google Maglev paper.¹²

One might ask if this “limiting ourselves” could lower overall performance compared to the synchronous versions. It is a valid question to ask, and often a synchronous implementation will beat an asynchronous one in a single-threaded, raw-throughput benchmark, for example. However, real-world workloads do not look like that. In an interesting performance analysis of RxNetty compared to Tomcat at Netflix, Brendan Gregg and Ben Christensen found that even with the asynchronous overhead and flow control, the asynchronous server implementation did yield much better response latency under high load than the synchronous (and highly tuned) Tomcat server.¹³

Streaming APIs and the Rise of Bounded-Memory Stream Processing

Ever-newer waters flow on those who step into the same rivers.

Heraclitus

Streaming, much like reactive, is in a phase where the community is trying to actually define what it means when one uses the word “stream.” Sadly, people are perhaps more confused about it than they are with “reactive.” The reason the streaming landscape has become so confusing is that multiple libraries that address very different needs have come to use the word. For example, Spark Streaming and Flink address the large-scale data-transformation side but are not well suited for small- to medium-sized jobs, nor for embedding as source of data for an HTTP response APIs that respond by providing an infinite stream of data, like the well-known Twitter Streaming APIs.

In this chapter, we’ll focus on what streaming actually means, why it matters, and what’s to come. There are two sides of the same coin here: consuming and producing streaming APIs. There’s a reason we discuss this topic in the chapter about reactive on the application level, and not system level, even though APIs serve as the integration layer between various systems. It has to do with the interesting capabilities that streaming APIs and bounded-in-memory processing give us. Most notably, using and/or building streaming libraries and APIs allows us to never load more of the data into memory than we actually need, which in turn allows us to build bounded-memory pipelines. This is a very interesting and useful property for capacity planning, as now we have a guarantee of how much memory a given connection or stream will take, and can include these numbers in our capacity planning calculations.

Let’s discuss this feature as it relates to the Twitter Firehose API, an API an application can subscribe to in order to collect and analyze all incoming tweets in the Twittersphere. Obviously, consuming such a high-traffic stream takes significant machine power on the receiving end as well. And this is where it gets interesting, what if the downstream (the customer, accessing the firehose API) is not able to consume it at the rate at which it is being emitted?

Let’s look at this design challenge from the server’s perspective. We have a live stream of data coming in from our backends, and we need to push those to downstream clients of the service. Some of them may be slow or may have trouble on their end which causes them to not consume the stream at all. What should we do? The usual answer is to buffer until the client comes back. That answer is both correct and scary at the same time. Of course, we want to buffer a little bit, to allow the client to recover from the slowness on their end and continue consuming the stream. However, we can’t be expected to buffer these tweets indefinitely; that would take unbounded amounts of memory (which would be lovely to have, but we’re not there yet). The solution is simple: we use bounded buffers. For example, if we see clients not being able to cope with the rate at which we’d like to emit events, we issue warnings to the clients (which in fact is an option available in all of Twitter’s streaming APIs) that “we’re queueing up messages for delivery to you. Your queue is now over 60% full.” This is a very nice strategy because we can estimate based on our bounded-size queues, how many slow clients we’re able to service, and balance our service levels and node utilization.

Monitoring the queue and buffer sized of clients of our APIs is a very interesting metric, and could even trigger some interaction with the customer. For example, we could offer them additional processing power, or suggest that they optimize their client in some way. Of course, once the buffer is full, we need to do something to save our application from any trouble. In the Twitter example, the answer is simple: we disconnect the client (the warning message always includes details about this). But this is not the only option. One could also drop the oldest or newest elements in the stream, especially if the oldest ones have become outdated. Needless to say, such decisions should be business driven, although the key enabler of making them trivial are streaming-first libraries such as Akka Streams/HTTP, where it is as simple as picking an overflow strategy, which Akka documents in its quick start guide.

Reactive Is an Architectural and Design Principle, Not a Single Library

The whole is other than the sum of the parts.

Kurt Koffka

You may have heard this quote phrased as the whole being “greater” than the sum of its parts. But it turns out that the actual quote was slightly different, as Koffka meant meant the whole being, in fact, something different than the sum of parts, not necessarily greater or larger (e.g., like the “invisible” triangle in Figure 2-3). The quote fits the situation we find ourselves in with the term “reactive.” You can easily find numerous libraries with the “reactive” prefix added to them, as well as some trying to change the meaning of reactive to be the only task that library does (e.g., stream processing). It is important to realize these are excellent building blocks, but they rarely address the entire problem (most likely omitting to handle resilience or elasticity). In the next chapter, dedicated to Reactive Systems, we’ll learn more about what else there is to reactive other than a nice programming model inside the applications we’re building. We’ll see how applications communicate and how we can scale them out and back down on demand.

 

There’s More to Life Than Request-Response-JSON-over-HTTP

HTTP/2 was meant as a better HTTP/1.1, primarily for document retrieval in browsers for websites. We can do better than HTTP/2 for applications.¹⁵

Ben Christensen, ReactiveSocket

Far too often are we seeing very synchronous request-response patterns dominate the communication patterns of our applications. This is not to say REST is a bad thing. In fact, it’s possible to implement asynchronous communication patterns using RESTful principles; the only problem here is that, in practice, they rarely are. The last few years have shown that many organizations are solely focused not as much on the REST mantra and ideology as they are on the “request-response JSON over HTTP” way of thinking. Thankfully many teams are slowly realizing that some use cases can be served better by newer upcoming protocols or messaging patterns, as showcased by recent developments of Twitter’s Finagle RPC, Google’s new GRPC (Google RPC) libraries, and finally Facebook’s and Netflix’s research into ReactiveSocket. The term “REST” has over the last few years deteriorated to mean JSON-over-HTTP, but it does not have to be this way. The reason I draw this difference is that in the original publication¹⁶ which coins the term REST, it does mention HTTP anywhere, yet somehow developers and architects came to understand REST as a strictly HTTP-bound architectural style. Part of the reason is that Fielding did participate in the URI, HTTP, and HTML IETF working groups, so obviously his work relates to them in some way.

Before we talk about communication patterns between services, let’s remind ourselves that a single service does not necessarily mean that it is handled by one instance of that service. For example, consider an image resizing service. The entry point is a single API endpoint, and users of the service would refer to it as “the image scaling service” (as in “is the image scaler down again?”). However, inside it may well be running multiple instances of the same service and balancing the incoming images across its internal cluster. It is important to realize that the tradeoffs we make for the external API and interactions of the service may be somewhat different than the ones we make for that service’s internal communication. This maps directly to bounded contexts from DDD terminology. Within a single bounded context, we’re communicating more freely; however, once we need to cross context boundaries, the required consistency guarantees change, depending on with whom and how we’re communicating. Those differences, more than anything, would be driven by the fact that the service is usually owned by a single team. In other words, we’ve rediscovered encapsulation, but this time it’s a result of organizational boundaries and the fact that internally each such team operates slightly differently. Instead of forcing all communication within a company into the same RESTful communication style, it often is beneficial to loosen up constraints for a service’s internal communication, since a single team controlling it can move way faster and use other technologies than the ones that make up its public API. Figure 3-1 illustrates the difference between outside and inside requirements regarding communication styles.

In the excellent paper titled “Data on the outside versus Data on the inside,” Pat Helland argues that there is an inherent difference in the requirements data has to face “inside” of a system (i.e., the internal datastore owned by your microservice) and “outside” (i.e., when emitting data representation to consumers of this service).¹⁷ Specifically, he argues that internal data representation be mutable, since, we may want to find more efficient ways to represent the same data (e.g., by changing your SQL schemas.) But the data’s external representation should be immutable. As long as users can get their list of contacts, it does not matter for them if internally we had to implement a JOIN or if it was fetched from a pre-calculated cache that we mutably refresh every now and then. I’d argue that the same idea applies to communication patterns. Externally, we want to expose a stable, well-defined RESTful API that other teams within (and possibly outside) our company can use, or we can just publish the facts for others to use at will (resulting in a more event-based architecture). Internally, however, we need the communication protocols best suited for our needs. For example, internally we may opt for fire-and-forget¹⁸ messaging using binary message formats.

 

The problem with strictly request-response APIs—RESTful APIs being just one of the examples; SOAP lures developers into a very similar approach as well—is that they are limiting in what developers can do. For example, in the HTTP world, subscriptions to event streams are somewhat of a hack, usually implemented by returning one “infinite” HTTP response that we then feed with line-by-line JSON data. A good example of such APIs are the Twitter Streaming APIs.

When discussing this topic, one may often hear the following counter-argument: “I can totally do async with REST.” This statement is, technically speaking, true. However, the cost of implementing it is really exponential when compared with a solution that supports messaging natively.

Let’s examine what an asynchronous job submission API looks like when implemented in a RESTful style. First, we’ll analyze the control flow, and then we’ll look at how many resources need to be allocated throughout its entire lifecycle. REST in today’s world usually is implemented on top of HTTP (though it does not have to; the original REST paper never mentioned any ties to HTTP). So to create a job we would perhaps PUT to a /jobs service, and it would reply with a 201 Created, or rather 202 Accepted, to signal that the request to run the job was accepted. It would also include a Location header to tell us where the result of the job will be located once it’s done. A nonreactive client would have to then poll that location for the job’s status. We also need to keep resources around to remember the mapping from our issued request to the process that handles the polling. Notice that even though the initial request/reply cycle does not tell us anything that we’re really interested in (e.g., the job’s result or progress), we still must wait for its response since we have no way to find the job’s result later.

The same task can be achieved trivially in messaging-based systems, such as Akka or Erlang, by sending a fire-and-forget message that we want to run a job. Eventually we can include in that message a flag that we want to be informed about its progress, and we’re done on the sending side. Once the job completes, we’ll get a message back with its results. We don’t need to care where or when it gets completed. Let’s analyze the overhead of what data or state we need to keep around in this solution. We just need an addressable entity that will receive the message (in Erlang terms, it would be a process; in Akka, those are called actors). Those can be very lightweight—approximately 400 bytes each. Although this differs among the various implementations, the key takeaway is that it’s very small. That’s all the overhead we pay for the reactive interaction with the other system—no connection establishment, no back-and-forth round trips, and no polling.

 

Alternative patterns in HTTP to address these kinds of issues have been invented over the years, but they are very heavy and simply won’t be able to survive the modern-day Internet, one where we have too many jobs, tasks, and clients happening at the same time. But let’s address some of these patterns to show their weaknesses in scalability. Proponents of HTTP may suggest that we could have used a long-running HTTP PUT call in the first place, so we’d keep the request “open” until the job has completed.

It is also important to realize that it’s not only about scalability. The number of decisions one has to make when designing such APIs with plain HTTP is growing exponentially. Will we block on the call? What method should we use? How do we tell the client to do polling? Should we use WebSockets instead? What about server-sent-events? With messaging, it’s trivial—you send a message when you want something, and you get a message when something has completed. So from that perspective it’s also a huge simplification and less headache when designing a system, and we can focus on the actual job instead of debating implementation details.

Surviving the Load…and Shaving the Bill!

The most interesting aspect of Reactive Systems is their ability to elastically scale in the face of varying incoming traffic. Scaling usually serves one of two purposes: either we need to scale out (by adding more machines) and up (by adding beefier machines), or we need to scale down, reducing the number of resources occupied by our application.

The word “elastic” has become more and more prominent; and even though its meaning is very similar to our well-known and loved term “scalability,” it is an important shift to notice. Especially during the early days of the cloud, you could easily go all in and end up being over-provisioned to the load you were actually handling. That is because scaling out in the cloud is rather trivial, such as adding new nodes, joining them up in a cluster, or hiding them behind a load balancer. Turns out what’s not that trivial is regaining a somewhat reasonable utilization level on the spun-up nodes. This is why the recent trends of containers and cluster orchestrators (such as Mesos) are now on the rise: we’ve been over-provisioning our applications over the last few years. Over-provisioning by itself is not a bad thing. It gives us a “buffer” that can survive sudden spikes in traffic, surviving while we add more nodes to the cluster (perhaps that new marketing campaign was really successful). However, over-provisioning by a lot is a huge problem, like a large invoice for things that we didn’t really use.

We found that when talking about scalability of systems, developers get fixated on scaling out or scaling up, and the scaling down part is somehow lost. I’d argue that scaling down is as important as scaling out/up: after all, once the intense traffic demands have laid off, there’s no need to pay those high bills, and of course we want to conserve energy and be environment friendly as well. New tools like Mesos and Kubernetes, together with a container-focused approach to application deployment, are paving the way toward a more ops-less future for normal operation of apps.

An interesting scaling pattern popularized by the likes of Netflix and Gilt (a popular flash-sale site) is predictive scaling, in which we know when spikes are going to hit so we can proactively provision servers for that period, and once traffic starts going down again, decrease the cluster size incrementally. One might get into a funny word-play here: is “proactive scaling” better than “reactive scaling”? No, not really, since Reactive Services are an enabeler for scaling of services. Proactive scaling, on the other hand, is a technique using which we can decide when to scale. For example, a flash-sale site, such as Gilt (which has been very open about its provisioning strategy), has a very high spike in traffic in the middle of the day, exactly when the flash-sale is active.¹⁹ So the proactive part of scaling here means that the nodes can be spun up before the high load hits the servers, because it is predictable. This goes hand-in-hand with these services being reactive—after all, you can’t proactively scale an app if you can’t scale it at all.

Without Resilience, Nothing Else Matters

Resilience is the ability of a substance or object to spring back into shape. The capacity to recover quickly from difficulties.

Merriam Webster

We discussed a lot around performance, scalability, messaging patterns, and various other aspects of reactive. Last, but not least, let’s now loop back to what, perhaps, is the most important trait of them all: resilience. After all, it does not help to have super fast and scalable system that becomes completely unavailable if anything goes wrong.

After all, it does not matter how fast, shiny and great your system is if it is unavailable. Reactive Systems most typically deal with failure by scaling out through replacing failed nodes with new healthy ones, or simply adding more nodes in the face of increased traffic which would have otherwise overloaded the system. In that sense, those systems can be thought of as antifragile, the property of not only being able to survive under stress, but actually improving under it.²⁰

Another anti-pattern that we implicitly discussed in the chapter about elasticity is that legacy systems sometimes tend to share access to a common expensive resource. On the application level, this would manifest as shared, concurrent access to a mutable state. On the system level, it is much easier to spot, as it often is simply a shared database or similar resource. A common theme of such resources is that all services need to access its shared state in order to function. In fact, the Reddit outage in 2016 was caused precisely by such shared services misbehaving.²¹ In other words, it was a potential single point of failure, and took down the entire site down.

Reactive Systems follow the “own your own data” pattern, which we already discussed when talking about actors. It is the same pattern, but on a system level now. If the systems had internally kept more state about their surroundings, instead of relying on ZooKeeper, they perhaps would have been able to survive the outage (for example, an “uncertainty” timeout could have bbn triggered if the shared resource failed, giving the ops team more time to react before these servers had shut down). Systems which are strictly peer-to-peer and masterless are even better as they inherently structure themselves to avoid the single, special-resource problem.

Introducing Reactive in Real-World Systems

Fully async architecture has [measurable] benefits. However I don’t expect to see a software system like that. Instead, we deal with mixed-codebases.²⁴

Ben Christensen

The reality of our daily work is that we rarely have the opportunity to start completely fresh. Even if we do have greenfield projects, to be relevant, they have to integrate with existing systems, and those may not have been built with the level of resilience and semantics we would have hoped for in this day and age. We don’t believe that looking down on legacy systems is a healthy approach. Legacy systems are in place because they succeeded at delivering something in their time—something important enough that kept them alive and running up until this day. Once we come to accept this simple truth, one can look for a productive way to move forward and adopt new techniques, such as reactive architecture. In this section, we aim to highlight a successful and proven approach for introducing new techniques into an existing ecosystem.

One of the ways to introduce change into existing code bases, especially when moving to different paradigms or languages, is to use the “Ivy Pattern” (sometimes referred to as the “Strangler Pattern”²⁵), which has been used in multiple projects in the field, however not often knowingly. The concept is rather simple, and is illustrated in Figure 4-1.

 

The idea here is to avoid rewriting the legacy code that works until you hit an actual problem with it, at which point you can consider rewriting it in the new style (but you’re not forced to.) This pattern received its name from how ivy plants strangle trees on which they grow. The idea here is the same, the new system grows around the old one, and without the need to rewrite anything in the old system; meanwhile, the rest of the system can move on to the more reactive style of development, rewriting the internal if (and when) needed, and not sooner. This helps to avoid high-risk “big bang” rewrites, which often end up in disastrous failures—mostly due to underestimating or not understanding the previous codebase.

Reactive, an Architectural Style for Present and Future

I know; you did send me back to the future. But I’m back. I’m back from the future.

Marty McFly, Back to the Future

It is important to realize early on that reactive is not about a single specific technology or library. Instead of spending this book on explaining a specific library, we spent the space and time to dive into the core concepts behind many of them. With this knowledge you should be able to continue your journey and decide on your own which tools suit you best and will help you move toward this new programming paradigm. It is also very interesting to see that we’ve learned from past mistakes²⁶ when developers tried to hide distributed systems, as if they’re just a special case of local execution. A mistake made with synchronous RPC systems such as CORBA and heavyweight SOA processes. We think that we’ve learned some good lessons from these experiments and now with embracing the network more than ever before. And instead of avoiding failure at all costs, we embrace it in our systems, letting them scale and accommodate it.

Of course, there are some great tools available; but used incorrectly, even the best tool or library won’t solve the problem by itself. You may recall the same situation when Agile was taking over the IT industry, and for good reason. But many teams back then applied “the daily standup” without thinking too much about how it should help their team. Instead, they applied it rigidly “by the book,” and when it didn’t solve the rest of the broken process they had in place, they blamed Agile as a whole. Nowadays it is hard to imagine software development without some of the Agile and Lean practices; however, as with any methodology, it still remains open to misinterpretation.

Thus, it is important when adopting reactive to give it some thought and to understand the principles before moving forward. I hope this report will prove to be useful in doing exactly that and in guiding you and your teams toward making the best decisions for the systems you build.