Building Real-Time Analytics Systems

Chapter 8. Joining Streams with Kafka Streams

Although the orders stream doesn’t currently contain detailed information about products, in the last chapter we set up Debezium to capture any changes to the MySQL products table and write them into the products stream. In this chapter, you’ll learn how to combine the orders stream and products streams by using a stream processor. We’ll put the new stream into Apache Pinot and update the dashboard with the top-selling products and categories.

Enriching Orders with Kafka Streams

In Chapter 4, we used Kafka Streams to create a windowed aggregation over the orders stream so that we could compute the number of orders and revenue in the last few minutes. In this section, we’re going to populate a new stream called enriched-order-items that will contain all the order items contained in the orders stream, hydrated with details from the products stream. Figure 8-1 shows what we’re going to build in more detail.

Figure 8-1. Architecture with enriched orders using Kafka Streams

We can break Figure 8-1 down further to visualize the different processors that will exist in the Kafka Streams graph to achieve this, as shown in Figure 8-2.

Figure 8-2. Kafka Streams processors

Now that we have an overview of what we’re going to build, let’s get ...