Architecting IoT Solutions on Azure

Chapter 10. Cold Path Data Processing

Recall back to Chapter 8, when you looked at the different architectures on Azure for data. In IoT, no matter how you slice it, inevitably, there will be a need to at least support a cold path, even if you adopt an architecture that is supposed to treat everything as hot path data. In Lambda Architecture, a cold path is inherent. Kappa Architecture is supposed to treat cold paths as a special case of hot paths. Although I prefer Kappa-style architecture, it’s possible only in the best scenarios. Therefore, I acknowledge that architectures still need to be at least open to the possibility of cold paths. Suffice it to say, the need for cold paths is frozen and here to stay.

In Figure 10-1, the cold path is represented as the bar at the bottom of the IoT Landscape.

Figure 10-1. The cold path on the IoT Landscape

Remember, cold paths are not so much about speed. The defining characteristics are higher latency, larger data sets, complex computations, and heavier use of persistent storage as part of the data processing cycle. These characteristics lean heavily into batch processes that run on larger batches of data rather than small datasets or atomic messages like hot path processes do. Many of these batches might run in intervals like five minutes. Others could be hourly, daily, weekly, monthly, yearly, or longer. The relative size of the data ...