Learning OpenTelemetry

Chapter 8. Designing Telemetry Pipelines

I have always found that plans are useless, but planning is indispensable.

President Dwight D. Eisenhower¹

In the previous chapters, we have focused on managing the components that emit telemetry: applications, libraries, services, and infrastructure. Let’s pivot now to managing the telemetry itself, once you have it. Collecting and processing telemetry from every single application, service, and infrastructure component is a sustained, high-throughput operation. Like any other significant component in a distributed system, it takes careful planning to design a telemetry pipeline that always makes sufficient resources available while also minimizing costs.

When telemetry is dropped, you lose observability. Because the volume of telemetry emitted by a system is directly proportional to the load on the system, operators need a clear playbook for scaling the telemetry pipeline in response to sudden traffic spikes and changes in application behavior.

If you plan on operating a telemetry pipeline, this chapter is for you. We discuss the most common telemetry pipelines you will want to adopt as your system grows. We also discuss the various kinds of processing you may want your telemetry pipeline to perform. At the end of the chapter, we focus specifically on managing Collectors within Kubernetes.

Common Topologies

Occasionally, a system is simple enough or new enough that no telemetry management needs to occur. But as systems grow in complexity ...