Chapter 2. The Domain Language of Telemetry Pipelines
There is a tide in the affairs of men
Which, taken at the flood, leads on to fortune;
Omitted, all the voyage of their life
Is bound in shallows and in miseries.
On such a full sea are we now afloat.
And we must take the current when it serves,
Or lose our ventures.
William Shakespeare, Julius Caesar
In Chapter 1, you saw how telemetry data has become a flood: a flood where the properties of your telemetry data need to be considered so that you can get the most value out of all the data that is available. In this chapter, you’re going to explore the foundational concepts of telemetry pipelines so that you can begin to do exactly that.
The Building Blocks of Telemetry Pipelines
Telemetry pipelines are made up of five concepts: sources, streams, processors, destinations, and, of course, the pipeline itself. Sources are your faucets for the flood of telemetry data being emitted from your systems. Each source produces a stream of structured telemetry data. Processors are the workhorses that do what it takes to manipulate those streams of telemetry data you’ve tapped into. Destinations are your channels to bring this refined telemetry data to downstream tools and other systems. Finally, the pipeline packages all these sources, streams, processors, and destinations into a unit you can work with and manage. Let’s take a closer look.
Starting at the Source
Most data pipelines start by answering the question, “Where are we going ...
Get The Fundamentals of Telemetry Pipelines, Revised Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
