Data Engineering Design Patterns

Chapter 2. Data Ingestion Design Patterns

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Data engineering systems always start by acquiring the data to work on. They’re rarely the data generators themselves and most often they need to interact with various data producers. Working with these producers is not easy as they can be completely different organizations, different teams within your company, or even different pipelines inside your team. Each of them will have dedicated constraints inherited from the technical and business environments that will make the interaction challenging.

But you have no choice. You have to adapt. Otherwise, you won’t get any data and as a result, won’t feed your data analytics or data science workloads. Or even worse, you will get some data, share it with your downstream consumers, but a few days later, you’ll get some complaints in return. The complaint may be about an incomplete dataset, an inefficient data organization, or a completely broken data requiring internal restore process and backfilling.

As you ...