3
Data Observability Techniques
As we have already learned, sometimes, data quality is still not considered a determining factor during the life cycle of data, or the design and development of new data pipelines.
When a data team has to design and build a new data pipeline, there are many aspects to focus on:
- Data sources – input and output
- Scalability, reliability, and performance
- Total Cost of Ownership (TCO)
- Security
- Operation and maintainability
- Compliance with data regulations
But, often, what is missing is the adoption of data quality and observability by design – in other words, defining the specifications regarding what and how to monitor from the beginning of the design of the new data pipeline. These expectations are often not well ...
Get Data Observability for Data Engineering now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
