Data Observability for Data Engineering

Data Observability for Data Engineering

by Michele Pinto, Sammy El Khammal
Released December 2023
Publisher(s): Packt Publishing
ISBN: 9781804616024

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Discover actionable steps to maintain healthy data pipelines to promote data observability within your teams with this essential guide to elevating data engineering practices

Key Features

  • Learn how to monitor your data pipelines in a scalable way
  • Apply real-life use cases and projects to gain hands-on experience in implementing data observability
  • Instil trust in your pipelines among data producers and consumers alike
  • Purchase of the print or Kindle book includes a free PDF eBook

Book Description

In the age of information, strategic management of data is critical to organizational success. The constant challenge lies in maintaining data accuracy and preventing data pipelines from breaking. Data Observability for Data Engineering is your definitive guide to implementing data observability successfully in your organization.

This book unveils the power of data observability, a fusion of techniques and methods that allow you to monitor and validate the health of your data. You’ll see how it builds on data quality monitoring and understand its significance from the data engineering perspective. Once you're familiar with the techniques and elements of data observability, you'll get hands-on with a practical Python project to reinforce what you've learned. Toward the end of the book, you’ll apply your expertise to explore diverse use cases and experiment with projects to seamlessly implement data observability in your organization.

Equipped with the mastery of data observability intricacies, you’ll be able to make your organization future-ready and resilient and never worry about the quality of your data pipelines again.

What you will learn

  • Implement a data observability approach to enhance the quality of data pipelines
  • Collect and analyze key metrics through coding examples
  • Apply monkey patching in a Python module
  • Manage the costs and risks associated with your data pipeline
  • Understand the main techniques for collecting observability metrics
  • Implement monitoring techniques for analytics pipelines in production
  • Build and maintain a statistics engine continuously

Who this book is for

This book is for data engineers, data architects, data analysts, and data scientists who have encountered issues with broken data pipelines or dashboards. Organizations seeking to adopt data observability practices and managers responsible for data quality and processes will find this book especially useful to increase the confidence of data consumers and raise awareness among producers regarding their data pipelines.

Table of contents

  1. Data Observability for Data Engineering
  2. Contributors
  3. About the authors
  4. About the reviewers
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Conventions used
    6. Get in touch
    7. Share Your Thoughts
    8. Download a free PDF copy of this book
  6. Part 1: Introduction to Data Observability
  7. Chapter 1: Fundamentals of Data Quality Monitoring
    1. Learning about the maturity path of data in companies
    2. Identifying information bias in data
      1. Data producers
      2. Data consumers
      3. The relationship between producers and consumers
      4. Asymmetric information among stakeholders
    3. Exploring the seven dimensions of data quality
      1. Accuracy
      2. Completeness
      3. Consistency
      4. Conformity
      5. Integrity
      6. Timeliness
      7. Uniqueness
      8. Consequences of data quality issues
    4. Turning data quality into SLAs
      1. An agreement as a starting point
      2. The incumbent responsibilities of producers
      3. Considerations for SLOs and SLAs
    5. Indicators of data quality
      1. Data source metadata
      2. Schema
      3. Lineage
      4. Application
      5. Statistics and KPIs
      6. Examples of SLAs, SLOs, and SLIs
    6. Alerting on data quality issues
      1. Using indicators to create rules
      2. The data scorecard
    7. Summary
  8. Chapter 2: Fundamentals of Data Observability
    1. Technical requirements
    2. From data quality monitoring to data observability
    3. Three principles of data observability
    4. Data observability in IT observability
    5. Key components of data observability
      1. The contract between the application owner and the marketing team
      2. Observing a timeliness issue
      3. Observing a completeness issue
      4. Observing a change in data distribution
    6. Data observability in the enterprise ecosystem
      1. Measuring the return on investment – defining the goals
    7. Summary
  9. Part 2: Implementing Data Observability
  10. Chapter 3: Data Observability Techniques
    1. Analyzing the data
      1. Monitoring data asynchronously
      2. Monitoring data synchronously
    2. Analyzing the application
      1. The anatomy of an external analyzer
      2. Pros and cons of the application analyzer method
      3. Advantages
      4. Disadvantages
      5. Principles of monkey patching for data observability
      6. Wrapping the function
      7. Consolidating the findings
      8. Pros and cons of the monkey patching method
    3. Advanced techniques for data observability – distributed tracing
    4. Summary
  11. Chapter 4: Data Observability Elements
    1. Technical requirements
    2. Prerequisites and installation requirements
      1. Kensu – a data observability framework
      2. kensu-py – an overview of the monkey patching technique
    3. Static and dynamic elements
    4. Defining the data observability context
      1. Application or process
      2. Code base
      3. Code version
      4. Project
      5. Environment
      6. User
      7. Timestamp
      8. The application run
    5. Getting the metadata of the data sources
      1. Data source
      2. Schema
    6. Mastering lineage
      1. Types of lineage and dependencies
      2. Lineage run
      3. What’s in the log?
    7. Computing observability metrics
      1. What’s in the log?
    8. Data observability for AI models
      1. Model method
      2. Model training
      3. Model metrics
      4. What’s in the log?
      5. The feedback loop in data observability
    9. Summary
  12. Chapter 5: Defining Rules on Indicators
    1. Technical requirements
    2. Determining SLOs
      1. Project versus data source SLOs
      2. Use case
    3. Turning SLOs into rules
      1. Different types of rules
      2. Implementation of the rules
    4. Project – continuous validation of the data
      1. Concepts of CI/CD
      2. Deploying the rules in a CI/CD pipeline
    5. Summary
  13. Part 3: How to adopt Data Observability in your organization
  14. Chapter 6: Root Cause Analysis
    1. Data incident management
      1. Detecting the issue
      2. Impact analysis
      3. Root cause analysis
      4. Troubleshooting
      5. Preventing further issues
      6. Applying the method – a practical example
    2. Anomaly detection
      1. Simple indicator deterministic cases
      2. Multiple indicators deterministic cases
      3. Time series analysis
      4. Case study
    3. Summary
  15. Chapter 7: Optimizing Data Pipelines
    1. Concepts of data pipelines and data architecture
      1. What is a data pipeline?
      2. Defining the types of data pipelines
      3. The properties of a data pipeline
    2. Rationalizing the costs
      1. Data pipeline costs
      2. Using data observability to rationalize costs
    3. Summary
  16. Chapter 8: Organizing Data Teams and Measuring the Success of Data Observability
    1. Defining and understanding data teams
      1. The roles of a data team
      2. Organizing a data team
    2. Data mesh, data quality, and data observability – a virtuous circle
      1. Data mesh
      2. Building the virtuous circle
    3. The first steps toward data observability and how to measure success
    4. Measuring success
    5. Summary
  17. Part 4: Appendix
  18. Chapter 9: Data Observability Checklist
    1. Challenges of implementing data observability
      1. Costs
      2. Overhead
      3. Security
      4. Complexity increase
      5. Legacy system
      6. Information overload
    2. Checklist to implement data observability
      1. Start with the right data or application
      2. Choosing the right data observability tool
      3. Selecting the metrics to follow
      4. Compute the return on investment
      5. Scaling with data observability
    3. Summary
  19. Chapter 10: Pathway to Data Observability
    1. Technical roadmap to include data observability
      1. Allocating the right resources to your data observability project
      2. Defining clear objectives with the team
      3. Choosing a data pipeline
      4. Setting success criteria with the team and stakeholders
      5. Implementing data observability in applications
      6. Continuously improving observability
      7. Scaling data observability
      8. Using observability for data catalogs
      9. Using observability to ensure ML and AI reliability
      10. Using observability to complete a data quality management program
    2. Implementing data observability in a project
      1. Resources and the first pipeline
      2. Success criteria for PetCie’s implementation
      3. The implementation phase at PetCie
      4. Continuously improving observability at PetCie
      5. Deploying observability at scale at PetCie
      6. Outcomes
    3. Summary
  20. Index
    1. Why subscribe?
  21. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share Your Thoughts
    3. Download a free PDF copy of this book

Product information

  • Title: Data Observability for Data Engineering
  • Author(s): Michele Pinto, Sammy El Khammal
  • Release date: December 2023
  • Publisher(s): Packt Publishing
  • ISBN: 9781804616024