Managing Data Downtime

Do your product dashboards look funky? Are your quarterly reports way off? Are you sick and tired of running a SQL query only to discover that the dataset you’re using is broken or just plain wrong? These errors are highly costly and affect almost every team, yet they’re typically only addressed on an ad hoc basis and in a reactive manner.

As companies increasingly rely on data to lead operations and drive decision making, you need to ensure that your data pipelines are consistently healthy and reliable. In the same way that software developers tackle application downtime, data professionals deal with their own set of availability challenges. In other words, data downtime—the periods of time when your data is partial, erroneous, missing, or otherwise inaccurate. To identify and eliminate data downtime, teams must leverage the five pillars of data observability and embrace automated checks to monitor pipeline performance.

Join experts Barr Moses and Ryan Kearns to learn how to minimize data downtime and increase observability into your data ecosystem. You’ll explore the concept of data downtime and see how to measure it to determine the quality and health of your data using SQL, a sample data table, and a Jupyter notebook. From there, you’ll apply software engineering principles of observability to your data through five key pillars of data health—volume, schema, lineage, freshness, and distribution—as you set service-level objectives for data observability in your data table and implement basic data observability checks. You’ll end by creating your very own anomaly detection algorithm that will help capture data downtime incidents in your data table.