Data Contracts

Table of contents

1. Brief Table of Contents (Not Yet Final)
2. 1. Why the Industry Now Needs Data Contracts
   1. Garbage-In Garbage-Out Cycle
      1. Modern Data Management
      2. What is data debt?
      3. Garbage In / Garbage Out
   2. The Death of Data Warehouses
      1. The Pre-Modern Era
      2. Software Eats the World
      3. A Move Towards Microservices
      4. Data Architecture in Disrepair
   3. Rise of the Modern Data Stack
      1. The Big Players
      2. Rapid Growth
      3. Problems in Paradise
   4. The Shift to Data-centric AI
      1. Diminishing ROI of Improving ML Models
      2. Commoditization of Data Science Workflows
      3. Data’s Rise Over ML in Creating a Competitive Advantage
   5. Conclusion
   6. Additional Resources
   7. References
3. 2. Data Quality Isn’t About Pristine Data
   1. Defining Data Quality
   2. OLTP Versus OLAP and Its Implications for Data Quality
      1. A Brief Summary of OLTP and OLAP
      2. Translation Issues Between OLTP and OLAP Data Worldviews
   3. The Cost of Poor Data Quality
      1. Measuring Data Quality
      2. Who Is Impacted
   4. Conclusion
   5. Additional Resources
   6. References
4. 3. The Challenges of Scaling Data Infrastructure
   1. How Data Development Is Not Like Software Development
      1. How Software Engineers Build Products
      2. How Data Developers Build Products
   2. Core Challenges for Modern Data Engineering Teams
   3. Why Data Development Needs a Design Surface
      1. Prevention first
      2. Communicative
      3. Contextual
      4. At the right time
      5. Including the right people
   4. The Cost of Large-Scale Refactors
      1. Large-Scale Refactor Considerations
      2. Use Case: Alan’s Large-Scale Refactor
   5. The Dangers of Database Migrations
      1. Data Loss
      2. Introduction of Data Quality Issues
      3. Massive Amounts of Change Management
      4. Staff Pulled Away From Main Roles
      5. Untangling Business Logic is Painful
      6. Data debt pain
      7. Changing business models :
      8. Regulatory changes:
      9. Skyrocketing cloud costs :
      10. Opportunities with new technologies :
   6. The Role of Change Management in Data Quality
      1. The Entropic Behavior of Data
      2. How Data Drifts from Established Business Logic
      3. Change Management Needs to Align with the Needs of the Business for It to Be Accepted
   7. How Infrastructure Needs Change at Scale
      1. Dunbar’s Number & Conway’s Law
      2. Case Study: Atlassian Engineering Team
      3. How Data Contracts Enable Change Management at Scale
   8. Conclusion
   9. Additional Resources
   10. References
5. 4. An Introduction to Data Contracts
   1. Collaboration is Different in Data
   2. The Stakeholders You Will Work With
      1. The Role of Data Producers
      2. The Role of Data Consumers
      3. The Impact of Producers and Consumers
      4. The Trials and Tribulations of Data Consumers Managing Data Quality
   3. An Alternative: The Data Contract Workflow
      1. The Data Contract Workflow
      2. An Overview of Where to Implement Data Contracts
      3. The Maturity Curve of Data Contracts: Awareness, Ownership, and Governance
      4. Outcomes of Implementing Data Contracts
   4. Data Contracts vs. Data Observability
   5. Conclusion
   6. Additional Resources
6. 5. The Data Contract Components: Data Assets and Contract Definition
   1. Overview of Components
   2. Data Assets
      1. Analytics Databases
      2. Transactional Databases
      3. Event Sourcing and Event Streams
      4. First-Party Data, Third-Party Platform
   3. Contract Definition
      1. Data Contract Spec
      2. Business Logic
      3. Schema Registry and Data Catalogs
   4. Conclusion
   5. Additional Resources
7. About the Authors