Implementing Data Mesh

Table of contents

1. Foreword
2. Preface
   1. Who Is This Book For?
   2. Overview of the Parts and Chapters
   3. What This Book Isn’t
   4. Conventions Used in This Book
   5. O’Reilly Online Learning
   6. How to Contact Us
   7. Acknowledgments
3. I. The Basics
4. 1. Understanding Data Mesh: The Essentials
   1. Making Data Agile
   2. Local Autonomy + Speed = Agility
   3. Solving Today’s Data Challenges
      1. Bridging Data Silos
      2. Shifting Toward Higher Quality Data
      3. Transforming Data Governance
   4. Data Volume, Variety, and Variability
   5. Turning Principles into Practice
   6. Summary
5. 2. Applying Data Mesh Principles
   1. Data Mesh Principles
      1. Data as a Product
      2. Decentralized Domain Ownership
      3. Self-Serve Data Platform
      4. Federated Computational Governance
   2. Defining a “Good” Data Product
      1. Defining a Principled Data Product
      2. Defining a FAIR Data Product
      3. Defining an Enterprise-Grade Data Product
      4. Defining a Valuable Data Product
      5. Defining a Balanced Data Product
      6. Defining a Modern Data Product—More than Just “Regular” Data
      7. Defining a Practical Data Product Lifecycle
      8. Defining a Practical Data Mesh Roadmap
      9. A “Good” Data Product Has an Empowered Data Product Owner
   3. Identifying a Data Product
   4. Summary
6. 3. Our Case Study: Climate Quantum Inc.
   1. Making Climate Data Easier to Find, Consume, Share, and Trust
   2. Introducing Climate Quantum Inc.
   3. Climate Quantum Data Landscape
   4. Applying Climate Quantum Inc. to Your Enterprise
   5. Summary
7. II. Designing, Building, and Deploying Data Mesh
8. 4. Defining the Data Mesh Architecture
   1. Data Product Architecture
      1. Data Product Artifacts
      2. Development Architecture Components
      3. Runtime Architecture Components
      4. Operations Architecture Components
   2. Data Mesh Architecture
      1. Data Mesh Marketplace and Registry
      2. Data Mesh Backbone Services
   3. Climate Quantum Use Case Considerations
   4. Summary
9. 5. Driving Data Products with Data Contracts
   1. Bringing Value Through Trust
   2. Navigating the Data Contract
      1. Going Through the Theory
      2. Stacking Up Good Information
      3. It’s All About Proper Versioning
      4. Keeping It Simple and Semantic
      5. Walking Through an Example: Complementing Tribal Knowledge
   3. What is Data QoS and Why Is It Critical?
      1. Representation
      2. Why Does It Matter?
      3. Why Data Quality Is Not Enough
      4. Service Levels Complement Quality
   4. Applying Data QoS to the Data Contract
      1. Checking Conformity of Measurements
      2. Completeness
      3. Accuracy
      4. Engaging Service Levels
   5. Summary
10. 6. Building Your First Data Product
    1. Anatomy of a Data Product
    2. The Data Contract to the Rescue
    3. Connecting Your Data Sources
    4. Ensuring Higher Data Quality with Observability
    5. Getting Faster Data Discovery
    6. Enabling Operability and Control
    7. Building and Packaging
    8. Summary
11. 7. Aligning with the Experience Planes
    1. The Three Planes
       1. The Infrastructure Plane Remains Key
       2. The Data Product Experience Plane Is Independent
       3. The Mesh Experience Plane Is About Synergy
    2. Building a Capability Model for Each Plane
       1. Capabilities of the Infrastructure Experience Plane
       2. Capabilities of the Data Product Experience Plane
       3. Capabilities of the Mesh Experience Plane
    3. Gathering Key Metrics Through Feedback Loops
    4. Summary
12. 8. Meshing Your Data Products
    1. Registering Your Data Product
    2. Connecting to Your Products in the Mesh
       1. Meshing Data Products Together
       2. Describing Your Data Lineage
       3. Notification Through the Mesh
    3. Summary
13. III. GenAI, Teams, Operating Model, and Roadmap for Data Mesh
14. 9. Running and Operating Your Data Mesh
    1. Making Data Products Discoverable, Observable, and Secure
       1. Data Product Discovery Interface
       2. Data Product Observability Interface
       3. Data Product Control Interface
       4. Data Product Security
    2. Climate Quantum Use Case Considerations
    3. Toward Dynamic Data Products
    4. The Power of Dynamic Data Products
    5. Summary
15. 10. Creating a Data Mesh Marketplace
    1. Challenges with Traditional Data Catalogs
    2. The Data Mesh Marketplace
    3. A Window into Data Mesh
    4. The Consumer User Journey
    5. The Producer User Journey
    6. Self-Publishing of Data Products
    7. The Power of Discoverability
    8. Bootstrapping Your Marketplace
    9. Climate Quantum’s Data Mesh Marketplace
    10. Summary
16. 11. Establishing Data Mesh Governance
    1. Data Governance
    2. Data Product Certification
    3. Federated Data Product Certification
    4. Implementing Data Product Certification
    5. Data Contracts and Certification
    6. Climate Quantum’s Certification Approach
    7. Summary
17. 12. Understanding Data Product Supply Chains
    1. The Modern Manufacturing Supply Chain
    2. The Modern Software Factory
    3. The Data Product Factory
    4. The Data Product Supply Chain
    5. Climate Quantum’s Data Product Supply Chain
    6. Summary
18. 13. Integrating Data Mesh and Generative AI
    1. Generative AI Background
       1. Large Language Models
       2. Embeddings
       3. Vector Databases
       4. Prompts
       5. Challenges
    2. Data Mesh and Generative AI
       1. Improve Data Quality with Generative AI
       2. Elevate Models as Primary Artifacts
       3. Consume Models Inside Data Products
       4. Generate Data Product Code
       5. Complement Data Product Capabilities
    3. Climate Quantum and Generative AI
       1. Simplifying Climate Data Search
       2. Analyzing and Summarizing Climate Data
       3. Creating Disclosure Reports
       4. Generating Code to Simplify Climate Data Processing
    4. Summary
19. 14. Establishing Data Mesh Teams
    1. Team Topologies in Data Mesh
       1. The Data Product Team
       2. Platform Teams
       3. Enabling Teams
    2. Climate Quantum Teams
    3. Summary
20. 15. Defining a Data Mesh Operating Model
    1. Characteristics of an Operating Model
    2. The Operating Model Continuum
       1. Centralized Organizations
       2. Matrixed Organizations
       3. Federated Organizations
       4. Distributed Organizations
    3. Data Mesh Operating Model
    4. Data Mesh Maturity Model
    5. Climate Quantum’s Data Mesh Operating Model
    6. Summary
21. 16. Establishing a Practical Data Mesh Roadmap
    1. Roadmap Structure
       1. Strategy and Roadmap Stream
       2. Technology Stream
       3. Factory Stream
       4. Operating Model Stream
       5. Socialization Stream
       6. Rollout Stream
    2. Climate Quantum’s Data Mesh Roadmap
    3. Summary
22. Index
23. About the Authors