Streaming Data Mesh

Table of contents

1. Preface
   1. Who Should Read This Book
   2. Why We Wrote This Book
   3. Navigating This Book
   4. Conventions Used in This Book
   5. Using Code Examples
   6. O’Reilly Online Learning
   7. How to Contact Us
   8. Acknowledgments
      1. Hubert
      2. Stephen
2. 1. Data Mesh Introduction
   1. Data Divide
   2. Data Mesh Pillars
      1. Data Ownership
      2. Data as a Product
      3. Federated Computational Data Governance
      4. Self-Service Data Platform
      5. Data Mesh Diagram
   3. Other Similar Architectural Patterns
      1. Data Fabric
      2. Data Gateways and Data Services
      3. Data Democratization
      4. Data Virtualization
   4. Focusing on Implementation
      1. Apache Kafka
      2. AsyncAPI
3. 2. Streaming Data Mesh Introduction
   1. The Streaming Advantage
      1. Streaming Enables Real-Time Use Cases
      2. Streaming Enables Data Optimization Advantages
      3. Reverse ETL
   2. The Kappa Architecture
      1. Lambda Architecture Introduction
      2. Kappa Architecture Introduction
   3. Summary
4. 3. Domain Ownership
   1. Identifying Domains
      1. Discernible Domains
      2. Geographic Regions
      3. Hybrid Architecture
      4. Multicloud
   2. Avoiding Ambiguous Domains
   3. Domain-Driven Design
      1. Domain Model
      2. Domain Logic
      3. Bounded Context
      4. The Ubiquitous Language
   4. Data Mesh Domain Roles
      1. Data Product Engineer
      2. Data Product Owner or Data Steward
   5. Streaming Data Mesh Tools and Platforms to Consider
   6. Domain Charge-Backs
   7. Summary
5. 4. Streaming Data Products
   1. Defining Data Product Requirements
   2. Identifying Data Product Derivatives
      1. Derivatives from Other Domains
   3. Ingesting Data Product Derivatives with Kafka Connect
      1. Consumability
      2. Synchronous Data Sources
      3. Asynchronous Data Sources and Change Data Capture
      4. Debezium Connectors
   4. Transforming Data Derivatives to Data Products
      1. Data Standardization
      2. Protecting Sensitive Information
      3. SQL
      4. Extract, Transform, and Load
   5. Publishing Data Products with AsyncAPI
      1. Registering the Streaming Data Product
      2. Building an AsyncAPI YAML Document
      3. Assigning Data Tags
      4. Versioning
      5. Monitoring
   6. Summary
6. 5. Federated Computational Data Governance
   1. Data Governance in a Streaming Data Mesh
      1. Data Lineage Graph
      2. Streaming Data Catalog to Organize Data Products
   2. Metadata
      1. Schemas
      2. Lineage
      3. Security
      4. Scalability
   3. Generating the Data Product Page from AsyncAPI
      1. Apicurio Registry
      2. Access Workflow
   4. Centralized Versus Decentralized
      1. Centralized Engineers
      2. Decentralized (Domain) Engineers
   5. Summary
7. 6. Self-Service Data Infrastructure
   1. Streaming Data Mesh CLI
   2. Resource-Related Commands
      1. Cluster-Related Commands
      2. Topic-Related Commands
      3. The domain Commands
      4. The connect Commands
      5. The streaming Commands
      6. Publishing a Streaming Data Product
   3. Data Governance-Related Services
      1. Security Services
      2. Standards Services
      3. Lineage Services
   4. SaaS Services and APIs
   5. Summary
8. 7. Architecting a Streaming Data Mesh
   1. Infrastructure
   2. Two Architecture Solutions
      1. Dedicated Infrastructure
      2. Multitenant Infrastructure
   3. Streaming Data Mesh Central Architecture
      1. The Domain Agent (aka Sidecar)
      2. Data Plane
      3. Control Plane
   4. Summary
9. 8. Building a Decentralized Data Team
   1. The Traditional Data Warehouse Structure
   2. Introducing the Decentralized Team Structure
      1. Empowering People
      2. Working Processes
      3. Fostering Collaboration
      4. Data-Driven Automation
   3. New Roles in Data Domains
      1. New Roles in the Data Plane
      2. New Roles in Data Science and Business Intelligence
10. 9. Feature Stores
    1. Separating Data Engineering from Data Science
    2. Online and Offline Data Stores
    3. Apache Feast Introduction
    4. Summary
11. 10. Streaming Data Mesh in Practice
    1. Streaming Data Mesh Example
    2. Deploying an On-Premises Streaming Data Mesh
       1. Installing a Connector
       2. Deploying Clickstream Connector and Auto-Creating Tables
       3. Deploying the Debezium Postgres CDC Connector
       4. Enrichment of Streaming Data
       5. Publishing the Data Product
    3. Consuming Streaming Data Products
    4. Fully Managed SaaS Services
    5. Summary and Considerations
12. Index
13. About the Authors