Data Mesh

Chapter 14. Design Managing, Governing, and Observing Data

Data products are long-lived, and with that comes the need to manage their state and govern, debug, and audit them over their lifetime.

This chapter concludes the design of the data product architecture, with a look at the final three affordances that enable managing a data product over the course of its lifetime.

In this chapter I discuss the design of how a data product affords:

• Data product developers to manage its life cycle from inception through many iterations of change, fixes, and evolution

• Being governed and in compliance with global policies

• Being monitored, debugged, and audited

Manage the Life Cycle

Never tell people how to do things. Tell them what to do and they will surprise you with their ingenuity.

George S. Patton

Data mesh’s promise of scale can be fulfilled only if the life cycle of a data product can be managed autonomously, when a data product can be built, tested, deployed, and run without friction and with limited impact on other data products. This promise must remain true while there is interconnectivity between data products—through their input and output data ports, sharing data, or schemas.

In Chapter 10 we looked at the journey of a data product developer in managing the life cycle of a data product with a focus on the use of platform services. It was then evident that the majority of the work around provisioning and managing infrastructure resources or complex mechanisms such ...