Preface

In the 1840s, a Hungarian physician named Ignaz Semmelweis encountered a perplexing challenge while working in the maternity clinic at the General Hospital in Vienna. A significant number of women were succumbing to a mysterious ailment known as “childbed fever,” which plagued many European hospitals.

Semmelweis made a striking observation: the maternity ward overseen by male doctors had a significantly higher mortality rate than the one managed by midwives. Furthermore, he noticed that doctors often proceeded directly from performing autopsies to examining expectant mothers.

After a colleague pricked his own finger while doing an autopsy, resulting in the colleague falling ill and eventually dying, Semmelweis had a revelatory moment: perhaps what killed his colleague might be also killing the women in childbirth.

Semmelweis theorized that contaminants from the cadavers that doctors were operating on and using to teach medical students might be transferring to the women, leading to the fever. To test this hypothesis, he implemented a policy in 1847 that required doctors to wash their hands with a chlorine solution to eliminate what he called “cadaverous particles,” before examining pregnant women.

Following the implementation of this handwashing policy, the maternal mortality rate in the doctors’ ward plummeted from 18% to a mere 2%. However, Semmelweis’s ideas were met with skepticism from the medical community because they challenged the scientific beliefs at the time, and germ theory had not yet been developed.

Semmelweis could offer no theoretical explanation for his findings, and he was mocked and ridiculed. In 1865, Semmelweis suffered from a nervous breakdown, resulting in his being committed to an asylum in Vienna by his colleagues, where he was beaten by guards and tragically died from a gangrenous wound on his right hand just 14 days later, at the age of 47.

The story of Ignaz Semmelweis offers a few valuable insights:

Human behavior is constrained by bias

Embracing new perspectives often challenges our deeply held beliefs. Such changes are frequently met with resistance—even from those equipped with knowledge and influence. It underscores the profound impact of cognitive bias and societal norms on human judgment.

Interconnected systems are impacted greatly by hygiene

The vast and intricate systems we see, such as hospital protocols or childbirth procedures, can be dramatically influenced by elements so minuscule they’re often invisible, such as germs. This highlights the delicate balance and interconnectedness of our world, from the microscopic to the grand scale.

Simple actions can have massive ripple effects

At times, the most straightforward measures, like handwashing, become our most potent solutions. Understanding the methods to mitigate tiny threats can prove pivotal, with ramifications felt on a monumental scale.

What You Can’t See Can Kill You, and the Same Is True for Data

The transformative shift in our understanding of disease causation can be dated back to the 1860s. Louis Pasteur’s revolutionary experiments demonstrated that microorganisms were responsible for fermentation and spoilage, laying the foundation for germ theory and paving the way for monumental advancements such as vaccines, antiseptics, and sterilization techniques.

In marked contrast, Ignaz Semmelweis made essential observations decades earlier but remained largely overlooked due to his lack of a robust scientific theory. The divergence in their legacies—Pasteur’s transformative influence versus Semmelweis’s limited recognition—emphasizes the critical need for both theoretical and practical foundations in tackling complex problems.

Unifying Business, Data, and Code seeks to bridge this very gap in the field of organizational data management and the design of intelligent systems. We aim to furnish you with both a robust theoretical framework and actionable practical tools, applicable whether you’re brainstorming strategies on a whiteboard or coding sophisticated algorithms.

Diverging from books that concentrate on either technical or managerial facets of data and intelligent system design, Unifying Business, Data, and Code takes a holistic stance that merges both strategic perspectives. We’ve discovered that a technically sound strategy lacking managerial integration is doomed to fail—and the reverse is equally true. This synthesis enables you to make better-informed decisions, effectively bridging the divide between IT and business strategy.

Just as neglecting basic hand hygiene had devastating repercussions in Semmelweis’s time, modern organizations face concealed yet significant risks from poor data management. In essence, the primary challenges compromising your organizational data hygiene can be distilled into three categories:

Ambiguity

There are multiple possible interpretations.

Knowledge gaps

Missing information obstructs problem solving.

Blind spots

There is a lack of awareness of ambiguity and knowledge gaps and their effects on organizational outcomes.

This book will guide you through the process of identifying poor data hygiene and the root causes of misalignment that it leads to within your organization. Armed with this understanding, you’ll be equipped to drive innovation and transformation through a strategic data management approach, unlocking the benefits of intelligent system design for superior results.

Hidden Threats to Organizations: A Modern Parallel

In the expansive world of organizational dynamics, hidden levels of granularity shape our actions and decisions, yet remain unseen in our daily routines. This book journeys into these enigmatic depths. True organizational coherence demands the dexterity to zoom out, transcending individual roles and looking at the vast networks that knit an organization together. At the same time, mastering the finesse to “zoom in” becomes crucial to tackle nuanced data challenges. Like the invisible germs Semmelweis grappled with, these subtle issues can reverberate and escalate unpredictably, leading to profound consequences.

As an example, imagine a retail company using data analytics to forecast demand. A column in the database is ambiguously labeled as Total Sales Revenue. An analyst assumes this means Net Sales, the revenue after returns and discounts, but it actually represents Gross Sales, the revenue generated before any expenses. This simple misunderstanding skews the demand forecast, as the report doesn’t take into account the item’s terrible quality and high return rates.

The company ends up overstocking the flawed items and understocking good ones. Inventory costs balloon, customers are left dissatisfied and lose trust in the brand. The flawed decision making results in a hunt for who to blame for the error, and as the culture becomes toxic, top performers who care about the company leave a vacuum of expertise and talent. Like a house of cards, a single ambiguous label can lead the entire organizational strategy to crumble.

Central to our discussion is the idea of concepts, shown in Figure P-1, serving as the foundation of our unifying methodology. While a deeper exploration awaits in later sections, for now you can think of concepts as the vital atoms whose unique configuration and combination creates the elements of our everyday experiences at the second level of granularity shown in Figure P-1: language, processes, and decision making.

Consider data products, shown in Figure P-1, to be our metaphorical “handwashing” solution. Although our unifying principles help pinpoint ambiguity, knowledge gaps, and blind spots, it’s data products that, much like a sanitizing solution, actively cleanse and address these issues in practice.

Imagine your data as a high-quality product on a store shelf. It should be well crafted, easy to use, and comprehensive. In this book, you’ll learn how to elevate your data to that level of quality. We’ll guide you through a standardized process that packages the structure, meaning, and context along with the data itself.

Once your organization begins designing high-quality data products, the benefits of implementing data hygiene can be quite transformative, freeing up teams from putting out fires in a chronically troubled system and enabling them to focus on creating business value, enhancing efficiencies, and innovation excellence.

Additionally, the principles and methodologies we’ve discussed so far set the foundation for something even more powerful: unified intelligence, which is applying the unifying methodology to human and machine learning system design. Chapter 15 introduces unified intelligence. However, before we can even begin to think of using the principles of unifying with AI, we need to get our data in good shape.

Your AI Is Only as Good as Your Data

The axiom “Your AI Is Only as Good as Your Data” serves as a critical pillar of this book, highlighting the inextricable link between data quality and AI efficacy. Our framework builds on the groundbreaking contributions of seminal figures in the field—Claude Shannon’s information theory, Alan Turing’s computational models, and Shane Legg and Marcus Hutter’s advancements in reinforcement learning. Their collective insights merge seamlessly into our comprehensive methodology, which we will explore in detail in Chapter 15.

Data scientists leverage rigorous methodologies and empirical reasoning to dissect complex challenges and represent them in a structured format. This facilitates the deployment of machine learning algorithms and the construction of predictive models. In this book, we introduce the concept of designing intelligence—a synthesized set of best practices aimed to equip both technical experts and managerial staff with a robust skill set in data-centric problem solving.

Figure P-1. Three levels of granularity are shown in this illustration, each with issues that the unifying methodology will address at each level of granularity. The key activity you will be learning is to identify and minimize ambiguity, knowledge gaps, and blind spots to align all three levels.

Adopting these best practices doesn’t merely set the stage for successful AI initiatives; it transforms your entire organizational data culture, cultivating a fertile ground for data-centric innovation across your organization grounded in principles of designing intelligence.

Aligning Problem-Solving Strategies, Data, and AI

Reinforcement learning serves as a critical pillar in understanding principles of designing intelligent systems, guiding decision-making strategies that oscillate between exploration for new knowledge and exploitation of existing knowledge. As illustrated in Figure P-2, this dynamic reflects human and organizational tendencies to balance effort against reward, thereby shaping the innovation and efficiency strategies of companies.

Figure P-2. This diagram presents a cycle of decision making based on outcomes from exploiting current data and exploring new data. Organizations or individuals can use this model to determine when to rely on existing knowledge (exploit) and when to seek out new information or try new approaches (explore).

Too often, organizational leaders are ensnared in a narrow, top-down mindset that prioritizes exploitation strategies over exploration. This culminates in vague visions that rarely manifest into tangible innovation. When these approaches fall short, it’s usually the workforce that suffers the consequences, from blame and job loss to unsettling structural shifts. This book offers a suite of strategic and technical tools aimed at breaking this detrimental cycle, moving beyond short-term fixes to achieve sustainable progress.

This book encapsulates our insights from personal exploration and exploitation journeys—knowledge we find crucial to share. We’re deeply grateful for your investment in this work. Our aspiration is that, by the end, the principles we unveil will resonate so deeply that their application becomes as intuitive and vital as washing your hands.

A New Paradigm to Optimize Data Management and Business Strategy for the Age of AI

Recognize that unlearning is the highest form of learning.

Rumi, Persian poet

Unifying challenges conventional approaches with a cutting-edge approach: it uses principles from data science used in problem solving to optimize data and knowledge for creating business value. This strategy ensures that your organization will be maximally primed for success in AI endeavors.

Whether you’re dealing with human decision making or computational systems, this book offers a practical blueprint for smarter operations:

  • Strategies and technologies unifying data management and business strategy are presented in Chapters 1–14.

  • The foundational theoretical principles from the fields of artificial intelligence, cognitive psychology, that were used to create the unifying methodology, are covered in Chapter 15.

  • Building upon your unified data management and business strategy and the principles of designing intelligent systems, Chapter 16 explores different ways to apply unifying with AI.

In the pursuit of understanding and harnessing the power of data for business strategy, it’s crucial to keep an open mind—to entertain various hypotheses and embrace the uncertainty created when experiencing new ways of thinking.

As Hala Nelson asserts in Essential Math for AI (O’Reilly, 2023), “Data is the fuel that powers most AI systems” and “What I did not know, and learned the hard way, was that getting real data was the biggest hurdle.”

The methodology elucidated in this book empowers you to apply data science principles and problem-solving strategies effectively without needing to be a data scientist, ensuring that the data you create and collect is not only more accurate and useful, but also a closer reflection of reality.

By embracing the principles you will learn in this book, you will not just be able to solve existing problems better than ever before—you’ll preempt future ones from existing in the first place.

The Origin Story of Unifying

Driven by his work in AI within the edtech sector, Ron harbored an insatiable curiosity to understand principles of designing intelligence that underpin both human and machine learning systems. He envisioned organizations not merely as static structures, but as dynamic ecosystems where information networks intermingle much like the notes in a symphony.

Enter Juan, a leading expert in JSON, JSON Schema, and data serialization. Juan wasn’t just technically proficient; he had the unique ability to take Ron’s grand vision and turn it into a finely tuned reality. Juan’s award-winning research in data serialization at the University of Oxford revealed he could apply the methodology all the way down to the binary level and all the way up to gold-standard protocols for a global-scale data specification.

Our partnership was nothing short of magical—akin to a musical band discovering perfect harmony among its members. Together, we embarked on an unceasing journey of growth and innovation, each challenging and enriching the other’s domain expertise. This book represents the zenith of our collaborative efforts, serving as a comprehensive guide that harmonizes overarching strategies with granular technical solutions for organizations.

We wrote this book with a singular, transformative purpose in mind: to empower people with bold guiding principles and technical strategies that can cut through seemingly impossible problems by unifying people, processes, and data across multiple, and seemingly invisible, scales. We want to democratize this knowledge, to make it accessible and actionable for all, unleashing waves of creativity and ingenuity to transform the world for the better.

The quest to explore and codify the principles of unifying led us into the realms of the mysterious and unknown. Sharing the wisdom we’ve garnered along this journey brings us the incomparable joy of serving a purpose far greater than ourselves.

Orchestrating Alignment at Organizational Scale

Historically, the paradigm shift from attributing illnesses to supernatural causes to understanding them as results of bacteria and viruses wasn’t just a leap in knowledge. It required a massive change in practices, behaviors, and beliefs. In a similar vein, organizations today need to shift from seeing challenges as unsolvable mysteries to recognizing them as tangible problems that can be addressed with the right strategies and methodologies.

Unifying serves as a vital framework designed to demystify the intricate challenges organizations face—challenges rooted in misalignment and silos among business, data, and coding teams. Informed by Figure P-1, our methodology orchestrates alignment across three crucial scales of granularity: the organizational scale, which encapsulates the broad view of roles and networks; the human experience scale, focusing on language, processes, and decision making; and the data product scale, the frontline where data hygiene and quality are actively managed. As you journey through this book, we’ll explore these scales in granular detail, guided by the following pillars:

Theory

Establishing the underpinning philosophical shift and vocabulary essential for evolving data management and intelligent systems. Think of this as the fundamental why and what that lays the foundation for change.

Strategy

Offering a blueprint for practical application, this high-level guidance navigates the how, outlining steps to implement the new paradigm across the scales.

Tools

These are your translators that convert business logic into actionable technical language. Comprising nontechnical, tactical activities, these tools serve as the bridge between strategy and implementation. Tools are tactics to eliminate ambiguity, knowledge gaps, and blind spots. However, there are multiple ways to do this, and we provide you with templates and suggestions.

Implementation

This is the doing phase where coding practices are employed to manifest the methodology in real-world, technical environments.

By seamlessly merging theory, strategy, tools, and implementation, unifying elevates your organization’s approach to data management and designing intelligent systems to unparalleled heights. This is not just about identifying the pitfalls of poor data hygiene—like ambiguities, knowledge gaps, and blind spots—but about systematically rectifying them at every scale of your organization.

Unifying transcends silos, enabling a holistic alignment that harmonizes the macro view of organizational roles and networks with the nuanced details of human experiences and data product quality. The ultimate takeaway? A transformative impact that not only optimizes your data for AI applications but also fuels a culture of ceaseless innovation and excellence. You’ll be able to navigate the labyrinth of challenges with the finesse of a maestro, orchestrating a symphony of meaningful change.

The question isn’t whether you can afford to implement these strategies; it’s whether you can afford not to.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This element indicates a warning or caution.

O’Reilly Online Learning

Note

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

We have a web page for this book, where we list errata, examples, and any additional information: https://oreil.ly/unifying_business_data_and_code_1e.

For news and information about our books and courses, visit https://oreilly.com.

Find us on LinkedIn: https://linkedin.com/company/oreilly-media

Follow us on Twitter: https://twitter.com/oreillymedia

Watch us on YouTube: https://youtube.com/oreillymedia

Acknowledgments

Ron Itelman would like to thank:

Stephanie Itelman, for showing me the strength of your character, the empathy of your heart, the wit in your mind, and the power of your being. You enabled, supported, inspired, and challenged me every step of the way. Thank you, baby, for giving me the gift of sharing life with you, and the experience of creating a family full of laughter and love.

Reuven and Zehava Itelman, for giving me the opportunity to develop experience in experimenting with innovation strategies holistically across a business.

Michael Kaplan, for being a mentor and a guide, teaching me the true meaning of wisdom.

Stephanie Golinveaux, for being a beacon of light which has transformed my life for the better.

Don Houde, for showing me what excellence in management means, and that there are those who will truly develop, nurture, and invest in their teams.

Ole Bagneux, for believing in me enough to introduce me to Aaron Black, creating this opportunity and mentoring this book and me along the authoring journey.

Jim Knickerbocker, PhD, for being a warrior and a thought leader; your belief and support in me has been pivotal in my professional development.

Sean Goodpasture, for believing in me, for being a champion, mentor, friend, and brainstorming partner. You’re also greatly appreciated when presenting at a data conference featuring a T. rex to demonstrate the intersection of UX, psychology, and AI ;).

Anthony Marquardt, for investing your time and efforts in mentorship, demonstrating apex qualities that blend the art of business, technology, and empathy.

Ben Rolnik, for opening up doors that have changed our trajectory and created opportunities to change the world.

Shawna Strickland, who is a brilliant ray of calming, grounded light with sharp business acumen, exemplifying qualities I aspire to emulate.

Karl Friston, whose encouragement banished my personal limitations that were holding me back from pursuing my personal missions in scientific inquiry with unabashed curiosity.

Laura Pionek, for seeing something in me and giving me the opportunity to express my creativity and curiosity, a major catalyst on this journey.

Aaron Black, for taking a chance on us; you’ve changed our lives. Thank you for giving us the opportunity almost no one gets—the ability to write a book about one’s passion.

Corbin Collins, for guiding us and this book through maelstroms to sunny shores. Dealing with authors is probably not easy, and you helped carry the weight of people’s dreams in your hands to make sure we succeeded.

Juan Cruz Viotti would like to thank:

Darlene Colque Roman, for being by my side every evening I spent writing this book. You make me a better man, and your love brings so much joy, purpose, and balance to my life.

Karina Viotti, for teaching me to believe in myself, have the courage to aim high, and that not even the sky’s the limit.

Perla Viotti, for cultivating the habit of reading in me since I was a child and buying me countless O’Reilly books when I was a teenager.

Julian Berman, Greg Dennis, Ben Hutton, Jason Desrosiers, Benjamin Granados, and Henry Andrews for welcoming me to the JSON Schema community and teaching me most of what I know about JSON and JSON Schema.

Aaron Black and Corbin Collins, for providing so much help and guidance, making the daunting process of writing a book so smooth, enjoyable, and fun.

Get Unifying Business, Data, and Code now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial