Creating a Data-Driven Enterprise with DataOps

Chapter 3. Data Warehouses Versus Data Lakes: A Primer

Chapters 1 and 2 introduced the idea of a data-driven organization and defined the concept of DataOps within the context of big data initiatives. Now, it’s time to take a step back and explore some other basic but important concepts. One of our most important tasks at this point is to clearly delineate the differences between data warehouses and data lakes.

When I give talks about self-service data, questions inevitably come up. What distinguishes a data lake from a data warehouse? Do I need to choose between them or do I need both? What are current best practices for setting up the relationship between a data warehouse and a data lake? This chapter answers these questions and more, and delves into a detailed explanation of why augmenting your existing data warehouse with a data lake is the best path to take given the current state of maturity of the various technologies.

Data Warehouse: A Definition

At its most basic, a data warehouse is a central repository for all the data that is collected in an organization’s business systems. Data is extracted, transformed, and loaded (known as ETL) into a data warehouse, which supports applications for reporting, analytics, and data mining on this extracted and curated dataset (Figure 3-1). The previous generation of data infrastructure centered on data warehouses and was based on technologies such as Teradata, Oracle, Neteeza, Greenplum, and Vertica, among others.

Figure 3-1.