Agile Data Warehousing for the Enterprise

Agile Data Warehousing for the Enterprise

by Ralph Hughes
Released September 2015
Publisher(s): Morgan Kaufmann
ISBN: 9780123965189

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Building upon his earlier book that detailed agile data warehousing programming techniques for the Scrum master, Ralph's latest work illustrates the agile interpretations of the remaining software engineering disciplines:

  • Requirements management benefits from streamlined templates that not only define projects quickly, but ensure nothing essential is overlooked.
  • Data engineering receives two new "hyper modeling" techniques, yielding data warehouses that can be easily adapted when requirements change without having to invest in ruinously expensive data-conversion programs. 
  • Quality assurance advances with not only a stereoscopic top-down and bottom-up planning method, but also the incorporation of the latest in automated test engines. 

Use this step-by-step guide to deepen your own application development skills through self-study, show your teammates the world's fastest and most reliable techniques for creating business intelligence systems, or ensure that the IT department working for you is building your next decision support system the right way.

  • Learn how to quickly define scope and architecture before programming starts
  • Includes techniques of process and data engineering that enable iterative and incremental delivery
  • Demonstrates how to plan and execute quality assurance plans and includes a guide to continuous integration and automated regression testing
  • Presents program management strategies for coordinating multiple agile data mart projects so that over time an enterprise data warehouse emerges
  • Use the provided 120-day road map to establish a robust, agile data warehousing program

Table of contents

  1. Cover image
  2. Title page
  3. Table of Contents
  4. Copyright
  5. Advance Praise for Agile Data Warehousing for the Enterprise
  6. List of Figures
  7. List of Tables
  8. Abbreviations
    1. Development Team Roles
  9. Foreword
  10. Acknowledgments
  11. Chapter 1. Solving Enterprise Data Warehousing’s “Fundamental Problem”
    1. The Agile Solution in a Nutshell
    2. Five Legs to Stand Upon
    3. The Agile EDW Alternative is Ready to Deploy
    4. Defining a Baseline Method for Agile EDW
    5. Plenty of Motivation to “Go Agile”
    6. Structure of the Presentation Ahead
    7. Summary
  12. Part I: Summaries of Generic Agile Development Methods
    1. Chapter 2. Primer on Agile Development Methods
      1. Defining “Agile”
      2. Agile Manifesto Values and Principles
      3. Scrum in a Nutshell
      4. Contributions from Extreme Programming
      5. Summary
    2. Chapter 3. Introduction to Alternative Iterative Methods
      1. Lean Software Development
      2. Kanban
      3. The Hybrid “Scrumban” Approach
      4. Rational Unified Process
      5. Summary
    3. Part I References
      1. Chapter 2
  13. Part II: Review of Fast EDW Coding and Risk Mitigation
    1. Chapter 4. Essential DW/BI Background and Definitions
      1. Primary Source for DW/BI Standards
      2. Basic Business Terms
      3. Data and Information Terms
      4. Information Services Terms
      5. Software Engineering Terms
      6. Basic Architectural Concepts
      7. Architectural Frameworks
      8. Additional Data Warehousing Concepts
      9. Traditional Project Management Terms
      10. Summary
    2. Chapter 5. Recap of Agile DW/BI Coding Practices
      1. Iterative Coding Alone Significantly Improves BI Projects
      2. New Roles for DW/BI Projects
      3. 80/20 Specifications
      4. Developer Stories
      5. Current Estimates
      6. Adding Techniques from Kanban
      7. Evidence-Based Service Level Agreements
      8. Proof that Agile DW/BI Works
      9. Summary
    3. Chapter 6. Eliminating Risk Through Nested Iterations
      1. EDW Programs Slip into “231 Swamps”
      2. Agile’s Fundamental Risk Mitigation Technique
      3. Agile Edw’s Extended Risk Mitigation Techniques
      4. Summary
    4. Part II References
      1. Chapter 4
  14. Part III: Agile EDW Requirements Management
    1. Chapter 7. Balancing between Two Extremes
      1. Building the Case for Effective Requirements Management
      2. Easy to Overinvest in Requirements Management
      3. Reasons Not to Overinvest in Requirement Work
      4. Agile’s Approach Centers on Balance
      5. Two Intersecting Requirements Management Value Chains
      6. Business Analysts Implicit in Two Project Lead Roles
      7. Summary
    2. Chapter 8. Redefining the Epic Stack to Enable Value Accounting
      1. Toward a Robust Epic Decomposition Framework
      2. Testing Whether Stories are Good Enough
      3. Clarifying Everything with Value Accounting
      4. Allocating Value Throughout an Epic Tree
      5. Value Buildups by Environment Provide Motivation and Clarity
      6. Summary
    3. Chapter 9. Artifacts for the Generic Requirements Value Chain
      1. Beware of Requirements Churn
      2. User Modeling/Personas
      3. End Users’ Hierarchy of Needs
      4. Mind Maps and Fishbone Diagrams
      5. Vision Boxes
      6. Vision Statements
      7. Product Roadmaps
      8. Summary
    4. Chapter 10. Artifacts for the Enterprise Requirements Value Chain
      1. The Generic Value Chain Can Overlook Crucial Requirements
      2. ERM as a Flexible RM Approach
      3. Focusing on Enterprise Aspects of Project Requirements
      4. Uncovering Project Goals with Sponsor’s Concept Briefing
      5. Identifying Project Objectives with Stakeholder’s Requests
      6. Sketching the Solution with a Vision Document
      7. Segmenting the Project with Subrelease Overview
      8. Providing Developer Guidance with Module Use Cases
      9. Summary
    5. Chapter 11. Intersecting Value Chains for a Stereoscopic Project Definition
      1. Intersecting the Two Value Chains
      2. Addressing Nonfunctional Requirements
      3. Supporting the Organization’s Software Release Cycle
      4. Techniques for the Elaboration Phase
      5. Prioritizing Project Backlogs
      6. Managing Incremental Precision
      7. Effort Levels by Team Roles
      8. Conquering Complex Business Rules with an Embedded Method
      9. Interfacing with Project Governance
      10. Not Returning to a Waterfall Approach
      11. Summary
    6. Part III References
      1. Chapter 7
  15. Part IV: Agile EDW Data Engineering
    1. Chapter 12. Traditional Data Modeling Paradigms and Their Discontents
      1. EDW at a Crossroads
      2. Models, Architectures, and Paradigms
      3. Normalization Basics
      4. Generalization Basics
      5. The Standard Approach and its Data Modeling Paradigms
      6. The Traditional Integration Layer as a Challenged Concept
      7. “Straight-To-Star” as a Controversial Alternative
      8. Four Change Cases for Appraising a Data Modeling Paradigm
      9. Summary
    2. Chapter 13. Surface Solutions Using Data Virtualization and Big Data
      1. Leveraging Shadow it
      2. Faster Value Delivery with Data Virtualization
      3. An Agile Role for Big Data
      4. Summary
    3. Chapter 14. Agile Integration Layers with Hyper Normalization
      1. Hyper Normalization Hinges on “Ensemble Modeling”
      2. Hyper Normalized Data Modeling Concepts
      3. Reusable ETL Modules Accelerate New Development
      4. Common Data Retrieval Challenges and Their Solutions
      5. Re-Architecting the EDW for Hyper Normalization
      6. Enabling Evolution of Existing EDW Components
      7. HNF-Powered Agile Solutions
      8. Evidence of Success
      9. Summary
    4. Chapter 15. Fully Agile EDW with Hyper Generalization
      1. Hyper Generalization Involves a Mix of Modeling Strategies
      2. HGF Enables Model-Driven Development and Fast Deliveries
      3. Loading Data into the Hyper Generalized Integration Layer
      4. Retrieving Information from a Hyper Generalized EDW
      5. Model-Driven Evolution and Fast Adaptation
      6. Supporting Derived Elements
      7. Addressing Performance Concerns
      8. Demonstrating Agility Through Four Change Cases
      9. HGF-Powered Agile Solutions
      10. Evidence of Success
      11. Summary
    5. Part IV References
      1. Chapter 12
  16. Part V: Agile EDW Quality Management Planning
    1. Chapter 16. Why We Test and What Tests to Run
      1. Why Test?
      2. An Agile Approach to Quality Assurance
      3. “What to Test?” Answered with Top-Down Planning
      4. A 2×2 Planning Matrix for Top-Down Test Selection
      5. “What to Test?” Answered Bottom-Up
      6. Summary
    2. Chapter 17. Designating Who, When, and Where
      1. Who Shall Write the Tests?
      2. When Should Teammates Perform Their QA Duties?
      3. Where Should Teammates Perform Their QA Duties?
      4. Key Quality Responsibilities by Team Role
      5. The Overarching Duties of the System Tester
      6. How Many Testers are Needed?
      7. Summary
    3. Chapter 18. Deciding How to Execute the Test Cases
      1. Good Agile Quality Plans Involve Numerous Test Executions
      2. Step 1: Update the Top-Down Plan
      3. Step 2: Start Building the Parameter-Driven Widgets
      4. Step 3: Plan Out the Test Data Sets
      5. Step 4: Implement the Engine, Whether Manual or Automated
      6. Step 5: Define the Project’s Set of Testing Aspects
      7. Step 6: Build and Populate the Test Data Repository
      8. Step 7: Quantify the Testing Objectives
      9. Step 8: Begin Creating Test Cases
      10. Step 9: Start Up the Engine
      11. Step 10: Visualize Project Progress with Quality Assurance
      12. Step 11: Document the Team’s Success
      13. Summary
    4. Part V References
      1. Chapter 16
  17. Part VI: Integrating the Pieces of the Agile EDW Method
    1. Chapter 19. The Agile EDW Subrelease Cycle
      1. Making the Release Cycle a Repeatable Process
      2. Traditional Notions of Data Governance
      3. The Agile EDW Subrelease Value Cycle
      4. Centering the Value Cycle on Data Governance and Quality
      5. Guiding the Agile EDW Transition
      6. Summary
    2. Part VI References
      1. Chapter 19
  18. Index

Product information

  • Title: Agile Data Warehousing for the Enterprise
  • Author(s): Ralph Hughes
  • Release date: September 2015
  • Publisher(s): Morgan Kaufmann
  • ISBN: 9780123965189