Effective Machine Learning Teams

Effective Machine Learning Teams

by David Tan, Ada Leung, David Colls
Released February 2024
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781098144630

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Gain the valuable skills and techniques you need to accelerate the delivery of machine learning solutions. With this practical guide, data scientists, ML engineers, and their leaders will learn how to bridge the gap between data science and Lean product delivery in a practical and simple way. David Tan, Ada Leung, and Dave Colls show you how to apply time-tested software engineering skills and Lean product delivery practices to reduce toil and waste, shorten feedback loops, and improve your team's flow when building ML systems and products.

Based on the authors' experience across multiple real-world data and ML projects, the proven techniques in this book will help your team avoid common traps in the ML world, so you can iterate and scale more quickly and reliably. You'll learn how to overcome friction and experience flow when delivering ML solutions.

You'll also learn how to:

  • Write automated tests for ML systems, containerize development environments, and refactor problematic codebases
  • Apply MLOps and CI/CD practices to accelerate experimentation cycles and improve reliability of ML solutions
  • Apply Lean delivery and product practices to improve your odds of building the right product for your users
  • Identify suitable team structures and intra- and inter-team collaboration techniques to enable fast flow, reduce cognitive load, and scale ML within your organization

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Who This Book Is For
    2. How This Book Is Organized
      1. Part I: Product and Delivery
      2. Part II: Engineering
      3. Part III: Teams
    3. Additional Thoughts
    4. Conventions Used in This Book
    5. Using Code Examples
    6. O’Reilly Online Learning
    7. How to Contact Us
    8. Acknowledgments
      1. From David Tan
      2. From Ada Leung
      3. From David Colls
  2. 1. Challenges and Better Paths in Delivering ML Solutions
    1. ML: Promises and Disappointments
      1. Continued Optimism in ML
      2. Why ML Projects Fail
    2. Is There a Better Way? How Systems Thinking and Lean Can Help
      1. You Can’t “MLOps” Your Problems Away
      2. See the Whole: A Systems Thinking Lens for Effective ML Delivery
      3. The Five Disciplines Required for Effective ML Delivery
    3. Conclusion
  3. I. Product and Delivery
  4. 2. Product and Delivery Practices for ML Teams
    1. ML Product Discovery
      1. Discovering Product Opportunities
      2. Canvases to Define Product Opportunities
      3. Techniques for Rapidly Designing, Delivering, and Testing Solutions
    2. Inception: Setting Teams Up for Success
      1. Inception: What Is It and How Do We Do It?
      2. How to Plan and Run an Inception
      3. User Stories: Building Blocks of an MVP
    3. Product Delivery
      1. Cadence of Delivery Activities
      2. Measuring Product and Delivery
    4. Conclusion
  5. II. Engineering
  6. 3. Effective Dependency Management: Principles and Tools
    1. What If Our Code Worked Everywhere, Every Time?
      1. A Better Way: Check Out and Go
      2. Principles for Effective Dependency Management
      3. Tools for Dependency Management
    2. A Crash Course on Docker and batect
      1. What Are Containers?
      2. Reduce the Number of Moving Parts in Docker with batect
    3. Conclusion
  7. 4. Effective Dependency Management in Practice
    1. In Context: ML Development Workflow
      1. Identifying What to Containerize
      2. Hands-On Exercise: Reproducible Development Environments, Aided by Containers
    2. Secure Dependency Management
      1. Remove Unnecessary Dependencies
      2. Automate Checks for Security Vulnerabilities
    3. Conclusion
  8. 5. Automated Testing: Move Fast Without Breaking Things
    1. Automated Tests: The Foundation for Iterating Quickly and Reliably
      1. Starting with Why: Benefits of Test Automation
      2. If Automated Testing Is So Important, Why Aren’t We Doing It?
    2. Building Blocks for a Comprehensive Test Strategy for ML Systems
      1. The What: Identifying Components For Testing
      2. Characteristics of a Good Test and Pitfalls to Avoid
      3. The How: Structure of a Test
    3. Software Tests
      1. Unit Tests
      2. Training Smoke Tests
      3. API Tests
      4. Post-deployment Tests
    4. Conclusion
  9. 6. Automated Testing: ML Model Tests
    1. Model Tests
      1. The Necessity of Model Tests
      2. Challenges of Testing ML Models
      3. Fitness Functions for ML Models
      4. Model Metrics Tests (Global and Stratified)
      5. Behavioral Tests
      6. Testing Large Language Models: Why and How
    2. Essential Complementary Practices for Model Tests
      1. Error Analysis and Visualization
      2. Learn from Production by Closing the Data Collection Loop
      3. Open-Closed Test Design
      4. Exploratory Testing
      5. Means to Improve the Model
      6. Designing to Minimize the Cost of Failures
      7. Monitoring in Production
      8. Bringing It All Together
    3. Next Steps: Applying What You’ve Learned
      1. Make Incremental Improvements
      2. Demonstrate Value
    4. Conclusion
  10. 7. Supercharging Your Code Editor with Simple Techniques
    1. The Benefits (and Surprising Simplicity) of Knowing Our IDE
      1. Why Should We Care About IDEs?
      2. If IDEs Are So Important, Why Haven’t I Learned About Them Yet?
    2. The Plan: Getting Productive in Two Stages
      1. Stage 1: Configuring Your IDE
      2. Stage 2: The Star of the Show—Keyboard Shortcuts
      3. You Did It!
    3. Conclusion
  11. 8. Refactoring and Technical Debt Management
    1. Technical Debt: The Sand in Our Gears
      1. Getting to a Healthy Level of Debt Through Tests, Design, and Refactoring
      2. Refactoring 101
    2. How to Refactor a Notebook (or a Problematic Codebase)
      1. The Map: Planning Your Journey
      2. The Journey: Hitting the Road
      3. Looking Back at What We’ve Achieved
    3. Technical Debt Management in the Real World
      1. Technical Debt Management Techniques
      2. A Positive Lens on Debt: Systems Health Ratings
    4. Conclusion: Make Good Easy
  12. 9. MLOps and Continuous Delivery for ML (CD4ML)
    1. MLOps: Strengths and Missing Puzzle Pieces
      1. MLOps 101
      2. Smells: Hints That We Missed Something
    2. Continuous Delivery for Machine Learning
      1. Benefits of CD4ML
      2. A Crash Course on Continuous Delivery Principles
      3. Building Blocks of CD4ML: Creating a Production-Ready ML System
    3. How CD4ML Supports ML Governance and Responsible AI
    4. Conclusion
  13. III. Teams
  14. 10. Building Blocks of Effective ML Teams
    1. Common Challenges Faced by ML Teams
    2. Effective Team Internals
      1. Trust as the Foundational Building Block
      2. Communication
      3. Diverse Membership
      4. Purposeful, Shared Progress
      5. Internal Tactics to Build Effective Teams
    3. Improving Flow with Engineering Effectiveness
      1. Feedback Loops
      2. Cognitive Load
      3. Flow State
    4. Conclusion
  15. 11. Effective ML Organizations
    1. Common Challenges Faced by ML Organizations
    2. Effective Organizations as Teams of Teams
      1. The Role of Value-Driven Portfolio Management
      2. Team Topologies Model
      3. Team Topologies for ML Teams
      4. Organizational Tactics to Build Effective Teams
    3. Intentional Leadership
      1. Create Structures and Systems for Effective Teams
      2. Engage Stakeholders and Coordinate Organizational Resources
      3. Cultivate Psychological Safety
      4. Champion Continuous Improvement
      5. Embrace Failure as a Learning Opportunity
      6. Build the Culture We Wish We Had
      7. Encourage Teams to Play at Work
    4. Conclusion
      1. Epilogue: Dana’s Journey
  16. Index
  17. About the Authors

Product information

  • Title: Effective Machine Learning Teams
  • Author(s): David Tan, Ada Leung, David Colls
  • Release date: February 2024
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098144630