Ten Things to Know About ModelOps

Ten Things to Know About ModelOps

by Thomas Hill, Mark Palmer, Larry Derany
Released June 2022
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781098133467

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

The past few years have seen significant developments in data science, AI, machine learning, and advanced analytics. But the wider adoption of these technologies has also brought greater cost, risk, regulation, and demands on organizational processes, tasks, and teams. This report explains how ModelOps can provide both technical and operational solutions to these problems.

Thomas Hill, Mark Palmer, and Larry Derany summarize important considerations, caveats, choices, and best practices to help you be successful with operationalizing AI/ML and analytics in general. Whether your organization is already working with teams on AI and ML, or just getting started, this report presents ten important dimensions of analytic practice and ModelOps that are not widely discussed, or perhaps even known.

In part, this report examines:

  • Why ModelOps is the enterprise "operating system" for AI/ML algorithms
  • How to build your organization's IP secret sauce through repeatable processing steps
  • How to anticipate risks rather than react to damage done
  • How ModelOps can help you deliver the many algorithms and model formats available
  • How to plan for success and monitor for value, not just accuracy
  • Why AI will be soon be regulated and how ModelOps helps ensure compliance

Table of contents

  1. 1. ModelOps
    1. Overview of ModelOps
      1. Data-Driven Organizations
      2. Extracting Information from Data via Models
      3. The Model Building/Authoring Life Cycle
      4. What Are ModelOps Tools?
  2. 2. Ten Things to Know About ModelOps
    1. 1. ModelOps Is the Enterprise “Operating System” for AI/ML Algorithms
      1. Many Types of Models
      2. Many Different People/Roles Build Models
    2. 2. There Are Many Algorithms and Many Model Formats: Deliver Them All
      1. Model Formats
      2. Support Your Model Builders, Don’t Constrain Them
    3. 3. First, Do No Harm: Anticipate Risks Rather Than React to Damage Done
      1. Offending Customers
      2. Risky Predictions for Manufacturing
      3. Considering Wrong Predictions
      4. Anticipating Risk
    4. 4. Aim to Build Your Organization’s IP Secret Sauce Through Repeatable Processing Steps
      1. Managing and Accumulating IP Through Models
      2. Disorganized Analytics Across Silos
    5. 5. ModelOps Is About Maintaining and Expanding Your IP
      1. Model Decay (Drift): Changing Populations, Data, and Relationships
      2. Flexible Monitoring of End-to-End Scoring Pipelines and Agile Deployment
      3. The Bane of AI/ML at Scale: Big Code
      4. Governed Inputs/Outputs; Small Reusable Steps; and Efficient, Targeted Monitoring That Enables Automation
    6. 6. The World Is Changing Fast: Think Agile Model Deployment and Streaming Data
      1. Batch and Real-Time Data—Use Cases
      2. Responding to a Dynamically Changing World
    7. 7. Plan for Success and Monitor for Value (Not Just Accuracy)
      1. Every Project Starts at the End—or It Is Likely to Fail!
      2. Planning for How Model Scoring Is to Be Consumed—and Valuable
    8. 8. The Future of AI Will Be Regulated—ModelOps Helps Ensure Compliance
      1. Requirements for Analytics in Medical Device and Pharma Manufacturing Offer Guidance for What’s to Come
      2. ModelOps Capabilities, Beyond Efficient Deployment and Scoring
    9. 9. Expect Quality and Process Best Practices from the ModelOps Operating System
    10. 10. ModelOps Is a Required Tool for Managing AI/ML Model Deployment: Start Small, but Start Now
  3. References
  4. About the Authors

Product information

  • Title: Ten Things to Know About ModelOps
  • Author(s): Thomas Hill, Mark Palmer, Larry Derany
  • Release date: June 2022
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098133467