What Is MLOps?

Chapter 4. Governance

In many ways, governance is the backbone of MLOps. It is the set of controls placed on a business to ensure that it delivers on its responsibilities to all stakeholders, from shareholders and employees to the public and national governments. These responsibilities include financial, legal, and ethical obligations. Underpinning all three of these responsibilities is the fundamental principle of fairness. All of these components are critical parts of a robust MLOps system.

This chapter will explore the many components of a modern AI governance strategy and how it’s inherently intertwined with MLOps efforts. It will close out with a template for governance in the context of MLOps, which may be particularly useful for business leaders looking to spearhead governance strategies in their own organizations.

Why Governance Matters to the Business

What most businesses want from governance is to safeguard shareholder investment and to help ensure a suitable return on investment (ROI), both now and in the future. That means the business has to perform effectively, profitably, and sustainably. The shareholders need clear visibility that customers, employees, and regulatory bodies are happy, and they want reassurances that appropriate measures are in place to detect and manage any difficulties that could occur in the future.

If businesses and governments want to reap the benefits of ML, then they have to safeguard the public trust in it as well as proactively address the risks. For businesses, this means developing strong governance of their MLOps process. They must assess the risks and determine their own set of fairness values, and they must implement the necessary process to manage these. Much of this is simply about good housekeeping with an added focus on mitigating the inherent risks of ML, addressing topics such as data provenance, transparency, bias, performance management, and reproducibility.

But governance isn’t a free lunch; it takes effort, discipline, and time.

From the perspective of the business stakeholders, governance is likely to slow down the delivery of new models, which may cost the business money. But it’s also important for the business side to recognize what governance looks like to a data scientist, which is a lot of bureaucracy that erodes their ability to get things done.

Types of Governance

Applying good governance to MLOps is challenging. The processes are complex, the technology is opaque, and the dependence on data is fundamental. Governance initiatives in MLOps broadly fall into one of two categories:

Data governance

A framework for ensuring appropriate use and management of data.

Process governance

The use of well-defined processes to ensure that all governance considerations have been addressed at the correct point in the life cycle of the model, and that a full and accurate record has been kept.

Data Governance

Data governance, which concerns itself with the data being used—especially for model training—addresses questions like:

• What is the data’s provenance?
• How was the original data collected and under what terms of use?
• Is the data accurate and up to date?
• Is there Personally Identifiable Information (PII) or other forms of sensitive data that should not be used?

AI projects usually involve significant pipelines of data cleaning, combination, and transformation. Understanding the data lineage is complex, and anonymizing or pseudo-anonymizing data is not always a sufficient solution to managing personal information. If not performed correctly, it can still be possible to single out an individual and their data.

In addition, inappropriate biases in models can arise quite accidentally despite the best intentions. The point is that making predictions based on past experience is a powerful technique, but sometimes the consequences are not only counterproductive, they are illegal.

Process Governance

The second type of governance is process governance, which focuses on formalizing the steps in the MLOps process and associating actions with those.

Today, process governance is most commonly found in organizations with a traditionally heavy burden of regulation and compliance, such as finance. Outside of these organizations, it is rare. With ML creeping into all spheres of commercial activity, and with rising concern about Responsible AI, we will need new and innovative solutions that can work for all businesses.

Those responsible for MLOps must manage the inherent tension between different user profiles, striking a balance between getting the job done efficiently, and protecting against all possible threats. This balance can be found by assessing the specific risk of each project and matching the governance process to that risk level. There are several dimensions to consider when assessing risk, including:

• The audience for the model
• The lifetime of the model and its outcomes
• The impact of the outcome

This assessment should determine not only the governance measures applied, but it should also drive the complete MLOps development and deployment toolchain.

The Right Level of Governance for the Job

A Self Service Analytics (SSA) project, consumed by a small internal-only audience, calls for relatively lightweight governance. Conversely, a model deployed to a public-facing website making decisions that impact people’s lives or company finances requires a very thorough process.

This process would consider the type of KPIs chosen by the business, the type of model-building algorithm used for the required level of explainability, the coding tools used, the level of documentation and reproducibility, the level of automated testing, the resilience of the hardware platform, and the type of monitoring implemented.

But the business risk is not always so clear-cut. An SSA project that makes a decision that has a long-term impact can also be high risk and can justify stronger governance measures. That’s why across the board, teams need well-thought-out, regularly reviewed strategies for MLOps risk assessment (see Figure 4-1 for a breakdown of project criticality and operationalization approaches).

Ultimately, it’s important to understand from the business side that in many ways, governance is not an overarching set of restrictions; rather, it’s a balance that depends on the use case at hand. It’s up to business and tech experts to work together to determine the proper governance standards for projects under an MLOps framework.

Figure 4-1. Choosing the right kind of operationalization model and MLOps features depending on the project’s criticality.

A Template for MLOps Governance

There is no one-size-fits-all solution across businesses, and different use cases within a business justify different levels of management, but the step-by-step approach outlined can be applied in any organization to guide the implementation process.

The process has seven steps:

1. Understand and classify the analytics use cases.

2. Establish responsibilities.

3. Determine governance policies.

4. Integrate policies into MLOps process.

5. Select the tools for centralized governance management.

6. Engage and educate.

7. Monitor and refine.

This section will go through each of the steps in detail, including a simple definition and the “how” of actually implementing the step.

Step 1: Understand and Classify the Analytics Use Cases

This step defines what the different classes of analytics use cases are and, subsequently, what the governance needs are for each.

Consider the answers to the following questions for a representative cross section of analytics use cases. Identify the key distinguishing features of the different use cases and categorize these features. Conflate categories where appropriate. Typically, it will be necessary to associate several categories to each use case to fully describe it:

• What regulations are each use case subject to, and what are the implications? Sector-specific regulations, regional, PII?
• Who consumes the results of the model? The public? One of many internal users?
• What are the availability requirements for the deployed model? 24-7 real-time scoring, scheduled batch scoring, ad hoc runs (self-service analytics)?
• What is the impact of any errors and deficiencies? Legal, financial, personal, public trust?
• What is the cadence and urgency of releases?
• What is the lifetime of the model and the lifetime of the impact of its decision?
• What is the likely rate of model quality decay?
• What is the need for explainability and transparency?

Step 2: Who Is Responsible?

Identify the groups of people responsible for overseeing MLOps governance as well as their roles:

• Engage the whole organization, across departments, from top to bottom of the management hierarchy.
• Peter Drucker’s famous line “Culture eats strategy for breakfast” highlights the power of broad engagement and shared beliefs.
• Avoid creating all new governance structures—look at what structures exist already and try to incorporate MLOps governance into them.
• Get senior management sponsorship for the governance process.

• Think in terms of separate levels of responsibility:

  Strategic

  Set out the vision

  Tactical

  Implement and enforce the vision

  Operational

  Execute on a daily basis

• Consider building a RACI matrix for the complete MLOps process (see Figure 4-2). RACI stands for Responsible, Accountable, Consulted, Informed, and it highlights the roles of different stakeholders in the overall MLOps process. It is quite likely that any matrix you create at this stage will need to be refined later on in the process.

Figure 4-2. A typical RACI matrix for MLOps.

Step 3: Determine the Governance Policies

With an understanding of the scope and objectives for governance now established and the engagement of the responsible governance leaders, it is time to consider the core policies for the MLOps process. This is no small task, and it is unlikely to be achieved in one iteration. Focus on establishing the broad areas of policy and accept that experience will help to evolve the details.

Consider the classification of initiatives from Step 1. What governance measures does the team or organization need in each case?

In initiatives where there is less concern about the risk or regulatory compliance, lighter-weight, cheaper measures may be appropriate. For example, “what if” calculations to determine the number of in-flight meals of different types has relatively little impact—after all, the mix was never right even before the introduction of ML.

Even such a seemingly insignificant use case may have ethical implications as meals are likely to be correlated to religion or gender, which are protected attributes in many countries. On the other hand, the implications of calculations to determine the level of fueling of planes carry substantially greater risk.

Governance considerations can be broadly grouped under the headings in Table 4-1. For each heading, there is a range of measures to consider for each class.

Table 4-1. Governance considerations. Example measures that businesses can take to ensure that they address important governance considerations.

Governance consideration	Example measures
Reproducibility and traceability	Full data snapshot for precise and rapid model reinstantiation or ability to recreate the environment and retrain with a data sample or only record metrics of models deployed
Audit and documentation	Full log of all changes during development including experiments run and reasons for choices made or automated documentation of deployed model only or no documentation at all
Human-in-the-loop sign-off	Multiple sign-offs for every environment move (dev, QA, pre-Prod, Prod)
Pre-production verification	Verify model documentation by hand coding the model and comparing results or full automated test pipeline recreating in production-like environment with extensive unit and end-to-end test cases or automated checks on database, software version, and naming standards only
Transparency and explainability	Use manually coded decision tree for maximum explainability or use regression algorithms explainability tools such as Shapley values or accept opaque algorithms such as neural networks
Bias and harm testing	“Red Team” adversarial manual testing using multiple tools and attack vectors or automated bias checking on specific subpopulations
Production deployment modes	Containerized deployment to elastic scalable HA multinode configuration with automated stress/load testing prior to deployment or a single production server
Production monitoring	Real-time alerting of errors, dynamic multi-arm bandit model balancing, automated nightly retraining, model evaluation, and redeployment or weekly input drift monitoring and manual retraining or basic infrastructure alerts, no monitoring, no feedback-based retraining
Data quality and compliance	PII considerations including anonymization Documented and reviewed column-level lineage to understand the source, quality, and appropriateness of the data Automated data quality checks for anomalies

The finalized governance policies should provide:

1. A process for determining the classification of any analytics initiative. This could be implemented as a checklist or a risk assessment application.

2. A matrix of initiative classification against governance consideration, where each cell identifies the measures required.

Step 4: Integrate Policies into the MLOps Process

Having identified the governance policies for the different classes of initiatives, the measures to implement these need to be incorporated into the MLOps process and the responsibilities for actioning the measures assigned.

While most businesses will have an existing MLOps process, it is quite likely that this has not been defined explicitly but rather has evolved in response to individual needs. Now is the time to revisit, enhance, and document the process. Successful adoption of the governance process can only happen if it is communicated clearly and buy-in is sought from each stakeholder group.

Understand all of the steps in the existing process by interviewing those responsible. Where there is no previous formal process, this is often harder than it sounds—the process steps are often not explicitly defined, and ownership is unclear.

Attempting to map the policy-driven governance measures into the understanding of the process will identify issues in the process very quickly. Within one business there may be a range of different styles of project and governance needs, such as:

• One-off self-service analytics
• Internally consumed models
• Models embedded in public websites
• Models deployed to IoT devices

In these cases, the differences between some processes may be so great it is best to think in terms of several parallel processes. Ultimately, every governance measure for each use case should be associated with a process step and with a team that is ultimately responsible (see Table 4-2).

Table 4-2. Governance steps throughout the AI life cycle process. Example activities and governance considerations for each step in the raw data to ML model process.

Process step	Example activities and governance considerations
Business scoping	Record objectives, define KPIs, and record sign-off: for internal governance considerations
Ideation	Data discovery: data quality and regulatory compliance constraints Algorithm choice: impacted by explainability requirements
Development	Data preparation: consider PII compliance, separation of legal regional scopes, avoid input bias Model development: consider model reproducibility and audibility Model testing and verification: bias and harm testing, explainability, sign
Preproduction	Verify performance/bias with production data Production-ready testing: verify scalability
Deployment	Deployment strategy: driven by the level of operationalization Deployment verification tests Use of shadow challenger or A/B test techniques for in-production verification
Monitoring and feedback	Performance metrics and alerting Prediction log analysis for input drift with alerting

Step 5: Tools for Centralized Governance Management

The MLOps governance process impacts both the complete ML life cycle as well as many teams across the organization. Each step requires a specific sequence of actions and checks to be executed. Traceability of both the development of the model and the execution of governance activities is a complex challenge.

Most organizations still have a “paper form” mindset for process management, where forms are filled in, circulated, signed, and filed. The forms may be text documents, circulated via email, and filed electronically, but the limitations of paper remain. It is hard to track progress, review many projects at once, prompt for action, and remind teams of responsibilities. The complete record of events is typically spread across multiple systems and owned by individual teams, making a simple overview of analytics projects effectively impossible.

While teams will always have tools specific to their roles, MLOps governance is much more effective if the overarching process is managed and tracked from one system. This system should:

• Centralize the definition of the governance process flows for each class of analytics use cases
• Enable tracking and enforcement of the complete governance process
• Provide a single point of reference for the discovery of analytics projects
• Enable collaboration between teams, in particular, the transfer of work between teams
• Integrate with existing tools used for project execution

The workflow, project management, and MLOps tools currently in use can only partially support these objectives. A new category of ML governance tools is emerging to support this need directly and more fully. These tools focus on the specific challenges of ML governance, including:

• A single view on the status of all models (otherwise known as a Model Registry).
• Process gates with a sign-off mechanism to allow ready traceability of the history of decision making.
• Ability to track all versions of a model.
• Ability to link to artifact stores, metrics snapshots, and documentation.
• Ability to tailor processes specifically for each class of analytics use cases.
• Ability to integrate health monitoring from production systems and to track the performance of models against the original business KPIs.

Step 6: Engage and Educate

Without a program of engagement and training for the groups involved in overseeing and executing the governance process, the chances of it being even partially adopted are slim. It is essential that the importance of MLOps governance to the business, and the necessity of each team’s contribution, is communicated. Building on this understanding, every individual needs to learn what they must do, when, and how. This exercise will require considerable documentation, training—and most of all—time.

Start by communicating the broad vision for MLOps governance in the business. Highlight the dangers of the status quo, an outline of the process, and how it is tailored to the range of use cases.

Engage directly with each team involved and build a training program with them. Do not be afraid to leverage their experience to shape not only the training, but also the detailed implementation of their governance responsibilities. The result will be much stronger buy-in and more effective governance.

Step 7: Monitor and Refine

Is the newly implemented governance working? Are the prescribed steps being implemented, and are the objectives being met? What actions should be taken if things are going poorly? How do we measure the gap between today’s reality and where the business needs to be?

Measuring success requires metrics and checks. It requires people to be tasked with monitoring and a way to address problems. The governance process and the way it is implemented will need to be refined over time, based both on lessons learned and evolving requirements (including, as discussed earlier in this chapter, evolving regulatory requirements).

A big factor in the success of the process will be the diligence of the individuals responsible for the individual measures in the process, and incentivizing them is key.

Monitoring the governance process starts with a clear understanding of the key performance metrics and targets—KPIs for governance. These should aim to measure both whether the process is being enacted and if the objectives are being achieved. Monitoring and auditing can be time consuming, so look to automate metrics as far as possible and encourage individual teams to own the monitoring of metrics that relate to their area of responsibility.

It is hard to make people carry out tasks that seemingly deliver nothing concrete to those doing the work. One popular tactic to address this is gamification. This is not about making everything look like a video game, but about introducing incentives for people to carry out tasks where the main benefit is derived by others.

Look to gamify the governance process in simple ways—publishing KPI results widely is the simplest place to start. Just being able to see targets being met is a source of satisfaction and motivation. Leaderboards, whether at the team or individual level, can add some constructive element of competition. For example, people whose work consistently passes compliance checks the first time, or meets deadlines for tasks, should be able to feel their efforts are visible.

For example, GE Aviation developed a low-cost program to have individuals contribute to data quality by rolling out a point system such that each time someone tagged a dataset, created new documentation, created a new dataset, etc., that person would receive a certain number of points. More points unlocked the possibility to pass levels and get exclusive laptop stickers, and they took the competition to the next level by adding a leaderboard so people could see the accumulated points of others. The interest and involvement due to this gamification was undoubtedly a huge piece of the organization’s overall success in removing data silos and becoming a data-driven company.¹

Excessive competition can be disruptive and demotivating. A balance must be struck, and this is best achieved by building up gamification elements slowly over time. Start with the least competition oriented and add new elements one by one, measuring their effectiveness before adding the next.

Monitoring changes in the governance landscape is essential. This might be regulatory, or it might be about public opinion. Those with responsibility for the strategic vision must continue to monitor this as well as have a process to evaluate potential changes.

Finally, monitoring of the process is only worthwhile if issues are acted upon. Establish a process for agreeing on change and for enacting it. Iteration is inevitable and necessary, but the balance between efficiency and effectiveness is hard to find, and many lessons can only be learned the hard way. Build a culture where people see iteration and refinement as a measure of a successful process, not a failed one.

Closing Thoughts

It is hard to separate MLOps from its governance. It is not possible to successfully manage the model life cycle, mitigate the risks, and deliver value at scale without governance. Governance impacts everything from how the business can acceptably exploit ML, the data and algorithms that can be used, to the style of operationalization, monitoring, and retraining.

MLOps at scale is in its infancy. Few businesses are doing it, and even fewer are doing it well—meaning it’s an opportunity for businesses to set themselves apart and get ahead in the race to AI. When planning to scale MLOps, start with governance and use it to drive the process. Don’t bolt it on at the end. Think through the policies; think about using tooling to give a centralized view; engage across the organization. It will take time and iteration, but ultimately the business will be able to look back and be proud that it took its responsibilities seriously.

¹ See “GE Aviation: From Data Silos to Self-Service Analytics” for the story of why (and how) the company upended its approach to analytics.