Kubeflow Operations Guide

Chapter 2. Kubeflow Architecture and Best Practices

In this chapter we’ll continue to build on the introductory concepts from Chapter 1 to dive into the architecture of Kubeflow, and then close out the chapter with best practices for using Kubeflow.

In the first part of this chapter, we’ll focus on Kubeflow’s architecture and how the components work together to form the platform. When supporting complex distributed systems, DevOps teams need to have an idea which components are talking to which other components in the system. This helps set context when triaging issues with a Kubeflow cluster.

This architectural background sets up the second part of the chapter, where we’ll introduce best practices for using Kubeflow. Subsequent chapters will focus on either installing Kubeflow on a specific platform or deploying models on KFServing. In both situations you are better served understanding the architecture and how best to use the system as you build out your custom Kubeflow platform.

Kubeflow Architecture Overview

In Figure 2-1, we can see the high-level architecture of Kubeflow. This diagram shows the request flow from the user into components of Kubeflow via the CLI or kubectl to operate the Kubeflow system.

We use the Kubeflow system to set up machine learning workflows that run a job or a series of jobs to build machine learning models. A job in Kubeflow can be a Jupyter Notebook or it can be a Python script in a series of pipeline-connected jobs. A job in Kubeflow can also ...