Introduction

Edith Harbaugh

Operating continuously is the nirvana of software development—a software delivery lifecycle in which planning, coding, building, testing, releasing, deploying, operating, and monitoring processes unite to produce value for end users and less stress and risk for those building the software. We are in a transition phase from traditional project-oriented waterfall processes to product-oriented continuous processes. This transition is exciting but, like any change, also scary. I hope this book explains this natural evolution and gives you examples of cultures, processes, and tooling that can help you successfully embrace operating continuously.

I started my career in the ’90s, working first as a software analyst/designer and then as an engineering program manager, when software was made in rigid waterfall processes. The product manager wrote a product requirement doc (PRD), which I, the analyst/designer in engineering, then translated into use cases and functional requirements (known as functional requirement specifications). Designers would make mock-ups, which engineers would subsequently implement. There were then several rounds of quality assurance (QA) to check for user acceptance (did the product manager think it functioned?), defects, and platform compatibility, since customers were installing the software on their own systems. Part of my role as program manager was to generate and maintain the supported platform matrix, which identified which combinations (and what versions) of operating systems, databases, and application servers were supported.

Shipping in the 1990s and early 2000s was an exercise in evaluating where in the triangle of scope, quality, and time the software was. If there were sufficient features and low enough defects (coupled with pressure from sales and marketing), a release would be deemed “ready.” Following that was a release-to-manufacturing stage in which the bits would be put up on a website for customers to download and install. Since customers adopted a new version probably only once a year or so unless they urgently needed a new feature or bug fix, there was much angst about what level of defects was acceptable, not to mention which features to build to begin with. All of this led to more and more documentation about requirements, and longer and longer release processes, due to software teams trying to pack in new features and bug fixes. As a result, software releases sometimes shipped not just months but years late, completely missing the mark for customers. Duke Nukem Forever is legendary for being 14 years in development.

The tandem rise of Agile and the cloud radically changed software development. If customers didn’t have to download and install a release, but rather the software was hosted, you could release frequently—not just once a year or once a quarter, but literally every minute. In 2001 the authors of the Agile Manifesto preached these four virtues:¹

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The combination of being able to release more quickly due to cloud hosting and a methodology that explicitly rewarded “responding to change” naturally resulted in software organizations rethinking how they viewed software development. This rethinking led to DevOps and to new software lifecycle methodologies such as the DevOps infinity loop, which represents the stages that take place during software development (see Figure I-1).

The software industry has focused an enormous amount of effort on the steps that precede release in the DevOps infinity loop. The Agile movement, the adoption of continuous integration/continuous delivery (CI/CD), and the introduction of DevOps have led to a set of modern practices that take software from concept to “ready to deploy.” While these practices have given us new ways to quickly develop and ship software, they’ve also created challenges in the ways these processes move between teams, and in the model itself. Entire solutions have been created around each stage in the infinity loop, but what if the problem isn’t the individual steps? What if the real problem is that the step-by-step model that the DevOps infinity loop represents has become outdated?

The next stage of evolution in software delivery is happening now. Cutting-edge software teams are developing and introducing new tools, technologies, and processes that extend continuous delivery beyond the deployment stage into software that’s live and running in production. No longer sequential handoffs, these interactions are instead often running as continuous processes within their own stages. I refer to this model as continuous operation.

The practice of DevOps largely grew from the learnings of software development teams, which were applied to broader operations throughout organizations. Similar to how infrastructure teams evolved to adopt developer practices, continuous operation improves the efficiency of the teams involved in operating software platforms. This book documents these shifts and provides guidance for how teams can adopt the approach of continuous operations. We’ll explore the specific stages and areas of the software delivery model and provide guidance on:

The discrete steps in continuous operation and how they fit together
The tools, processes, and people needed to make each step continuous
The business value continuous operation provides

As we detail these emerging practices, it will become clear how important it is to revisit the way teams are building and shipping software and to rethink existing models, such as the DevOps infinity loop and the traditional software development lifecycle (SDLC) model. The end result is a refreshed framework that takes a continuous approach to the key components of software delivery, focusing on deploy, release, operate, and measure and experiment (see Figure I-2).

Continuous operation has three distinct differences from earlier models:

It’s not just about dev and ops: Additional personas, including product managers, designers, security managers, and more, have a role in monitoring and operating software in production.
The steps need a refresh: When viewed narrowly as a DevOps task, the idea of release as a packaging step that occurs before deployment is dated, as is the idea of monitoring.
The steps happen in parallel: Unlike the left side of the DevOps infinity loop, where, for example, planning precedes development, the steps of continuous operation all happen in parallel while the system is in operation.

It’s Never Been Just About Dev and Ops

One of the stated goals of the DevOps movement is to reduce siloing and handoffs. By integrating the practices of developers and operations teams, we can create teams with shared objectives and responsibilities. We can collaborate continuously and eliminate the step-by-step processes and dependencies that create the “baton handoff” motion between teams.

Note

For the purposes of this book, we’ll use the generic term integrated software delivery team, or simply software delivery team, to refer to an organizational structure that integrates all the functions within the software development lifecycle into a collaborative unit.

Developers do not build software in isolation. When I worked as a product manager at TripIt in the 2010s, my team comprised myself, a designer, and engineers. This model of operation is commonly referred to as a squad model, where a cross-functional team comes together to develop as an integrated team. Increasingly in the 2020s, security and operations are baked into every step of the process instead of being bolted on at the end.

One of the key observations I’ve made in my career in engineering, product, and marketing and as a founder of LaunchDarkly is that high-performing software teams invite significantly more members into the software delivery team. These expanded teams can often include broader stakeholders across development, operations, product management, security, and other areas, engaging them in observing, controlling, monitoring, and measuring what happens in production software. In other words, continuous operation is a shared responsibility for the entire software delivery team, not just developers and operations. For example, developers can use runtime error monitoring software to detect bugs. Operations engineers use observability tools to measure performance and mitigate incidents. But in addition to that, product managers can use feature management tools to control feature releases and run experiments. Designers can use session replay software to observe user interactions and improve end-user experiences. Security engineers can use threat monitoring tools to continuously monitor security risk. The continuous operation of a software product across an integrated software delivery team creates incredibly efficient feedback loops and rapid responses to change failure, incidents, and other forms of risk.

When you consider the ways these teams interact with one another in an integrated software delivery team and then try to fit that interaction into the sequential model represented by the infinity loop, the challenges in that legacy model immediately become visible.

The Steps Need a Refresh

The idea of release before deploy, as depicted in the DevOps infinity loop image in Figure I-1, has been updated. In the previous model, development teams would build their software independently, testing locally and in their own environments. Upon that code becoming “development complete,” the code would be handed off to the teams responsible for building and packaging the code. This metaphorical baton-pass represented the handoff to “code release” before actual deployment out to destination infrastructure. Postdeployment, the baton would then be passed to QA teams to validate the success of the build.

Release processes in this model involved release managers who would decide what was included in a release and eventually distributed out to users. While this model is still relevant (e.g., for firmware, operating systems, and open source libraries), software as a service (SaaS) models demand and thrive in a more continuous process. This enables a more concurrent model of deployment and release, which can be improved even further by leveraging feature management platforms to separate the deployment process from the release process. The separation allows the release process to become a more continuous operation of its own, independent of continuous integration/deployment.

Shifting to a continuous operation also forces us to shift our thinking around many of the stages involved in the SDLC. A great example is the monitoring step (which I cover in greater detail later in this book). In traditional organizations, the task of monitoring is typically a function of operations teams ensuring the availability of a system, implying a specific function performed by a specific team.

Continuous operation, however, involves more stakeholders, and it’s necessary to expand our notion of monitoring appropriately to reflect the concerns of those stakeholders. While monitoring accurately reflects the needs of operations engineers concerned with availability, it doesn’t encompass product managers monitoring user behaviors and running A/B tests and experiments, or growth teams monitoring the impact of changes on their key performance indicators (KPIs). In short, it’s too limited a notion to capture everything that broader integrated software delivery teams watch postdeployment and postrelease. I believe measuring and experimenting is a better name for this stage; it encapsulates the need to quantitatively observe and react to both system and user behaviors occurring in a production system. In continuous operation, everyone involved in the software release measures relevant aspects of their domain, and does so continuously.

The Steps Happen in Parallel

One of the differences between the pre- and postdeployment steps in the SDLC is that in postdeployment, all steps happen in parallel. The DevOps infinity loop implies that the loop is continuous (meaning a cycle in which the process continues and repeats as teams iterate on what they’ve built). However, it’s still restrictive in that it implies that the handoffs happen sequentially. In truth, many of these steps have evolved to have their own continuous loops within the DevOps loop—resulting in this model of continuous operation. For a given product change, predeployment steps have a defined sequence. In other words, planning occurs before implementation.

In contrast, the stages we’re introducing as essential in continuous operation all happen concurrently. There are no sequential steps postdeployment because modern software services are always running. When teams deploy, they are upgrading the airplane while it’s in flight. They’re constantly measuring everything about the system, constantly deploying changes to it, and constantly reacting to changing inputs and traffic. These factors necessitate that they be able to measure, react, and control the system quickly—ideally in real time, so that they can adapt to changing conditions.

When the traditional SDLC diagram is updated with our proposed changes, the result looks something like Figure I-3.

What do these changes look like in practice? What tools and processes facilitate continuous operation? In this book, we’ll cover each stage of continuous operation and share observations on emerging best practices and tools.

Software development has changed dramatically in the 25 years since my first programming job in Visual Basic. Software is more omnipresent in how businesses operate. As software has become how the world runs, the roles involved in software development, the timing of tasks, and the tools that are utilized have all evolved. With changes in people, processes, and ways of working together, a new framework is needed to visualize the modern ways teams ship and deliver software. This book aims to spell out a refreshed framework for visualizing this lifecycle.