DevOpsSec: Securing software through continuous delivery

Introduction

Some people see DevOps as another fad, the newest new-thing overhyped by Silicon Valley and by enterprise vendors trying to stay relevant. But others believe it is an authentically disruptive force that is radically changing the way that we design, deliver, and operate systems.

In the same way that Agile and Test-Driven Development (TDD) and Continuous Integration has changed the way that we write code and manage projects and product development, DevOps and Infrastructure as Code and Continuous Delivery is changing IT service delivery and operations. And just as Scrum and XP have replaced CMMi and Waterfall, DevOps is replacing ITIL as the preferred way to manage IT.

Learn faster. Dig deeper. See farther.

Join the O'Reilly online learning platform. Get a free trial today and find answers on the fly, or master something new and useful.

Learn more

DevOps organizations are breaking down the organizational silos between the people who design and build systems and the people who run them—silos that were put up because of ITIL and COBIT to improve control and stability but have become an impediment when it comes to delivering value for the organization.

Instead of trying to plan and design everything upfront, DevOps organizations are running continuous experiments and using data from these experiments to drive design and process improvements.

DevOps is finding more effective ways of using the power of automation, taking advantage of new tools such as programmable configuration managers and application release automation to simplify and scale everything from design to build and deployment and operations, and taking advantage of cloud services, virtualization, and containers to spin up and run systems faster and cheaper.

Continuous Delivery and Continuous Deployment, Lean Startups and MVPs, code-driven configuration management using tools like Ansible and Chef and Puppet, NoOps in the cloud, rapid self-service system packaging and provisioning with Docker and Vagrant and Terraform are all changing how everyone in IT thinks about how to deliver and manage systems. And they’re also changing the rules of the game for online business, as DevOps leaders use these ideas to out-pace and out-innovate their competitors.

The success that DevOps leaders are achieving is compelling. According to Puppet Labs’ 2015 State of DevOps Report:

• High-performers deploy changes 30 times more often than other organizations, with lead times that are 200 times shorter.

• Their change success rate is 60 times higher.

• And when something goes wrong, they can recover from failures 168 times faster.

What do you need to do to achieve these kinds of results? And, how can you do it in a way that doesn’t compromise security or compliance, or, better, in a way that will actually improve your security posture and enforce compliance?

As someone who cares deeply about security and reliability in systems, I was very skeptical when I first heard about DevOps, and “doing the impossible 50 times a day.”¹ It was too much about how to “move fast and break things” and not enough about how to build systems that work and that could be trusted. The first success stories were from online games and social-media platforms that were worlds away from the kinds of challenges that large enterprises face or the concerns of small businesses.

I didn’t see anything new or exciting in “branching in code” and “dark launching” or developers owning their own code and being responsible for it in production. Most of this looked like a step backward to the way things were done 25 or 30 years ago, before CMMi and ITIL were put in to get control over cowboy coding and hacking in production.

But the more I looked into DevOps, past the hype, I found important, substantial new ideas and patterns that could add real value to the way that systems are built and run today:

Infrastructure as Code

Defining and managing system configuration through code that can be versioned and tested in advance using tools like Chef or Puppet dramatically increases the speed of building systems and offers massive efficiencies at scale. This approach to configuration management also provides powerful advantages for security: full visibility into configuration details, control over configuration drift and elimination of one-off snowflakes, and a way to define and automatically enforce security policies at runtime.

Continuous Delivery

Using Continuous Integration and test automation to build pipelines from development to test and then to production provides an engine to drive change and at the same time a key control structure for compliance and security, as well as a safe and fast path for patching in response to threats.

Continuous monitoring and measurement

This involves creating feedback loops from production back to engineering, collecting metrics, and making them visible to everyone to understand how the system is actually used and using this data to learn and improve. You can extend this to security to provide insight into security threats and enable “Attack-Driven Defense.”

Learning from failure

Recognizing that failures can and will happen, using them as learning opportunities to improve in fundamental ways through blameless postmortems, injecting failure through chaos engineering, and practicing for failure in game days; all of this leads to more resilient systems and more resilient organizations, and through Red Teaming to a more secure system and a proven incident response capability.

The more I talked to people at organizations like Etsy and Netflix who have had real success with DevOps at scale, and the more I looked into how enterprises like ING and Nordstrom and Capital One and Inuit are successfully adopting DevOps, the more I started to buy in.

And the more success that we have had in my own organization with DevOps ideas and tools and practices, the more I have come to understand how DevOps, when done right, can be used to deliver and run systems in a secure and reliable way.

Whether you call it SecDevOps, DevSecOps, DevOpsSec, or Rugged DevOps, this is what this paper will explore.

We’ll begin by looking at the security and compliance challenges that DevOps presents. Then, we’ll cover the main ideas in secure DevOps and how to implement key DevOps practices and workflows like Continuous Delivery and Infrastructure as Code to design, build, deploy, and run secure systems. In Security as Code: Security Tools and Practices in Continuous Delivery, we’ll map security checks and controls into these workflows. Because DevOps relies so heavily on automation, we’ll look at different tools that you can use along the way, emphasizing open source tools where they meet the need, and other tools that I’ve worked with and know well or that are new and worth knowing about. And, finally, we’ll explore how to build compliance into DevOps, or DevOps into compliance.