Chapter 5. Compliance as Code

DevOps can be followed to achieve what Justin Arbuckle at Chef calls “Compliance as Code”: building compliance into development and operations, and wiring compliance policies and checks and auditing into Continuous Delivery so that regulatory compliance becomes an integral part of how DevOps teams work on a day-to-day basis.

A way to achieve Compliance as Code is described in the “DevOps Audit Defense Toolkit”, a free, community-built process framework written by James DeLuccia, IV, Jeff Gallimore, Gene Kim, and Byron Miller.1 The Toolkit builds on real-life examples of how DevOps is being followed successfully in regulated environments, on the Security as Code practices that we’ve just looked at, and on disciplined Continuous Delivery. It’s written in case-study format, describing compliance at a fictional organization, laying out common operational risks and control strategies, and showing how to automate the required controls.

Defining Policies Upfront

Compliance as Code brings management, compliance, internal audit, the PMO and infosec to the table, together with development and operations. Compliance policies and rules and control workflows need to be defined upfront by all of these stakeholders working together. Management needs to understand how operational risks and other risks will be controlled and managed through the pipeline. Any changes to these policies or rules or workflows need to be formally approved and documented; for example, in a Change Advisory Board (CAB) meeting.

But instead of relying on checklists and procedures and meetings, the policies and rules are enforced (and tracked) through automated controls, which are wired into configuration management tools and the Continuous Delivery pipeline. Every change ties back to version control and a ticketing system like Jira for traceability and auditability: all changes must be made under a ticket, and the ticket is automatically updated along the pipeline, from the initial request for work all the way to deployment.

Automated Gates and Checks

The first approval gate is mostly manual. Every change to code and configuration must be reviewed precommit. This helps to catch mistakes and ensure that no changes are made without at least one other person checking to verify that it was done correctly. High-risk code (defined by the team, management, compliance, and infosec) must also have an SME review; for example, security-sensitive code must be reviewed by a security expert. Periodic checks are done by management to ensure that reviews are being done consistently and responsibly, and that no “rubber stamping” is going on. The results of all reviews are recorded in the ticket. Any follow-up actions that aren’t immediately addressed are added to the team’s backlog as another ticket.

In addition to manual reviews, automated static analysis checking is also done to catch common security bugs and coding mistakes (in the IDE and in the Continuous Integration/Continuous Delivery pipeline). Any serious problems found will fail the build.

After it is checked-in, all code is run through the automated test pipeline. The Audit Defense Toolkit assumes that the team follows Test-Driven Development (TDD), and outlines an example set of tests that should be executed.

Infrastructure changes are done using an automated configuration management tool like Puppet or Chef, following the same set of controls:

  • Changes are code reviewed precommit

  • High-risk changes (again, as defined by the team) must go through a second review by an SME

  • Static analysis/lint checks are done automatically in the pipeline

  • Automated tests are performed using a test framework like rspec-puppet or Chef Test Kitchen or ServerSpec

  • Changes are deployed to test and staging in sequence with automated smoke testing and integration testing

And, again, every change is tracked through a ticket and logged.

Managing Changes in Continuous Delivery

Because DevOps is about making small changes, the Audit Defense Toolkit assumes that most changes can be treated as standard or routine changes that are essentially preapproved by management and therefore do not require CAB approval.

It also assumes that bigger changes will be made “dark.” In other words, that they will be made in small, safe, and incremental changes, protected behind runtime feature switches that are turned off by default. The feature will only be rolled out with coordination between development, ops, compliance, and other stakeholders.

Any problems found in production are reviewed through post-mortems, and tests added back into the pipeline to catch the problems (following TDD principles).

Separation of Duties in the DevOps Audit Toolkit

In the DevOps Audit Toolkit, a number of controls enforce or support Separation of Duties:

  • Mandatory independent peer reviews ensure that no engineer (dev or ops) can make a change without someone else being aware and approving it. Reviewers are assigned randomly where possible to prevent collusion.

  • Developers are granted read-only access to production systems to assist with troubleshooting. Any fixes need to be made through the Continuous Delivery pipeline (fixing forward) or by automatically rolling changes back (again, through the Continuous Delivery pipeline/automated deployment processes) which are fully auditable and traceable.

  • All changes made through the pipeline are transparent, published to dashboards, IRC, and so on.

  • Production access logs are reviewed by IT operations management weekly.

  • Access credentials are reviewed regularly.

  • Automated detective change control tools (for example, Tripwire, OSSEC, UpGuard) are used to check for unauthorized changes.

These controls minimize the risk of developers being able to make unauthorized, and undetected, changes to production.

Using the Audit Defense Toolkit

The DevOps Audit Defense Toolkit provides a roadmap to how you can take advantage of DevOps workflows and automated tools, and some of the security controls and checks that we’ve already looked at, to support your compliance and governance requirements.

It requires a lot of discipline and maturity and might be too much for some organizations to take on—at least at first. You should examine the controls and decide where to begin.

Although it assumes Continuous Deployment of changes directly to production, the ideas and practices can easily be adapted for Continuous Delivery by adding a manual review gate before changes are pushed to production.

Code Instead of Paperwork

Compliance as Code tries to minimize paperwork and overhead. You still need clearly documented policies that define how changes are approved and managed, and checklists for procedures that cannot be automated. But most of the procedures and the approval gates are enforced through automated rules in the Continuous Integration/Continuous Delivery pipeline, leaning on the automated pipeline and tooling to ensure that all of the steps are followed consistently and taking advantage of the detailed audit trail that is automatically created.

In the same way that frequently exercising build and deployment steps reduces operational risks, exercising compliance on every change, following the same standardized process and automated steps, reduces the risks of compliance violations. You—and your auditors—can be confident that all changes are made the same way, that all code is run through the same tests and checks, and that everything is tracked the same way: consistent, complete, repeatable, and auditable.

Standardization makes auditors happy. Auditing makes auditors happy (obviously). Compliance as Code provides a beautiful audit trail for every change, from when the change was requested and why, to who made the change and what that person changed, who reviewed the change and what was found in the review, how and when the change was tested, to when it was deployed. Except for the discipline of setting up a ticket for every change and tagging changes with a ticket number, compliance becomes automatic and seamless to the people who are doing the work.

Just as beauty is in the eye of the beholder, compliance is in the opinion of the auditor. Auditors might not understand or agree with this approach at first. You will need to walk them through it and prove that the controls work. But that shouldn’t be too difficult, as Dave Farley, one of the original authors of Continuous Delivery explains:

I have had experience in several finance firms converting to Continuous Delivery. The regulators are often wary at first, because Continuous Delivery is outside of their experience, but once they understand it, they are extremely enthusiastic. So regulation is not really a barrier, though it helps to have someone that understands the theory and practice of Continuous Delivery to explain it to them at first.

If you look at the implementation of a deployment pipeline, a core idea in Continuous Delivery, it is hard to imagine how you could implement such a thing without great traceability. With very little additional effort the deployment pipeline provides a mechanism for a perfect audit trail. The deployment pipeline is the route to production. It is an automated channel through which all changes are released. This means that we can automate the enforcement of compliance regulations—“No release if a test fails,” “No release if a trading algorithm wasn’t tested,” “No release without sign-off by an authorised individual,” and so on. Further, you can build in mechanisms that audit each step, and any variations. Once regulators see this, they rarely wish to return to the bad old days of paper-based processes.2

Get DevOpsSec now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial