Injecting security into Continuous Delivery

Before you can begin adding security checks and controls, you need to understand the workflows and tools that the engineering teams are using:

• What happens before and when a change is checked in?
• Where are the repositories? Who has access to them?
• How do changes transition from check-in to build to Continuous Integration and unit testing, to functional and integration testing, and to staging and then finally to production?
• What tests are run? Where are the results logged?
• What tools are used? How do they work?
• What manual checks or reviews are performed and when?

And how can you take advantage of all of this for security and compliance purposes?

Let’s map out the steps involved from taking a change from check-in to production and identify where we can insert security checks and controls. See the figure below for a model that explains how and where to add security checks into a Continuous Delivery workflow.

Precommit

These are the steps before and until a change to software or configuration is checked in to the source code repo. Additional security checks and controls to be added here include the following:

• Lightweight, iterative threat modeling and risk assessments.
• Static analysis (SAST) checking in the engineer’s IDE.
• Peer code reviews (for defensive coding and security vulnerabilities).

Commit stage (Continuous Integration)

This is automatically triggered by a check in. In this stage, you build and perform basic automated testing of the system. These steps return fast feedback to developers: did this change “break the build”? This stage needs to complete in at most a few minutes. Here are the security checks that you should include in this stage:

• Compile and build checks, ensuring that these steps are clean, and that there are no errors or warnings.
• Software Component Analysis in build, identifying risk in third-party components.
• Incremental static analysis scanning for bugs and security vulnerabilities.
• Alerting on high-risk code changes through static analysis checks or tests.
• Automated unit testing of security functions, with code coverage analysis.
• Digitally signing binary artifacts and storing them in secure repositories.^[1]

Acceptance stage

This stage is triggered by a successful commit. The latest good commit build is picked up and deployed to an acceptance test environment. Automated acceptance (functional, integration, performance, and security) tests are executed. To minimize the time required, these tests are often fanned out to different test servers and executed in parallel. Following a “fail fast” approach, the more expensive and time-consuming tests are left until as late as possible in the test cycle, so that they are only executed if other tests have already passed.

Security controls and tests in this stage include the following:

• Secure, automated configuration management and provisioning of the runtime environment (using tools like Ansible, Chef, Puppet, Salt, and/or Docker). Ensure that the test environment is clean and configured to match production as closely as possible.
• Automatically deploy the latest good build from the binary artifact repository.
• Smoke tests (including security tests) designed to catch mistakes in configuration or deployment.
• Targeted dynamic scanning (DAST).
• Automated functional and integration testing of security features.
• Automated security attacks, using Gauntlt or other security tools.
• Deep static analysis scanning (can be done out of band).
• Fuzzing (of APIs, files). This can be done out of band.
• Manual pen testing (out of band).

Production deployment and post-deployment

If all of the previous steps and tests pass, the change is ready to be deployed to production, pending manual review/approvals and scheduling (in Continuous Delivery) or automatically (in Continuous Deployment). Additional security checks and controls are needed in production deployment and post-deployment:

• Secure, automated configuration management and provisioning of the runtime environment.
• Automated deployment and release orchestration (authorized, repeatable, and auditable).
• Post-deployment smoke tests.
• Automated runtime asserts and compliance checks (monkeys).
• Production monitoring/feedback.
• Runtime defense.
• Red Teaming.
• Bug bounties.
• Blameless postmortems (learning from failure).

Depending on the risk profile of your organization and systems, you will need to implement at least some of these practices and controls. Leaders in this space already do most of them.

---