9 tips for a more secure continuous delivery pipeline

It’s important not only to secure the application and its runtime environment, but to secure the Continuous Delivery tool chain and build and test environments, too. You need to have confidence in the integrity of delivery and the chain of custody, not just for compliance and security reasons, but also to ensure that changes are made safely, repeatably, and traceably.

Your Continuous Delivery tool chain is also a dangerous attack target itself: it provides a clear path for making changes and pushing them automatically into production. If it is compromised, attackers have an easy way into your development, test, and production environments. They could steal data or intellectual property, inject malware anywhere into the environment, DoS your systems, or cripple your organization’s ability to respond to an attack by shutting down the pipeline itself.

Continuous Delivery and Continuous Deployment effectively extend the attack surface of your production system to your build and automated test and deployment environment.

You also need to protect the pipeline from insider attacks by ensuring that all changes are fully transparent and traceable from end to end, that a malicious and informed insider cannot make a change without being detected, and that they cannot bypass any checks or validations.

Do a threat model on the Continuous Delivery pipeline. Look for weaknesses in the setup and controls, and gaps in auditing or logging. Then, take these steps to secure your configuration management environment and Continuous Delivery pipeline:

1. Harden the systems that host the source and build artifact repositories, the Continuous Integration and Continuous Delivery server(s), and the systems that host the configuration management, build, deployment, and release tools. Ensure that you clearly understand—and control—what is done on-premises and what is in the cloud.
2. Harden the Continuous Integration and/or Continuous Delivery server. Tools like Jenkins are designed for developer convenience and are not secure by default. Ensure that these tools (and the required plug-ins) are kept up-to-date and tested frequently.
3. Lock down and harden your configuration management tools. See “How to be a Secure Chef” for example.
4. Ensure that keys, credentials, and other secrets are protected. Get secrets out of scripts and source code and plain-text files and use an audited, secure secrets manager like Chef Vault, Square’s KeyWhiz project, or HashiCorp Vault.
5. Secure access to the source and binary repos and audit access to them.
6. Implement access control across the entire tool chain. Do not allow anonymous or shared access to the repos, to the Continuous Integration server, or confirmation manager or any other tools.
7. Change the build steps to sign binaries and other build artifacts to prevent tampering.
8. Periodically review the logs to ensure that they are complete and that you can trace a change through from start to finish. Ensure that the logs are immutable, that they cannot be erased or forged.
9. Ensure that all of these systems are monitored as part of the production environment.