Securing your multi-tenant CI/CD platform
5 questions for Binu Ramakrishnan: Insights on the threats of centralized multi-tenant CI/CD environments, common protection methods, and tips for improved security.
Apartment buildings (source: Unsplash via Pixabay)
I recently sat down with Binu Ramakrishnan, principal security engineer at Yahoo, to discuss the dangers associated with a centralized multi-tenant continuous integration and continuous delivery (CI/CD) platform, and methods to protect it. Here are some highlights from our talk.
1. How can a multi-tenant CI/CD platform impact the overall security posture of an organization?
Lately, the software industry has started moving away from single-tenant environments in favor of multi-tenant CI/CD environments. A major factor behind the platform shift is the economies of scale. In single-tenant build environments, each team owns and operates a dedicated set of machines to build and deploy their applications. The security risk exposure from these deployments is limited because systems are distributed and dedicated to each team, and the impact of an attack is likely confined to a few related hosts.
Learn faster. Dig deeper. See farther.
Unlike dedicated systems, shared platforms enable pooling of development and support resources, and IT infrastructure to gain operational efficiency. Centralization of platform creates a build corridor that acts as a gateway to production and pre-production networks. The ability to directly reach a large number of machines inside an enterprise makes this platform an attractive target for attackers. By compromising the CI/CD pipeline, an attacker can steal private keys, bypass firewalls, and gain access to machines inside the enterprise networks that are otherwise protected. An attack could result in exposing sensitive data, such as personally identifiable information (PII), user passwords, and denial of service (DoS), that directly affects the availability of applications and the build platform itself.
In cloud-based hosted environments, build pipelines often span across multiple trust boundaries, and are operated by different organizations. Dependence on multiple operators to support build pipelines brings an additional set of security and compliance challenges to an enterprise. A careful evaluation of security risks associated with the use of a multi-tenant CI/CD platform helps an enterprise identify and mitigate those risks to an acceptable level.
2. What are some common security challenges in a multi-tenant CI/CD environment?
From a platform provider’s angle, a major challenge is preventing unauthorized use of the CI/CD platform from reaching target hosts, tampering with applications or mounting DoS attacks. Since build jobs execute untrusted arbitrary commands, host and network isolation of jobs becomes necessary. A job should not break out from their sandbox, interfere with other jobs and degrade the whole environment.
Moving to multi-tenant environments also makes some existing security controls less effective. Imagine your build job needs to access an IP whitelist-protected endpoint. Whitelisting build slave IPs allows other jobs to spoof and access the endpoint. Such systems need to move to a key-based authentication model, but this could be a challenge depending on your environment.
3. What are the most common approaches to protecting CI/CD platforms?
Due to its critical nature, multi-tenant CI/CD platforms are treated as trust anchors, and security controls are developed based on that trust model. To function, build jobs require permissions to push a newly built package to artifact repository, or SSH target hosts to deploy applications. A common practice followed by in-house platforms is to use shared headless credentials. These credentials are stored locally and shared by build jobs.
The cloud-based hosted platforms take a different approach. These platforms collect and store user OAuth tokens, and use them on behalf of their jobs to access other cloud-based applications. Most platforms allow users to upload secrets and other key materials, and make them available to build jobs during their execution.
To provide isolation, platforms generally use containers and virtual machines. They act as a trust boundary between host OS and the build job. Networks are often segmented to separate build machines from other internal machines.
4. What are some suggestions for moving beyond the existing model for protecting large-scale CI/CD platforms?
Modern CI/CD environments spread across multiple trust boundaries and are operated by different organizations. The current practice is to use locally stored long-lived credentials to establish trust between connected services. To further reduce the risk, the recommendation is not to store job-specific credentials locally in the build systems. Instead the platform should adopt a stateless model and receive delegation tokens as part of build trigger events. Since the delegation of permissions is vital to automated deployments, service providers are encouraged to support ephemeral tokens for authentication.
5. You’re speaking about securing application deployments in multi-tenant CI/CD environments at the O’Reilly Security Conference in New York this November. What presentations are you looking forward to attending while there?
It is hard to choose from a great lineup of talks. “The future of strong online identities: Simple, open, and mobile” by Stina Ehrensvard and “Benefits of isolation provided by containers” by Jessica Frazelle are the two I’m most looking forward to attending.