What Is Poisoned Pipeline Execution (PPE)?

The D-PPE attack example illustrated in figure 1 transpires in the following succession of steps:

1. An adversary initiates a new remote branch within the repository, altering the pipeline configuration file with harmful instructions to retrieve AWS credentials stored within the GitHub organization and transmit them to an external server under the attacker's control.
2. The code push activates a pipeline that pulls the code, inclusive of the malicious pipeline configuration file, from the repository.
3. The pipeline operates according to the configuration file, now tainted by the attacker. The malicious instructions command the AWS credentials, stored as repository secrets, to load into memory.
4. Following the attacker's instructions, the pipeline carries out the task of transmitting the AWS credentials to a server under the attacker's control.
5. In possession of the stolen credentials, the attacker gains the ability to infiltrate the production environment.

Indirect PPE

Indirect PPE occurs when the possibility of D-PPE isn’t available to an adversary with access to an SCM repository:

• If the pipeline is configured to pull the CI configuration file from a separate, protected branch in the same repository.
• If the CI configuration file is stored in a separate repository from the source code, without the option for a user to directly edit it.
• If the CI build is defined in the CI system itself — instead of in a file stored in the source code.

In these scenarios, the attacker can still poison the pipeline by injecting malicious code into files referenced by the pipeline configuration file, such as scripts referenced from within the pipeline configuration file, code tests, or automatic tools like linters and security scanners used in the CI. For example:

• The make utility executes commands defined in the Makefile file.
• Scripts referenced from within the pipeline configuration file, which are stored in the same repository as the source code itself (e.g., python myscript.py — where myscript.py would be manipulated by the attacker).
• Code tests: Testing frameworks running on application code within the build process rely on dedicated files, stored in the same repository as the source code. Attackers who can manipulate the code responsible for testing can then run malicious commands inside the build.
• Automatic tools: Linters and security scanners used in the CI commonly rely on a configuration file residing in the repository that typically loads and runs external code from a location defined inside the configuration file.

Rather than poisoning the pipeline via direct PPE, the attacker launching an indirect PPE attack injects malicious code into files referenced by the configuration file. The malicious code is ultimately executed on the pipeline node and runs the commands declared in the files.

In this I-PPE attack example, the chain of events unfolds as follows:

1. An attacker creates a pull request in the repository, appending malicious commands to the Makefile file.
2. Since the pipeline is configured to trigger on any PR against the repo, the Jenkins pipeline is triggered, fetching the code from the repository — including the malicious Makefile.
3. The pipeline runs based on the configuration file stored in the main branch. It gets to the build stage and loads the AWS credentials into environment variables — as defined in the original Jenkinsfile. Then, it runs the make build command, which executes the malicious command that was added into Makefile.
4. The malicious build function defined in the Makefile is executed, sending the AWS credentials to a server controlled by the attacker.
5. The attacker can then use the stolen credentials to access the AWS production environment.

Public PPE

Public PPE is a type of PPE attack executed by anonymous attackers on the internet. Public repositories often allow any user to contribute, usually by creating pull requests. If the CI pipeline of a public repository runs unreviewed code suggested by anonymous users, it’s susceptible to a public PPE attack. The public PPE can also expose internal assets, such as secrets of private projects, in cases where the pipeline of the vulnerable public repository runs on the same CI instance as private ones.

Importance of Secure Pipeline Execution in CI/CD

Executing malicious unreviewed code in the CI via a successful PPE attack provides attackers with the same level of access and ability as the build job:

• Access to secrets available to the CI job, such as secrets injected as environment variables or additional secrets stored in the CI. Being responsible for building code and deploying artifacts, CI/CD systems typically contain dozens of high-value credentials and tokens — such as to a cloud provider, to artifact registries, and to the SCM itself.
• Access to external assets the job node has permissions to, such as files stored in the node’s file system, or credentials to a cloud environment accessible through the underlying host.
• Ability to ship code and artifacts further down the pipeline, in the guise of legitimate code built by the build process.
• Ability to access additional hosts and assets in the network/environment of the job node.

But organizations can safeguard their software products and infrastructure with a secure pipeline execution that ensures all code compiled, tested, and deployed is legitimate and untampered.

Risks Associated with Poisoned Pipeline Execution

The implications of PPE can be severe, ranging from unauthorized data access, compromised software integrity, system disruptions, to data breaches or even a total system takeover. These risks pose significant threats to both the business and its clients, underscoring the gravity of PPE.

In the eight-step supply chain compromise operation seen in figure 3, the attacker gains access to the CI pipeline and poisons components of the SaaS application. Via the poisoned component, the attacker builds backdoor functionality into the application and sends the poisoned plugins to downstream clients. Since the downstream organizations likely perceive the poisoned package as legitimate, they build it into their cloud or on-premises infrastructure.

From one poisoned CI pipeline, the attacker achieves exponential collateral damage, having created backdoor access to countless organizations. This was the case with the SolarWinds attack.

READ MORE: Anatomy of a CI/CD Pipeline Attack

Preventing Poisoned Pipeline Execution

Preventing and mitigating the PPE attack vector involves multiple measures spanning across both SCM and CI systems:

• Ensure that pipelines running unreviewed code are executed on isolated nodes versus exposed to secrets and sensitive environments.
• Evaluate the need for triggering pipelines on public repositories from external contributors. When possible, refrain from running pipelines originating from forks and consider adding controls such as requiring manual approval for pipeline execution.
• For sensitive pipelines, for example those that are exposed to secrets, ensure that each branch that is configured to trigger a pipeline in the CI system has a correlating branch protection rule in the SCM.
• To prevent the manipulation of the CI configuration file to run malicious code in the pipeline, each CI configuration file must be reviewed before the pipeline runs. Alternatively, the CI configuration file can be managed in a remote branch, separate from the branch containing the code being built in the pipeline. The remote branch should be configured as protected.
• Remove permissions granted on the SCM repository from users who don’t need them.
• Each pipeline should only have access to the credentials it needs to fulfill its purpose. The credentials should have the minimum required privileges.

Poisoned Pipeline Execution FAQs