Mitigating known security risks in open source libraries

Major Upgrades

While most issues are fixed, very often the fix is only applied to the latest and greatest version of the library. If you’re still using an older version of the library, upgrading may mean switching to a new major version. Major upgrades are typically not backward compatible, introducing more risk and requiring more dev effort.

Another reason for fixing an issue only in the next major version is that sometimes fixing a vulnerability means reducing functionality. For instance, fixing a certain XSS vulnerability in a jQuery 2.x codebase requires a change to the way certain selectors are interpreted. The jQuery team determined too many people are relying on this functionality to deem this a non-breaking change, and so only fixed the vulnerability in their 3.x stream.

For these reasons, a major upgrade can often be difficult, but if you can accept it, it’s still the best way to fix a vulnerability.

Indirect Dependency Upgrade

If you’re consuming a dependency directly, upgrading is relatively straightforward. But what happens when one of your dependencies is the one who pulled in the vulnerable package? Most dependencies are in fact indirect dependencies (a.k.a. transitive dependencies), making upgrades a bit more complex.

The cleanest way to perform an indirect upgrade is through a direct one. If your app uses A@1, which uses a vulnerable B@1, it’s possible that upgrading to A@2 will trigger a downstream upgrade to B@2 and fix the issue. Applying such an upgrade is easy (it’s essentially a direct upgrade), but discovering which upgrade to do (and whether one even exists) is time consuming. While not common, some SCA tools can determine and advise on the direct upgrades you need to make to fix an indirect vulnerability. If your tooling doesn’t support it, you’ll need to do the searching manually.

Old vulnerabilities in indirect libraries can often be fixed with a direct upgrade, but such upgrades are frequently unavailable for new issues. When a new vulnerability is disclosed, even if the offending package releases a fix right away, it takes a while for the dependency chain to catch up. If you can’t find a path to an indirect upgrade for a newly disclosed flaw, be sure to recheck frequently as one may show up soon. Once again, some SCA tools will do this monitoring for you and alert you when new remediations are available.

Conflicts

Another potential obstacle to upgrading is a conflict. Many languages, such as Ruby and Python, require dependencies to be global, and clients such as Ruby’s bundler and Python’s pip determine the mix of library versions that can co-exist. As a result, upgrading one library may trigger a conflict with another. While developers are adept at handling such conflicts, there are times when such issues simply cannot be resolved.

On the positive side, global dependency managers, such as Ruby’s bundler, allow the parent app to add a constraint. For instance, if a downstream B@1 gem is vulnerable, you can add B@^2 to your Gemfile, and have bundler sort out the surrounding impact. Adding such constraints is a safe and legitimate solution, as long as your ecosystem tooling can figure out a conflict-free combination of libraries.

Is a Newer Version Always Safer?

The conversation about upgrading begs a question: can a vulnerability also be fixed by downgrading?

For the most part, the answer is no. Vulnerabilities are bugs, and bugs are typically fixed in a newer version, not an older one. In general, maintaining a good upgrade cadence and keeping your dependencies up to date is a good preventative measure to reduce the risk of vulnerabilities.

However, in certain cases, code changes or (more often) new features are the ones that trigger a vulnerability. In those cases, it’s indeed possible that downgrading will fix the discovered flaw. The advisory should give you the information you need about which versions are affected by the vulnerability. That said, note that downgrading a package puts you at higher risk of being exposed to new issues, and can make it harder to upgrade when that happens. I suggest you see downgrading as a temporary and rarely used remediation path.

There Is No Fixed Version

Last on the list of reasons preventing you from upgrading to a safe version is such a version not existing in the first place!

While most vulnerabilities are fixed, many remain unfixed. This is sometimes a temporary situation—for instance, when a vulnerability was made public without waiting for a fix to be released. Other times, it may be a more long-term scenario, as many repositories fall into a poor maintenance state, and don’t fix reported issues nor accept community patches.

In the following sections I’ll discuss some options for when you cannot upgrade a vulnerability away.

Sourcing Patches

To create a patch, you first need to have a fix for the vulnerability! You could write one yourself, but patches are more often sourced from existing community fixes.

The first place to look for a patch is a new version of the vulnerable package. Most often the vulnerability was fixed by the maintainers of the library, but that fix may be in an out-of-reach indirect dependency, or perhaps was only fitted back into the latest major version. Those fixes can be extracted from the original repo and stored into their own patch file, as well as back-ported into older versions if need be.

Another common source for patches are external pull requests (PRs). Open source maintenance is a complicated topic, and it’s not uncommon for repos to go inactive. In such repos, you may find community pull requests that fix a vulnerability, have been commented on and perhaps vetted by others, but are not merged and published into the main stream. Such PRs are a good starting point—if not the full solution—for creating a patch. For instance, an XSS issue in the popular JavaScript Markdown parsing library marked had an open fix PR for nearly a year before it was incorporated into a new release. During this period, you could use the fix PR code to patch the issue in your apps.

Snyk maintains its own set of patches in its open source database. Most of those patches are captures or back-ports of original fixes, a few are packaged pull requests, and even fewer are written by the Snyk security research team.

Depend on GitHub Hash

In very specific cases, you may be able to patch without storing any code changes. This is only possible if the vulnerable dependency is a direct dependency of your app, and the public repo holding the package has a commit that fixes the issue (often a pull request, as mentioned before).

If that’s the case, most package managers allow you to change your manifest file to point to the GitHub commit instead of naming your package and version. Git hashes are immutable, so you’ll know exactly what you’re getting, even if the pull request evolved. However, the commit may be deleted, introducing certain reliability concerns.

Fork and Patch

When patching a vulnerability in a direct dependency, assuming you don’t want to depend on an external commit or have none to use, you can create one of your own. Doing so typically means forking the GitHub repository to a user you control, and patching it. Once done, you can modify your manifest to point to your fixed repository.

Forking is a fairly common way of fixing different bugs in dependencies, and also carries some nice reliability advantages, as the code you use is now in your own control. It has the downside of breaking off the normal version stream of the dependency, but it’s a decent short-term solution to vulnerabilities in direct dependencies. Unfortunately, forking is not a viable option for patching indirect dependencies.

Static Patching at Build Time

Another opportunity to patch a dependency is during build time. This type of patching is more complicated, as it requires:

1. Storing a patch in a file (often a .patch file, or an alternative JAR file with the issue fixed)

2. Installing the dependencies as usual

3. Determining where the dependency you’d like to patch was installed

4. Applying the patch by modifying or swapping out the risky code

These steps are not trivial, but they’re also usually doable using package manager commands. If a vulnerability is worth fixing, and there are no easier means to fix it, this approach should be considered.

This is a classic problem for tools to address, as patches can be reused and their application can be repeated. However, at the time of this writing, Snyk is the only SCA tool that maintains patches in its DB and lets you apply them in your pipeline. I predict over time more and more tools will adopt this approach.

Dynamic Patching at Boot Time

In certain programming languages, classes can also be modified at runtime, a technique often referred to as “monkey patching.” Monkey patching can be used to fix vulnerabilities, though that practice has not become the norm in any ecosystem. The most prevalent use of monkey patching to fix vulnerabilities is in Ruby on Rails, where the Rails team has often released patches for vulnerabilities in the libraries it maintains.

Removal

Removing a dependency is a very effective way of fixing its vulnerabilities. Unfortunately, you’ll be losing its functionality at the same time.

Dropping a dependency is often hard, as it by definition requires changes to your actual code. That said, such removal may turn out to be easy—for instance, when a dependency was used for convenience and can be rewritten instead, or when a comparable alternative exists in the ecosystem.

Easy or hard, removing a dependency should always be considered an option, and weighed against the risk of keeping it.

External Mitigation

If you can’t fix the vulnerable code, you can try to block attacks that attempt to exploit it instead. Introducing a rule in a web app firewall, modifying the parts of your app that accept related user input, or even blocking a port are all potential ways to mitigate a vulnerability.

Whether you can mitigate and how to do so depends on the specific vulnerability and application, and in many cases such protection is impossible or high risk. That said, the most trivially exploited vulnerabilities, such as the March 2017 Struts2 RCE and ImageTragick, are often the ones most easily identified and blocked, so this approach is definitely worth exploring.

Log Issue

Last but not least, even if you choose not to remediate the issue, the least you can do is create an issue for it. Beyond its risk management advantages, logging the issue will remind you to re-examine the remediation options over time—for instance, looking for newly available upgrades or patches that can help.

If you have a security operations team, make sure to make them aware of vulnerabilities you are not solving right now. This information can prove useful when they triage suspicious behavior on the network, as such behavior may come down to this security hole being exploited.

Ignoring Issues

If you choose not to fix an issue, or to fix it through a custom path, you’ll need to tell your SCA tool you did. Otherwise, the tool will continue to indicate this problem.

All OSS security tools support ignoring a vulnerability, but have slightly different capabilities. You should consider the following, and try to note that in your tool of choice:

• Are you ignoring the issue because it doesn’t affect you (perhaps you’ve mitigated it another way) or because you’ve accepted the risk? This may reflect differently in your top-level reports.

• Do you want to mute the issue indefinitely, or just “snooze” it? Ignoring temporarily is common for low-severity issues that don’t yet have an upgrade, where you’re comfortable taking the risk for a bit and anticipate an upgrade will show up soon.

• Do you want to ignore all instances of this known vulnerability (perhaps it doesn’t apply to your system), or only certain vulnerable paths (which, after a careful vetting process, you’ve determined to be non-exploitable)?

Properly tagging the reason for muting an alert helps manage these vulnerabilities over time and across projects, and reduces the chance of an issue being wrongfully ignored and slipping through the cracks.

Fix All Vulnerable Paths

For all the issues you’re not ignoring, remember that remediation has to be done for every vulnerable path.

This is especially true for upgrades, as every path must be assessed for upgrade separately, but also applies to patches in many ecosystems.

Track Remediations Over Time

As already mentioned, a fix is typically issued for the vulnerable package first, and only later propagates through the dependency chain as other libraries upgrade to use the newer (and safer) version. Similarly, community or author code contributions are created constantly, addressing issues that weren’t previously fixable.

Therefore, it’s worth tracking remediation options over time. For ignored issues, periodically check if an easy fix is now available. For patched issues, track potential updates you can switch to. Certain SCA tools automate this tracking and notify you (or open automated pull requests) when such new remediations are available.