Attacking Containers and runC

This week a new vulnerability was published (CVE-2019-5736) that highlights everything bad and good about containers. Simply put, this vulnerability can be exploited using an infected container to attack the host. It’s a real world example of a breakout attack that has long been a major concern in virtualized and container environment.

Here, the attack highlights the biggest security weakness of containers: they are loosely isolated sharing the same host operating system. This is in stark contrast to virtual machines which are isolated instances of a complete operating system.

CVE-2019-5736

The vulnerability itself can be exploited by an attacker using a custom container or by gaining write access to an existing container. They then can manipulate the symbolic process link (/proc/self/exe/) in order to overwrite the runC library. runC is portable, lightweight container runtime. It’s a critical piece of container infrastructure.

In this attack, once runC is overwritten and under the attackers control, they own the host and—potentially—any container running on it.

That’s a devastating foothold and is why this vulnerability has a CVSSv3 score of 7.2 or “high”. A score this high means that you should mitigate or fix the vulnerability as soon as possible.

For Trend Micro customers using Deep Security to protect their container hosts, this knowledge base article explains the rules that you can use to both detect and prevent this issue until you have the opportunity to deploy a patch to your infrastructure.

A Container Refresher

When reading about a vulnerability like this, the natural question to ask is, “Why isn’t there a firmer line between containers on the same host?”. The answer is a complicated one.

To start with, containers are not designed to solve security challenges. They were designed to tackle a very specific development challenge: dependency nightmares.

Any application you write is built on layers of other teams code. Whether it’s the framework you’re using directly, standard libraries provided by your programming language, services made available by the OS, or even resources provided in hardware, you code does not stand alone.

This leads to a web of interdependencies and requirements for your code to run. For a very long time, developers faced a challenge documenting all of these dependencies and ensuring they were met in production environments.

If you’ve ever heard a developer exclaim, “It worked on my machine!”. You understand the problem.

Containers were designed to make it easy to package all of an applications dependencies in a portable fashion. This helps with deployment, versioning, and a number of other delivery challenges.

In this respect containers are a fantastic step forward for developer efficiency.

The Downside of Containers

This efficiency for developers comes at the cost of infrastructure complexity. Often overlooked is the security of the container host, network complexity, and the integrity of the build pipeline.

In the case of CVE-2019-5736, the container host’s security is paramount. Hardening the hosts operating system by reducing the number of available services—it should only run the container runtime, host security controls, and host monitoring applications—to the bare minimum is critical to security success.

Furthermore, using security controls like integrity monitoring, log inspection, and application control will ensure that you hardened configuration stays that way.

This vulnerability demonstrates that each container can be risk to the host. The easiest analogy here comes from noted container expert Kelsey Hightower, he compared virtual machines to single houses (isolated, rarely impacting their neighbours) and containers to apartments. If you upstairs neighbour is always banging on the floor, you have a problem.

CVE-2019-5736 is the distinct possibility of having a neighbour who throws a crazy party that trashes not only their own apartment but the hall, elevator, and lobby. Everyone has to deal with that mess.

The Upside

This issue also demonstrates the upside of the container model. Containers are designed for a highly automated and dynamic environment. In order to resolve this issue, the container runtime will need to be protected and then patched.

These measures may impact the availability of each host. The advantage? You can simply spin up a new version of your container on an already protected or patched host.

Take for example the list of affected AWS services. In each of these cases, a rolling update or blue/green deployment is possible in order to address the issue within impacting your users.

If your CI/CD pipeline is setup—and if you’re using containers, it should be—a simple re-deployment to known good hosts will mitigate the issue. This is a prime example of the advantages of a highly automated build pipeline.

No special processes are required. Simply mitigate or patch the hosts and run your build again. DevOps culture FTW.

Next Steps

This won’t be the last security issue in your container environment. Containers were designed to improve developer efficiency. Security is a priority for the teams working on the projects—like runC—that make containers work but there will always be security issues that pop up.

If you’re following best practices and have automated your build and deployment pipeline, these issues shouldn’t impact your end users. At worst, it should mean adding a new security rule or two to your tool set, adding a new security test to your build (to prevent recurrence), and a rolling update.

It’s also a reminder that the security of your container host is paramount to the security of your container infrastructure. Take this opportunity to review the security posture of these hosts and if you haven’t already, deploy a strong set of security controls that include integrity monitoring and application control.