Most container images in production have serious vulnerabilities | Hacker News Reader

Most container images in production have serious vulnerabilities

darkreading.com

59 pointsCharlesW29 comments

> An alarming 87% of container images running in production have critical or high-severity vulnerabilities, [...] Yet only 15% of those unpatched critical and high-severity vulnerabilities are in packages in use at runtime when patches are available. [...] Further, only 2% of the vulnerabilities are exploitable.

And this is why the numbers are so high -- because all of those "vulnerability reports" are complete junk most of the time. SSHD warnings for systems which do not run sshd. ImageMagick vulnerabilities for internal code docs generator. A high severity sudo vulnerability which requires a specific very uncommon configuration.

A great idea on paper, but it is achieving an opposite effect, teaching people that most of automated vulnerability detections are junk.

I’ve worked doing data analysis on vulnerabilities, and it’s really really bizarre.

Some times the most secure systems are running insanely old software.

Sometimes software that’s a bit out of date correlates to fewer attacks.

When people are attacked, they upgrade their software. So a company that has gone a long time with no updates, followed by frantically updating everything could be a warning sign.

Maybe some weirdness is cleared up by noting that besides being an indication that upgrades haven't been necessary for a while, really old software often gains additional resilience to attacks by becoming more obscure with age. Targeted efforts are warded off somewhat by the unfamiliar attack surface and mass scanners typically won't be looking for ancient services. This is only tangentially related, but Im curious if you found anything about the typical length of time between systems being compromised to when intrusions are discovered as part of your work - if you would like to and are at liberty to share :)

Very old software may gain an edge. However, that takes a while and depends on the number of people using the old software as well. For instance, Windows XP was a big target for decades after it had been superseded by newer versions of windows. If your software stack is from the 90s, I'd say you might just getting obscure enough by this point.

Scanners are frequently not looking for the breakthrough 0-day vulnerabilities, but just for the low hanging fruit of old and well known vulnerabilities.

The local university maintained DECNet routing to support systems still in use in the physics department until the early 2000's (when they also ceased inter-building IPX routing). Guess which systems were never exploited for distributing Warez and porn?

All exploited systems before then were either windows nt or Linux. Commercial Unix (Dec, sgi, sun) rarely came up and generally only when there were accounts using identical credentials with a more exploitable platform.

What I'm also seeing is a knee-jerk reaction to COTS/OSS software, and people using these scores to justify building software in-house. Can't be vulnerable if you're not in vulnerability scanners, right?

Which seems a bit weird, as they wouldn't have paid a cent before for security auditing OSS codebases.

I cannot empasis how true this is. The classic UNIX problem was that the LPT printer daemon has an issue (it had lots and lots). But, none of your systems were running LPT, but you still had to patch 1000+s of systems just to maintain a security policy.

What's different between full on UNIX systems and Docker, the possibility of deploying code based on scratch images. Imagine a system which only had the pieces necessary to run in production, your security exception reports would go to zero.

It’s almost free to rebuild and redeploy from a Dockerfile if you have a good devops culture. This would replace the traditional unattended upgrader with scheduled reboots.

This drives me nuts. And I always remind myself not to fall into this behavior but can't help it with the sheer volume of alerts.

No they don't. Container scanning is a load of shit, sorry. Just because something in the container contains a vulnerability, doesn't mean the application itself is vulnerable. It is not the same thing.

Instead of doing the right thing, which would be static and dynamic analysis of the entrypoint to determine the runtime dependencies, the industry has just resorted to saying anything you bundle with your app needs to be 100% invulnerable at all times. It's stupid and practically unachievable if your container is big.

Container scanners and dependency bots like Dependabot are snake oil solutions sold to people too lazy to actually secure their application.

Here is the source report cited in the article:

https://sysdig.com/2023-cloud-native-security-and-usage-repo...

It is provided by:

"Sysdig Secure: Cloud, container, and Kubernetes security that closes the loop from source to run."

It's the best interest of security companies to sell scareware.

I have a container that contains some vendor-provided software used to support GPU-driven machine learning in a high performance cluster. This vendor didn’t do the cleanest build ever, and so the package they supply installs kernel headers.

Based on those headers, the container scanner is convinced that my container is vulnerable to many exciting things, like Atheros WiFi chipset driver vulnerabilities.

A counter point when you run 1000’s of containers or even 10’s of 1000’s it’s hard to near impossible to validate what can effectively be exploited and what can’t which is why most vulnerability management program take a zero vulnerabilities approach and deal with exceptions on a case by case basis.

One of the easiest ways of dealing with these vulnerabilities is by actually building your images correct there is rarely a need to take on every dependency of your base OS so by either manually crafting images or by using image minimizers like DockerSlim you can make your life much easier.

Vulnerability scanners especially for container images are indeed problematic since they are often only doing static analysis on the image SBOM and comparing it to a list of vulnerabilities, this is where half of the problem begins which is again where tools like DockerSlim come really handy.

DockerSlim also has another benefit of forcing teams to have good test coverage for the minimization process to work correctly and also produces runtime policy files which can be used to restrict containers further.

A "zero vulnerability" approach is probably harder.

It is not, at least not as far as a policy goes. Zero by default is really the only effective way to handle these situations, especially when you’ll have to otherwise prove that it’s not vulnerable as the cost of getting it wrong hangs over the security team.

All for the small price of paying attention to the lines in Dockerfile, which is presumably already being looked at/maintained

You are underestimating the problem, looking at a few lines in a docker file will not solve this issue.

Take for example various framework vulnerabilities the amount of effort require to ensure that these are not exploitable is monumental even if the specific containerized workload can be proven not to invoke any vulnerable code it doesn’t mean that other workloads won’t today or tomorrow.

Then you have other vulnerabilities that may not be relevant to your workload at all however if your container isn’t locked down properly, you don’t have runtime protection and there are other vulnerabilities they would allow an attacker to chain them together.

This is why reducing the attack surface of the underlying image itself is one of the most cost effective preventative measures you can employ.

I'm totally with you, I think my short/snide remark wasn't taken quite properly.

Containers shouldn't have anything more than what they need. I downplay the scrutiny a bit, boiling it down to "don't put the bad thing in, to begin with"

The trick is identifying it, of course - but a good start that's commonly missed is simply stripping (or not supplying) the build tools

Oh, but I am going to defend Dependabot. It’s not there to secure your app. It’s there so your smallcorp team can get one or two PRs a day against your codebases, updating your software dependencies one at a time — rather than an unmanageable mess or slow rot on old systems. That’s literally the original design goal. Any security implications are indirect.

> Container scanners and dependency bots like Dependabot are snake oil solutions sold to people too lazy to actually secure their application.

Many years ago I'd have agreed with this so hard I'd hurt myself. These days I've moved on to a much more nuanced "maybe".

Also many years ago the code might've had half a dozen dependencies and the developers knew quite well how they were used and what code paths were relevant now or in the near future. So it was fairly tractable to say: Nope, that doesn't impact us, ignore.

These days there are thousands of dependencies and if I grab a random developer from the team and ask them "How is libfoo being used?" the response is most likely along the lines of "libfoo? never heard of it" because it was brought in as a third-degree dependency. So is the vulnerable functionality conceivably relevant or not? Nobody in the house knows.

In such a world, it is quite often easiest to have a blanket policy of update/patch everything if at all possible. Only if some upgrade causes trouble, then spend human cycles to review and analyze what's up.

I've seen teams that try to analyze everything fall further and further behind because vulnerabilities come in faster than they can analyze.

I understand the dread people have for these scans, more noise than signal

With that said, if your application doesn't need these things, why are they in there?

Time and effort, usually. So there's a choice; continue chasing the vulns or you can improve your builds.

You get out what you put in, the process can be valuable. It's not because cargo-culting.

Man, I'm torn on this one. I work at a reasonably large shop where we are required to take these CVE scores seriously. I do see a bunch that are really not applicable every single day. (The dude from Mitre has earned a special place in hell for https://nvd.nist.gov/vuln/detail/cve-2016-1000027)

When I look at images where there are thousands of issues... it almost always is something end of life that is not supported. Our folks should not be using it. General rule of thumb, if you see an issue that you can patch - do so. There should be enough automation/testing so that minor version ++ is not a scary event. The real challenge is getting the business side to understand they need to allow time for updating. A SpringBoot 1.2 app might work just fine, from a functionality perspective. We try to use the CVE cleanup as a tech debt indicator to make sure folks are finding the time to upgrade from Java 5. Crazy, right?

Most folks just use a docker image. Few actually build them or know enough to maintain someone elses. Surprisingly rare skill.

Seemingly all businesses of any size are using applications in production that are no longer being developed.

Yea, I can take that application and upgrade that library... Will it work? Might crash. Might work fine. Or in the worst case silently corrupt data in an almost unnoticeable way that we don't notice for till 10 million transactions down the line.

Containers are often used to get around the fact that a lot of languages and environments make it hard or impossible to generate a single statically linked binary.

Containers are at least as problematic for security updates as a statically linked binary and more inefficient.

In my experience, the servers that hosted an application before it was in a container was much more insecure. An unfortunate amount of companies don't patch servers or upgrade the OS over time.

In the past year, I've worked with Solaris from 2005 and CentOS 5, and migrated them to either Kubernetes or modern CentOS versions.

How many of these containers were pulled down from the web vs. being built in-house and slimmed down or built slim to begin with, meaning they only have the binary files absolutely required by the application?

Using builder images and copying the bare essentials is usually what I aim to do. Not just for security, but because it's so much nicer to work with tiny images, no matter how much caching you do.

Sure. And 99.99% of them aren't exploitable.

I have a Python-based pod that polls an internal API endpoint. Why should I care if the Python container has a critical vulnerability completely unrelated to what it does?

PRO:

1.) https://en.wikipedia.org/wiki/Swiss_cheese_model

2.) Our own blind spots + unknown unknowns

CON:

1.) There are many false alarms or not critical issues :

see: https://github.com/search?q=org%3Adocker-library+CVE&type=is...

Reply on news.ycombinator.com