Canonical's 'distroless' Linux images are a game-changer for enterprises | Hacker News Reader

Canonical's 'distroless' Linux images are a game-changer for enterprises

zdnet.com

55 pointsCrankyBear80 comments

"Distroless docker image"

We already have this from docker itself.

https://hub.docker.com/_/scratch/

I don't see how adding cannonical to our minimalist null distro adds any value. The idea of minimalism is to take away. What would they be doing anyway?

Sounds like a corp just wanting to get in a business. Antithetical to the idea of removing everything but the kernel.

From scratch will not include CA certificates, timezone data etc.. which is necessary for some applications.

Also Google has distroless images based on Debian for long time. Surely they have some use for it.

It's basically canonical is moving towards image building supply chain ala bitnami. We get trusted source to base image building and support from canonical.

Where are the old bitnami images now?

Does Docker offer 12 years LTS?

"scratch" is an empty image, so you technically get support forever

This isn’t a very useful comment. It’s about as useful as an empty container image by itself

sure, it’s “distroless” and also has the fantastic property of having 0 security updates required forever, but surely you can see that the moment you add anything to it to yourself neither of those properties may apply.

the security updates occur in the kernel, which is outside docker control. You are free to install an LTS or SLTS kernel :).

In the first paragraph of the article that you’re commenting about:

> These custom "distroless" Linux images are set to come with 12 years of security support for Linux, and any included open-source application or dependency within the container

They're offering to build custom container images, with the applications configured to customer specification? And the customer is free to host the resultant container on any OS / cloud provider of their choice. So it's a paid service, not a product?

Am as skeptical of this as I am of 'serverless' which also doesn't mean much and isn't true.

What does this actually mean and why do they hide it behind a nonsensical term?

I haven't seen ubuntu's version of distroless but the general term implies a container image with an empty shell of what a linux distro provides. Leaving behind only the minimum files for running your app rather than all the cruft around running and maintaining a distro, which is normally all done at container build time.

From the Debian based https://github.com/GoogleContainerTools/distroless/blob/main...

Statically compiled applications (Go) that do not require libc can use the gcr.io/distroless/static image, which contains:

    - ca-certificates
    - A /etc/passwd entry for a root user
    - A /tmp directory
    - tzdata

Applications that require libc but do not need libssl can use the gcr.io/distroless/base-nossl, which contains all of the packages in gcr.io/distroless/static, and

    - glibc

Most other applications (and Go apps that require libc/cgo) should start with gcr.io/distroless/base, which contains all of the packages in gcr.io/distroless/static, and

    - glibc
    - libssl

What I have found is that at least Google's distroless images do not get updated frequently enough to handle security issues.

Because of this, I ended up switching to Ubuntu images and doing apt-get dist-upgrade before anything else to try to pick up fixed binaries.

If Ubuntu truly is able to keep an up to date cadence for its distroless images, then I'd definitely want to investigate to see if I should switch.

Looking at the dependency listing I'm trying to figure out why this would be. There are like 4 dependencies here in the largest of these containers, so why is it so difficult to keep them up to date?

The issues I've seen are around updating the run times to point to new release versions in a timely manner. The project maintainers don't actively work on (the endless) security updates and defer people to commercial projects that can spend the time doing that.

The images are tiny, and trivial to update. You don’t really need theirs, just take the bazel rules and build it yourself.

I'm using rules_distroless+rules_oci with bazel to build small Ubuntu based images. It takes some effort, but ends up being pretty nice.

They are not hiding behind a non sensical term. Distroless is a established term in the container space. Many images are often offered in both "normal" and "distroless" flavor, where distroless is stripped of anything that is not needed to run the application, for example curl and shells etc.

Since distroless are lacking all these tools other techniques needs to be used during troubleshooting and debugging.

I think distroless images are great. Less stuff to patch and smaller images.

A term can be both established and nonsensical, like grandparent's "serverless".

It doesn't make sense in multiple ways.

"Distro" means "distribution", so "something that is distributed". Aren't "distroless" images being widely distributed?

It seems that "distroless" images are being built from a distribution, and then by removing some of the contents. So you must have a distro first, otherwise you can't have "distroless". That doesn't add up semantically.

Words can have different meaning than its etymological parents.

If I seed Ubuntu it doesn't become blckoil distro. So the word distro means different from distribution.

The issue is you could make and maintain a blckoil distro tomorrow that only keeps the bare minimum to run a go application. It would be bit for bit the same as the "distroless" version, except it will be a distro.

The "something that is distributed" is a package manager and package repositories. What makes a distribution is what choices it makes in letting you install software.

A distroless image doesn't include a package manager. You cannot use anything in the provided distroless image to install software.

That doesn't make it not a distro, that makes it a different kind of a distro. I'm not sure if we've had ones that absolutely had no package manager, but we've definitely had TONS for which the intent ISN'T installing extra/different software,e.g. a bunch of the specialized raspberry pi ones.

Do you think Linux From Scratch https://www.linuxfromscratch.org/ is a distribution? That project is not a distribution, it helps you build your own distribution. The Linux kernel by itself, although the source and binaries are "distributed" to you (i.e. English-dictionary definition of distribution), is not a distribution.

There is no essential difference that makes Linux From Scratch not a distribution. None.

Some distros come with a lot of stuff. Some with very little.

That it can be used to make another distro has NO BEARING on this. What would you call that thing if it's not a distro? What is that noun?

> A term can be both established and nonsensical

Not at all. The establishment of a term, and giving it a definition that's widely agreed upon, by definition (heh) makes it not nonsensical.

> "Distro" means "distribution", so "something that is distributed"

No. In the Linux world a distribution means something much more restricted than that. It's a full system that can be installed, complete with a package manager, init system, etc.

"Distroless" simply means it's a minimal set of software and libraries such that it can run a particular binary that you copy into it, but it doesn't act like a distribution. You can't install libraries and programs into the container using a package manager, for example, like you'd be able to if your container held a stock Debian install.

> It seems that "distroless" images are being built from a distribution, and then by removing some of the contents. So you must have a distro first, otherwise you can't have "distroless". That doesn't add up semantically.

I don't see where you're getting that conclusion from. If you take a distro, install it into a container, and then strip away everything that makes it a distro, then it is indeed distroless. How you got there is irrelevant.

You could even build a distroless container without installing the distro into the container first. You could install the distro into a chroot (or into another container) and then selectively copy things out into the target container. Or if you were a masochist, I suppose you could even download source tarballs of everything you wanted in your distroless container, and manually build them on a host system and copy them into the container.

Again, how you get to the final state is irrelevant. The idea is that the final state is runnable, but does not contain all the bits you'd expect to see in an installed distro.

I've seen nothing to indicate that your definition of "distro" is THAT restrictive.

I'm confident saying a linux distro is anything with a linux kernel that you can slap on bare metal or a VM system and get something up and running.

You're merely describing a minimalist/specialized distro, a not particularly new or unique concept.

I mean in the beginning you'd be downloading the kernel code yourself and building a distribution with core utils. Distros came along and did all that work for you: they were a system that packaged up the kernel with a lot of useful utilities as well as an packaging and update mechanism.

Of course you can find a way to be upset with the term if you want to, but I think it's in this sense that these container images are distroless.

The term makes a lot of sense when you read it as having the package manager at it's core, which also happens to be the update mechanism. But in the container world, that would be an idle update mechanism and package management (as in selecting the subset of installed stuff) is also handled through image selection. But an image created from scratch by whoever would be considered end user in the container world would still contain the package manager of the distribution used to set up whatever is eventually imaged. With this as the baseline, distributionless is either an image that has this umbilical removed, or, as purists would probably prefer, an image where the package manager has never existed because everything was set up without involving one, or set up from the outside, without ever running anything "inside".

So they could have called it "minimalist", "barebone", "stripped".

I share parent's view that it's marketed with the most provocative term to attract attention, where an appropriate name wouldn't have stirred enough controversy to their taste ?

I'm quite surprised by this descriptive term also; in fact I've always (for years) called this type an "application container" since it only contains the files required by the primary process and doesn't have a standard init system (usually it is expected to handle PID 1 responsibilities in its primary executable or use a stub init (as when using e.g: systemd-nspawn --as-pid2 ... ).

In contrast a "system container" has a regular init system and likely also package management (although that is optional) but was created using a package manager so the miscellaneous package-installed files are usually in there.

They could, but "distroless" is already an estabilished name, I think pioneered by Google[1]. As someone close to this field, I immediately know what "distroless" means in this context, and it's unclear what "minimalist" mean (does it mean it's based on Ubuntu Minimal? Alpine? Distroless?).

[1]https://github.com/GoogleContainerTools/distroless

those terms have already been used in various distro installers for installations that contain very little, but still all the setup for you to expand it into a full blown distro with the package manager.

The distroless images specifically don't give you a package manager, only specific packages needed for you to run.

It means that your containers only have what you explicitly put there. There's none of the niceties that the distro's typically include.

It means you have to stop treating your containers a little less like VM's and start treating them a bit more like OS processes. For instance, you can't get a shell on one unless you bother installing that shell yourself.

How would you install a shell without a package manager?

Edit: @mhio That's what I suspected, thank you.

I assume the images are pre built based on a manifest provided by the developer. The developer decides whether or not to include a shell and which shell to include.

What are these manifests? I presume you are correct - the images are prebuilt (as typical for docker), but then it's like coreos, what you get is whatever you `docker pull'ed, most likely very basic (by design).

Great in cases where the app is a self-contained Go program only requiring libc, but not useful when you need to apt/yum/dnf install anything.

You don't.

There are tools like `nsenter` to act on the container namespaces.

https://manpages.debian.org/bookworm/util-linux/nsenter.1.en...

Is this that hard to do on your own? You can base from scratch if you want. Then you’d have to add just what you need which I guess could be hard.

Probably not, I've never tried it. But I assume you can waste a bit of time puzzling out the few things to include which are actually needed every time, and a distroless image has that pre-puzzled for you.

If I wanted to be so specific about what's going into my images I'd probably just build them with nix.

I guess the key thing is that they promise the security fixes for 12 years.

This is interesting, because in large org you ofren have lot of stuff you just want to keep on running with minimal changes.

I would say a lot of developers don't really know their way around a Linux distro as well as you might think?

It's the same logic as frameworks or other building blocks: why leave all these decisions up to individual teams, when everyone should be doing things more or less the same way?

If need to do something custom, you'll know what you're doing and you can do that. Otherwise, use good defaults provided by people who know better than you.

I mean... one certainly could do this. And then as soon as anything breaks with that container, the person trying to fix it will curse the day you were born. People already go way too far with stripping container images, and it's awful for troubleshooting. The last thing we need as an industry is taking that silly trend even bother.

If you're going to use minimal container images you definitely need to know how to inject tools into the container.

Yeah and it's a true pain. Double that if your container security policy requires running in fully rootless mode (i.e. not just the usual CAP_ALL drop but also container uid != 0), quadruple if you're in a Kubernetes environment, and even more if you can't access the nodes themselves directly (AWS EKS).

I'd say you need to know how to properly instrument your containers so that you'll never need to get inside. If your team is on the level of "we need tools inside the container for troubleshooting", then don't use minimized containers like these.

It's about security and reducing the probability of getting CVEs for your running containers. Less software you bundle there, less security issues you will have. And this in turn means less work on maintenance as you don't need to update, test and deploy new versions of images so often.

When the cow gets sick, you shoot it.

You still want to know why the cow got sick, in order to prevent all other cows in the stable from falling sick as well.

And ideally, you want to keep the cow alive, at least if you fed it artificially.

You're not looking at this from a DevOps perspective.

In real-world cloud environments, logging into the container and expecting to have a complete Unix environment to poke around in just isn't an option. You might not even have the required permissions to do so, as a matter of corporate policy. You need to build observability directly into the application -- or you haven't built a complete cloud application.

> You need to build observability directly into the application

This is not objectively better than external observability.

If the "DevOps perspective" requires it, then it's a pointlessly inflexible perspective.

> just isn't an option

You don't get to blame the system when you made the system.

The option is there.

That's the key thing why DevOps is frankly a pile of bullshit and why so many companies ridiculously overpay on cloud infrastructure.

When you got clueless morons who were in a 6 week nodejs bootcamp now "developing" containers, with no one even having the permissions to troubleshoot crashes - that may very well originate from someone trying to get remote code execution - it is absolutely no surprise that hacks, ransomware, infrastructure so brittle that it only doesn't fail because EBS / EKS keeps restarting the pile of dung every minute and infrastructure pieced together from 40 AWS services where one classic EC2 VM would be sufficient are plentiful.

Developers having shell access to production container is not the best practice, to say the least. For debugging locally, it's very easy to nsenter your container.

We use Google's distroless containers for production, but we also have alternate Dockerfiles available in our CI/CD for building a more traditional container that can be logged into if there are application issues to debug.

It's not an exact copy of prod, but so far (a few years, and thousands of containers) it hasn't really been an issue as far as I've heard. We don't do anything fancy though. Node, Go, Java and Python apps..

I think the point is that you should have a plan (which may or may not require debug tooling in the image) and you should include what you need for that plan and nothing more.

Finding something useful that the distro left lying around but that wasn't part of the plan might feel like a win here and there, it's just as likely to be a win for your adversary as it is for your friend.

Think in the way that even without kernel and toolkit, many deployments still need libraries and frameworks. Like python deployment needs python runtime, AI deployment needs Python and pyTorch.

Canonical will support that.

It's actually make sense as you don't need package manager or other things (basically distro toolkit) to build and run container image.

I suppose now that they've containerized a huge amount of their repositories' userspace applications and they do much less work there (all pushed to upstream container makers) they can have those employees working to support other software than their open source OS.

> Canonical plans to maintain the 2,000 widely used AI/ML libraries and tools, including heavy hitters such as PyTorch, TensorFlow, and Rapids, as source code instead of as Debian/Ubuntu deb packages.

That's rough. Hopefully the desktop ecosystem doesn't suffer. That said, 12 years sounds amazing! I wish this were applied to their actual desktop LTE and ESM and not commercial docker container using non debian package management.

This actually sounds like an amazingly simple (at least in principle), and probably effective solution to a problem that does exist that I didn't realize needed to be solved.

https://www.fedramp.gov/assets/resources/documents/Vulnerabi...

This seems like a spec. Did you mean to say that there are similar products/services that already implement this?

no. i am saying that this is actually a problem. especially for companies that need to implement certification like fedramp, which is very strict in areas of vulnerability management/keeping things updated/secured/hardened per specifications. (each individual cve vulnerability must be detected and remediated within specific timelines, based on it's severity)

i am currently working on fedramp certification for a big system, and all requirements are serious pain in the ass to comply with, in case your organization not used to it

Is this the beginning of the end of free Ubuntu?

Already, Canonical keeps pushing updates for "Ubuntu Pro".

I don't think so. Ubuntu pro is support package offered by canonical, including longer LTS support until 10 years.

So no, it's not Ubuntu becoming not free.

Also, ubuntu pro is free if you have 5 or fewer servers.

> Also, ubuntu pro is free if you have 5 or fewer servers.

Just worth pointing out that, its 5 free for PERSONAL (or your own business) use.

    "Free, personal subscription for 5 machines for you or any business you own"

So if you're sysadmin'ing for someone else, pony up buttercup. :)

P.S. ALSO read the small print. Virtual instance = Machine ("on up to five physical or virtual Ubuntu systems ")

Yes, and I think that is a very reasonable policy overall. I was screwed by Centos when they suddenly dropped the support from Centos 8 and moved to Ubuntu. So far I've been very satisfied. Canonical has figured out that you can make money without screwing over your users.

I don't really get why Ubuntu is still popular. It was a wonderful idea at first, but has gone downhill quite a bit since then. Debian with the contrib and non-free dists enabled is plenty capable enough for most people, and then you don't have to deal with all of Ubuntu's and Canonical's baggage.

> I don't really get why Ubuntu is still popular. It was a wonderful idea at first...

With you 100% here.

I think sadly its a case of so many internet tutorials starting with "download Ubuntu".

Like you, I used to use Ubuntu a lot back in the day.

Then the enshittification started, the nods and pushes towards commercial, the various "features" I didn't want or need etc etc etc

Ubuntu are a profit-making commercial company, I just wish the rest of the community would understand Ubuntu are moving more and more towards the Redhat end of the spectrum stop writing tutorials with the words "download Ubuntu".

Its also annoying that, for example, Github built-in runners only support Ubuntu and not Debian builds.

Ultimately, like you, I went to Debian and never looked back.

> Then the enshittification started

Even in open source.

It's a problem on Android. All the "Simple ..." tools have been bought up by someone who's coming out with "Pro" versions.

I used Ubuntu on my servers for a long time, because they had recent-ish tools and the experience was nice.

Then I got into a situation where a server wasn't updated often enough and the do-release-upgrade script told be to go pound sand.

Switched to Debian testing, which is a rolling release. Haven't had any issues since.

Debian testing is a rolling release like the Pacman shape is a wheel. Every once in a while, it stops rolling and just drags along the surface for a while. And when it starts rolling again, it's very bumpy.

Do you really see that much of a runtime gain with scratch images (or distroless, same general idea) to justify using them? Not a criticism, I’m genuinely curious to hear from people experienced with both sides of this.

Main goal of using distroless containers is increased security. Runtime gain is kind of a nice benefit you get for free. Security gains from distroless containers come in two flavours:

* Reduced attack surface from some vectors, especially they make the container resistant against many common RCE exploits (almost all RCE attacks end up executing shell from the host, and shell is missing by design from distroless containers). Similarly, many in-container privilege escalation attacks are thwarted by this. * Drastically reduced numer of vulnerabilities reported by security scanners. This makes compliance people happy, even though it doesn't really increase security by itself.

The runtime gain is probably near zero. But there are a handful of advantages.

1. Smaller images. This reduces storage cost, pull time, setup time and can potentially allow more active images to be cached on your nodes.

2. Less installed means less for security scanners to trip on. Even if some software is unused it is good practice, and me be a compliance requirement to update with a fix. If it isn't installed there is no worry.

3. Simplicity. You know that things that aren't installed aren't causing issues. So tiny images can be easier to debug.

Is this a response to Chainguard? https://www.chainguard.dev/

If there is no distributed binaries, or executable files .. what exactly are they in the hook for supporting?

Everything. From the actual horse's mouth [0]:

> “Everything LTS means CVE maintenance for your entire open source dependency tree, including open source that is not already packaged as a deb in Ubuntu” said Mark Shuttleworth, CEO of Canonical

The idea is that you just focus on producing the code and outsource the maintenance of the entire container image to Canonical.

[0] https://canonical.com/blog/canonical-offers-12-year-lts-for-...

I don't see how this is "distroless".

Clearaly, this is a Canonical distribution.

Someone has to choose the exact combination of userland s/w to include with the kernel. Whoever/whatever chooses this s/w is creating a distribution.

To this boomer yelling at the cloud, this is just more nonsensical hype, equivalent to "serverless".

CrankyBear aka sjvn?

Honestly? Is another way to quickly push crap in an infra, crap because you can't maintain nor know it enough, it will so stay for a looooong time, no one will know how to recreate it as needed, all will be done in hackish, quick and undocumented/untraceable ways and so on.

Oh, I've no doubt some devs in Silicon Valley Mode and some managers will like it of course, as I've no doubt at a sudden point in future they'll regret.

Ladies and gentleman's it's 2024, the era of containers MUST end for the sake of humanity, similarly to the previous full-stack virtualization on x86, it's about time to go declarative like NixOS/Guix System. The current typical infra is a big load of unmanageable, fragile crap with a so big attack surface that you have only to choose how to get TFU.

Reply on news.ycombinator.com