Wolfi: A community Linux OS designed for the container and cloud-native era | Hacker News Reader

Wolfi: A community Linux OS designed for the container and cloud-native era

github.com

169 pointsthunderbong86 comments

I've attended a local CNCF meetup where the chainguard folks presented Wolfi and their related tools to create container images and SBOMs.

I was already skeptical of the product, having heard of it before. Unfortunately, attending that talk confirmed to me that they've just re-invented a severely limited version of Nix powered by yaml files that can output SBOMs in a standard format.

Their software repository only has around 500 packages vs the 80k+ in nixpkgs. When I read "Alpine Linux", I was immediately reminded of its use of the musl libc which has endless DNS resolution issues[1][2] in and outside of Kubernetes, however, thankfully, it seems like they realized bundling musl was a bad idea and ship glibc instead.

I'll stick to Nix and nixpkgs to build my reproducible container images with full dependency graphs back to the source code.

1: https://martinheinz.dev/blog/92

2: https://news.ycombinator.com/item?id=35058094

We're rapidly approaching 10k packages, here's today's count:

/ # apk update fetch https://packages.wolfi.dev/os/aarch64/APKINDEX.tar.gz [https://packages.wolfi.dev/os] OK: 9494 distinct packages available

We're definitely coming at this from a different angle from Nix, but the approaches are pretty complementary. I'm a big fan of all the work they do.

musl vs. glibc is one of the big departures we make from Alpine though, we use glibc everywhere because of those issues you pointed out.

> OK: 9494 distinct packages available

I opened that apkindex file and it had duplicate entries for a ton of packages with different versions, taking a look at https://github.com/wolfi-dev/os I only see about 840 yaml files which I assume define the packages. I don't think claiming to have 10k packages when only 10% of them are actually different pieces of software is a good claim to make. Nixpkgs would have millions of packages if we added up every single unique package from every revision.

The real number is probably somewhere in the middle - one yaml file can define many packages - see the gcc or clang or argocd ones for examples of that.

glibc explodes into a few dozen, for example.

I think it's more subpackages such as -dev, -lib and -doc variants. These are defined as part of the parent package but count as distinct packages.

Even so, I did a quick search on repology and Nix derivations with multiple outputs (the nix lingo analogous to the subpackages you mentioned) are counted as a single package. For example, bash has 5 outputs but only counts for 1 package in the 85k figure, so I think comparing 900 packages to 85k is a valid comparison.

Anyway, this is all besides the point I was trying to make which is that I don't see why I should use _yet another_ software distribution that has 1% of the amount of packages found in a mature distribution that already has frequent automatic updates and bleeding-edge software revisions.

We needed Wolfi to be able to create minimal (distroless if you like) container images based on glibc with 0 vulnerabilities. Turns out a lot of other people are interested in Wolfi for various reasons, and we're more than happy to work with them.

You definitely don't need to use Wolfi! But I would say, if you run containers you might want to check out Chainguard Images: https://github.com/chainguard-images/images

Hey, thanks for chiming in. How do they complement each other?

Can anyone help me understand musl libc and DNS issues? Note that I’m most interested in this “DNS over TCP” issue, since the other case I’ve heard of is for custom DNS setup—not for resolving host names in a default configuration.

My reading indicates that DNS resolution simply might not work in certain cases. This seems like a huge problem, yet Alpine Linux is widely deployed and I think Zig uses musl libc as well. In fact , every fully static binary I’ve seen (except for Go?) relies on musl.

For what it’s worth, I’ve seen DNS errors on Alpine (specifically I was getting EAGAIN), but I assumed this was unrelated to musl (and am still unsure). In general, I feel like I see a lot more transient networking errors on Alpine, and I wonder if this is related.

Edit: I also didn’t think DNS would be in libc, so I’ve got a lot to learn…

It sounds like DNS-over-TCP should be supported now: https://wiki.musl-libc.org/functional-differences-from-glibc...

Edit: Hacker News link from last month:

https://news.ycombinator.com/item?id=35964717

If you use Musl 1.2.4+ (or Alpine 3.18+), there are no longer the same DNS fallback issues: https://www.openwall.com/lists/musl/2023/05/02/1

To summarize the issue: DNS is done optimistically over UDP because it's faster, but this doesn't work when DNS responses are large because of the design of UDP. TCP should be used as a fallback mechanism when responses are large. This is uncommon normally, but increasingly DNS responses are large in special scenarios; for instance when you're querying an internal DNS for service discovery (read: k8s or nomad deployments, most commonly).

Musl's maintainer interpreted the spec for a libc's resolver to not require TCP fallback (source: https://twitter.com/RichFelker/status/994629795551031296?lan...), so for a long time Musl simply didn't support this feature, justifying it as better UX because of the more predictable performance.

I don't agree with the maintainer on this interpretation, but I am glad the feature was added and the issue is no longer a concern as an otherwise very happy Alpine user!

I’d found bits and pieces of this, but I didn’t have all the context. Thank you for summarizing!

I'd say he was wrong here, and his assumption was incorrect.

RFC2181 specifically says 'Where TC is set, the partial RRSet that would not completely fit may be left in the response'

'may be' being the key words. This would mean that it's up to the implementation to decide whether to include any records at all, and many do not.

As mentioned correctly in [2], DNS issues with musl stem from the fact that it follows the DNS specs strictly and exposed bugs in certain DNS servers.

For an HTTP server replying with 200 OK instead of an 404 Not Found upon ENOENT, the situation is clear that this is not a client-side bug. The same should be assumed of DNS/NXDOMAIN.

As much as I agree with your reasoning, I have experienced this kind of breakage first-hand due to cloudflare and DNSSEC and will continue using DNS resolving code that doesn't cause me pain and suffering (anything that isn't musl) because making this work with musl is quite literally out of my control.

Wait, are you me? I tried to convince musl to not break in this case (https://www.openwall.com/lists/musl/2022/12/04/1), but mostly got a "we're right, cloudflare's wrong" answer. Tweeted at Cloudflare about the issue (https://twitter.com/KennyMacDermid/status/160055878578481971...) and never heard back. I really don't care who's right, and can't change who's using Cloudflare+DNSSEC, so instead I just don't use musl.

[Previous comment](https://news.ycombinator.com/item?id=35058094): ---

My personal 'musl broke it' story comes from resolving domains from Cloudflare that use DNSSEC in a K8 cluster. Basically this:

  - K8 sets container to use `ndot:5`, causing the search list to be used
  - Musl walks that search list looking for domain
  - Cloudflare does not set the NXDOMAIN flag on a DNSSEC domain but does include an NSEC record (if you query with the dnssec flag).
  - Musl takes this 'NOERROR' reply and returns an EAI_NODATA.

Is Cloudflare wrong? I don't know, maybe. They say some things about the standards[0] and that it's technically 'right'. I don't see why they couldn't change the behaviour for queries without the dnssec flag, but I digress.

The issue is that every other libc I tested will continue searching and actually resolve the domain. Musl is the odd one out, and _only_ in the case where the search list ends up with domain using Cloudflare and dnssec.

Even if Musl is 'right' here, when it disagrees with major implementations and a major DNS nameserver does it really matter?

[0]: https://blog.cloudflare.com/black-lies/ ---

> Wait, are you me?

If you hadn't made that HN post I would probably still be scratching my head as to why I have intermittent DNS resolution issues in my personal Kubernetes cluster. I've been having them for years only for some specific software (my mediaserver stack, packaged by linuxserver.io). Since this isn't mission critical stuff I just shrugged it off and dealt with it. Eventually I read your post and it all clicked. I can't find any maintained distributions for that software that don't use alpine/musl sadly so I added[1] dnsConfigs to all of my deployments using musl to force ndots to 1 and accept the fact that all of my cluster name resolution for those deployments will have to be fully qualified. It's a really frustrating situation and made me realize how badly musl plagues the container ecosystem due to its use in Alpine Linux and alpine's popularity because of the small images it produces. Alpine Linux and musl are on my shitlist for life.

1: https://github.com/starcraft66/infrastructure/commit/3b53bb0...

musl doesn't implement TCP DNS at all. When it gets a respond with the truncation bit set, it just assumes the completed records in the truncated response were sufficient, the best records even, and proceeds as if the lookup succeeded. It's hard to take seriously claims that it's strictly implementing the protocol.

It does as of a month or so ago. https://news.ycombinator.com/item?id=36494177

> endless DNS resolution issues

should now be resolved (UDP fallback to TCP), even though it's arguable that this is actually rooted in issues elsewhere in the stack.

I would love to see non-trivial examples of using the nix toolchain to build images with multiple OS, architectures, SBOMs. As someone unfamiliar with the nix ecosystem, it seems like a tough ask for contributors to require nix knowledge rather than just changing out my existing base image.

I'm not sure what you mean by "non-trivial" but here's a simple discord bot I wrote in python, that I distribute as an OCI image and that is built with Nix for both x86_64 and aarch64 linux via GitHub actions: https://github.com/starcraft66/attention-attention

There is no SBOM because I didn't bother publishing one but the way Nix builds derivations, you basically get the SBOM for free. You could use a tool like sbomnix[1] to trivially generate an SPDX-format SBOM from the nix derivation that builds the container image.

Edit: Since you mention swapping out base images, I think there is a misconception about how building images with Nix works. There is no such thing as a "Base" image, nix builds images from the Dockerfile equivalent of "scratch". You would ditch the Dockerfile completely and use only Nix to build the image.

1: https://github.com/tiiuae/sbomnix

Not sure an OS created to meet the specific need of VC funded startup, created exclusively by employees of that company should be called a "community Linux".

It's almost the opposite of that in fact.

You're welcome to join the regular community meeting: https://github.com/wolfi-dev/community

We do fully intend for wolfi to be a community project, but it will take some time. We do say on the home page:

>What are the plans for long-term Wolfi governance? We intend for Wolfi to be a community-driven project, which means over time it will have multi-vendor governance and maintainers. For now we're focused on building the project and community, and will revisit this in several months when a community has formed.

++the next call is July 5th at 12 pm EST. Come with questions! Here is the direct link to the public calendar :o)

https://www.google.com/calendar/event?eid=aWpydnZxc2VhaHBlMz...

Totally understand the concern. We've tried to draw a very clear line between Wolfi (the project), and Chainguard Images (our product). They're in different GitHub organizations, have different maintainers, and are documented differently.

We (Chainguard) still represent most of the maintainers to Wolfi, but there are external folks too. We're going to need help scaling Wolfi to get it to where we want it to be, and that will require a real community.

We've also tried to be very clear on the business model for us, to avoid any "rug pulls" or "bait and switches" later. Here's some more context on that too: https://www.chainguard.dev/unchained/scaling-chainguard-imag...

We use Alpine Linux in all our docker builds for AWS ECS / Fargate. I noticed the FAQ says this:

> Wolfi is a Linux undistro designed from the ground up to support newer computing paradigms such as containers. Although Wolfi has a few similar design principles as Alpine (such as using apk), it is a different distribution that is focused on supply chain security. Unlike Alpine, Wolfi does not currently build its own Linux kernel, instead relying on the host environment (e.g. a container runtime) to provide one.

Is it possible to expand on this more? Why should I, a technical expert, use this over an already established distribution (Alpine)? What are good talking points I can bring to management / architects that would sell them on a switch?

Can you expand on what “supply chain security” actually means?

In this case, supply chain security mostly means compliance. Alpine and other distros already do a great job at most aspects of supply chain security, but we make package/image signatures and SBOMs very easy to get in Wolfi.

We're also more flexible on packaging extra software in Wolfi than other distros are, which has an effect on your overall supply chain security posture. We're aiming to package the entire Cloud Native landscape (to start!), so you can simply "apk add" anything you need, without having to drop back to "curl | bash" in your Dockerfile.

By making it easier for devs to get the software they need "the right way", they'll have a reduced need to work around the already existing secure distribution mechanisms built into their distros.

If you're already on Alpine and like it, that's great! If there are some packages you can't find in Alpine, or musl vs. glibc ever causes you issues, or you need SBOMs for some reason, give Wolfi a try!

I’ve worked in a few industries that were crazy about compliance. Do you have any certifications for different compliances? Ie FedRamp, IL4/5/6, HIIPA, etc.

I don’t think any other container images are strictly compliant with these standards - they’re not certified, but the technology stack (ECS, EKS, etc) is certified. Either way, if you can get this certified with different compliance platforms that would be a big selling point for B2B customers.

We're working on FIPS builds and will self-attest to the NIST SSDF stuff later this fall, but there aren't too many other requirements that directly apply to our images.

The images are very useful for other organizations working on FedRAMP or HIPAA though, even though we don't need to have those ourselves. We wrote a bit more about that here: https://www.chainguard.dev/unchained/fortify-comply-and-conq...

Do you provide an OVAL feed?

Alpine is out of the picture for us because the guy that works on their security tracker just doesn't care, and responds half a year after filing an issue. The tracker itself is broken for over a year and the response was to basically rebuild our own package index and host our own security tracker.

So I would not say that Alpine has security as a high priority, even though in theory there are the secfixesdb.

Redhat Enterprise, Debian and Ubuntu are used because they provide an OVAL feed that are easily integrated with zero development overhead. So if compliance is your focus, I'd heavily recommend generating an OVAL feed when you're regenerating the secfixes json files.

Source: Am building a cross-linux-distro vulnerability database and I am scraping _all_ linux security trackers. [1]

[1] https://github.com/tholian-network/vulnerabilities

Thanks for the pointer! We update the JSON feed now but should be able to generate an OVAL feed too.

OVAL: https://oval.mitre.org/

Probably because "the guy" is not a guy, but who cares, it's easier to just be salty on HN and spread lies about the project.

> it's easier to just be salty on HN and spread lies about the project.

https://gitlab.alpinelinux.org/ariadne/secfixes-tracker/-/is...

Check response time (in months or years) and also closed issues.

What specifically was a lie in my comment?

For one, you refer to me as "a guy." I am, in fact, not a guy.

Secondly, the issue you opened was against an internal project used by the security team to do continuous CVE triage of the distribution. It is not meant to be used for public security data. We are tired of security vendors scraping our internal tool, as generating those reports in real time is very expensive on resources.

For public security data, you can use the secdb: https://secdb.alpinelinux.org/

No, it's not OVAL, but we would accept patches to generate an OVAL feed as well.

I'm sorry I did not ask you beforehand about your preferred pronouns. If you want people to use the correct pronoun and it's such a big deal for you, add it to your profiles.

Regarding the rest: don't call it a security focussed distro if you don't even care about providing the data for vulnerabilities.

If you can't see the constructive criticism I tried to provide in the issues (and pull requests) that I filed then I made a good choice avoiding your distribution ecosystem from now on.

Maybe @dang wants to chime in here to prevent more chan-level escalations.

We do provide the data for vulnerabilities, at secdb.alpinelinux.org.

The security tracker is a tool for the security team to remediate vulnerabilities.

The data provided by it is not particularly useful nor intended for consumption by people other than the security team and alpine package maintainers: it generates reports for possible CVEs to review and possibly mitigate in the package collection. The presence of data in the tracker that is not present in the secdb (either as an ACK or NAK) is just an indication that there is a vulnerability to investigate, not that anything has been confirmed or denied. Really, the data is not relevant as a product for end users to consume.

The secdb outlines what package versions fix what CVEs, and what CVEs have been formally NAKed. Speculative data from a distribution-wide vulnerability scanning tool is not useful data to be making security-related decisions with.

Wasn't really sure where this was going, but the background to this approach adds some details - namely what they mean by distro-less and how they pull it off.

https://www.chainguard.dev/unchained/minimal-container-image...

Seems to have a confusing lack of "getting started" documentation. On the plus side, I now know what the world's smallest octopus is, so the important stuff is covered!

I'm thankful that the important stuff is covered , there is helpful documentation over here btw https://edu.chainguard.dev/open-source/wolfi/overview/

They should probably have it linked in the readme but this is their getting started guide: https://edu.chainguard.dev/open-source/wolfi/wolfi-with-dock...

From my side I'd love to see clear comparison to distroless[1] images, the best info I could find was this Github issue comment[2], although I would expect to find it in https://github.com/wolfi-dev#faq. Or maybe my google-fu failed me :(

[1]: https://github.com/GoogleContainerTools/distroless

[2]: https://github.com/tektoncd/pipeline/issues/5507#issuecommen...

I was (one of) the original creators of the Google Distroless project. The main difference is that Original Distroless uses Debian as the upstream. This is great in many ways, but makes it so stuff outside of Debian is hard to package in.

Wolfi is its own upstream - packages are sourced and built directly from source, without another Linux distro.

The rest of the differences stem from here - we get more control over what we package and how, and are able to more quickly add new versions of packages, or roll out CVE patches.

Do you see any significant changes in comparison to the "gcr.io/distroless/static" image, which is commonly used in Go ecosystem? Thanks for answer btw, really appreciate that.

For static, not really just because there's so little in it. For apps that need CGO there's a benefit as more dependencies are required. CGO apps are broken right now IIUC in distroless/glibc-dynamic because of the Debian glibc bump.

I like the idea of having an SBOM as a standard, baked-in feature from the outset. That kind of thing is becoming critical in software supply chain management.

Thanks! By building the SBOMs as part of the build process we can ensure full coverage, vs. the other approaches that rely on "guessing" the contents after. We wrote a bit more about this approach here: https://www.chainguard.dev/unchained/make-sboms-not-guessbom...

If nothing else, it's at least useful to ensure you are able to comply with all the open source license terms. Now how many copies of the MIT and BSD licenses do I need to include with my software? ;)

To save some people a click, the headline made me assume this was something like CoreOS, but this is in fact the opposite.

It's not a traditional Linux distro you can actually run standalone, it's just another base for building OCI containers.

Yeah, it's definitely not a platform for running containers. Apologies for any confusion there.

But regarding being a full linux distro, we're really just missing the kernel and I could see that happening at some point in the future.

What the hell is an OS without a kernel?

Is this less an OS and more like a toolchain to build container images?

99% of Wolfi is the package repository. We've compiled all the software to run an OS from source against glibc and made it available as APKs (apart from the kernel). The major use case for this is composing container images using tools like apko.

When you run Linux containers, you're using the kernel from the host, but everything else is provided in the container i.e. file system, utilities etc. So you can effectively run Debian or Alpine on top of a Arch kernel by just doing "docker run debian" etc. And now you can also do "docker run cgr.dev/chainguard/wolfi-base".

Distroless or not, maintainers are required… but for me running docker pull debian still does the job.

Whats wrong with current distros vuln assesment and security reporting to have another one?

Debian does a great job! But here's one example where their packaging system makes container workloads hard. Debian, like many other distros has a strict "one version of every package" rule, meaning that Debian only ships one version of common things like programming languages or webservers.

If you "apt-get install nodejs", you can only have Node.js v18, in the very very recently released Debian bookworm. If your devs need Node.js v20, or even v22 which will both be LTS releases during the bookworm release cycle, you're out of luck.

You can go install those outside of the Debian package manager, but then you're on your own for vuln assessment and security fixes. Worse - many scanners don't actually even "see" packages that are installed outside of package managers, so you may not know that these exist or are installed.

We designed Wolfi to be more flexible around package versions. This is a very very hard problem to solve in general, but since we're focused on immutable containers, we can skip a lot of the complexity around conflicts, upgrade/downgrade scenarios, and cross-version compatibility.

That's cool, but aren't containers considered to be a workaround for exactly that problem?

Yep, but workarounds come with a cost. We're trying to package basically everything, so you don't need to pick between "up to date software" and "software from a trusted distro".

It's going to be hard to scale, but we're going to at least try! We have a lot of ideas on how to make this work that I'm excited to try out.

So you want to provide all the up to date software, but as battle tested as the software included in the debian repos, with a few people?

Roughly yes! I think we have a few advantages - mainly that we're focused on container workloads - that simplify the problem. But otherwise, yes!

I'm betting that with enough automation we can get there :)

We can always come back to the discussion of statically linked vs dynamically. If your build system is strong an can adapt to these changes thats another option.

If I need node.js 20 I will just use the node:20-bookworm image.

And the fun part there is that this image is effectively a workaround - node.js 20 is installed via curl | bash. That means CVE scanners and SCA tools can miss node itself in that image.

As silly as it sounds, try running "snyk container test --print-deps" on that image, and look around for Node.

This approach works fine, but means that you might not be able to rely on most container security scanners to let you know when there's an issue.

You can scan the file system, probably thats what AWS ECR does?

At the end of they you need to keep an eye on file integrity, because of rootkits, config integrity…

Yeah there are many wats to approach to this… with its corresponding costs of course

Sounds to me like CVE scanners aren't doing a great job if they can't pick up a nodejs installation from the official nodejs image distribution. Just looking at package manager metadata effectively won't give you the full picture.

Wouldn't then using nix be better? Or at least base the distro on nix?

Before reading the linked page, I thought this was about a Linux distribution optimized to run both local but containerized applications and remote/cloud web applications, kind of a ChromeOS like distribution with the same focus on simplicity and security, but with an addition local execution approach based on containers, and less tied to Google.

So, this is not at all this, but now I wonder if a distribution like the one I imagined exists?

Sounds like you're looking for the CoreOS Linux successor FlatCar https://www.flatcar.org/

It's actually based on some ChromeOS update tools under the hood but is a regular Linux distro, just super minimal and designed to run containers.

VMware's PhotonOS is designed for use as a stripped-down container host. It's pretty good, and they're open to community contributions too.

To add to the other excellent answers, I would recommend adding Bottlerocket to your reading list: https://github.com/bottlerocket-os/bottlerocket#readme

I'm also aware of (but haven't used) https://github.com/siderolabs/talos#readme

I just realized your question may have implied a desktop os, whereas Bottlerocket, Flatcar, and likely the others in this specific thread are server-side. I don't have much experience with trying to solve that problem on the desktop except for the horror-show that is snap

They could've just improved Alpine's supply chain (or built on top of it) rather than making a new distro :( But they're a company, so pushing out new products faster is more important than the community.

We work with much of the Alpine community and some of the maintainers. The glibc choice instead of musl alone means that a separate toolchain and distro is required, but there are also a lot of other distinctions.

We still work with them upstream on apk and apk-tools, and would be happy to collaborate on more tooling.

We have contributed the majority of our improvements in such a way that other APK distributions, including Alpine, can make use of them, and the original plan was to just leverage Alpine for all of this. Indeed, we even sponsored the refactoring of musl's DNS implementation that allows it to support DNS over TCP now.

But we heard overwhelming demand for glibc support from customers. So we built a companion distribution for customers who needed to run GNU/Linux workloads, but still wanted to leverage the advantages of Alpine.

While I do appreciate the great work than Dan and team do, I feel like posting this headline, at this time, is click-baiting and playing off the recent Red Hat shake up to try and grab attention.

The fun part of HN is that you don't get to pick your turn on the front page. This was a surprise to me too, but I'm guessing it's up here for the same reasons you are.

Is this the first and only submission about Wolfi? It may just enjoy traction because of the recent news. Also, posting and upvoting timely and relevant links is something that makes HN valuable, among other things.

What’s the new news? The last major RedHat drama I’m aware of was the CentOS shakeup.

Ah, they changed RHEL Source visibility from Public to subscribers only to make it harder for Rocky Linux, et al to provide strong compatibility. Lame.

https://www.phoronix.com/news/Red-Hat-CentOS-Stream-Sources

What does un-distro mean?

Think of a circle with a fine split in it. At one end there's using a distro. You go around the circle to distroless, and on the other end of the circle, close to using a distro, but not using a distro, is un-distro.

Paraphrased in jest from https://www.youtube.com/watch?v=_OEO79B-CZ4

Seems they use it to highlight it's not just distroless but also kernel-less[1].

[1]: https://www.chainguard.dev/unchained/introducing-wolfi-the-f... (last section)

It was meant to be a tongue-in-cheek way of pointing out that a it's a Linux distro without the Linux kernel, so it's really an "unLinux distro", or an (un)distro for short.

The last section in the link helped, but I have to say, the circle analogy made very little sense to me. Thank you for clarifying nonetheless.

> the circle analogy made very little sense to me

Yeah, was just a silly connection I made to the sample used in the track[1]. From a documentary on LSD[2].

[1]: https://www.psydb.net/release/hallucinogen-the-lone-deranger...

[2]: https://www.youtube.com/watch?v=gm6-THvnf44

What is the benefits of using the container kernel? Does alpine generally do this if it installed in a container?

You don't really have a choice - containers always use the host kernel. So packaging the full kernel itself is a waste of space at best, confusing to CVE scanners, and something you just don't need to bother with if you work with containers only.

Why would I use that instead Clear Linux?

Wolfi is for running inside the container instead of on a VM or bare metal host. They're complementary - you'd run something like Clear Linux to boot up into a container host, then run this inside the containers.

Reply on news.ycombinator.com