I Will Never Use Alpine Linux Ever Again | Hacker News Reader

I Will Never Use Alpine Linux Ever Again

martinheinz.dev

34 pointsnalgeon30 comments

The only real gripe I see in here is 'Alpine uses muslc'. And the only problem they have with musl is DNS resolution over UDP.

I've written a fair amount of C over my career. And let me tell you, shipping statically compiled binaries that cal getaddrinfo with musl "Just works". That is not the case with glibc which expects to be dynamically linked.

That and reasoning about the libc source is easier to do with musl as well.

Both drepper and felker are highly skilled individuals, but felker makes solid calls based on kiss. I love it and think he does great work.

You about summed up the problem, neither was ever caught suspending their dogma for the good of users.

True, I'd imagine it's nearly impossible to find a libc maintainer who isn't highly opinionated. One of those things that seems to come with the territory.

The weird thing about DNS-over-TCP is that the maintainers of musl know about it, there were patches for that long time ago, but they intentionally want it to be broken and they refuse the patches. It’s not a known bug, a design decision.

I’m however not sure if this is correct

> Another example is Golang whose standard library - or more specifically net/http or os/user modules - depend on C libraries and therefore on glibc.

To my knowledge, no go standard library depends on cgo? Can you show me in the docs/source code where is stdlib using cgo? I am very surprised by this claim

edit: I might be wrong about musl declining patches. I swear I saw it happen but now I cannot find it

That's odd. What's the stated justification for it?

You might not be aware, but dns-over-tcp has been added to musl[0] and will be part of the next release.

[0]:https://git.musl-libc.org/cgit/musl/commit/?id=51d4669fb9778...

okay then I am double wrong and I’m sorry.

What is so wrong with DNS over UDP? It's worked like that since it was invented. Surely making sure your cluster has a UDP-accessible DNS service can't be that hard. I think this is not an unreasonable decision for the musl maintainers to take.

Is there an RFC that deprecated DNS over UDP?

If your dns response won’t fit in a udp packet dns needs to flip to tcp or dns just silently breaks. A docker image that wont work in large k8s deployments is broken. The thing that breaks it is broken. Broken by design is still broken.

I think it’s a pretty unreasonable design decision.

No, that’s why UDP is default and TCP is expected to be a fallback.

DNS over TCP has been standardized since before many people on HN were born. The "problem" with UDP is that there are limitations on UDP datagram sizes. DNS is designed to deal with this: there's a "truncation" bit in the header that tells resolvers to repeat their queries over TCP.

Alpine has bitten me on several occasions as well.

Nowadays, both on my previous and current job, we go for Debian slim. Works perfectly and is worth the extra bytes.

I will also never use alpine again :-)

My personal 'musl broke it' story comes from resolving domains from Cloudflare that use DNSSEC in a K8 cluster. Basically this:

* K8 sets container to use `ndot:5`, causing the search list to be used * Musl walks that search list looking for domain * Cloudflare does not set the NXDOMAIN flag on a DNSSEC domain but does include an NSEC record (if you query with the dnssec flag). * Musl takes this 'NOERROR' reply and returns an EAI_NODATA.

Is Cloudflare wrong? I don't know, maybe. They say some things about the standards[0] and that it's technically 'right'. I don't see why they couldn't change the behaviour for queries without the dnssec flag, but I digress.

The issue is that every other libc I tested will continue searching and actually resolve the domain. Musl is the odd one out, and _only_ in the case where the search list ends up with domain using Cloudflare and dnssec.

Even if Musl is 'right' here, when it disagrees with major implementations and a major DNS nameserver does it really matter?

[0]: https://blog.cloudflare.com/black-lies/

the solution is for the resolver to not use DNSSEC because it is broken. I have a domain that we hit and looking at the query stats we have for it I have a strong suspicion that it is returning invalid DNSSEC signatures when it is rolling over the signatures. I suspect it is either not rolling them fast enough and serving stale signatures or rolling them too quickly and serving signatures that are not valid for the current time. we do parallel queries to public recursive nameservers and the name servers we were using we returning SERVFAIL errors. its possible when making queries to recursive name servers to disable checking DNSSEC and we have enabled that option now.

Maybe. The problem is that I couldn't control the DNS settings where this code was run, nor could I control what ended up in the search domain. The query did not have the +dnssec flag, but that didn't actually help because Cloudflare wasn't setting NXDOMAIN either way.

This was in golang code, so I ended up switching to `netdns=go` because like all the other libc's, the golang resolving code worked fine.

Last time I checked, also - Java. Using some PDF / Graphics related stack with Alpine can be difficult and might fail in some not obvious ways.

Maybe this situation improved a bit now.

Even those java/jdk-*-alpine images, claiming being able to run JDK actually support only subset of a (gigantic) APIs available in Java.

There are a few problems.

First, common C standard library implementations also include POSIX implementations, unnecessarily conflating the two. POSIX does not require `getaddrinfo` to implement DNS over TCP, indeed it doesn't even have to use DNS at all.

Second, many programs rely on the OS-provided DNS when using a more capable library would be a better choice. Then they don't read the documentation for `getaddrinfo`, fail to check the `hints` field returned to see that it's UDP only, and still try to fall back to TCP and return an inevitably confusing error.

POSIX `getaddrinfo` is usually not what you want, but unfortunately it often works well enough that you don't notice until someone tries to run your program on a different OS.

musl definitely causes heartache but I'll gladly accept it for an image that is less than 20MB and comes baked with busybox sh and an awesome little package manager.

>If the above issues prompted you to reconsider using Alpine, then you might be wondering what to use instead.

Come on over to the BSDs, both Open/NetBSD are very small and tight.

For example, ls(1) size:

NetBSD: 35616

Linux: 138816

So because they have a weirdass edge case because of their broken design (ridiculously outsize DNS replies), they're going to stop using a particular distro that nearly everything in Docker is based on?

Oooh-kaaaay.... sure, I guess

> musl (by design) doesn't support DNS-over-TCP

I find this surprising. Most DNS hostname lookups do work via UDP but I would expect TCP to also be supported. Caveat emptor.

There's something to be said about swapping the glibc with musl and just hoping for the best. Maybe there's other musl oddities which will cause issues in Alpine Linux?

I think its fair to say that most Linux developers test on glibc. Now I hear of some programs using ulibc or so forth (ex: Busybox) where space / embedded is key... but that's a purposeful choice with tradeoffs.

But for a general application... is there really any reason to port off of glibc?

You get a static library that always works on myriads of distros. You can more easily cross-compile. When you want to cross-compile for Linux on macOS, or crosscompile for arm on intel, much simpler to use musl.

I honestly think it's worth testing how your code builds against musl, if you think it's likely to be something that ends up in Docker.

RFC 7766 [1] actually requires TCP support:

> Section 6.1.3.2 of [RFC1123] is updated: All general-purpose DNS implementations MUST support both UDP and TCP transport.

For stub resolvers like the ones provided by glibc and musl:

> Stub resolver implementations (e.g., an operating system's DNS resolution library) MUST support TCP since to do otherwise would limit the interoperability between their own clients and upstream servers.

[1]: https://datatracker.ietf.org/doc/html/rfc7766#section-5

`getaddrinfo` is a POSIX function, and the POSIX specification for it[1] allows implementations that only support UDP (it even allows implementations that don't use DNS). Such implementations should (but are not required to) set the `struct addrinfo` `hints` field to `IPPROTO_UDP`. Note that POSIX only mentions "In many cases it is implemented by the Domain Name System, as documented in RFC 1034, RFC 1035, and RFC 1886." and not any later RFCs, so RFC 7766 changes aren't required for POSIX-compliant implementations.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/

The Docker containers I run are a mixed bag of base images. I often reach for Alpine first. If that doesn't work for some reason I think "Oh well" and switch to Debian or Ubuntu.

Wouldn't a similar approach work for most people? I don't know how it works in other companies, so perhaps I'm just privileged to not need a specific base image for everything.

> I often reach for Alpine first. If that doesn't work for some reason I think "Oh well" and switch to Debian or Ubuntu.

No, the moment something doesn't quite meet your needs you must abandon it immediately.

Like for example I used to cycle a lot but then one day it snowed and my feet got cold and my bike got stuck in the snow, so now I will never cycle again and I just drive everywhere in a massive 6x6 7-tonne army truck, except I don't because that would be fucking idiotic.

I don't get why everyone has to be so polarised these days.

I expect the author of the article is polarized due to the pain inflicted by the bug, the unreasonable design decision that led to it (seems like the bug exists by design) and the fact that there’s not really a workaround that would prevent the problem from happening again.

I’d personally like to see a fix, but my thinking might be different if I’d gone through what sounds like several days of downtime that could have been avoided.

“Weirdly outsized” which happens regularly k8s which is in place in most professional docker deployments. Docker being the primary use case for alpine…

I’d say this is a pretty reasonable gripe.

Reply on news.ycombinator.com