Hacker News Reader

Everything you describe already exists. Executables do list their dependencies, and we have well-defined conventions for indicating ABI breaks. It is entirely normal to have multiple major versions of a library installed for ABI compatibility reasons, and it is also entirely normal to expect that you can upgrade the dependencies out from under a binary as long as the library hasn't had an ABI break.

The bigger dependency management problem is that every distro has their own package manager and package repository and it's tough for one application developer to build and test every kind of package. But if they just ship a binary, then it's up to the poor user to figure out what packages to install. Often the library you need may not even be available on some distros or the version may be too old.

That's why distros ask you to provide just the sources and we'll do the packaging work for you. The upstream developers shouldn't need to provide packages for every distro. (Of course you can help us downstream packagers by not having insane build requirements, using semantic versioning, not breaking stuff randomly etc).

This is only realistic for established applications with large userbases. For new or very niche apps, distros are understandably not going to be very interested in doing this work. In that case the developer needs to find a way to distribute the app that they can reasonably maintain directly, and that's where containers or statically-linked binaries are really convenient.

This isn't really true, Fedora, Debian and Arch have huge numbers of packages, many very niche. You might well need to make the distro aware that the new program exists, but there are established routes for doing that.

Arch particularly has the user repository where anyone can submit a package and vote on the ones they use most often to be adopted into the community repository, yes.

It’s a great way to start contributing to the distribution at large while scratching an itch and providing a service to individual projects.

I agree with everything you said up to this. We're talking about a software library, for which the user is a software developer. IMO a software developer should be able to package a library for their own distro (then they can share that package with their community and become this package's maintainer).

As the developer of an open source library, I don't think that you should distribute it for systems that you don't use; someone else who uses it should maintain the package. It doesn't have to be a "distro maintainer". Anyone can maintain a single package. I am not on a very mainstream distro, and I still haven't found a single package that I use and is not already maintained by someone in the community (though I wish I did, I would like to maintain a package). My point is that it really works well :-).

I disagree with the idea that we should build a lot of tooling to "lower the bar" such that devs who don't know how to handle a library don't have to learn how to do it. They should learn, it's their job.

For proprietary software, it's admittedly a bit harder (I guess? I don't have much experience there).

This is not grounded in reality. Look at popcon or something like it. It is a nearly perfect "long tail" distribution. Most software is niche, and it's packaged anyway. It's helped by the fact that the vast majority of software follows a model where it is really easy to build. There are a lot more decisions to take with something like Chromium, which perhaps ironically is also the type of software which tends to package its own dependencies.

>Executables do list their dependencies

They list paths to libraries, but not the exact version that the executable depends on. It is a common occurrence for executables to load versions of libraries they were not designed to be used with.

If you're talking about ELF for desktop Linux, they for the most part don't contain file paths, and may specify the version but usually just have the major version (to allow for security updates). You can use ldd to read the list of deps and also do a dry run of fulfilling them from the search path, for example:

  $> ldd $(command -v ls)
    linux-vdso.so.1 (0x00007ffd5b3a0000)
    libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f6bd398c000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f6bd3780000)
    /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f6bd39e5000)

Libraries can cause bugs even if they have the same exact version as it may be compiled in a way that is not expected by the program. Ideally the list of libraries should be some form of a hash of the library to ensure it is loading exactly what it expects.

Yes, if someone actually did dependency management in Linux properly then I agree - dynamic linking would be fine. It works pretty well in Nixos as I understand it. But it’s called dependency hell for a reason. And the reason is almost no operating systems handle C dependencies well. There’s always weird, complex, distribution specific systems involving 18 different versions of every library. Do you want llvm18 or llvm18-dev or llvm-18-full-dev or something else entirely? Oh, you’re on gentoo? Better enable some USE flags. Redhat? It’s different again.

If Linux dependency management worked well, there would be no need or appetite for docker. But it works badly. So people just use docker and flatpak and whatnot instead, while my hard drive gently weeps. I don’t know about you, but I’m happy to declare bankruptcy on this project. I’d take a 2mb statically linked binary over a 300mb Linux docker image any day of the week.

This isn't really an "operating system" problem. Particularly in the open-source world, there are a number of fairly core libraries that refuse to provide any kind of API compatibility.

Then, when there are a couple dozen applications/etc that depend on that library, it's almost an impossible problem because each of those applications then needs to be updated in lockstep with the library version. There is nothing "clean" about how to handle this situation short of having loads of distro maintainers showing up in the upstream packages to fix them to support newer versions of the library. Of course, then all the distro's need to agree on what those versions are going to be...

Hence containers, which don't fix the problem at all. Instead they just move the responsibility away from the distro, which should never really have been packaging applications to begin with.

> away from the distro, which should never really have been packaging applications to begin with.

I disagree here: the whole point of a "software distribution" is to "distribute" software. And it does so by packaging it. There is a ton of benefit in having distro/package maintainers, and we tend to forget it.

> If Linux dependency management worked well, there would be no need or appetite for docker.

I kindly disagree here. Linux dependency management does work well. The problem is the bad libraries that don't do semver properly, and the users who still decide to use bad libraries.

If people stopped using libraries that break ABI compatibility, then the authors of those libraries would have to do it properly, and it would work. The reason it doesn't work is really just malpractice.

If Linux dependency management works well in theory but not in practice then it doesn't work. It works in nix because it can literally use multiple minor versions of a library when it needs to with no problem. Most distro's can't or won't do that.

You can call it malpractice but it's not going to stop so in practice you need a way to deal with it.

Well, by calling it "malpractice", I say that it works for "true professionals". Then we could say that "it doesn't work in practice if people who don't know what they are doing cannot use it", of course.

The question then is where we want to put the bar. I feel like it is too low, and most software is too bad. And I don't want to participate in making tooling that helps lowering the bar even more.

And by the way it does work really well for good software. Actually most Linux distros use a system package manager and have been doing it for decades.

So I think it would be more accurate to say that "it doesn't work for lower quality software". And I agree with that.

Semver only controls API compatibility, not ABI compatibility. You can make an ABI break in a Semver minor (or patch) version update. Semver is nice, but it's not enough for ensuring compatibility when dynamic linking.

SONAME is here for ABI compatibility, right?

Linus Torvalds disagrees: https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...

I should have been more balanced or nuanced a bit: I also don't think static linking is to be forbidden or completely shunned. As Linus himself says, a combination of both may be ideal. For basic system libraries like GUI libraries the current approach works well. But you should be free to static link if you want, and if there are serious issues if you don't. Maybe dynamic linking should be focused on a smaller number of well curated libraries and the rest should be left to static. Library archeology seems like a potential serious problem years from now.

I still think better listing dependencies (perhaps with the option to pin an exact version?) would be helpful, as well as better usage of something like semver. Someone mentioned binaries include paths to dependencies, but as far as I know, there is no tool to automatically try to resolve those dependencies or standard interface, maybe some more tooling in this area would help.

Another nice point about how it current works is that I think it relieves work from programmers. The policy of "Don't worry about distribution (just tell us it exists)" from distros seems like one less headache for the creator (and you can provide static linked binaries too if you want).

As most things in life, the ideal is somewhere in the middle...