For context: I've been around in the Debian Rust team since 2018, but I'm also a very active package maintainer in both Arch Linux and Alpine.
Rust packaging is absolutely trivial with both Arch Linux and Alpine. For Debian specifically there's the policy of "all build inputs need to be present in the Debian archive", which means the source code needs to be spoon-fed from crates.io into the Debian archive.
This is not a problem in itself, and cargo is actually incredibly helpful when building an operating system, since things are very streamlined and machine-readable instead of everybody handrolling their own build systems with Makefiles. Debian explicitly has cargo-based tooling to create source packages. The only manual step is often annotating copyright attributions, since this can not be sufficiently done automatically.
The much bigger hurdle is the bureaucratic overhead. The librust-*-dev namespace is for the most part very well defined, but adding a new crate still requires an explicit approval process, even when uploads are sponsored by seasoned Debian Developers. There was a request for auto-approval for this namespace, like there is for llvm-* or linux-image-*, but back then (many years ago) this was declined.
With this auto-approval rule in place it would also be easier to have (temporarily) multiple versions of a crate in Debian, to make library upgrades easier. This needs to be done sparsely however, since it takes up space in Packages.xz which is also downloaded by all users with every `apt update`. There's currently no way to make a package available only for build servers (and people who want to be one), but this concept has been discussed on mailing lists for this exact reason.
This is all very specific to Debian however, I'm surprised you're blaming Rust developers for this.
What’s the additional security benefit of the explicit approval? A major problem with Rust (for me) is the multitude of dependencies required even for trivial software, which complicates supply-chain monitoring.
Auto-approval for crates gives me a bad feeling.
Guix is also a distro that allows for any number of versions of the same package globally, something that language specific dependancy managers do not.
Distors are there for a reason, and anyone who doesn't understand that reason is just another contributor to the ongoing collapse of the tower of abstractions we've built.
In Debian stable ripgrep on amd64 is currently on version 13.0.0-4+b2.
The relevant buildinfo file can be found here:
https://buildinfos.debian.net/buildinfo-pool/r/rust-ripgrep/...
It encodes the entire Rust dependency graph that was used for this binary, with exact versions of each crate.
for me: make an os out of the kernel
Distros (and the people who run most scales of IT org) want to be able to deploy and verify that the fix is in place - and its a huge advantage if it's a linked library that you can just deploy an upgrade for.
But if it's tons and tons of monolithic binaries, then the problem goes viral - every single one has to be recompiled, redeployed etc. And frequently at the cost of "are you only compatible with this specific revision, or was it just really easy to put that in?"
It's worth noting that docker and friends also while still suffering from this problem, don't quite suffer from it in the same way - they're shipping entire dynamically linked environments, so while not as automatic, being able to simply scan for and replace the library you know is bad is a heck of a lot easier then recompiling a statically linked exe.
People are okay with really specific dependencies when it's part of the business critical application they're supporting - i.e. the nodejs or python app which runs the business, that can do anything it wants we'll keep it running no matter what. Having this happen to the underlying distributions though?
(of note: I've run into this issue with Go - love the static deploys, but if someone finds a vulnerability in the TLS stack of Go suddenly we're rushing out rebuilds).
What I'm specifically suggesting is:
* Distributions package *binaries*, but not the individual libraries that those binaries depend on.
* Distributions mirror all dependencies, so that you can (in principle) have a completely offline copy of everything that goes into the distribution. Installing a binary uses the language-specific install tools to pull dependencies, targeting the distribution's mirror.
* Enough dependency tracking to know what needs to be rebuilt if there's a security update.
* Any outside dependencies (e.g openssl) will continue to depend on whatever the distribution packages.
* Dependencies are not globally installed, but use whatever isolation facilities the language has (so e.g, a venv for python, whatever npm does)The problem here is upstream dev's saying "my dependency needs are absolute". And a security update ruins that: because as soon as one happens, now no matter what we're going to be replacing libraries anyway. Even your prosposal includes this: we're going to strip out openssl librares and use distro ones.
At which point everything might break anyway, because whether a security hole can be fixed at all depends on which versions of a library it affects and how. Not to mention problem's like finding the issue in one version, but it's changed enough that it's not clear whether a different version is impacted the same way.
Is that ultimately the responsibility of the application developer or the OS developer?
The issue is one way or another it needs to happen ASAP: so either the distro is haranguing upstream to "approve" a change, or they're going to be going in and carrying patches to Cargo.toml anyway - so this idea of "don't you dare touch my dependencies" lasts exactly as long as until you need a fix in ASAP.
And even if they are, for rust software that doesn't come from debian, there is no way to ensure it all gets rebuilt and updated with the fix.
Also, projects are generally slow (taking several months) to accept patches. When a distribution has fixed something and the users notice no issue, the upstream project if downloaded and compiled would be a different matter entirely.
...they fail to integrate with dependencies written in any other language.
It's fine if you just want to sit a monoculture language software stack on top of a multilingual base platform. You can't make a functional system with one language alone, yet those who criticise distribution packaging architecture do so while simultaneously depending on this ability that language-specific package managers do not have. There is no viable alternative today. Most critics think they understand the general problem but only have narrow practical experience, so end up believing that their solution is superior while not considering the general multilingual software supply problem.
Nix isn't a solution either, because in the general case Nix isn't security-supporting arbitrary and multiple dependency versions either.
In C, it is generally assumed that libraries maintain compatibility within one major version, so programs rarely have tight version intervals and maintainers could just use the newest available version (in each major series) for all packages depending on it.
If the build system (for Rust and some other languages) makes it easy to depend on specific minor/patch versions (or upper-bound intervals of versions), it encourages developers to do so instead of working on fixing the mess in the ecosystem.
openssl has made several rounds of incompatible changes. ffmpeg and related av* libs make major releases frequently. libx264 is famous for having "just pin a commit" approach to releases.
It's common for distros to carry multiple versions of major libraries. Sometimes it's a frequent churn (like llvm), sometimes it's a multi-year migration (gtk, ncurses, python, libpng).
C libraries aren't magically virtuous in their versioning. The language doesn't even help with stability, it's all manual painstaking work. You often don't see these pains, because distro the maintainers do heroic work of testing the upgrades, reporting issues upstream, holding back breaking changes, and patching everything to work together.
----
Cargo packages almost never pin specific dependency versions (i.e. they don't depend on exact minor/patch version). Pinning is discouraged, because it works very poorly in Cargo, and causes hard dependency resolution conflicts. The only place where pinning is used regularly is pairs of packages from the same project that are expected to be used together (when it's essentially one version of one package, but had to be split into two files for technical reasons, like derive macros and their helper functions).
By default, and this is universally used default, Cargo allows semver-major-compatible dependency upgrades (which is comparable to sover). `dep = "1.2.3"` is not exact, but means >=1.2.3 && <2.0.0. The ecosystem is quite serious about semver compatibility, and there is a tooling to test and enforce it. Note that in Cargo the first non-zero number in the version is the semver-major.
True, not every lib keeps long-term stable API.
> It's common for distros to carry multiple versions of major libraries. Sometimes it's a frequent churn (like llvm), sometimes it's a multi-year migration (gtk, ncurses, python, libpng).
This is consistent with what i wrote as these are changes in major versions.
> C libraries aren't magically virtuous in their versioning.
My statement was not categorical statement about C libraries, but it was about incentives shaping the ecosystem. Using libraries has costs and benefits. Frequent churn means higher costs, so it is acceptable when bringing higher benefits. In the C ecosystem, the cost for major version churn (or general library API incompatibility) is high, so people avoid it when possible. If sophisticated tooling like Cargo makes building against specific versions easy, it lowers the cost of churn, so everyone is less concerned with it.
It is true that in C ecosystem, keeping stability is not effortless, for software developers it means running CI with tens of distros / distro versions, to test their software against diversity of library versions. Not 'just use my Cargo.lock' (i am not sure how widespread this approach is in Rust ecosystem, but at least some people argues for it in this discussion).
I release foo 1.2.3 as a package, when doing it depending on bar 4.5.6. Perhaps foo still compiles with bar 4.5.2 or 4.6.8... but that's _not_ foo 1.2.3 right? And if there are bug fixes in some of those but not others, if I had my deps set up to just package with "bar 4.x" , suddenly I'm getting bug reports for "1.2.3" without a real association to "the" 1.2.3.
I'm saying this, perhaps this is as easy as having two sets of deps (like deps for repackagers, deps for people wanting to reproduce what I built)... but if a binary is being shipped _not_ being explicit about the exact dependencies used is upstream of a whole lot of problems! Is there a great answer from OS distro package managers for this?
Is the thing that should happen here be that there's a configure step in rust builds that would mess around with deps based on what you have?
Do you rewrite every API call in your software every time you bump dependencies?
Do you think that a library that changes API very often and thus remains insecure because bumping it breaks the program should be running on people's computers?
Software A and B depend on C (version V). A has a bug that gets fixed by shipping with C version V+1. B unfortunately has a latent bug that appears when shipping with C (version V+1).
You’re now going to have to do a bunch of work, outright not ship stuff, or even introduce bugs because of this policy!
Nobody wants buggy libraries, and in this example it’s not the library’s fault (B might just be misusing C, or in fact compensating for a bug).
Reading through these threads I’m much more empathetic to the software bill of materials argument than before as a justification for this policy. I do not think there’s an intrinsic quality argument to the global single version policy.
Bug fix releases are proof that sometimes you do really just need to put out new things to fix a bug.
I mean… you're welcome to write a book and not publish it, just don't expect anyone to read it no? Same thing with libraries.
If it has a 0.x.y version number? Yes, that's what 0.x means to begin with. In fact Rust/cargo provides a stricter interpretation of semver than the original, which allows you to express guarantees of API stability (by keeping the minor version number fixed) even in 0.x-version projects.
It was a mess, and linux package managers were first to actually solve the problem. I fully believe that the explosive growth of linux in the early years had little to do with the kernel, and was much more about the proper package managers introduced by Debian, Red Hat, and friends.
But just because package managers were massively better than nothing, and nothing (universally) better has come along since, doesn't mean they are actually good. The user experience is pretty good. But for developers, the drive to dynamically link every single dependency, and only have one (maybe two) versions of each dependency, and to randomly update dependencies causes massive problems for developers. It only works as well as it does for C/C++ applications because they have gotten used to working within the restrictive framework. And maybe a bit of Stockholm syndrome.
-----------------
Let me tell you a story:
Back in 2014, I was working on Dolphin Emulator (which is entirely c++), trying to fix regressions for a Dolphin 5.0 release. One of these regressions was that on linux, launching a game a second time (or opening the controller config window twice) would cause a crash. I quickly tracked the bug down to a bug introduced in SDL 2.0.3. Most consumers of SDL weren't impacted by this bug, because they only ever initialised SDL once.
The fix was simple enough, it had already been fixed in the latest reversion and so would be fixed whenever 2.0.4 was released. But we wanted to release before that, so how to fix the bug? If this was cargo, I would just point it at a git revision (possibly a custom branch, with just the fix back ported). But that wasn't an option.
I spent ages trying to work out a workaround that avoided the bug, but couldn't find one. I considered including a patched version of SDL with dolphin; Dolphin's source tree already contained a bunch of libraries which we statically linked in. Some patched, others were just there for easier builds on Windows/Mac). But I knew from experience that any distro would simply patch our build system to use the distro version. Dolphin's build system already had an extensive set of manual checks to use the distro version of a library if it existed. Many of these of these checks were contributed upstreams by distro package maintainers.
I considered throwing an error message (twice, both at build and runtime) if dolphin was linked with a bugged version of SDL 2... But I figured the distro maintainers would probably patch out both error message too. Remember, we are talking about a bug that wouldn't even be fixed to the next release of SDL, with an uncertain release date. And there would probably be distros which stuck with the buggy version for way too long (Debian actually avoided the bug, because it was still shipping SDL 2.0.1, before the bug was introduced)
Eventually I got frustrated. And I realised that SDL was only used for controller input, and only on linux/bsd. SDL already wasn't meeting our needs, we had a dedicated core input backend for MacOS and both DInput and XInput backends for Windows because they worked better. I had also just gone though the entire SDL input code, back when I was trying to find a workaround, and had learned how it work.
So... to workaround the limitations of Linux Disto packaging (and a stupid, easy to fix SDL bug), I simply deleted SDL out of the Dolphin codebase. Only took me three days to completely reimplement all the controller input functionality, directly accessing the lower-level linux libraries: libudev for controller detection/hotpugging, open the correct `/dev/input/eventX` file, libevdev for parsing the raw input events.
Three days to implement it. A few weeks of testing to find bugs... Not only did this workaround the bug. But it allowed Dolphin to access controller functionality that SDL wasn't exposing at the time (suddenly, the Dual Shock 4's touchpad would work, and the pressure sensitive buttons on Dual Sock 3). I don't think SDL was even exposing the gyroscope/accelerometers of those controllers at the time.
---------------------
I was very happy with the result, but it was crazy that I was forced to go to such extremes, just because of the linux package manager insistence on enforcing one shared version of a dependency. I'm not sure how common it is for developers to rip-out and replace an entire dependency because of distro packaging, but workarounds for buggy library versions are common.
These days, I'm really enjoying programming with rust; The fact that cargo always gives me the exact version of the package I asked for is a massive win. And why should we give it up, just because distros want to do something worse. And it would be so hard to make the transition, because a lot of rust code has been written under the assumption that cargo will give you the version (or version range) than you ask for.
If you think about it, it's kind of insane that packages will be build and tested against one version of a library, just to have it randomly replaced with a later version, potentially introducing bugs. In some edge cases, packages even get downgraded to an older version between build/test and being installed on a user's system.
---------------------
What's the solution for rust? I have no idea. Distros do have a point that statically linking everything isn't the best idea. Maybe we can reach some kind of middle ground where some dependencies are statically linked and others are dynamically linked, but with explicit support for multiple versions of each dependency co-existing on the system.
Maybe we also add explicit support to package managers to track which dependencies have been statically linked into them, so we can force a rebuild of them all when there is a security issue.
Distro maintainers will not patch in a bug into a program, they will patch out bug from the SDL. :-/
But I still would have required an interim solution for everyone building from source until these patched/fixed versions of SDL started shipping in distros. And then how would the warning message tell the difference between the original SDL 2.0.3 and a patched version of SDL 2.0.3?
And we are talking about dozens of distros, multiple chances for something to go wrong.
My point is that I quickly came to the conclusion that ripping out SDL and reimplementing the functionality was the simplest, most conclusive solution.
$ rpm -q SDL2
SDL2-2.30.9-1.fc40.x86_64
SDL2-2.30.9-1.fc40.i686
:-/This to me sounds like the distro packaging model isn’t a good one.
I wish people would interrogate this more deeply: why do we see so many Dockerfiles with random binary artifacts? Why do people download mystery meat binaries from GitHub releases? Why is curl-pipe-to-bash such a popular pattern?
The answer to the questions is that, with few exceptions, distributions are harder to package for, both in first- and third-party contexts. They offer fewer resources for learning how to package, and those resources that do exist are largely geared towards slow and stable foundational packaging (read: C and C++ libraries, stable desktop applications) and not the world of random (but popular) tools on GitHub. Distribution packaging processes (including human processes) similarly reflect this reality.
On one level, this is great: like you, I strongly prefer a distro package when it's available, because I know what I'm going to get. But on another level, it's manifestly not satisfying user demand, and users are instead doing whatever it takes to accomplish the task at hand. That seems unlikely to change for the better anytime soon.
Distributions using a "model of global shared libraries".
If you have to run 5 different docker images each with their own “global shared library” set you clearly no longer have system wide globals. You have an island of deps potentially per program or possible a few islands for a few sets of programs.
Which, once again, completely defeats the entire purpose of the Linux global shared library model. It would have been much much much simpler for each program to have linked statically or to expect programs to include their dependencies (like Windows).
Containers should not exist. The fact that they exist is a design failure.
If you’re going to say something so spicy, at least back it up with some reasons, otherwise you could have just said it out loud and saved everyone from reading it (blah blah HN guidelines blah blah).
Naw. Lack of reliability in running programs is a distinctly Linux phenomenon. Windows culture is much much much better.
I know I’m dealing with programmers but the level of precision required on internet comments is absurd.
I work on Windows and don’t use Docker so I too known it’s not Everyone. But when I said everyone I would like to think not Everyone assumed I meant literally Everyone. Capitalization intended.