Death to shared libraries. The headaches they cause are just not worth the benefit. I have been dreaming of someone cresting a system devoid of them.
Completely disagree. Even though one size does not fit all, anyone who makes sweeping statements about static libraries is just stating to the world how they are completely oblivious regarding basic software maintenance problems such as tracking which software package is updated, specially those who are not kept up to date on a daily basis.
You just gave a non-sequitur answer that rings as rational and true as stating that if the Pope would fly he would be an Airbus A330.
Limiting which static library you can link to means absolutely nothing to the problems created by static linking, and how shared libraries solve all of them without any meaningful tradeoff at all.
Pre-linking is a way to statically specify the lookup for a given shared library in an ELF executable. It's still kinda dynamic linking because the file is loaded at runtime but there's no library lookup, it's just "load this file at this path into address space while we load the rest of the main executable". Before PIC there would have been a bit more of a distinction but nowadays static and dynamic libraries are really only distinguished by where the symbols are (unless you want to be able to dynamically load and unload after startup).
It's a more computing power and bandwidth that is required, but the maintenance burden is mostly solved by now by modern package managers or tooling on top of not so modern ones.
People pay out the ass in droves via Docker (huge-ass images, clunky composability, cgroup/namespace complexity) and Nix (horror-movie learning curve) and I'm sure other things to get around it / emulate static linking. `sudo apt-whatever upgrade` and pray is pretty much DOA in even low-complexity settings. People are voting with their feet.
You can disagree, and if you're not affiliated with Debian or GNU I might even believe that you come by that opinion honestly, but this gigantic bloc of people who hate it are not all "oblivious".
I take the authors of a disaster more seriously when they say it's a disaster, not less.
Talk about being serious.
It's strange when an app is packaged with its own private copy of all of the "shared" libraries that it uses.
Many apps that could be a single static binary seem to be packaged as docker images largely for dependency inclusion and some cgroup/namespace isolation.
As for a system devoid of it, I have grand plans, but for now: people do it with Docker every day! That's what Docker is. It's not broadly technically feasible right now to get a low-friction static link on a mainstream distro with mainstream libraries, but life, erm, technology Finds a Way. For now that's Docker, but the modest </s> enthusiasm around it shows that people want a way out pretty fucking bad and there will be better options soon.
No, Docker really isn't what you think it is.
Docker isolates processes, and provides a convenient platform independent way to deliver stand alone applications, complete with software defined networking and file system abstraction, at the expense of shipping the whole tech stack, including everything plus the kitchen sink like full blown web servers.
Thus a .DEB package which otherwise would be 10MB ends up being a Docker image often weighing over 100MB.
And pray tell what's Docker's solution to software updates? Well, it's periodic builds that force downstream dependencies and users to re-download payloads of well over 100MB a pop.
And you advocate this as an alternative to shared libraries?
Who in their ri
But in a world where native applications worked on any kernel >= the one it was built on, you'd rapidly find out that bin-packing the webserver and webcrawler into a finite list of SKUs (which is where all this container mania originated) via namespaces and cgroups are really not problems that most Docker users have, and there are much simpler ways to get two versions of `node.js` looking at two different sets of libraries.
Because you can't reliably know what devilry just stomped all over your `/usr/lib/x86_64/...` or whatever, you kind of have to eat that 100MB image if you want software that worked on Tuesday to still work on Friday.
It's a sad state of affairs, but This Too Will Pass. Getting the word out that the reason Docker saves your ass on reproducibility is emulation of static linking behind the FSF's back will accelerate that process.
Can you expand on this? It’s an interesting perspective I haven’t heard before.
I get the gist but even without a Dockerfile, a container image is much less opaque and much more deterministic than a binary with a statically linked library where the compiler might inline entire functions and perform optimizations across module boundaries with LTO. It’s not as simple as setting LD_LIBRARY_PATH but it’s not like trying to reverse the information lost during compilation.
Bundling dependecies is essentially static linking that doesn't violate the GPL. Static linking means all dependencies are compiled into the binary. Modern technology like docker, appimages, snaps, etc. all emulate that by creating a static image with all the dependencies and then dynamically linking to those specific libraries. So now we have a secondary linker and loader on top of the operating system's.
But power corrupts, and once you've got everyone linking to your `libc`, the temptation to get some combination of lazy around standards compliance at best, and seduced by what amounts to proprietary vendor API lock-in to push your politics in at least some cases seemingly becomes overwhelming. Add in LLVM kicking your ass from here to FreeBSD with Stallman freaking the fuck out about `gcc`'s plunging relevance and there's understandable but still not acceptable instinct to go further into incompatible-by-design. People on here routinely lose their shit because NVIDIA or whoever makes it a nightmare to interoperate, and I don't like it either, but vendor lock-in is vendor lock-in.
`emacs` talks to the LSP servers via JSON for Christ's sake, which only makes it about 100x laggier, but you can see why they did it, because just like Docker even RMS can't convince anyone that What We Went With is "linking". God bless the VSCode people for accepting the performance penalty so that the rest of us can come to the party, they could have muscled LSP out of the way IMO and didn't. So `rust-analyzer` gets to shovel megabytes of JSON around and Docker gets to (figuratively) tarball a bunch of `.so` files so that `ldd` can "dynamically" link them.
The thing is, this way lies irrelevance. Even the GNU userland is pretty optional these days. Is Alpine, as I've taken to calling it, "GNU plus Linux"? Is NixOS? What are we putting up with this shit for? Bash?
And now we have this "hybrid linking" thing: static image containing dynamically linked libraries.
Hermetic builds (Bazel, Nix, etc.) are in some sense orthogonal to packaging and delivery via e.g. a Docker container.
It's true that it takes great care to get bit-for-bit identical output from a C/C++ toolchain, and it's also true that very subtle things can send the compiler and linker into a tailspin of "now I inline this and now I don't have room to inline that and now other optimization passes do something different" and whoa, my program just got a lot faster or slower, yikes. I forget the title now, but there was a great Strange Loop talk about how if you do good statistics on repeated builds or even just stuff more or less into `argv` that the difference between `-O2` and `-O3` doesn't hold up (on either `gcc` or `clang`).
But whether the final executable is delivered to the target system via Docker or `scp` doesn't change any of that. If you want identical builds the Bazel and Nix people have a high-investment/high-return sales pitch they'd like you to hear :).
And it's also true (as another commenter pointed out) that it's at least a little bit reductionist of me to say "Docker gives you static linking". Docker does a zillion things, and for the people using the other stuff, more power to them if it's working well.
I would contend that most of them (especially since QEMU emulating x86_64 on your Intel Mac or vice versa isn't really a great story anymore) are not the real selling point though: you already had `virtualenv` and `nvm` and `rustup` and just countless ways to isolate/reproduce everything except the contents of your friendly neighborhood `/usr/lib/...`. If the native compiled code underpinning all of this stuff had all it's symbols linked in, and your kernel is the same or newer, you could use a tarball (and I've worked places where we did just that)!
`git` was probably the tool that really sold everyone on: I want a goddamned cryptographic hash of everything in the thing, otherwise I Don't Trust Like That. We don't accept mystery meat from our source code anymore, why we feel any differently about what actually runs is Stockholm's syndrome IMHO.
Isn't that the world we live? Linux kernel has a stable binary interface. Problems almost always arise in user space.
Yeah. I've had some success just dropping all that user space stuff and targeting Linux directly with system calls. I boot up Linux straight into my program. Not viable for every use case but it's amazing when the system is small.
I've seen other people commenting here on HN saying they're using the same approach in production so it's defenitely not my invention. Hopefully they'll see this thread and reply.
I published some of my work in the form of a liblinux that I use to make system calls:
https://github.com/matheusmoreira/liblinux
I'm not developing it anymore though because I found out the kernel itself has a superior nolibc library:
https://github.com/torvalds/linux/tree/master/tools/include/...
It used to be a single header but it looks like they've recently organized it into a proper project!
I wonder if it will become some kind of official kernel library at some point. I asked Greg Kroah-Hartman about this and he mentioned there was once a klibc:
https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah...
The quote about "glibc maintainers being non-helpful" slayed me. I guess at least some of the kernel people aren't exactly thrilled about the tireless efforts of the GNU project to gatekeep, take credit for, be obstructionist around, and otherwise subsume Linux into The GNU Operating System.
I hate that it's gotten to the point where I'm the "anti-GNU" guy, I never wanted that, but they need some fresh blood in the leadership or something.
Nice idea. I think a kernel libc in the form of a vDSO would unnecessarily force compilers to use the C ABI though. The Linux system call interface is really unique in how it's not only stable but language-agnostic and it would be a shame to lose that.
I think compilers are the right place for system calls. The programming language should have a system_call keyword that makes the compiler emit the syscall instruction with the parameters in the right registers. It was my intention to eventually create that language, liblinux was something of a prelude... Then I ran out of free time. :)
Then it turned out smarter people than me have gone much farther than I did:
https://news.ycombinator.com/item?id=28283632
Really cool stuff.
> The quote about "glibc maintainers being non-helpful" slayed me.
> I guess at least some of the kernel people aren't exactly thrilled about the tireless efforts of the GNU project to gatekeep, take credit for, be obstructionist around, and otherwise subsume Linux into The GNU Operating System.
I understand what you mean. I think it's really weird how often I get glibc manual pages when I look up Linux kernel documentation. I look up some system call and get glibc wrappers instead. Aren't they separate projects?
In the first AMA someone asked Greg whether glibc was blessed by the kernel:
https://old.reddit.com/r/linux/comments/2ny1lz/im_greg_kroah...
He said no, but it still seems like the official libc of Linux.
I also get systemd manuals when I look up Linux init system documentation. I expected to read about what the kernel itself expects from PID 1, information useful for making my own system. Turns out it's just a process like any other, it just needs to bring up the system by doing things like mounting procfs and sysfs.
Linux should be its own thing. It's only "GNU/Linux" because of POSIX which says an operating system has to have binaries like cp, mv, etc. Things don't have to be that way. POSIX is just paper.
> I hate that it's gotten to the point where I'm the "anti-GNU" guy, I never wanted that, but they need some fresh blood in the leadership or something.
I think it was a quote from Lennart Poettering that inspired me to start dropping this "standard" GNU stuff. I think he wrote something like "read TLPI and ignore all the POSIX stuff". So I did and I discovered a really nice system that actually turned out to be easier and more fun to program with than dealing with C libraries. No global state anywhere!
GNU is just a POSIX compliant user space. It doesn't have to be that way... Wouldn't it be cool if some awesome person made a pure Lisp user space? A pure Rust user space? It doesn't have to be the same old POSIX tools... Linux enables that. We can trash the entire user space and start fresh if we want. We can't do that on Windows, Mac, the BSDs.
./nginx.bin
People go to ridiculous lengths just to avoid the shared library mess, and it only makes the argument against shared libraries stronger.
Some libraries managed to preserve API contracts throughout the years just fine.
Even in C++, a language which does not establish a standard ABI, has frameworks which have not been breaking the ABI throughout multiple releases, such as Qt.
libstdc++ errors? Seen them.
Surely, if you care, you have a mitigation plan for this?
One headache over another.
What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?
With the dynamic one you have both problems…
> What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?
It's easy to see which commands are given to the compiler and if there is a copied library if you have sources.
If we made all builds reproducible (like oasis) we could setup an automated trusted system that does builds verifiably automatically (like what Nix has).
Doing automated, local patching and rebuilding is far from enough. You need experts to correctly backport patches and test them thoroughly.
Oh, just like Gentoo, Slackware and LFS. ;)
Seriously, maybe we shouldn’t ignore history and learn why things are like this today.
It’s not just a simple switch you flick and recompile.
Do you even understand that you are advocating in favour of perpetually vulnerable systems?
Let's not touch the fact that there are vague and fantastic claims about shared libraries being this never ending sources of problems (which no one actually experiences) that when asked to elaborate the best answer is wand waving and hypotheticals and non-sequiturs.
With this fact out if the way, what is your plan to update all downstream dependencies when a vulnerability is identified?
Linux has never been anywhere close to resistant to privilege escalation bugs.
But statically linked systems are perpetually vulnerable by design, aren't they?
I mean, what's your plan to fix a vulnerable dependency once a patch is available?
Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?
Because with shared libraries, all it takes to patch a vulnerable library is to update just the one lib, and all downstream consumers are safe.
What's your solution for this problem? Do you have any at all?
Do you actually have an answer any of the points I presented you, or are you going to continue desperately trying to put up strawmen?
If any of your magical static lib promises had at bearing in the real world, you wouldn't have such a hard time trying to come up with any technical justification for them.
Unfortunately, this is very far from being true in the real world.
And even if all libraries did maintain 100% ABI compatibility forever, and even if there were never any compiler or processor bugs that needed to be mitigated by recompiling applications, dynamic linking would still add runtime complexity and thus surface area for security vulnerabilities.
> Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?
Yes? I mean, are you just throwing random untrusted binaries onto your servers, not keeping track of what their dependencies are, and hoping to God that you will never ever need to recompile them? (And if your explanation for that is "I use proprietary software from vendors who refuse to share source code", have you chosen incompetent vendors who cannot respond to security vulnerabilities in a timely fashion?)
1. For each application and library in the system, maintain a copy of the source code and the scripts necessary to rebuild it.
2. Keep track of the dependency tree of each application and library in the system.
3. If there's a problem with a dependency, update it and rebuild all dependent applications.
4. If I'm using binaries from a vendor (whether that's a Linux distribution or a proprietary software company), the vendor needs to be responsible for (1) to (3).
In practice, your package manager will generally implement almost all of (1) to (3) for you. (Newfangled systems like NPM or Cargo usually do it using lockfiles and automated tools like Dependabot.)
If you're using dynamic linking, guess what? You still have to do these four things, because even with dynamic linking, there are still problems that can only be solved by recompiling your software. Even if you think reasons like ABI incompatibility or processor bugs aren't compelling, there might be, y'know, bugs in the applications themselves that you've gotta patch.
I'm not even going to bother pointing how unfeasible all your other points are. I'm just going to point this Fack: you do understand this does not work and never worked at any point in time, don't you?
Do you understand the explicit reference to perpetually vulnerable systems? Do you realize where it cames from?
You only have the power to rebuild the packages you own personally, and even so static libs offer zero ways to keep track of which version went into which build.
Once your fantastic panacea starts to rely on your idea to force third parties to follow your personal orders to make new releases under your own personal terms, you should be very aware that you will not get your wish. You'll instead just keep on using the same vulnerability-riddled release.
Do you understand the importance and value of static libs? You do not need four-point authoritarian and deeply impractical and unfeasible plans to keep your system safe. With shared libraries you just patch the one lib, and all your system is safe.
And again what tradeoff do you want to achieve for this? Nothing?
The amount of frustration dynamic linking causes is just not worth it in today's day.
I think that calculus looks very different 20, 10, maybe even 5 years ago, when getting packages rebuilt and distributed from a zillion different vendors was much harder.
Docker/containers do far more than just bundle shared libraries. We use a container for almost all of our CI builds. Our agents are container images with the toolchains, etc installed. It also means it's trivially easy to reproduce the build environment for those occasional wtf moments where someone has an old version of <insert tool here>.
Tools aren't the only part of the environment though. Setting the JAVA_HOME environment variable for a build can dramatically change the output, for example. Path ordering might affect what version of a tool is launched.
They also mean that our developers don't need to run centos 7, they can develop on windows or Linux, and test on the target platform locally from their own machine.
The nice thing about fully statically linked executables can just be copied between machines and ran. If Java worked this way, the compiler would produce a (large) binary file with no runtime dependancies, and you could just copy that between computers.
Rust and Go don’t give you a choice - they only have this mode of compilation.
Docker forces all software to work this way by hiding the host OS’s filesystem from the program. It also provides an easy to use distribution system for downloading executables. And it ships with a Linux VM on windows and macos.
But none of that stuff is magic. You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.
The killer feature of docker is that it makes anything into a statically linked executable. You can just do that with most compilers directly.
> But none of that stuff is magic.
I never said it was, I simply said that docker does more than statically link binaries.
>You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.
https://news.ycombinator.com/item?id=9224
> The killer feature of docker is that it makes anything into a statically linked executable.
The killer feature of docker is it gives you an immutable*, versionable environment that handles more than just scp'ing binaries.
The benefit docker provides here is giving users a simple, reproducible, downloadable environment with no external dependencies.
I'm not saying Docker is useless in the current build environment. On the contrary - Having a convenient way to sandbox, ship and execute arbitrary execution environments and statically linked executables is really useful.
I think this is such a great idea we should make debian work closer to this model. Nix is a fantastic step in this direction and it'd be great if more distributions followed suit.
Debian and friends fill your hard drive with of weird, non-reproducible versions of various dynamically linked libraries. The fact I can't take a binary on my x86_64 linux machine and run it on your x86_64 linux machine is an embarrassment. Moving toward static linking and fully reproducible builds would be a tremendous help here.
Docker feels like a kludgy hack. I'd love a world where docker isn't necessary because its best ideas have been lowered into the OS itself.
It's plenty common for binary distributions of software to come as tgz files. That's usually including a bunch of .so files.
Yes, and compatibility across distros / releases is very hit-and-miss, because they usually ship certain libraries and rely on the OS for others.
I’m not sure whether it’s possible to ship every library without running into ODR issues, but at that point, you’d loose all of the advantages of dynamic linking anyway.
Static linking is not a new thing. It was there since a beginning. Dynamic linking is a new thing compared to it.
The very opposite. You can do all the sandboxing you want without needing docker and you will avoid docker's big attack surface.
Docker has cleared become popular because it drastically eases application deployment and cleanup.
Static linking also solves a huge number of these problems, and in a generally better way.
Docker has overhead, an associated attack surface, bulky artifices, etc but it yields very nice benefits for workload isolation (env configs, storage mounts, etc) as well as cross OS/arch builds.
Is there a solution that combines the best of these options? A super-slim docker for statically linked apps that combines the ease of deployment with a much smaller footprint?
I feel like this could be a simple shell script, at least for 99% of real-world use cases.
Once you bundle multiple libraries into blobs you always have the same security nightmare:
https://www.researchgate.net/publication/315468931_A_Study_o...
(and on top of that docker adds a big attack surface)
My feeling is that dynamic linking is a pain because sometimes one needs, say, 1.2.5 for some part of the code and 1.3.1 for another part, and they are not compatible. Which hints towards the library author having messed up something, right?
Yay, we get to keep on using old Log4j now because one program holds it back.
Yes! A thousand times yes. I'd rather reinstall every single compromised program than deal with the ridiculous complexity of dynamic linking.
Anyway that's not why we have shared libraries. They were invented so that memory managers could load the same library once, then map it into the address space for every process which used it. This kind of savings mattered back when RAM was expensive and executable code represented a meaningful share of the total memory a program might use. Of course that was a very long time ago now.
To the extent that shared libraries economize on memory, or are claimed to, it's when a lot of different processes link the same version of the same library, e.g. your `libc` (or the original motivating example of X). The tradeoff here is that the kernel has an atomic unit of bringing data into RAM (AKA "core" for the old-timers), which is usually a 4kb page, and shared libraries are indivisible: you use `memcpy` and you get it and whatever else is in the 4kb chunk somebody else's linker put it in. Not a bad deal for `memcpy` because it's in everything, but a very bad deal for most of the symbols in e.g. `glibc`.
The counter-point is that with static linking, the linker brings in only the symbols you use. You do repeat those symbols for each `ELF`, but not for each process. This is often (usually?) a better deal.
A little poking around on my system to kind of illustrate this a bit more viscerally: https://pastebin.com/0zpVqA0r (apologies for the clumsy editing, it would be too long if I left in every symbol from e.g. `glibc`).
Admittedly, that's a headless machine, and some big clunky Gnome/Xorg thing might tell a different story, but it would have to be 100-1000x worse to be relevant in 2022, and almost certainly wouldn't offset the (heinous) costs. And while this is speculation, I think it would be negative worse, because there just aren't that many distinct `ELF`'s loaded on any typical system even with a desktop environment. Chromium or whatever is going to be like, all your RAM even on KDE.
There are couple of great CppCon talks that really go deep here (including one by Matt Godbolt!): https://www.youtube.com/watch?v=dOfucXtyEsU, https://www.youtube.com/watch?v=xVT1y0xWgww.
Don't be put off by the fact it's CppCon, all your GNU stuff is the same story.
This personal assertion has no bearing in the real world. Please do shine some light on what exactly do you interpret as "expensive", and in the process explain why do you feel that basic software design principles like modularity and the ability to fix security vulnerabilities in the whole system by updating a single shared library is "trouble-prone".
I mean, the whole world never saw a real world problem, not the UNIX comunity nor Microsoft and it's Windows design nor Apple and it's macOS design not anyone ever, but somehow here you are claiming the opposite.
"DLL hell" [1] was absolutely a thing. It was solved only by everyone shipping the world in their application directories.
A single product can have a modular design, but modularity across unrelated products designed by different designers is a different concept that should have a different name.
In my opinion it’s just not something that really works. This is similar to the “ability” of some software to have multiple pluggable database engines. This always yields a product that can use only the lowest common denominator… badly.
But now imagine having to coordinate the sharing of database engines across multiple relying products! So upgrading Oracle would break A, B and fix C, but switching A and B to MySQL would require and update to MySQL that would break D, E and F.
That kind of thing happens all the time and people run away screaming rather than deal with it.
Shared libraries are the same problem.
And nearly all libraries eventually become "security-oriented libraries", given enough time. Everything is a hacking target eventually.
Doesn’t that pretty much already exist now with immutable system builds, eg Fedora Silverblue, GUIX, NixOS? Each package pulls in its own dependencies with the specified version and config, and then no changes can be made once the system is built. Only way to update is to rebuild the system with the updated versions. Costs more disk space but that is cheap these days, and solves shared dependency hell.
Lack of storage and bandwidth in the 1970s
I’m pretty sure the reason is “X11 was large and badly factored, and people who should have known better thought it would save disk space.”
The problems they claim to solve are updates across applications at once, but in the enterprise how often does that actually happen? Application authors still have to test against the new version, and sysadmins still have to roll out the changes required to support the update. In practice they may as well statically link.
Let’s not forget the problems they introduce are significant. Implementations are complex. Applications become less portable. ABI versioning is a pain (versioned symbols are gross). Semantic changes in the shared lib cause chaos. Exec is slower because of the linking done at load time. Dynamic libs are bigger than static ones, AND they’re loaded into memory. More difficult to verify the correctness of a given executable.
The alleged “benefit” here really does not outweigh the cost. If we were doing static linking from the beginning, it’s difficult to say if we would have wound up with things like docker or nix in the first place.
We were doing static linking from the beginning, at least on Unix. Dynamic linking came later.
Would static linking conceivably make sandboxing, jailing, etc easier?
Compared to all the effort which goes into dynamic linking, I suspect it would be better spent on sandboxing (both mechanism and UX).
Oh, cry me a river. 30 years ago the performance impact was known and was very well worth the trade-off. Machines are orders of magnitude faster now and consequently the impact less significant. Next thing, you're telling me X11 is slow.
Those reasons were floppy disks and RAM that was measured in kilobytes though.
Assuming that you pin the version of the library you need, two applications using the same library /shouldn't/ cause issues.
In most *nix systems - each instance of a shared library has its own data.
Then why bother?
However, until that happens there is nothing wrong with shared libraries. They exist for good reason. They save bandwidth and ensure that you're not getting old libraries.
It's not actually "code" which is being shared, but instead the memory pages containing that code (the sharing granularity is a single page, since sharing works by pointing the page table entries on each process to the same physical page). So either you'd have to force page alignment for each and every function (highly wasteful, since the page size is at least 4096 bytes), or you're reinventing shared libraries but using a hash instead of a version.
Static linking again and again has proved to be a security nightmare. It makes security updates dependent from upstream release or full rebuilds making things difficult.
You get targeted security updates without changes in any other functionality.