You don't link all of Libc | Hacker News Reader

You don't link all of Libc

flak.tedunangst.com

30 pointsingve26 comments

If you use dynamic linking, you will get all of libc mapped into your address space. It won't all be faulted in; just the parts you call. The parts you fault in should point to the same physical frame of memory that other processes have faulted in that have also called the same part of the library. That's why it's a "shared library".

Yeap. I roll my eyes at the people who want to static link everything and think they're somehow better or purer when they're just wasting RAM for nothing.

Static linker supporter here, not to start a flame war but I like static linking because it means I don't have to worry about version mismatches of shared libraries between what I ship and what the user gets.

Your complaint is more related to how the OS you're running on handles dynamic libraries, rather than anything inherent to dynamic libraries themselves. It is possible to version libraries and serve up the correct version(s) to different applications simultaneously.

Now, consider the case of a Linux distro with a few thousand binaries. Should a defect be found in a common library, the burden of updating, say, 10,000 servers might rapidly become a headache. In this case, a shared library probably makes sense.

Also, if you want to create extensible software (like a plugin system), dynamic libraries are one of the most expedient and well supported ways to do so. This is not necessarily the traditional use case for shared libs, but it's not uncommon either. And this use case normally doesn't have those versioning problems either (as it leans more to the dynamic side than the shared side).

All that being said- static linking also has its advantages; as you stated, you can be guaranteed that your users are executing the library code you intended and that nothing breaks. It's also faster, because you're not waiting for ld.so to resolve your imports at startup & first use (nit: yes, I know you also can force all resolution to finalize at startup at the expense of slightly longer startup time). Further, you can be sure that someone hasn't hooked your library calls to do something nefarious. The downside, of course, is that you have to re-build and re-deploy your entire application on every library's bugfix/security updates.

As with anything related to computing, everything has tradeoffs. Some tradeoffs may be dealbreakers for your use case, but they might not be for someone else's.

Static linking does not guarantee that you can escape this. There are certain functions which require the same version of libc in the system, which was used for linking. At least this applies to glibc, and sticking with musl or Cosmopolitan might help.

Specifically, the ones that dynamically link.

Such as getaddrinfo, or pwent.

It's not for nothing. Function calls into shared libraries are pretty disgusting.

You'll need to elaborate that, because I have read the assembly for a function call in shared libraries and wasn't too disturbed by it... what do you mean ?

The first call is through a function pointer to a symbol resolution function, in any subsequent call the function pointer has been patched to directly call the desired function. Seems pretty good to me.

It's pretty gross. The shared lib call keeps going through a stub in the procedure link table; only the stub has improved after the first call so that it doesn't have to keep fetching the address of the function via the global offset table. The details are architecture specific, but it has overhead pretty much everywhere. On top of that, a register is dedicated to pointing to the global offset table. It has to be saved reassigned and restored as the control flow path takes trips through multiple shared libraries.

Static linking does not have to waste any RAM by duplicating code, even if it is frequently implemented in this way.

It is possible and not complicated to implement static linking in a way where the executable file retains the metadata about the component object files and also the OS retains some information about the object files that are already loaded in memory as parts of other executables.

In this case, when loading an executable, the component objects that are already present in memory can be omitted and the references to them in the executable can be adjusted.

While this is not done in the more popular operating systems, due to their preference for the more inefficient dynamic linking, the alternative is still possible.

Moreover, it is impossible to say in general which of static linking and dynamic linking wastes more memory. Static linking will have multiple copies of some code, while dynamic linking will have a lot of unused unreachable code, which is superfluous. Moreover, dynamic linking also has some execution time overhead.

In practice, with the typical bloated dynamic libraries that are available, I have seen much more memory wasted by dynamic linking than by static linking.

When only static linking is used, there is rarely much code duplication besides a small subset of the standard C library, which includes things like the wrappers for system calls and the functions for formatted I/O.

Therefore most of the benefit in reducing memory use from dynamic linking would be achieved with a single dynamic library, which should contain only a subset of libc.

Then your eye will keep rolling when all people asking for a “docker image ” for any existing software.

With the GNU linker, you would have to go out of your way to link all of a library, like:

  ld ... --whole-archive libwhatever.a --no-whole-archive

Into the 1990s, it was still a big deal whether a linker could eliminate individual unused functions rather than just remove unused object files. I recall this ability is called function linking.

In GCC they wanted function linking for the libgcc.a functions that provide run-time support, like numeric functions (e.g. division) needed by some generated code to make up for what the machine's instruction set does not provide.

GCC's build process simulated (maybe still does?) function linking for libgcc.a by using the preprocessor to compile the file with these routines multiple times, separating the output into multiple .o files containing just one function.

These days we have a libgcc as a shared library.

You can do better than working at the object file level. With --gc-sections and --ffunction-sections, the linker can include only the relevant functions into your binary (whether your libc tries to minimise internal dependencies is another question)

I think even better than functions it can optimize out data sections too. For example if I declare a huge array and don't use it, it doesn't show up in the binary. Not always true but the are times it is.

When I boot OpenBSD, I see:

  reordering: ld.so libc libcrypto sshd.

How does that play into this?

Not related.

The blog post is a mini-rant complaining about how other people are complaining about having to link "all of libc" on BSD when their standard libraries are not based on libc (libc is the interface to the kernel on BSD). He's talking about static linking.

Honestly, I'm not terribly sure I get it, as almost all "other languages" link to libc in some way on most OSes, and don't call native kernel interfaces directly at all. But maybe it's more common than I think. Famously, Go went this route and it caused them no end of consternation with their FFI, and they've gone back in the other direction of linking with libc directly.

In your case, reordering has to do with ASLR (address randomization) in dynamic libraries. OpenBSD takes ASLR a step further and so not only randomizes the libraries themselves, but the archives inside the library and the functions themselves- they call it KARL. That's what it's doing when you see that message.

Right, but does that mean that /bin/ls changes at every boot, because some of what it took from libc is dynamic?

The article states that /bin/ls is statically linked. That means that the parts it uses from libc have long-since been taken and merged directly into the ls binary. So this randomization has no effect.

Even if ls were dynamically linked, the ls binary itself wouldn't change at every boot; instead its import/jump table, generated dynamically by ld.so as part of the OS's executable loading process, would simply point to different (and random) offsets within libc. It would have a different in-memory representation after each boot, but its on-disk representation would remain the same.

On Linux, your language run-time can call into the kernel directly with itsown syscall assembly language stubs. The kernel interface is something public and stable, independently of the C library (of which there are several available, incidentally: glibc, uClibc, musl, ...).

Right, I understand that. You can do the same on Windows too.

My point was that most languages' runtimes don't do that, not that they can't do that.

I think NT syscalls can change. https://j00ru.vexillium.org/syscalls/nt/64/

You're meant to go through ntdll.dll, but even that interface is documented as private/unstable.

You're absolutely right, they can.

And they have.

But the reality is more like this: https://xkcd.com/1172/

You're meant to go through published interfaces, such as the ones in kernel32.dll. ntdll is an implementation detail and so are syscalls.

Because then they would face additional portability hurdles that they can avoid by targeting POSIX. Even decent porting to Windows is possible (speaking of which) via POSIX.

Not directly related, but note what is mentioned in the post about the purpose of individual object files for each function. What might be considered an anachronism is being leveraged for improved security.

What's happening (IIRC) with the reordering is that all those otherwise dynamically linked binaries (sshd is a PIE binary) are being themselves relinked from scratch (OpenBSD ships with the .o files), but shuffling the order of .o files passed to the linker. This effectively enhances ASLR. The way ASLR normally works is that each binary (program or shared library) is loaded into a random base address, but all the functions defined in each binary are fixed relative to each other. In stock installations libc's fopen is at a fixed offset from libc's snprintf, for example, and this will be true for every program loading libc.so. In the above, libc.so is relinked with a randomized ordering to the .o object files. The result is that relative offsets for libc functions on your individual system will be different from every other system, including from your previous boot, hopefully making exploits (e.g. utilizing ROP chains) more difficult. More modern projects like libcrypto and sshd tend to group related functions into the same .o file, so all the functions in each .o will still have a fixed offset from each other, but they still have quite alot of individual .o files to shuffle.

Note that OpenBSD does the same thing for the kernel.

Reply on news.ycombinator.com