It’s not that libc is used to fix kernel problems/compatibility issues, but rather, the kernel ABI is not a supported, stable public API.
On macOS, there are cases where using the syscall ABI directly will result in your code breaking when interacting with any other code that does correctly use libc, due to out-of-sync userspace state maintained by libc.
For example, the fork(2) implementation in macOS’ libSystem will invalidate the cached copy of the current process’ pid used by getpid(2).
If you fork(2) by directly trapping to a syscall, the cached pid won’t be invalidated; any future calls to getpid(2) through libSystem will return the stale pid.
If you’re only using syscalls and no libc at all, you’re fine (on Linux, and only Linux).
Yes. As a glibc maintainer once said, if you use clone you're on your own. And that is fine.
Meanwhile OpenBSD was discussing ideas to extend their system-call-origin verification to only allow syscalls from libc.
What’s so crazy about an OS vendor wanting to exercise control over that userspace shim layer, rather than assuming that user programs will be poking directly at the kernel?
You can argue that having multiple libc implementations is good, sure, but it’s not the only way of doing things and it’s not without tradeoffs.
Looking at this through a KERNEL32 lens, I can actually somewhat see the alternative point of view: in NT < 4 it was mostly an in-process RPC proxy for the subsystem server in CSRSS, and one can argue that libc on Unix-likes is in the same position (cf the vDSO on Linux). You can build RPC to a trusted server either way: define a stable wire protocol; or require the client to load proxy code into its address space. COM on Windows and hardware-accelerated graphics on Linux both take the second approach.
But then why the hell does the proxy also have opinions about memory allocation, assignment of TLS slots, or floating-point formatting? (KERNEL32 has them on the first two points as well, mind you.) History[1] aside, does this really look like good engineering? (KERNEL32 makes a bit of a point there, but given how much HeapAlloc sucks as an allocator compared to the interoperability benefits it brings, I’m not sure it should be taken too seriously.)
[1] https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRun...
I think my ideal system would look like Linux in some ways. Any given Linux system has a 'standard' shared library ecosystem, with a single shared libc, and usually a shared C++ standard library as well. Executables and libraries that participate in that ecosystem can assume they'll use the same standard libraries and so can pass those libraries' types across module boundaries. Alongside them are 'hermit' executables, like Go programs or C programs statically linking to musl, that avoid the standard libc. But those executables typically also avoid all other system shared libraries (or greatly constrain their usage), so the difficulty in using their APIs (due to mismatched standard library types) isn't a big deal. Now, Linux sort of forces this approach by having the dynamic linker itself be part of libc. But that has some downsides, such as forcing the vDSO to work differently from every other library. I think I'd prefer to have a universal dynamic linker, like on Windows, but to still have 'ecosystem shares a libc, while hermits stick to themselves' as a convention.
As described elsewhere in this thread, that is the policy of MacOS X and Windows.
It is also a policy which OpenBSD is moving towards:
https://lwn.net/Articles/806863/
According to the Go devs, this is also the policy on Solaris-family OSs (Go uses libc there):
https://github.com/golang/go/issues/24357#issuecomment-37300...
AFAIK, it is only Linux which explicitly supports the kernel system call interface, and so where alternative libcs are possible.
You can do this on Windows.
The syscall numbers change sometimes.
I use it to evade EDR hooks :)
Could you elaborate on this? Specifically I'm not understanding what exactly is "explicit" with regards system calls on Linux compared to other OSes.
Libc is seen as user land by the Linux kernel, it provides an API for programs to abstract them away from syscalls. Libc dies syscalls for programs.
By "never breaking userland", Linux promises to never break libc by never changing syscall numbers/args/types - new syscalls are added in ways that don't break older syscalls.
Other platforms don't make this explicit promise and reserve the right to shuffle syscalls around, change them, remove them, etc as desired - making maintaining a libc for them a pain in the ass, and programmers are told to only use the libc as syscalls don't come with guarantees.
You can (and I often do) program on windows using only syscalls, but your program tends to need rebuilding across major versions as the syscall interface isn't stable.
Fuck me typing that on a phone was painful.
You're allowed, of course, it's just not supported. We pop out of the woodwork to note that it's not supported and will break.
Linux's approach of treating the syscall ABI as a long-term supported, stable interface is somewhat unusual.
> Cosmopolitan Libc is a libc, so it can't very well depend on six other c libraries
At least on macOS, trapping to the kernel without going through libSystem first is simply not a supported use-case.
"Apple does not support statically linked binaries on Mac OS X. A statically linked binary assumes binary compatibility at the kernel system call interface, and we do not make any guarantees on that front. Rather, we strive to ensure binary compatibility in each dynamically linked system library and framework."
More to the point, platform vendors decide what their ABI boundaries are. Historically Unix vendors (back in the late 80s and early 90s) had no ABI boundaries... the expectation was every new OS release would require a recompile of all software. Obviously Linus has very different ideas about stability than the other Unix like systems of the time (which was a good thing), and focused on syscall stability. That made a lot of sense since he only wanted to maintain a kernel, not a full OS distribution.
When modern macOS and Windows developed their ABIs they both were relatively mature OS distributions including a dynamic linker and default runtime libraries, and the ABI boundary chosen was well above the kernel as that is an easier place to define and maintain it.
How do we actually know that placing the boundary above the kernel makes it easier to define and maintain?
Or: Life can be easier for the kernel team if they're allowed to make breaking changes. Then task the lib team with writing shims / wrappers / etc. to fix all the problems which that causes. Then the manager of the lib team may have a perfect reason to boost his headcount. Then...
Without it, you end up maintaining system calls that nobody should use.
Now, you could argue that moves the mess to the c library, which would still have deprecated functions that used to call old system calls, but now are built on top of better ones, but there’s more flexibility there. Application programmers can, one by one, move to newer c libraries that remove that cruft.
Having a stable syscall interface is one thing... not supporting static linking is another... Even on linux, statically linking libc, GTK, ... is not a great idea.
I also had a Mathematica binary (statically compiled except for libc) that ran on Linux from 1998 to 2010, including X windows (at some point, somebody moved the X files to a different location, so I had to set an env var).
(I'm not arguing for nor against syscalls-as-kernel-API, I'm honestly just curious. Without knowing too much about it, it seems pretty sensible to support statically linked executables, but I might be missing something. I guess linking against an "as old as you need to support" libc dynamically isn't the end of the world, but it does constrain the build environment somewhat.)
I can't speak for Windows, but on macOS, you build for older targets by passing `-(mmacos|ios)-version-min=` to the compiler to specify your deployment target version; this is used to determine symbol visibility, API visibility, toggled #ifdefs, etc.
The provided version is also stored in the Mach-O load commands of your executable, and will used to select compatible symbols (and enable compatibility shims) when loading your binary's image at runtime.
No need for static linking — or jumping through hoops to build against an "as old as you need to support" set of installed libraries.
For example, look at [1] and search for e.g. 0x01aa, to see how the meaning of that syscall number changes in a pretty systematic way over releases.
1: https://github.com/j00ru/windows-syscalls/blob/master/x86/cs...
Use of DLLs is baked VERY deeply into windows. The kernel is not even a monolithic file. It is an exe like most others pulling in dlls that implement other functionality. It even features like API-set DLL redirections which mean the bootloader for the kernel needs to implement a fair bit of the PE loader functionality of the kernel just to load the kernel. (I've no real clue if there is shared code between the two loaders, or if they are two separate loaders that implement similar things. Many of the options/features of the full kernel executable loader are not really needed in the bootloader.)
So it is not much of a surprise that making the numbers stable to support fully statically linked executables was not really something they cared about. From their perspective you could always just pull in the ntdll.dll for your syscalls like they intended (or more likely a higher level win32 dll that uses the syscall).
Except you're not doing that consistently though.
On OpenBSD, mincore(2) was removed 3 years ago, and it's UNIMPL syscall 78 was eventually recycled by a different system call: mquery(2), which your library calls expecting mincore and passes bogus arguments to.
I would be very surprised if there aren't more serious mistakes lurking in your library, in fact I know there are.
https://github.com/openbsd/src/commit/54e4f6b9a1dc183e1dc7c4...
https://github.com/openbsd/src/commit/1d60349d0b961891264d42...
https://web.archive.org/web/20220921050525/https://justine.l...
First the code to say hello:
const char \*const str = "Hello World!\n";
HANDLE standardOutput = GetStdHandle(STD_OUTPUT_HANDLE);
WriteFile(standardOutput, str, strlen(str), NULL, NULL);
Then the compiler switches: /merge:.rdata=.text cuts out an entire section 0.5K
/nocoffgrpinfo cuts out about 256 bytes from your .text section
/emittoolversioninfo:no is supposed to omit the RICH header, but doesn't seem to work anymore.
/stub:stub.bin will let you replace the default DOS EXE stub with something smaller. Using a 64 byte file will get the EXE header and PE header to fit in the first 512 bytes (when combined with merging the .rdata and .text sections)
Set "Entry Point" to main. This completely bypasses the C standard library and CRT, none of that gets initialized or called.Then you end up with an EXE containing 3 sections: The EXE header (512 bytes), the .text section (512 bytes, but only 141 bytes actually used in there), and the .idata section (512 bytes, for importing DLLs, only 41 bytes actually used)
Is that really the case? Given it’s specifically intended as a highly portable libc, in some cases maybe it could be implemented by calling through to the blessed system libc, rather than kernel calls.
I mean, that likely wouldn’t fit with your goals for the project, and it would likely need some horrible linker hackery, but in principle it seems technically possible. (I’ll believe you if you say it’s completely impossible, though!)
It's true that we don't need to load the vDSO into the address space of the process since the kernel does it for us. We do need to manually link the vDSO functions though: the kernel uses the auxiliary vector to pass the address of the vDSO ELF image to the process, allowing it to parse the ELF header and find the addresses of the functions. In most cases, the libc will do this while initializing itself prior to calling main.
Nope, it is built into the kernel image - at least on my system, but there doesn't seem to be a config option to disable that.
> When mapped into the process address space, it's just a normal ELF image. The functions all conform to the normal C ABI for the platform.
That's purely for convencience because it allows ASLR for the vDSO and there is no point in making it a new ad-hoc format if every C library already needs an ELF dynamic linker - if the vDSO was a thing before ELF it would surely have a different format.
> We do need to manually link the vDSO functions though: the kernel uses the auxiliary vector to pass the address of the vDSO ELF image to the process, allowing it to parse the ELF header and find the addresses of the functions. In most cases, the libc will do this while initializing itself prior to calling main.
You need to look up the entry points in the vDSO before you can call them but that is really no different than checking the kernel version before deciding which syscalls to invoke. And no, the vDSO is not loaded like any other .so - it is very much a special case [0] even if the libc ends up reusing some of the normal .so code and structures. For example, it needs to be loaded even for statically linked binaries [1].
[0] https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/sys...
[1] https://sourceware.org/git/?p=glibc.git;a=commit;h=1e8bdc3a2...