Behind “Hello World” on Linux

Considering adding a few things that are happening in the kernel and trying to use `bpftrace` to trace them, but I need to figure out how to use bpftrace/kprobes and hunt down the relevant kernel functions.

The software here is RTLD, the GNU RunTime LoaDer. In the ELF object there's a header labeled PT_INTERP which is short for the "program interpreter." Because yes, the python interpreter has itself an interpreter whose job it is to interpret the ELF file and make a block of memory that can be executed correctly (aka "loading" which is why this program is called the "loader" outside of ELF-world).

When you execve, after the kernel creates the program's address space it reads the contents of the file pointed to by PT_INTERP and writes it to the top of the program's address space, then calls RTLD_MAIN which is accessible by a pointer in the new address space - now the new process is actually doing something in user space. RTLD_MAIN sets up the rest of the loader state, reads the contents of the ELF binary its supposed to be loading, and only then does it start resolving dynamic library paths (if they exist). This is because the ELF object contains not only the libraries that need to be loaded alongside the executable, but also additional library include paths (via RPATH). The precedence order and cache behavior are documented (man ld-linux). When talking about LD_LIBRARY_PATH, LD_PRELOAD, rpaths, etc, it's important to remember that there's a distinction between linking (compile time behavior, used to create the ELF, Mach-O binaries that contain the references to other objects) and loading (run time behavior, used to control how the information in the objects are interpreted). As a nit this is why I don't like calling it the dynamic linker - it's a dynamic loader. The linking has already been done.

The loader maps the executable's binary into the rest of program memory. For external symbols from libraries, it resolves those symbols by searching the loaded dynamic libraries, loads that code into program memory, and then overwrites the callsites of those symbols with the now correct address of the symbol (which isn't known until this point).

(sidenote: even if you don't have a dynamically linked program, you still will need dynamic loading - modern statically linked binaries use a feature called Address Space Layout Randomization which (ab)uses the implementation of the dynamic loader to randomize where the symbols of your "statically" linked binary are going to be once the program is loaded into memory).

> I’m not going to talk about this because here I’m interested in general facts about how binaries are run on Linux, not the Python interpreter specifically.

Ah, but where is _start, and what happens before it? This is actually important to understand, because its ultimately up to the loader - RTLD from glibc is one loader and it does things quite differently than python3 - which is also a loader, and just happens to be loaded by the GNU libc loader if it was compiled and linked to, or a different loader (eg: musl libc's loader, or the kernel's loader). Because loaders can be loaded by other loaders, and "what is start" becomes important - and "what is being loaded" is deeply tied to the implementation language.

This is why there are those stray syscalls after loading: you're seeing what C programs are allowed to do before main. It's code inserted by the C compiler (for example, for the glibc runtime). Libraries are also allowed to insert code before main (for example, libpthread).