That's an interesting historical fact, and one that we've locked ourselves out of. I looked into stripping Linux down to the bare minimum, and it's extremely difficult to untangle the codebase. There are inter-module dependencies that aren't specified anywhere, the build just mysteriously fails. The kernel config tool pretends to give you a lot of options, but in reality a lot of configurations will never actually compile, and a lot of things that appear to be removable actually leave huge chunks of code behind because they're used by other things.
And that's just the kernel. If you want X server, browser, text editor, you'll be surprised at what dependencies they require. If I recall correctly, Xorg requires MESA which requires... drumroll please... clang and llvm (about 1GB of code if you clone from github). It's madness.