Ask HN: Why are all major operating systems monolithic?
Why are all major OS's (Windows, macOS, Linux, *BSD) all monolithic kernels? Why aren't there more microkernel operating systems, like for example Redox or Minix?
Performance of microkernels is or was a hotly debated topic back in the '90s; microkernels generally had lower performance than their monolithic counterparts. There was a famous argument comparing Linux (monolithic) to MINIX (micro) [0]. Wikipedia has simple explanations for the differences between hybrid, mono, and micro kernels that serve as a decent primer [1] and a visual that I find helpful [2].
[0] https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
[1] https://en.wikipedia.org/wiki/Kernel_(operating_system)
[2] https://en.wikipedia.org/wiki/Hybrid_kernel#/media/File:OS-s...
Microkernels still have lower performance than their monolithic counterparts. It makes sense really: the major difference between the two is protection barriers between components. All else being equal, a system without those barriers is faster than one with them unless crossing those barriers has zero cost -- which it only really does in the case of proof-carrying code and similar software solutions, where the "barrier" is more of a guarantee that the subsystem behaves as it should.
A lot of microkernel research in the past focused on the "all else being equal" aspect, attempting to come up with innovative ways to reduce protection domain crossings in microkernels. For example, mapping graphics card memory or network device buffers directly into the address space of the client means that the hardware can deliver data directly to the client. However, monolithic kernels can also benefit from these sorts of innovations!
As to why consumer OSes are all monoliths (yes, this includes Darwin and NT -- "hybrid" is a marketing term as all the important stuff runs in a single kernel address space; see https://en.wikipedia.org/wiki/Hybrid_kernel): in consumer OSes, the 5% (or lower) performance hit of a microkernel is a big deal. In other spaces, such as embedded, the benefits of microkernels (reliability, security guarantees) are sometimes more important than raw performance.
This is what I've understood, that the idea sounds good but in practice isn't as good as it seems like it would be.
The Mill is one, that has been in development for a while, but its company hasn't published much lately and its current status is a bit unclear. It would have had something very similar to Spring/Solaris "Doors" in hardware, where the caller could have passed temporary permissions to memory ranges. A "kernel" would have been a process like any other, just with more permissions.
And L4 on every Apple device with a secure enclave: https://support.apple.com/nl-nl/guide/security/sec59b0b31ff/....
But not as the actual system OS. Minix is doing a very little, running on a little side processor that also does very little. It's not running on the main CPU, nor having to interact with the insane number of possible peripherals that real CPUs and OSes have to interact with, nor with real system loads, nor with many of the other constraints that real OSes have to deal with.
Yes, it's there. Yes, it's there in really massive numbers. But, compared to what the OSes are doing on the main CPU, Minix is doing a toy job. So I'm not sure that it counts as evidence for much of anything.
My understanding is that (perhaps except for virtualization/address space partitioning on the hardware device to be able to use it with many programs at once) this is approximately how/why user-mode networking is used for high performance today?
https://en.wikipedia.org/wiki/Exokernel
No production exokernels were built, but the concept lives on as virtualization. Most hypervisors (eg. Xen) work on a substantially similar principal: the hypervisor is responsible for securely multiplexing the hardware between guest OSes, and then the guest OS talks to an interface that basically looks like the hardware.
A step beyond this is to get rid of the guest OS and have application code talk directly to the hardware interface. This is a unikernel (or called a "libOS" in the original exokernel papers), and as you predict, unikernels running on a hypervisor are often faster than traditional monolithic kernel OSes. The main thing blocking their adoption is that they require significant rewrites to application-level code: most applications (and the libraries they depend on) generally assume POSIX syscalls are available and would need to be largely rewritten to work in terms of direct hardware access. You can rewrite POSIX as a library in userspace (this was the approach taken by the original exokernel research), but most of the performance gains come from bypassing inappropriate abstractions, so if you use the same abstractions you get basically the same performance.
You could call NT a 'modified microkernel implementing monolithic kernel concepts', if you wish. The word 'hybrid' is much shorter, though.
in consumer OSes, the 5% (or lower) performance hit of a microkernel is a big deal
https://devstreaming-cdn.apple.com/videos/wwdc/2017/707h2gkb...
So the whole TCP/IP stack is just a regular userspace library on that platform. Only the wifi driver runs in-kernel.
These platforms are made up of literally hundreds of Mach servers. Relatively little code runs in the kernel when you examine the structure of the whole OS. The kernel is really only running some hardware drivers where moving them to userspace is hard, and some basics like memory management, process scheduling, IPC and code signature verification. The kernel just isn't that big.
It's important to realize though that moving even more stuff to processes does not automatically improve security or availability. Imagine there's an RCE vuln in your root filesystem code. Being in a userspace server doesn't help you:
1. If you can control the FS you can make a root owned file appear anywhere and then just run it, or alter system configuration in any other way.
2. If you can crash the FS the core kernel can't easily recover without special cases because where would it restart the FS process from? If you say "initramfs" then you're back to having an FS in the kernel.
In practice there are lots of cases like that. Sometimes it's worth it - modern OS' can generally recover from a display server crash and run graphics mostly in user space. Sometimes it's not.
Finally, bear in mind that Linux is very insecure due to its huge feature set and reliance on C, but kernel exploits are often not the biggest security problem on other platforms. The most serious bugs are all in userspace already on most platforms.
I think the real reason is they were also designed at a time when security was not as advanced as it is now, especially for Linux, and microkernels are more secure but also harder to write.
> In 2017, it was revealed that Intel was running MINIX inside the Management Engine, a separate processor incorporated in Intel processor chipsets since 2008. As a response to this, Tanenbaum wrote an open letter to Intel claiming MINIX to be "the most widely used computer operating system in the world".[17]
Hilarious.
I only wish I had a bucket of popcorn before I began reading this.
https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
In the mid 1980's, the ICON microcomputer system, perhaps known mostly for being used in Canadian schools.
In the early '00s, QNX got jealous of Linux's success and started opening it up to try to attract app developers from the open source community.. but closed down the program before it could gain traction.
In the early '10s, it was bought by Blackberry and used for the OS of a new wave of Blackberry smartphones and a tablet.
My opinion summed up ridiculously concisely is that the reasons are largely accidents of timing (i.e. there was room in the market for new entrants), mixed with trivial-at-the-time but high-impact choices. Had Tanenbaum been permitted by his publisher to give away copies of minix freely, Linux may never have been started at all. Had Tanenbaum chose raw performance over teachability (the kernel was a teaching tool after all), it may have made a difference. (warning: this part is controversial) There's also a lot of question about whether the GPL licensing kept Linux relevant, or if it was largely inconsequential. My opinion is that GPL made a big difference. Had the licensing of Minix been GPL flavored, it probably wouldn't have been used by companies like Intel for the Management Engine, but there might have been enough "forced sharing" to keep it relevant. Impossible to say for sure, but that's my two cents.
I really like the approach of separating, in some way, the kernel, from the user space and interchange them but why you would do that, except for playing and research, if the kernel doesn't give you an advantage and it just an alternative?
[1] https://unix.stackexchange.com/questions/282046/can-gnu-linu...
My understanding is it can't really run on actual hardware at all in it's current state, it's mostly run virtualized.
In practice, though, GPUs are the least of its concerns; AFAIK, HURD still doesn't support USB:(
> I really like the approach of separating, in some way, the kernel, from the user space and interchange them but why you would do that, except for playing and research, if the kernel doesn't give you an advantage and it just an alternative?
Well I think a lot of it is that we thought it would be easier; in theory, separating concerns and putting drivers in userspace sounds like it should make it a lot easier to write the OS. That didn't really turn out, but it seemed like a good idea at the time. On the more practical side, microkernels can have security and reliability benefits (if drivers run in userspace, they have less access to exploit, and crashing a driver doesn't crash the whole machine). I mean, diversity is also useful for its own sake, but there are more immediate benefits.
https://en.wikipedia.org/wiki/Tanenbaum–Torvalds_debate
Which in a nutshell comes down to Linus's view that producing a system that meets current needs (for example by offering a simple API) and is reasonably performant -- even if it ran on but a single architecture such as the i386 -- was more promising than a design offering "theoretical" benefits (such as portability) but which either didn't deliver the goods today or just wasn't reasonably performant.
This is a vast simplification of a much broader set of issues but if one digs into the notes of why subsequent projects (e.g. Windows NT) moved away from microkernel designs, or the roughly analogous microservices debate of the past decade -- one often finds echoes of the same basic considerations.
> A sports car is a car designed with an emphasis on dynamic performance, such as handling, acceleration, top speed, the thrill of driving, and racing capability.
How would lowering suspension turn a passenger bus to a sports car?
It's a bad analogy unless the button makes the bus change Transformers style into a completely different car.
Just because you've delegated some small parts into userspace, does not mean that you don't still have a massive monolithic kernel with metric tons of infrastructure and subsystems all running in the same kernel address space, oftentimes not even as dedicated threads.
A monolithic kernel is a tiny kernel that does not do much more than message passing. Linux is very far away from that, your "userspace servers" for FUSE and similar things are better called "delegates" for subsystems that still reside in the kernel itself.
> It's a bad analogy unless the button makes the bus change Transformers style into a completely different car.
That is not what the analogy was about at all.
> Just because you've delegated some small parts into userspace, does not mean that you don't still have a massive monolithic kernel with metric tons of infrastructure and subsystems all running in the same kernel address space, oftentimes not even as dedicated threads.
That's up to you to keep around. No need to compile in subsystems and drivers you don't need. HID is pushed userspace by HID-BPF and it works great. USB, I2C, SPI can all be handled by direct messages to the device.
These objections are pedantry around what exactly is micro about a microkernel. UIO lets you drive any hardware via userspace memory maps, it's just usually easier to slap it into a pre-built kernel subsystem more than any need for code to be in kernel mode.
> A monolithic [sic] kernel is a tiny kernel that does not do much more than message passing. Linux is very far away from that, your "userspace servers" for FUSE and similar things are better called "delegates" for subsystems that still reside in the kernel itself.
Your message passing is my delegation through, say, netlink. Perhaps you are describing an exokernel concept where only the barest minimum is run in kernel mode? Mach runs a lot more in kernel mode than necessary, even L4 isn't a "true" microkernel if you are ultra-pedantic and run on processors with, say, userspace interrupts.
I am not familiar with this project. It looks to be a VFS-like framework for userspace? If so, I don't see how that changes the nature of a kernel. I can implement any amount of kernel-like features on top of any operating system, provided I have Turing completeness.
By that argument, I can take pre-NT enhanced mode Windows, arguably one of the least microkernel kernels there is (AFAIK it even has large part of its UI system running in kernel space), run an L4 kernel within it as a userspace application, and declare that "Windows 3.11 is now a microkernel"?
As long as this SPDK thing does not rid me of VFS, it's just adding stuff in userspace?
> That's up to you to keep around. No need to compile in subsystems and drivers you don't need.
I genuinely did not follow Linux kernel development for quite a long time (as opposed to other OSes, like OpenBSD for example). Are you saying that nowadays (or maybe back then even), I can compile out VFS, the POSIX layer, the VM layer, kernel memory allocators?
If so, that's indeed a big step towards microkernel-ness!
> These objections are pedantry around what exactly is micro about a microkernel.
You're saying I'm pedantic about what is a microkernel, I'm saying your definition of "microkernel" seems surprisingly broad, and question the merit of the term under that definition.
According to you, where does a microkernel stop, and a monolithic kernel start, then?
> Your message passing is my delegation through, say, netlink.
netlink is not only one of many communication paths in Linux, it isn't even a raw message passing mechanism in the microkernel sense! It's built on top of POSIX and the socket layer, which are surfaced through many individual system calls into the kernel, and whose implementation is a pretty thick sandwich to get down to the actual mechanism that passes the message, i.e. that implements how the actual data is made available from one address space to another (zero-copy or not).
Compare this with e.g. QNXs very simple, low level message passing protocol in its microkernel: https://www.qnx.com/developers/docs/7.1/#com.qnx.doc.neutrin...
This is the actual low level mechanism, on top of which POSIX and pretty much everything else is built, outside of the microkernel. So it's actually quite the opposite of netlink!
> Perhaps you are describing an exokernel concept where only the barest minimum is run in kernel mode?
Liedtke's Minimality Principle states: "A concept is tolerated inside the microkernel only if moving it outside the kernel, i.e., permitting competing implementations, would prevent the implementation of the system's required functionality."
So it seems that your "exokernel" definition matches my "microkernel" definition.
Now, that definition represents an ideal that may not always be achieved in practive, but even then, the word "micro" in "microkernel" alone indicates that what should run in kernel mode should, at the very least, be extremely small. For my senses, I don't think "all of POSIX and VFS" is cutting it for that.
> even L4 isn't a "true" microkernel if you are ultra-pedantic and run on processors with, say, userspace interrupts.
I am actually open on what the "proper" definition of a microkernel is, but if the current state of the Linux kernel is included in that, and again if my present understanding of Linux is accurate and up to date (i.e. POSIX, VFS, allocators, all of the vm layer, and many many more things are still integral and inseparable parts running in kernel mode), i.e. if the term "microkernel" is not sufficient to separate such a kernel from QNX's Nucleus, or L4, then that term strikes me as rather useless.
FWIW, I don't think I have ever seen any Linux kernel maintainer, including Linus, referring to their kernel as a microkernel. I have seen the exact opposite in the famous debate with Tanenbaum, but that was decades ago, so I have to be open to the possibility that the shape of the Linux kernel may have changed significantly since then.
Windows and the Unix OSes just have a lot more market penetration, strong network effects, and a big first-mover advantage. That means that when you are buying your OS or your hardware or your software, you're probably stuck with a monokernel.
A few % performance hit? (if that) Who cares. Security & reliability is where it's at these days.
I have friends who use their desktop for gaming only, literally nothing else, I guarantee they don't care about the extra security a microkernel claims to offer.
Some kinds of code are inherently in the core failure domain and trying to use restartable processes for them is impossible. For example if the code configuring your PCI controller crashes half way through, well, you need to reboot. You can't just restart the driver because you have no idea what state the hardware is in. A reboot gets you back into a known state, a server restart doesn't. Likewise if your root filesystem server segfaults, you can't restart it because your process start code is going to involve sending RPCs to the filesystem server so it'd just deadlock.
Finally, it's a spectrum anyway. Microkernel vs monolithic is not a hard divide and code moves in and out of those kernels as time and hardware changes. The graphics stack on Windows NT started out of kernel, moved into the kernel, then moved back out again. In all OS' the bulk of the code resides in userspace libraries and in servers connected via IPC.
Also the model of a Unix kernel with services (daemons) in user space is kind of similar to the microkernel model. The graphical desktop is not in the kernel, the mail server isn't in the kernel, crond isn't in the kernel, the web server isn't in the kernel, the database server isn't in the kernel ... and maybe that's as close as we need to get to the microkernel.
Speaking of web servers in user space: at least for serving a static page, it's faster for that to be inside the monolithic kernel, where it can access the protocol stack and filesystem code without crossing a protection boundary. Over the history of Linux, we have seen experimentation both with moving kernel things into user space (like with FUSE) as well as user things in to the kernel (TUX server, now evidently unmaintained).
Also, it was never really clear that, in the end, a microkernel would offer significant practical benefits over what could be accomplished (with less work) within one of the established non-microkernel OSs.
- the people required to work on it
- in a timely fashion
- in a coordinated fashion
- the resources required
- bugs, updates, maintenance
2. The competition for aforementioned talent
3. The competition for aforementioned resources
4. Profit motive
Having an "owner" is a strong incentive/coordination point to get stuff done. You need the right owner, but the 95% of projects that don't get it right, we don't hear about.
We hear all the time about how code structure mirrors org structure. A distributed project encourages fiefdoms, a centralized one asks for a taskmaster with final say.
This isn't web development, where "microservices" came up as a way to distribute complex software across stakeholders. There are very hard purely technical reasons for (or against!) microkernels, and the majority of microkernels (if not all) are still released and distributed in a "monolithic" model, i.e. an update distributes all components of it (modulo delta updates, which is also just a technicality).
Nothing inherent to the software is preventing anything. You could be right about the technical justifications. But OP's question wasn't "which is better?" it was "why?"
The person I replied to said, I quote:
> A distributed project encourages fiefdoms, a centralized one asks for a taskmaster with final say.
What does the distinction between microkernel and monolithic kernel have to do with whether the project is “distributed”, or whether it is a “fiefdom”?
time
effort
money
buy-in
morale
credit
pride
What you say is akin to saying that you don’t want to divide the storage in your basement into multiple crates instead of one large one, because you somehow would need multiple people to manage and access each individual crate?
My hunch is that you’re somehow applying micro services in web development, and their motivation for existing, to OS design.
This used to be a site for great technical discussion.
If this is true why doesn't it happen more often?
See my first comment; there's more to building software than just the spec - it's political. I don't like it either.
> This used to be a site for great technical discussion.
Technical discussion has no utility if its done in a vacuum. Maybe academia. But it's not reality.
I don’t understand your very premise: Why does a monolithic kernel imply “central control” when developing, and a microkernel not? How does one or the other change the “politics” of the kernel? Can you state what you think the difference between a monolithic and a microkernel actually is?
probably will be different when humans have to stop writing code
I'm merely offering a different explanation for why software, which is presently built by humans, might express different configurations based on that fact. It's not a qualitative statement as to the merits of one over the other.
You can write a rocket OS as a "proper" microkernel, but there's no way you could write a full-blown Windows OS as a "proper" microkernel.
That said, Windows or Linux can take leaves out of the microkernel playbook and apply them judiciously: KVM's architecture is arguably an example of this, where the core kernel does only the minimum necessary, and everything else is pushed to the user-mode component (QEMU, Firecracker, etc).
Now the question becomes why these components are built in the kernel and not in userspace? The answer is clear for each individual component. See how the in-tree NTFS driver eventually replaces NTFS-3g. Basically when an in-tree solution exists it just gets preferred over userspace solutions because of performance and reliability etc.
A real microkernel OS with stable interfaces would mean its developers do not want control of all the kernel-like servers, and allow even basic users to easily install and use many alternative user servers, as opposed to Linux kernel services sanctioned and controlled by the Linux distribution kernel maintainers.
One big point of making a microkernel as opposed to making a monolith is that you want to give other developers more freedom to develop, distribute, fix bugs on their own time schedule, while keeping compatibility with new kernels easily. In other words, not wanting to control what and when they do, and what users run on the kernel.
Monolithic kernel like Linux, with its constant stream of little changes in interfaces, creates problems for developers of modules, and this leads to selection - those who keep up and satisfy the kernel requirements get in, those that don't keep up or do not satisfy requirements, can't get in, and are degraded to developing against some old kernel version. This is a kind of control.