Amoeba is a famous example of a distributed OS https://en.wikipedia.org/wiki/Amoeba_(operating_system)
- Inbreeding of ideas: researchers inventing and solving problems noöne actually has, without observing the discipline necessary to remain interesting and avoid emotional attachments (as is done in maths, with varying success). In the distributed OS case, the model in people’s heads (including Pike’s with Plan 9!) seems to have been a closed campus network, which is close to some of the areas people care now, but still isn’t quite any of them. The social problem of the system not being able to take over the world, thus necessitating explicitly specified protocols and loose coupling, also does not seem to have received much consideration. (Except for Microsoft, who has tried—deliberately or accidentally—to weaponize the homogeneity requirements of some of that research in building Windows domain networking.) Some of the kludges aren’t—they are solving social problems.
- Being before its time: sounds glorious and heroic, in reality makes people cold, lonely, and more often than not broke. The most commonly known instances of this are for individuals (or other actors capable of having a coherent volition, such as companies), but the distributed OS story feels like this happened to the community as a whole.
I think the main issue is actually more related to the trend in academia in which the 'least publishable unit' is ever more important. Pike's essay touches on this. Coming up with an interesting new OS design is hard, and implementing one is worse, so people don't try because it'd take too long and there wouldn't be enough papers in it for anyone. Companies still do OS research (see Singularity and Midori by Microsoft, Fuschia by Google) but don't publish much about it.
The problem is to become sustainable you need users, and to get users you really have to be above the core OS these days because that's where you can focus on concepts instead of getting distracted by kernel and driver writing (not much interesting to do there). If you want to experiment with microkernels, do it as a collection of processes in the Linux userland and if you want to extend that to get users, make it run on Windows too. Chrome is in effect a kind of microkernel-ish operating system, seen from this perspective. It consists of a pile of sandboxed components connected by a sophisticated IPC system called Mojo. It also handles properly cases that microkernel research sometimes ignores or punts to the user, like how to gracefully handle if the other end of a connection crashes.
I feel strongly that there's still lots of interesting research to do in the operating systems space but if I were to do it, I would not start by writing an actual OS. I'd start by exploring the ideas as an set of ordinary programs that could run on a regular OS. Maximize the window and start gradually eliminating dependencies on high level APIs provided by the host, if you really want to. Then you can explore concepts like single address space OS's, ultra-fine-grained microkernels, better shells, polyglot programming, distributed computing etc, without getting bogged down by irrelevant detail.
If you want novel computer systems you can't build them on top of the old ones. That's the whole point of Pike.
An OS is the thing in about Linux (kernel) scope. The whole point is that this is something that is by no mean "done". If you want innovation in this space you need to build exactly such a thing from scratch!
But the OS is only one part of systems design. Doing something novel in this area would be to continue where we stopped around 1970. There was no progress since than. Only one computer architecture with only one OS left. Since than nothing truly innovative was even tried. (OK, to be fair, almost nothing. There been things like Intel's iAPX 432, or Microsoft's Midori, or things like the Connection Machine. But mainstream is still on the std. tech form the 70'. Even mainframes are actually technological more advanced, as they had for example multiprocessing, VMs and containers already decades ago).
I for my part think that if we would rethink computers form the ground up in light of today's possibilities they would be very very different. Alone the security hell today is only there because we still want to run "normal C programs". Everybody knows that this is not fixable without throwing away some of the old concepts. Still nobody wants to move. (Almost perfectly secure computers would be possible. You "just" need to build capability security into CPUs, the OS, and all apps…). Or another example: Scaling to many many cores would be much easier if computers would be fully data-flow aware. But currently we like to pretend that our CPUs are still PDP-7s. They need to offer the exact same interface (conceptually) or else this would not match the C abstract machine any more 1:1… But the CPU than needs to use a built in "hardware JIT" to translate the sequential command streams it receives to its internal data-flow operations. Somethings that's extremely inefficient and overly complicated—besides that it conceptually can't even help much. There is only so much parallelism extractable form an inherently sequential description of computation… (The most crazy part about this is that we have plenty information available on the high level which we happy erase during compilation, so we can present a "nice flat command stream" to the CPU. That's imho pure insanity). But nobody dares to rethink such systems as a whole: which means form the languages and their abstract machines, through the concrete hardware, the OS level, up to application design. Things like the von Neumann bottleneck were discussed already almost 50 years ago. The issue remains open to this day. Just put another layer of caches on top. Nobody cares about efficiency anyway… Because rewriting software would be to expensive. So the hardware can't be improved. So software needs to optimize for the old hardware concepts. Which leads to the dead-lock in which we're currently, where no substantial improvement can be made any more.
We're in this trap since 50 years. And that's why systems research is absolutely dead by now.
My 2ct. Sorry for the long post. But I have strong feelings about that topic.
Free doesn’t just succeed because people don’t want to pay. It succeeds because you can just copy it with zero friction. It’s viral.
Nothing beats "free & good enough".
But it blocks now any meaningful progress. That's a pity.
Problem is likely that people just don't know anything else by now.
If all you've seen is Windows and a Unix clone, well, you could indeed come up with the idea that Unix is "well designed". Even it's actually not designed at all, and more or less everything was just a random accident (and usually even the worst, but "simplest", option around).
Still I don't think that it has "won" because it's amazing on the tech level. It really isn't!
It's a big hack. With ridiculously stupid / primitive ideas behind it. (Just to name one of my favorite elephants in the room: A stream of "chars"—whatever a "char" is—is a "data format"? Even a "universal" one? Alone saying such things out loud is absurdly foolish, imho).
But by now legends have formed around the whole story. Most of the "Unix philosophy" for example is a post-hoc invention by people who have some strong urge to "look smart" in retrospective…
The truth is Unix started as a toy without any serous planing behind it. But than business discovered it by chance, and business came to to the conclusion that "a Bobbycar is good enough" as long as it doesn't cost anything. Business is never interested in excellent technical solutions! It's always only interested in "the cheapest shit that gets the job done"™ (barely). Unix fits this pattern very well.
A small reminder where we're still standing mostly:
Nonsense
ca 30 years ago there were (serious!) efforts to fix this on the CPU level - this whole VLIW CPU/smart compiler with explicit parallelism. Intel Itanium, Transmeta, Elbrus to name a few. It didn't work out well.
System research might produce something looking good but not working out well at the end.
But I tried already to explain why I think those things failed: It's all about the software.
You can't build a "sufficiently smart compiler" if the task is to parallelize an inherently sequential command stream.
This endeavor failed because it was part of the wrong effort.
The whole point is that people didn't think in terms of systems design any more at this time! They looked on "the hardware problem" in separation.
The expectation was that you can take a program written for "a fast PDP-7" add some "smart compiler" magic on top, and let this run (hopefully) faster on an improved hardware architecture. This didn't work out. Imho to no surprise.
I think the problem lies in the mental model behind computers which dominates the perception is since some time. We're almost exclusively (at least when it comes to general computation) left with the the model of a sequential command stream machine. CPUs execute (interpret) a sequence of instructions. Those instructions may alter an "infinite" (or at least "gigantic enough") global state in arbitrary ways. That's the basic model.
As long as your software is basically written with this model as background you can't "fix this on the hardware level", at least without real magic.
What we no do is to mitigate the problem as best as we can with methods that reached imho completely insane levels.
I think that we could find much better solutions if we would take a step back and rethink the machine in whole. I mean, software hardware co-design is a thing, only not when it comes to general computation especially not together with also OS / runtime (software) design.
My current personal favorite idea how thing could be improved and simplified is to see a computer as a graph of event-stream processors. This mental picture scales nicely on different "zoom levels". From raw hardware to the runtime level to (distributed?) application design. Only that it would require languages that compile down to such a model in an efficient way. A command stream language wouldn't map good or likely even acceptably (same problem as with the old "more exotic" hardware designs).
BTW: For number crunching VLIW and compiler guided parallelism works quite fine. Vector CPUs anybody? That were the fastest computes once, afik. And since MMX up to AVX the whole idea is a big mainstream success. (We're not even talking about GPUs here of course, that's not the topic; nobody tried to replace "typical" CPUs with GPUs so this does not count).
I'm using the standard street definition in which the OS is everything that you get on a fresh (desktop) OS install, not just the kernel.
There's very little research available on kernels because their shape is largely dictated by the hardware design, and the hardware design is largely dictated by physics and what the manufacturers give us. But as you go further up the stack the options widen considerably. You can start to consider user interfaces, application programming models, etc.
"You just need to build capability security into CPUs, the OS, and all apps…"
Pure cap systems are an old idea that have never worked. They're too cumbersome for programmers and affect API design in problematic ways.
But here's the thing. If you disagree, you can go ahead and prove it. You don't need a custom CPU. Figure out the higher level software issues first, and if you ever get to the point of needing actual CPU changes then fork qemu and virtualize your intended design. Make it so custom hardware is just an optimization once the core ideas are worked out.
That's what I mean by doing stuff at a higher layer. The interesting and tractable problems are all at the level where you can just throw up a fullscreen window, use direct IO to grab a block of disk space and off you go. Start from the top and work down, not bottom and work up.
BTW you'd probably like the Mill architecture. It tackle a lot of the things you're talking about. Comes out of the commercial world though, not academia.
>Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines
It's a regular user-mode executable.
The idea is that it provides a kind of "virtual OS" to the applications build against it.
But Mesos needs a regular OS as runtime on the nodes of the cluster.
I don't think that necessarily excludes something from being an OS. Historically, a number of Unix ports ran as applications on top of a non-Unix. AT&T's early 1980s port of Unix to IBM S/370 mainframes, ran Unix as an application on top of IBM's TSS/370 operating system. [0] The Interdata 7/32 port done at Wollongong University in Australia, initially ran under Interdata's OS/32 operating system, although once the port had advanced sufficiently to run on bare metal, OS/32 was ripped out from under it. [1] The first versions of Cray's UNIX port, UNICOS, supported running as a guest under their earlier COS operating system (which was a batch processing operating system, modelled after that of CDC mainframes.)
I still don't think Mesos counts though. Applications running under Mesos access resources by calling the Linux APIs; Mesos configures the Linux environment the application runs under, but at runtime it steps out of the way. If, hypothetically, Mesos had exposed its own API to applications running under it – separate from the Linux API, even if ultimately implemented using it – and forced them to use that rather than call the host OS APIs directly – then I would count it as an OS. But, that would be a rather different thing from what Mesos actually is. (And all those points apply to Kubernetes too.)
With talk about WebAssembly as a container format – if you had something like Mesos/Auroroa/K8S/etc which ran WebAssembly containers only, so that applications running under it could only access WASI and other WebAssembly-based APIs – that would be closer to a true OS, even if it happened to run on top of the Linux kernel. You could swap out the Linux kernel for another – BSD, Windows, Illumos, XNU, GNU Hurd, L4, Fuschia, z/OS, whatever – and the apps shouldn't be able to tell the difference.
[0] https://ieeexplore.ieee.org/document/6771917
[1] http://bitsavers.informatik.uni-stuttgart.de/bits/Interdata/...
This would make more or less all VM runtimes "OSes".
In a sense, they are "operating systems", yes, but it's more about how the APIs bind and constrain the applications that build on top of them. For that reason I call K8s "Windows for the cloud". But that's another story.
I would say that, by a "strict definition", OS describes the software that runs on the metal, and operates all the different kinds of (application) runtimes and (OS) services under itself. A type 1 hypervisor would be by this definition an OS for example. But say a bare metal implementation of Doom would be not, I think, as it would not provide any implementation of a ("universal") app runtime. [Please don't say Doom is Turing complete. I hope not!]
And Mesos? Mesos is really something else. It's a distributed "task manager". But even without a scheduler. Everything else should be provided by "frameworks". One could say Mesos is a kind of micro kernel for the data center (and the frameworks are the OS servers). But that is very figuratively speaking.
Mesos is also less a "Windows for the cloud" compared to K8s because it supports different frameworks as runtimes. You can bring our own framework (or plugins for some existing). K8s on the other hand tries to be (more or less) "the one true" all encompassing API. Vendor lock-in works than as with Windows (or Linux, or macOS / iOS)^^.
I argued that Mesos is not an OS because it exposes so much of the underlying host OS API - well, the same is true of most language runtime VMs. Java’s famous slogan was “write once run anywhere”, but in practice it exposes a huge amount of the host operating system, such that Java apps often end up with multiple code paths (even entirely separate classes) for different host platforms. The same is true for node.ha, Python, Perl, Ruby, Tcl, most Lisp implementations, etc
But, imagine a language runtime which didn’t do that - that hid the host OS so well, apps written for it couldn’t know or care what the host OS was (or whether it was actually bare metal). Such a runtime would have greater claim to be an actual OS.
Imagine a language runtime which included its own filesystem implemented in userspace - which was a single file on the host filesystem. Maybe it even has its own user space network stack, using TUN/TAP. It treats the host OS basically as a microkernel, only consuming a small subset of services provided by it (such as memory management). Hence you could swap the host OS for an actual microkernel running on bare metal and no app would notice.
> I would say that, by a "strict definition", OS describes the software that runs on the metal
Nowadays, many OS don’t actually run on the “metal” directly, rather under a hypervisor. On some architectures, a hypervisor is architecturally mandatory, there is no ability to run without one. Whether some facility is implemented in the real hardware or the hypervisor may be an implementation detail which can change over time, oblivious to the OS. Given all of this, I don’t think “runs on bare metal” is necessary to be an OS, especially given it is not always clear any more what actually counts as “bare metal”
Even when they run on metal they don't talk "directly" to it..they often just talk to the firmware's.
I'm not sure about such a definition.
This would make more or less any ISA implementation on higher level hardware an "OS":
On advanced chips the ISA is just an API. The code gets JIT compiled by the hardware into its actual machine code.
I would not really agree an ISA (implementation) is an OS…
> […] only consuming a small subset of services provided by it (such as memory management).
Well, memory management, I/O, task scheduling, HAL, and all the other things an actual OS provides. ;-)
A runtime (or usually a few of them) is / are part of an OS. But a runtime isn't an OS by itself.
The parts that talk to the hardware are what makes some runtime an OS…
> Nowadays, many OS don’t actually run on the “metal” directly, rather under a hypervisor. On some architectures, a hypervisor is architecturally mandatory, there is no ability to run without one. Whether some facility is implemented in the real hardware or the hypervisor may be an implementation detail which can change over time, oblivious to the OS.
I think I've clarified this point already.
A (type 1) hypervisor is an OS (according to my previous definition).
> I don’t think “runs on bare metal” is necessary to be an OS, especially given it is not always clear any more what actually counts as “bare metal”
Well, the question is more whether it (potentially) can run on bare metal.
You can run any OS in a VM, sure. But you could never run a "non-OS" on bare metal. That's the difference.
A runtime that can't run without an OS (the part that runs on metal) is not an OS. But sure, such a runtime could be in fact a very large part of an OS.
I think – in practice – one of the most distinctive things an OS supplies is file management – APIs to open/close/read/write/etc files/directories, pathname syntax parsing, etc – both the VFS layer and the filesystems (if they are distinguished). I've never seen a hardware ISA (as opposed to a bytecode implemented in software) with instructions to do stuff like that. If that's the yardstick, no "ISA implementation on higher level hardware" counts as an "OS".
Most language runtimes provide those kinds of facilities, but they bulk of the implementation work is actually done by the host OS, with the language runtime often doing little more than providing pass-thru access to host OS services; sometimes adding some virtualisation layer on top (for example, Tcl VFS), but still the host OS file APIs are the by-far most common backend.
Whereas, a language runtime which actually implemented its own filesystem internally, and just asked its host for some block storage, is much closer to being an OS - how much does it matter whether the block storage is files on a host OS filesystem, or a paravirtualized block device, or a "real" block device such as SATA/SAS (which might be virtualized anyway)? The code change to switch between them is relatively small, it could even be pluggable modules. You can even talk SCSI/ATA from userspace: over network (iSCSI or ATAoE) or using SCSI/ATA-passthru support provided by the host OS.
Other features I've never seen in an ISA – a TCP/IP stack, user/group management (as opposed to some capability primitives which that could be built on top of). A language runtime which provides those itself, rather than relying on the host OS, is really a guest OS.
> Well, memory management, I/O, task scheduling, HAL, and all the other things an actual OS provides. ;-)
A lot of those things are at least partially implemented in hardware, although the hardware/software balance varies widely from architecture to architecture. Although a rare feature nowadays, several CISC ISAs include hardware-support for task switching (x86 TSS are most famous example although never saw much use; IMPI, the underlying ISA of the IBM System/38 and CISC AS/400s, is another). Some ISAs have even gone so far as to add hardware acceleration for garbage collection (Lisp machines, Intel iAPX 432). An operating system running on an ISA which provides many of those functions in hardware may delegate most of them to the hardware; what's the difference between delegating those functions to hardware versus to a host OS?
A guest OS asks for relatively few services from its host ("give me 4GB of memory, 100GB of block storage, an Ethernet interface, and here's some threads to run"). A mere language runtime expects the host to provide a filesystem (as opposed to just block storage), Berkeley sockets (as opposed to just a TUN/TAP device), a security database (NSS, PAM, /etc/passwd, Windows LSASS, etc), executable loading (here's the name of an ELF/PE-COFF/Mach-O, please load and run it as a subprocess with these arguments) – a guest OS does all that itself.