"we've only been at it for a bit over a year with three people"
By now, probably a lot more. I'm only talking about original architecture and implementation that possessed the reliability features.
a) The MINIX 3 developers had an enormous head start from reusing MINIX 2 in its entirety. In fact, that's how the project began: it was a refactoring of MINIX 2. The same cannot be said for the Hurd, which for the first few years was literally one guy (Thomas Bushnell) writing everything from scratch based on the stock CMU Mach 3.0 he was only getting acquainted with (he was much more familiar with 4.4BSD instead), and that the GNU project spent about 3 years procuring due to its proprietary licensing. Further, the MINIX 3 people later imported NetBSD, again heading on from that.
b) I ran GitStats on MINIX 3 and got a lifetime contributor count of 70, with around 13 major developers. Contrast to Hurd's lifetime of 51 with never more than 4-5 major devs, first few years being only 1.
EDIT: The EU grant no doubt helped, too.
"According to Thomas Bushnell, the initial Hurd architect, their early plan was to adapt the 4.4BSD-Lite kernel and, in hindsight, "It is now perfectly obvious to me that this would have succeeded splendidly and the world would be a very different place today"."
That was an epic hit. :) Far as microkernels, the few in existence were tied up in I.P.. So, I wouldn't rag them if they had to do that part custom. Mach was available with painful work. I believe Chorus was, but commercial. Only alternative I would consider was cloning Hydra's capability-secure microkernel. It ran on PDP-11's that UNIX was built on. CMU might have even given up the code since the project was dead. So, possibilities there. Past that, doing a microkernel at that time would be foolish and 4.4BSD integrated with GNU components was obvious choice.
After certain amount of time passed, there were a number of usable microkernels that could be split off and turned into working systems. There had also been numerous published projects... TrustedMach, Distributed TrustedMach, DTOS... by DOD to improve Mach's failures with UNIX compatibility without much success. Several commercial microkernels, including QNX, achieved success in in the market while publishing concrete details on how. Given all this, I think there's little justification for Hurd to keep trying failed strategy on the very microkernel that gave the rest a bad name. They're better off trying to rewrite parts from scratch on a L4-based kernel or QNX clone. Or joining MINIX 3. ;)
Past that, doing a microkernel at that time would be foolish and 4.4BSD integrated with GNU components was obvious choice.
I wouldn't have expected such a defeatist attitude from you. Really - trying to push the boundaries of the mainstream was foolish and instead we should have just gotten a GNU-BSD? By the way, they did try adopting an MIT research kernel called TRIX, which had in-kernel RPC and was partially user mode, but that flunked due to the costliness of porting it from the 68000 at the time.
After certain amount of time passed, there were a number of usable microkernels that could be split off and turned into working systems.
Ah yes, sure. After you're done with a monolithic Unix that the entire industry will settle on, go do a Perl 6 with "we microkernel now" and expect everyone to follow. Ain't happening.
Given all this, I think there's little justification for Hurd to keep trying failed strategy on the very microkernel that gave the rest a bad name. They're better off trying to rewrite parts from scratch on a L4-based kernel or QNX clone. Or joining MINIX 3. ;)
Retroactively improving Mach is actually a more interesting undertaking than just cloning QNX or doing yet another L4 flavor (which they tried, but found serious issues with L4 and object-capability systems: https://www.gnu.org/software/hurd/faq/which_microkernel.html). Also MINIX 3 doesn't really have the translator concept going for it, nor capability-based security through port rights.
If interested, Timeline of Operating Systems is a good start. Just pick summary of anything major (minus DOS's) and dig into references if it interests you.
https://en.wikipedia.org/wiki/Timeline_of_operating_systems
The capability systems were also forward-thinking with much modern work on OOP, isolation, etc slowly reinventing its methods. Easy extension plus strong security/reliability were their thing.
https://homes.cs.washington.edu/~levy/capabook/index.html
Here's a paper vezzy-fnord shared with a lot of nice examples with features that defined future work and some still ahead of modern systems:
http://brinch-hansen.net/papers/2001b.pdf
Lots of old stuff that still beats modern, mainstream stuff on key metrics. Hopefully, you'll dig into some of it and enjoy the "arcane" stuff yourself. It's arcane, but sometimes awesome too. :)
http://aesec.com/eval/NCSC-FER-94-008.pdf
Note: Mainly see design, layering, and assurance activities.
The most modern use of them is CHERI processor and CheriBSD at Cambridge.
https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/
https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-864.pdf
Note: Paper references quite a few past uses of them in detail.
A great, modern example in software is Code Pointer Integrity (see 2014 paper link):
http://dslab.epfl.ch/proj/cpi/
So, there you go for tonight. Have fun. :)
It's really nothing special. Copious amounts of time coupled with an interest in system software research that motivates me to read research papers, dive into project wikis and the like.
If you'd like to discuss things further, just reply affirmatively and I'll send you an email.
I've started to read your piece, but I've honestly been interrupted constantly so haven't been able to give it my full attention, which I intend to do as soon as I can!
My disagreement with you over systemd, incidentally, never had anything to do with my assessment of your abilities. It was pretty obvious we both have technical skills :-)
It was part of it. Their main goal, per their site, was to replace UNIX('s) with a GNU OS. They failed that one totally far as I can tell and didn't do too well on others for over 10 years. I pick the 2007 papers as the cutoff point since knowledgeable people (eg you) can look at them and assess if they've eliminated those issues since. If not, the failure has lasted over 20 years. So, it's a failure and Linux was a success for the main goal.
If the goal is just exploration, then I agree that trying to do something with Mach and UNIX would be an interesting challenge. Still not best idea given all the failures up to that point of people trying that exact thing and always due to inherent incompatibilities in their nature. This was true in 1996. But it could be interesting and challenging work from a research perspective that might lead to innovations in handling such things. I've never doubted it would be interesting to read on how they handle everything.
"I wouldn't have expected such a defeatist attitude from you."
Maybe it's me going on 4 hrs of sleep for days in a row due to work and whatnot. I decided to give it a fresh look. I spent past hour reading its original paper, skimmed the papers on its mechanisms, read the 2007 critique in your microkernel link (do read that one), their position paper on fixes, and so on. Brain still tired but here's a semi-fresh take on it.
GNU was heavily invested into UNIX but not proprietary-related BS surrounding it. They wanted to maintain compatibility with their tools, POSIX, and especially a filesystem-centric model but with more flexibility, security, and reliability. Bushnell and supporters decided on a microkernel approach, built around Mach specifically, kept privileged the minimum apps needed, kept as much in the standard library as possible, and tried to rework that combination into something largely compatible with UNIX but different & distributed underneath. All systems up to that point had failed to even emulate UNIX on Mach without compatibility, performance, or security issues. However, they managed to mostly integrate it with better performance, decent compatibility, better (but not good) security, and unknown [to me] reliability. The proxying approach to service construction, used prior in capability & security kernel work IIRC, was also independently invented in Hurd with lots of improvements in usability and consistency. Met their goal for flexibility and debugging in non-kernel, OS modifications.
Now I contrast it to the other work a bit. Work prior to it, esp KeyKOS, managed to decompose the system further with the desired properties, strong security, good reliability, flexibility, and high performance. QNX created an integrated, UNIX-like OS that was fast, real-time, flexible, multi-server, and lacked Mach's weaknesses. They made it distributed & SMP-aware in their post-2000 version. L4 and separation kernels stayed with custom components directly on microkernel and virtualizing away POSIX/Linux/BSD as user-mode process with lightweight spawning (eg Dresden TUD:OS). The latter partly due to perceived impossibility of integration without API-related problems like in old Mach+UNIX work.
Further, many of these assumed and maintained POLA in much of their design while Hurd didn't. Two examples: too trusting of UNIX assumptions, necessitating full enumeration for defensive programming against many unknowns (Walfield & Brinkman 2007); its own assumptions like "you can't harm a process by giving it extra permission" (Bushnell 1996). So, doing the comparison, it's design has serious and often fundamental problems that others avoid which stem from the specific requirements of the authors mandating certain components and interfaces.
Let summarize. It's interesting work given the challenges and what they achieved. It has too many problems that are designed in which alternatives designed out. Mostly due to trying to shoehorn monolithic, insecure UNIX/POSIX API into a multi-server, robust microkernel with tight integration. Hard to say if further work is worthwhile given inability by 2007 to fix fundamental issues caused by mismatches in their goals, Mach, and UNIX/POSIX API. (Have they been since?) Although some precedent existed, the project did seem to make a lasting contribution via their translators. These were effective and UNIX-like enough that they got uptake that led to FUSE and later Rump kernels. Both are being used in many application areas ranging from development to desktop/server use to cloud applications.
So, that's the overall take. It was a failure in many respects, but not a total waste. The translator concept was a signficant win in FOSS with long-term benefits. The battles at mixing two incompatible systems might benefit other work extending UNIX's or even non-UNIX's using their RPC model. People wanting Hurd to succeed should read the critique and solution papers while knocking out as many of these issues as possible. By 2007 & maybe currently, numerous attributes of the system are inferior to other works. Anyone wanting to meet the goals of the project are better off building on other approaches, esp non-Mach, while considering selectively breaking POSIX model or even entirely virtualizing some aspects. That's my current take unless 2007-2015 brought in amazing, fundamental improvements.
Thoughts on the semi-fresh analysis?
Currently, the goal is indeed exploration. RMS and GNU have given up on it and morale is low. Can you blame them?
In fact, RMS' decision to go with Unix was a pragmatic one. He himself was always more a fan of ITS and Lisp Machines (see his October 1986 speech at KTH), but rightfully sensed no one would end up using such a solution, and thus decided a clone of Unix with non-breaking semantic improvements where possible and a Lisp-y userspace was the best way to bridge adoption with wanting to push forward. Hurd satisfied the first, the second unfortunately stagnated as Guile never broke the chasm until very recently now with GuixSD.
QNX was proprietary and didn't have its architectural unveiling until work on Hurd was started. It again did not have the translator concept, but like MINIX 3 had static servers. KeyKOS' main selling point was the single store and the orthogonal persistence, which nominally could be done on Hurd e.g. by integrating Mach VMM policy into a libdiskfs. It was not a priority then. L4 was too spartan for the experiments on object-capability systems that the Hurd devs embarked: https://www.cs.cmu.edu/~412-s05/projects/HurdL4/hurd-on-l4.p...
It's true that security was never a primary orientation of the Hurd, but its security is still better than monolithic Unixes. And do keep in mind QNX and MINIX 3 are not nominally security-oriented, either. It's the communication boundary structure (and in Hurd's case, port rights and auth server) that intrinsically create superior cases.
Walfield and Brinkmann's 2007 critique was done from the perspective of developers themselves. It wasn't a general criticism of the Hurd, but one of specific problems related to resource accounting that other u-kernel and most monolithic kernel designs suffer from. Relative to the state of the industry, it is still a leap forward. Relative to the state of the art, it is not, but then so do QNX and MINIX 3 make sacrifices.
Implementing the POSIX API has nothing to do with the Hurd's problems. And, indeed, Hurd does break the POSIX model in several aspects (e.g. allowing multiple uids per process). But then we must disqualify QNX and MINIX 3 by this token, too.
The fundamental issues with the Hurd then have little to do with the 2007 critique. More pressing is its incompleteness, which hopefully rump integration will finally break its samsara. If all else fails, using rump drivers on a heavily stripped Linux with Mach module emulation to create a true hybrid kernel, coupled with functional package management and system state management via Guix with added on orthogonal persistence and network transparency could be another future direction for the Hurd I might end up exploring. Perhaps tripling both as a multi-user OS, a unikernel and a container OS via subhurds.
The Mach improvements have mostly been surface, but again, I would not see it as a huge bottleneck (only an inelegance) and in the hybrid case it becomes irrelevant beyond as a programming API, which it isn't that bad of one.
I said it was a failure and they should give up on it outside research. So, no, I support their decision. ;)
" But then an explicit goal of the Hurd specifically was freedom from tyrannical sysadmin policy, which necessarily included freedom to host one's own private namespace in a multi-user system that doesn't affect the global namespace."
That makes sense, esp given the time it was made. Clouds, etc are still experimenting with models on that due to demand.
"QNX was proprietary and didn't have its architectural unveiling until work on Hurd was started."
And then they had the details they needed for the next 10-20 years. So, my memes about applying proven methods or not learning from the past may apply here.
"KeyKOS' main selling point was the single store and the orthogonal persistence,"
KeyKOS's main benefits they touted were security/reliability w/ fine-grained POLA, high performance, and flexibility. One could even extend and share things in such a way that untrusted code couldn't swamp CPU or memory. KeyKOS did in many use cases what Mach projects failed to do thanks to its design decisions and complexity. That's despite experienced, brilliant kernel programmers working full-time on it. KeyKOS also had the two benefits you mention. So, building a KeyKOS-style kernel had huge benefits over a Mach kernel. That's why Shapiro cloned it with EROS. I expect porting such attributes to this Mach project will also be (or were) hard.
"It's the communication boundary structure (and in Hurd's case, port rights and auth server) that intrinsically create superior cases."
It's isolation, resource management, and communication. They're necessary for reliability and security as both involve components behaving badly. Also, for reliability, add in self-healing capabilities with at least monitoring and restarting of drivers. The last is a feature but the first three are baked into underlying architecture. The paper noted it lacked one for sure. Further investigation might find problems with the others.
"Walfield and Brinkmann's 2007 critique was done from the perspective of developers themselves. It wasn't a general criticism of the Hurd, but one of specific problems related to resource accounting that other u-kernel and most monolithic kernel designs suffer from."
"Implementing the POSIX API has nothing to do with the Hurd's problems."
We might be reading different papers as there were several. Here's the one I'm referring to:
http://walfield.org/papers/200707-walfield-critique-of-the-G...
It's entirely about Hurd except when it's critiquing Mach. It first describes the architecture and mechanisms. Then, it goes with the critiques. I'm ignoring those that show up anywhere and just require a tweak. The biggest, for reliability and security, is that it's trust model is discretionary on app/process level: program inherits user's privileges with no further limits and any reductions are at program's discretion. Capability, MAC models, and even Windows Vista techniques counter threats in that space that Hurd wouldn't challenge.
The next come from UNIX/POSIX compatibility & integration style. The filesystem shows how vanilla UNIX apps might break or require significant debugging due to UNIX assumptions that don't apply. The server allocation problem with open is another but might be fixed if INTEGRITY RTOS's brilliant defence is used: calling process donates its own CPU/memory for external functions. Would that break POSIX/UNIX behavior? Idk. Author follows up, though, by pointing out most things in Hurd have lots of ambient authority and would take modifications. These are usually due to UNIX compatibility. Note that ambient authority, along with its resulting downtime and hacks, we're one of reasons UNIX was ditched for capability architectures by projects that built them.
They then look at Mach. One problem, optimization w/ predictable patterns, is indeed a general problem for microkernels this flexible and dynamic. This flexibility also makes for unpredictable delays. That's why most microkernels are static and/or optimized on critical parts. Mach also lacked secure management of global resources in such a dynamic system. Authors mention KeyKOS/EROS which didn't have that problem. There's others but those were best at this IIRC.
So, the paper and it's problems were mostly specific to Hurd and its usage of Mach. In each case, it's an open problem about how to fix that in Hurd while each are a done deal in one or more competing technologies. Mach and UNIX compatibility either prevent the fix or make it quite difficult. So, these should count as Hurd/Mach problems caused by their specific design(s).
"Relative to the state of the art, it is not, but then so do QNX and MINIX 3 make sacrifices."
They do. Not calling them perfection, either. However, they intentionally make choices that make their job easier at kernel, architecture, and API levels. Good engineering involves making right tradeoffs rather than setting oneself up to fail. That's my problem with Hurd if judging against its goals but in exploration mode they just need to make progress. Or make changes.
Note: I do disqualify the two from the secure attribute due to their compatibilities and QNX's UNIX code got it smashed first time an online presence was publicized. Reliability, performance, resource management, and UNIX/POSIX integration? Among best in class.
"The fundamental issues with the Hurd then have little to do with the 2007 critique. More pressing is its incompleteness"
That's another issue on top of the fundamentals I referenced earlier. We're probably going to learn about some more side effects of them when they add Rump kernels assuming monolithic UNIX to the mix. It's still a good move on their part, though, and would bring lineage full circle.
"f all else fails, using rump drivers on a heavily stripped Linux with Mach module emulation to create a true hybrid kernel, coupled with functional package management and system state management via Guix with added on orthogonal persistence and network transparency could be another future direction for the Hurd I might end up exploring."
Now, that sounds like the hacked together, against-all-odds, balls-to-the-wall solution I like to see! Although above issues will affect multi-user part: specific examples were already cited. Might be fine if they're benign users. So, build the monstrosity and make it work! :)
"I would not see it as a huge bottleneck (only an inelegance) and in the hybrid case it becomes irrelevant beyond as a programming API, which it isn't that bad of one."
In the past, it was a bottleneck but people thought the API was fine. So, it will be interesting to see what happens on containerized, multi-tenant, IPC-heavy workloads. On it vs alternative schemes.
A microkernel design seemed sane from experience in the UK. I don't know if it was technically a microkernel, but the GEC OS4000 Nucleus <https://en.wikipedia.org/wiki/OS4000> supported a system that made VMS VAXen look slow in the 80s. (Latterly, OS6000, with a software Nucleus, still compared well with System V on the same hardware.)
L4 was already tried for Hurd, surely.
Well, one company did build on that work that I can tell. They built on it so well they dropped bombs on the formal verification community. You might be surprised which company that was.
Safe to the last instruction: Automated verification of a type-safe operating system (2010)
I don't know to what extent security was the reason for the hardware nucleus rather than a by-product of the fast system. (I recall French colleagues thought the context switching was slow, thiking it must be in milliseconds, not microseconds.) However, the software versions for the later OS6000 and whatever the 88k implementation was called, seemed OK.
I don't understand the reference to the paper. I don't think there was any formal verification in OS4000 (or the hardware, given the 4190 arithmetic bug), or GC in Nucleus, and the typed assembler doesn't look to have much in common with Babbage. Which company?
Good to see an OS4000 user who appreciates its significance after all this time, anyhow.
Interesting. I never thought about a connection to RC 4000: just figured it was another example of how things were named back then. Could've been some influence.
"I don't know to what extent security was the reason for the hardware nucleus rather than a by-product of the fast system. (I recall French colleagues thought the context switching was slow, thiking it must be in milliseconds, not microseconds.) "
I doubt security was the reason as well. Few were thinking of it then. It was just a benefit of the design.
"I don't understand the reference to the paper. "
The original system designed a core component to make everything else easier to implement and consistent. That was called Nucleus. It had specific functions key to any OS. The rest was built around it.
The Microsoft system designed a core component to make everything consistent and verification easier. That was called Nucleus. It had specific functions key to the OS and many potential others. The rest was built around it.
I just thought whoever designed Verve might have heard of GEC 4000 and copied it to some degree. Or independently invented the concept. GEC 4000 still has advantages, though.
"Good to see an OS4000 user who appreciates its significance after all this time, anyhow."
I've never used it or any machine from 1960's-1970's. Someone asked vezzy-fnord recently why he has so much arcane knowledge about computing and programming. I answered for myself on that as I similarly research old work for the wisdom to be gained:
https://news.ycombinator.com/item?id=10488298
I have literally hundreds of papers on old stuff with many solving key problems and some doing better than today's stuff by at least one metric. I expressed the problem and potential solution here:
https://news.ycombinator.com/item?id=10269245
So, referencing Nucleus is just me doing my part. Neat that you got to use the thing for real. How robust was the system compared to others of the time? Did you find the kernel design or process separation to have practical benefit in its day-to-day operation or administration?
Meanwhile, Tanenbaum's grants from the EU total approx $6 million for the express purpose of producing a reliable microkernel and to build an operating system on top of it[3], all under the auspices of a research university. And as others have noted, most of MINIX's userland is the result of splicing in large parts of the existing NetBSD userland, so the skew in attention that the kernel received is pretty significant, cf GNU's Hurd.
1. http://tech.mit.edu/V110/N30/rms.30n.html
2. http://www.gnu.org/press/2001-12-03-Takeda.html
3. https://www.bsdcan.org/2015/schedule/speakers/258.en.html
On top of their design decisions that made the project ridiculously hard in the first place.
https://news.ycombinator.com/item?id=10486120
Note the quote from the head of the project who confessed to the specific mistake. That's not tautological at all: it's what 5-10 seconds of Google would've earned you. It also matched the first idea that came to my head about where I would've started at the time. Linus and Tannenbaum both started with Minix codebase, one indirectly and one directly, to achieve success.
So, Hurd was just Doing It Wrong. Further, that starting with a poor architecture or not building off a good codebase often leads to failure are proven heuristics.
I think the simple architecture is their main benefit, though. They only put in complexity where the most benefit. Then they reused NetBSD's work. A common cheat.