Or are you asking how we create more people who will do this in their free time?
I've attempted something similar myself - a refactoring of a very dirty Java project that was orders of magnitude smaller than the Linux kernel. I didn't have good refactoring tools, nor even a good SCCS; my month-long effort had to be backed-out. I didn't enjoy it - it was boring, but needed doing, because the accumulating cruft was impacting my ability to work on the codebase (and presumably everyone else's). The project stayed dirty, and my reputation suffered.
Credit to Ingo for his perseverance. That is massive commitment.
That said, in your case it felt a bit like an organizational failure also. As you could read in Ingo's mail he had to start over after plenty of work put in, for him there wasn't any reputation damage because he seems to have had free reign to start this as he wished.
It makes a big difference for the thousands of kernel developers [0]. There's also more than a million linux users globally[2], assuming only 1% might compile their own kernel, that still leaves tens of thousands of linux users benefiting.
Not to forget about the side benefits of a refactor. There could be runtime improvements coming out of this too down the road.
[1] https://en.wikipedia.org/wiki/Linux_kernel [2] https://www.zdnet.com/article/how-many-linux-users-are-there...
Also, as a Clojure dev, you cannot recompile the running kernel while using it at the same time using something akin to REPL driven development. That would be the other side of the live-patch productivity boost in so many areas.
If I am not mistaken, the CPU and other devices don't care, if everything is compiled ahead of time and a few functions tacked on later, as they are changed by e.g. patches, or if the kernel is completely recompiled. Obviously, I am simplifying a lot but it is a very serious question. We really should strive to build maintainable systems that don't have to reboot just to apply updates.
Here, you would need to involve a compiler of course and do more due diligence and then run ftrace or something to exchange the code. Also, you would probably in some cases need to make sure, the ordering of code is right as to not hurt performance. (See Emery Berger's talk https://www.youtube.com/watch?v=7g1Acy5eGbE which probably applies here as well.)
Of course, with great power comes great responsibility. You could add some optional checking facility, that would be able to roll back the changes to code if some conditions were met.
The rest could probably be solved by a very short "stop the world and switch a few pointers" routine.
I seem to get satisfaction from things that a lot of others might find tedious/boring.
People like us though just need to watch ourselves that what we’re doing is actually useful opposed to just scratching an itch. Which of course is not exclusive to us as developers often find many other ways of scratching itches without actually adding value.
On my best projects, I've done the first 90% so that a great many other can collective do the 2nd, 3rd, and 4th 90%s :). Without being "goal oriented" you can feel good about unlocking that part that would be too annoying for others, and they in turn can do the finishing work that wouldn't be fun for you.
Don't feel like you're not doing important work just because others are slotting in the keystones.
Reading code is fun and it's a skill that took me time to develop. People need to told that's a skill you need and yes (shock) not emphasize creativity as much as we do. Creativity is great but it has enough encouragement in society already. We need people willing to read what already exists as well.
If you do a from-scratch rewrite, you could stay in igorance and might make those mistakes. But if you do a refactor, especially one broken up into many steps like this Linux one, it in fact demonstrates a better understanding the existing code and any nuance than merely leaving it in place!
Not touching scary-looking code is just bad, and not a wise form of humility.
That or they advocate doing a from-scratch rewrite.
But I think throwing away the changes should be an option if there's no objective improvement. Very often those changes end up on the main branch but they only cause changes, not improvements.
The real outcome is in someone's head.
It may well be that same sense of ownership creates an "inner circle" effect that is a barrier to entry for creating more such developers.
Good team builders will find a variety of different employee talents and interests and let people do what they like and develop those skills.
So what do I win?
In other words, in a perfect world, someone recognizes the value-add of your passion, and offers you money to do what you would do for free.
Evidence of this is all around us, and while most won't get this, some significant proportion of people pull it off.
So we need to restructure our society to have economy the promotes leisure not output.
That will at least align the incentives right with a lot of more extreme engineering that just doesn't make economic sense today.
On a completely different level, yes, this is the sort of thing that does require passion. There is a lot of obscure stuff that is hard to plan or otherwise validate society. I'm a bit of a utopian I suppose in thinking if we had more leisure and guaranteed consumption, people would be able to do more passion projects, and we could give people hindsight recognition just instead like this.
Basic needs can be unconditional, renown can be the reward for "extra" work.
Finally, I hope as more monoliths are broken into libraries, we get more of this sort of stuff organically. Beyond the economics, conways law holds us back. (Even within the kernel!) Need to make sure people feel free and safe to really get down in other people's code, to develop the macro view, to see stuff like this.
This is the sort of "macro" refactor you can't get to with just local optimizations alone. We can and should break it into steps, but by not means does every commit have its own clear perf benefit.
Health problems -> Money problems -> Fun problems.
You get the privilege to work on "fun problems" once you don't have "money" or "health" problems (otherwise, the "fun" problems are the least of your "problems").
Linus Torvalds worked on Linux for a long time before being directly compensated for it, but he probably would never have done so had he needed to worry about whether he could afford his next meal and rent that month.
I had free time, and an interest. Often the same characteristics of people who contribute to FOSS.
> The fact that they were still using mailing lists
In my personal experience, going back some 30 years or so online with my own personal paid email account – which is still live and still works – I find that most people I've worked or interacted with who dislike using email for workflow or other important comms do not know how to use email effectively. Strange as it may seem, this now applies to the developers of most email clients and online email services.
Email and mailing lists are remarkably powerful and capable tools, if used properly. Most people do not use it or them properly. I have yet to see anything else by anyone that is an improvement in every way on email.
> I'm not sure how it will progress when they retire or die off.
TBPH, as a Linux user and professional, I rather hope it does not.
Linux is an amazingly useful tool, but OTOH it's now, in and of itself, a vast hairball of technical debt. The traditional UNIX model itself is.
It is long past time that we should have moved past it on to better things. There have been many attempts but none have ever achieved critical mass.
At the rate things are going it looks quite likely that our technological civilization will collapse due to out-of-control global warming and the ongoing and accelerating mass extinction event. As the parent of a 2YO I very much hope I am wrong.
But if I am and it doesn't, the world needs better tech, and a ½ century old UNIX clone already does not really cut it today.
If Linux and all the other monolithic Unixes die when their dev teams die, we'll have to move on to newer, smaller, simpler systems that a new generation of programmers can actually read from top to bottom, understand, and do useful work on.
If monolithic Unices become the mid-21st-century COBOL, just kept around for a few old systems and only critical bugs patched, that will be a big win for us all.
Linux, perhaps, but "the model"? I'm not so sure.
> It is long past time that we should have moved past it on to better things.
That's what they said about SQL RDBMSes too.
> There have been many attempts but none have ever achieved critical mass.
Maybe for good reason.
> the world needs better tech, and a ½ century old UNIX clone already does not really cut it today.
That's what they... Eh, I'm repeating myself.
> If Linux and all the other monolithic Unixes die when their dev teams die, we'll have to move on to newer, smaller, simpler systems that a new generation of programmers can actually read from top to bottom, understand, and do useful work on.
That's probably what they... No, not quite. But it may have been what Linus thought -- at least, judging from what he made.
> If monolithic Unices become the mid-21st-century COBOL, just kept around for a few old systems and only critical bugs patched, that will be a big win for us all.
I doubt that'll happen. I'll be happy if they become the mid-21st-century SQL -- which I suppose will also be still going strong.
But the alternatives are there.
The fast-headers thing put me in mind of the way that Plan 9 changed the C language, not only formalising indentation and things, but to forbid nested #includes, which apparently vastly reduced compilation times.
Inferno moved on to largely supplant C with Limbo, which is one of the ancestors of Go. I'd say Go, D and Rust all show that there's demand for a newer, better C and that C++ is not it.
Apple's xnu is widely held not to be a true microkernel because of its big in-kernel Unix server, but Minix 3 is... and for all that Minix 3 is not quite there yet, QNX shows this is possible and doable and performant.
It's 2022. We have boxes with terabytes of RAM now and hardware-assisted virtualisation even on £5 ARM boards. We don't need a total clean-sweep replacement; we can virtualise the old stuff and slot in underneath something smaller, simpler and cleaner that mere mortals can understand in less than a lifetime.
3D Xpoint memory is on the market now and makes non-volatile RAM doable, multiple orders of magnitude faster and longer-lasting than flash memory. Computers don't even need drives any more: we could just have a few terabytes of persistent memory, and no more shuffling stuff into memory. True single-level store is doable now. Who needs files when everything is in RAM forever?
ISTM that much of the computer industry has its eyes on the ground now, and is just shuffling round and round in the same old groove, rather than trying new stuff. In the middle of my working life there were dozens of exciting new OSes trying to do things differently. Now, we have 30 million lines of C instead, and need a billion-dollar industry to aim all those many eyes at all those many bugs.
As Molnár himself said: “Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away."