For example, think of LLVM. GCC has a lot of functionality, and implements a lot of important, difficult algorithms. But because GCC is in userspace, LLVM/Clang have been able to get better and better until they can realistically challenge GCC in this space. People are starting to use LLVM more and more; it offers really compelling new tools like ASAN (http://clang.llvm.org/docs/AddressSanitizer.html), Clang Static Analyzer (http://clang-analyzer.llvm.org/), and LLDB. LLVM is also offering an embeddable and modular architecture, which GCC has opposed on philosophical grounds for years.
Because this is in user-space, LLVM was able to form and grow alongside GCC. People didn't have to make a big switch just to try out LLVM; it's not intrusive like it would be to switch from Linux to FreeBSD or anything like that. That's why I think the network effects of user-space are better. In the kernel it's "get merged or die." In user-space, similar projects can compete and people can vote with their linker lines.
There are tons of examples of this. In the last 10 years we've seen a lot of displacement in userspace, where next-gen technologies have made a lot of inroads against the incumbents:
- Apache has been challenged by nginx
- Perl has been challenged by Python/Ruby
- screen has been challenged by tmux
None of these upstart competitors had to ask permission or get buy-in from the incumbents, they just did it. Now compare this with when Con Kolivas had an idea in 2007 for how to improve the Linux scheduler. He was really passionate about this and was maintaining his own patch against the Linux kernel -- he called his scheduler the Staircase Deadline scheduler (http://lwn.net/Articles/224865/). He showed positive results and had a lot of fans of his patches. But then Ingo Molnar, the established Linux scheduler guy, took Con's ideas and created his own separate rewrite of the scheduler that he called the "Completely Fair Scheduler." Linus picked Ingo's scheduler instead of Con's, which left Con frustrated and he stopped working on Linux.We'll never get to find out what might have happened if Con could have realistically offered his work as an alternative. The scheduler is in the kernel, which means that Linus picks winners and losers. In userspace, no one has that power.