The first obvious answer (also mentioned in other comments) is because "Systems Software Research is Irrelevant". See Rob Pike's paper by that name: http://herpolhode.com/rob/utah2000.pdf
To the reasons Pike offers there, I'll add:
1. The companies that owned the software had poor strategies for encouraging widespread adoption, if that was even ever their goal.
2. The first dot-com boom ca 2000 greatly expanded the IT / programmer workforce and (I claim) significantly dumbed it down. The advantages of a new operating system lack economic impact until a huge number of programmers are trained to use those advantages. We have huge sunk costs invested in maintaining a huge supply of mostly weakly skilled programmers and admins, a sort unlikely to adapt to and adopt a new OS.
3. In a related way, the sectors that could in principle drive something like Plan 9 adoption are heavily invested, by now, in massive amounts of bloatware that, dysfunctional as it is, is both critical and non-portable.
4. Modern hardware is fast enough that fairly high level programming languages and environments tend to dominate. These often include a "least common denominator" view OS capabilities so that programs port easily among Windows, Linux, Unix, MacOS, etc. This hurts demand for OS features other than the "least common denominator".
-----------------
Why it might get better:
Notice that none of the reasons listed above apply to a market niche like "the OS for Google's clusters" or, say, massive clusters providing an SQL-based RDMS.
On clusters like that:
1) You don't need a mass audience of buyers for a new OS. One or a few big customers will be plenty.
2) You don't care about hordes of cheap, weakly skilled programmers. Paying experts is peanuts compared to your hardware, power, and real estate costs.
3) You're not tied to bloatware. You need only run a few things very, very well.
4) You don't need to do "least common denominator" programming and, in fact, any new OS feature that can save you some $s per server-hour is potentially a huge win.
My betting money is that Pike et al. will produce YANOS (yet another new OS), quite possibly mostly written in the Go programming language, really well suited for huge compute clusters.