First mistake to avoid; writing it in C.
For a POSIX-compatible operating system, more exotic choices are probably not available.
If you use a stable language, you can concentrate on the challenges of the actual OS design.
Was C ready when Ritchie & Thompson wrote Unix with it?
> For a POSIX-compatible operating system, more exotic choices are probably not available
Why limit to POSIX compatibility?
No need to limit things to POSIX compatibility, but merely achieving it is no mean feat — and POSIX is seriously C-oriented. Some people invent a new OS architecture, then implement POSIX on the top of it (see NT) — but this is an order of magnitude harder, I suppose.
When you implement a POSIX-compliant OS, you suddenly have a lot of ready-made open-source software that both serves its primary purpose and as an "integration test suite". For a new OS, you have to write most basic userland things yourself. This is again a lot of effort.
Given its open nature, there's no reason (except limits of time/skill) someone writing an OS in Rust couldn't also be advancing Rust and its libraries in the directions needed to support that effort, even if they are outside of Mozilla.
I won't mind to be a billionaire with superhuman abilities and unlimited lifespan, too! Unfortunately, we mere mortals often have to limit the scope of our endeavors somehow.
POSIX? We don't need no steenking POSIX...
Modula-2 and of course Oberon the language were rather fine for their times, and I'm glad to see some pieces of them reused in e.g. Go.