It’s just tricky because writing a new rust compiler is a monster amount of work because of how complex Rust is. And rearchitecting the existing compiler would also be a massive undertaking because there’s so much existing code, and it’s spread over rustc (written in rust) and llvm (in C++).
A faster compiler probably wouldn’t bother with all of llvm’s intermediate representations. Refactoring across a code boundary that spans two languages (and two projects) sounds utterly exhausting.
Rust is the first language in its category. It won't be the best language we ever make in that category.
For me the biggest hurdle to adopting Rust is that rust-analyser turns my laptop's fans up to 11. I don't know whose problem that is, but it doesn't happen with other languages, and my black box impression of rust's compilation infrastructure is that it's fucked up.