Rust could have been designed for faster compilation without sacrificing runtime speed, safety, good abstractions, expressiveness and whatnot in any regard. The people designing Rust just either did not care or did not have the experience to do so. And it can not be fixed anymore because that would require too much breaking changes.

And yes, a substantial time is also spent for things like the borrow checker but no all of it.

An no, it is not a niche requirement. Short compile times are absolutely critical for developer productivity. One main reason Golang exists and got popular is that people got fed up with how slow C++ is to compile. Not to mention that most people on this earth are not as privileged as to have a beefy machine.