I don't get the complaints either, especially because so many people seem to treat it as an actually serious issue and not just the meme it is. Yes, many other compilers are faster and produce smaller binaries. But it's not that bad and I get incredible value out of all the checks done by rustc. Incremental builds are reasonably quick and produce excellent error messages.
When it comes to efficiency optimizations I see much lower hanging fruits in the web space, some TypeScript projects I'm working on take longer to fully start the dev setup than a fresh optimized Rust build. I hope Bun will make the difference it promises.