The slowest part is LLVM. (You can pass "-Z time-passes" to rustc to get detailed timing info.)
This isn't just an "LLVM is slow" problem, though. It's also a "rustc generates extremely verbose LLVM IR" problem. Optimizing the IR before it gets to LLVM is part of the plan for solving it.