What makes you think it is slower because of the borrow checker?
On the other hand, that extra time is traded off against not having to deal with bugs at runtime that the type system is able to catch. Rust is one of the better languages when it comes to making illegal states unrepresentable.
But he does specifically break down how much of compilation time is spent in the borrow checker, LLVM, macro expansion, etc.