It’s not because of the borrow checker specifically, but the type system design that makes the borrow checker possible is a big part of what leads to slower compilation times.
On the other hand, that extra time is traded off against not having to deal with bugs at runtime that the type system is able to catch. Rust is one of the better languages when it comes to making illegal states unrepresentable.