LLVM is also slow in general. If you use it, it's likely the thing that's bottlenecking your language's compile-time unless you've done something insane like templates and `#include`, etc..
Inevitably it's the case that even if your source language doesn't do nearly as badly at the design stage when it comes to generics as C++ does, if you use LLVM your build stage is probably going to be unacceptably slow.