It is a trade off. However, you can control those trade offs via the optimization switches. You can compare either builds with no optimization vs full optimization.
That said, all the major compilers can deliver adequate compilation speed on the order of 50-100 kloc/sec for well-structured C code bases. Speed is much more of a serious concern if you're working on large C++ code bases. Or if you're using LLVM as a backend from a language like Julia which does the moral equivalent of C++ template instantiation at runtime and feeds it to LLVM.
[1]: https://github.com/UU-ComputerScience/uhc [2]: https://en.wikipedia.org/wiki/Attribute_grammar