LLVM's thin LTO should at least not make your build times (much) longer. Have you tried that?
For our (Rust) project, thin LTO doesn't cost much extra in compile time and gives a few percent speedup. But it's not a terribly optimized code base, and we don't have any manual inline annotations etc.