Of course part of the trick is that work has been distributed across a cluster, so a lot longer was spent compiling than that...
Your CMake makes incremental builds safe, and the ccache saves you when switching between branches.
Also from what I've seen, large header-only codebases tend to be external and thus generally don't change that much, so if you can limit your own internal dependency tree ccache can still save you.
I've seen tens of thousands of lines of code compile in 30 minutes.
I've seen a hundred thousand lines of code compile in 2 minutes.
It really depends on how the project is laid out. If you're using lots of header-only libraries and try to compile monolithically (like soooooo many C++ applications do these days), you're going to have a bad (compile) time.