Change a basic thing in a header of a core library, or even add a flag to the compiler, and you need to recompile everything.

most of the builds are incremental, but even incremental builds still take awhile when it's that massive (linking, modifying a header / template code, etc)

True, but linker times still suck and don't parallelize well.

Also, sometimes you need to iterate on some very core .h file and touching any of those brings the whole house of cards down and triggers a full or nearly full rebuild.