If you have all your application code in headers you'll get the fastest build times (for full rebuilds at least) by including all headers into a single main.cpp file and build just that, since that way there is no redundant code for the compiler to build at all.

Of course the downside is that every tiny code change triggers a full rebuild then, but it's quite likely that the most time is spent in the linker anyway, so maybe worth a try.