To piggy back on this comment: what build system are you using? I've found that bazel has greatly reduced compile times, even when using boost and opencv.
¹ I used it in a couple projects and never had a single incorrect build. With most other build tools, be it CMake or Gradle, these happen regularly for me.