If that's the case, an optimized build with clang is also often faster than a debug build with... clang itself.

The reason is that many of the optimization passes that run first, like dead code elimination, can remove a lot of code early on, so "optimized" builds end up processing significantly less code, which is inherently faster.

The OP might just not be aware of what a "debug build" is. The goal of a debug build is for the binary to execute your code as closely to how you wrote it as possible, so that you can easily debug it.

Their goal isn't "fast compile-times". If you want fast compile-times, try using -O1. At that level, both clang and gcc do optimizations that are known to be cheap and remove a lot of code, which speeds up compile-times significantly. Another trick to speed-up compile-times is to use -g0, and if you do not need exceptions, use -fno-exceptions, since those make the front-end emit much less data, which results in less data having to be processed by the backends.