Shouldn't people expect compile times to be slower at -O3? Presumably the improvements are better optimizations, many of which trade compile time for execution time. Now if it's getting slower at -O0, that might be a concern.
In this particular case though, no it does not seem it matters, as the slowdown is related to C++ headers in the newer standard lib, not to the low-level compilation.
"One of the most common mistakes made by new language frontend projects is to use the existing -O2 or -O3 pass pipelines as is. These pass pipelines make a good starting point for an optimizing compiler for any language, but they have been carefully tuned for C and C++, not your target language. You will almost certainly need to use a custom pass order to achieve optimal performance."