That's somewhat true, but much of the unpredictability could be removed if compilers provided annotations saying "I expect this loop to be vectorized" where the compiler would be forced to report an error if it didn't manage to do it.
So... these annotations don't actually exist? ;)
The whole point of such a feature is that the compiler will fail if it can't do what you want.
#pragma clang loop vectorize(enable) warning: loop not vectorized: failed explicitly specified loop vectorizationThis isn’t a hypothetical, it happens in real life. Your only option at that point is roll back compilers and hope someone cares about the regression enough to fix it.
The point is the better model is to build the semantics into the language rather than relying on the whims of implementation.
Of course the semantic guarantees will likely be somewhat weak because of differences in ISAs, memory hierarchies, etc.
Linux has good support for performance counters, Windows requires a bit of work.
So in this case, the compiler update would show a regression in the test suite, which needs to be addressed as part of the compiler upgrade.
Of course, having this in the compiler would remove the need for unit tests.