There is a pervasive issue in that most code written today is tacitly designed for the assumptions of microarchitectures from a decade ago, and most code is even older and is rarely rewritten.

Generally speaking, compilers have a limited ability to improve the basic design of code or deviate from the assumptions about hardware embedded in that code. More often than not, taking advantage of different microarchitectures is not just about the code generation, it is also about algorithm design and selection, which falls directly on the programmer. I currently design software that assumes the Haswell microarchitecture. While the code is written in C++ and the high level architecture is generally the same, the lower level idioms are different because they reflect the strength of the microarchitecture. If the code was for the latest ARM core, it would need to be quite different to be efficient. And I am not even talking about things like AVX2. Idiom changes to match microarchitectures are not "micro-optimizations" in the traditional sense; it will often buy 2x performance improvement overall versus well designed generic code tacitly optimized for another microarchitecture.

The reality is that almost no one writes code for e.g. the Haswell microarchitecture. The idea of writing code for a microarchitecture at the level of, say, C++ does not even enter most programmers' minds. Broadly speaking, a compiler will not allow you to take advantage of advancements in microarchitecture except by accident, which works sometimes but is not efficient in terms of return on the new microarchitecture. Compilers learn some micro-idioms, but most of the macro-idioms escape them (like designing algorithms for the ALU parallelism of a particular microarchitecture).

The last big microarchitecture change for Intel was Nehalem, the original "i7". A lot of code optimization assumptions from prior CPUs went out the window with that microarchitecture. The latest, Haswell, added more usable ALU parallelism and the BMI2 extensions, which are quite useful if you know how to exploit them but nothing Earth-shattering. AVX2 was nice, but too limited to have much value for most normal code.

That said, I am pretty excited about AVX-512, which will be a major microarchitecture extension once that it is available. The giant caveat is that no one will be able to really take advantage of it without, once again, redesigning their code for that microarchitecture. A compiler won't be able to do that for you, which has been the real stumbling block for the adoption of advanced features in new microarchitectures. What constitutes an optimal algorithm or data structure is microarchitecture dependent.