Our collective "deal with the devil" is that, with all the various caches and predictors, ordinary code runs much faster than it has any business running, but it is impossible to be certain, up front, of the effects of any particular choice. Doing something clever is as likely to make code slower as it is to make it faster.
Still, it is true that practice improves intuition, and a habit of measuring helps in learning to avoid spending extra time to actually make code worse. But microarchitectures have become so complicated and so poorly documented that it amounts to a guessing game to discover why a change had the effect it did.
Compiler optimizers are always a few steps behind, often doing something that would once have been clever, but is now foolish. For example, it was once smart to intersperse arithmetic operations between branch instructions to fill pipeline holes, but those can now prevent micro-op fusion and make an innermost loop take two cycles where it could take one, resulting in a 2x penalty overall. Telling Gcc to optimize for Zen2 seems to result in much slower code, on Zen2, if the code had been tuned before.
I have recently got a 2x improvement in Quicksort, applied to random input, which on its face should have been impossible for such a mature algorithm. We have learned to avoid pointer chasing, and special-casing, but now a branch over extra work may take much longer than the work skipped, if it is predicted badly.
With the end of Dennard Scaling, just waiting doesn't make our programs faster anymore. As Moore's law, which still operates, lets microarchitecture complexity grow without bound, the improvement possible given extra attention increases, but the amount of skill needed to extract it grows too. As it grows, the number of people who can do it falls faster, and skepticism that any particular individual has it grows, particularly around languages that offer less direct control. As it gets harder, the range of applications that merit such extra work and attention narrows. Often, now, in places where better performance is most needed, no one on hand understands how to get it, or even knows it is available.