But it rarely matters how much work you put into something to make it faster. What matters is what you can avoid to do at all.
I have pretty much zero knowledge of CPU-level issues and barely know enough x86 assembler to read my compiler's output. I have never cared for optimizations like inlining code, vectorization or tail call elimination. I have tried inlining a few times, and the speed-ups it brought were negligible. (And the slow-downs that too much of inlining brings, by increasing code size and thereby increasing cache pressure, must be very subtle and basically unmeasurable, while being potentially enormous. And the disadvantages from a maintainability standpoint are considerable as well).
If you have to care about these things, then maybe you're doing too much work. (Well, I guess I would use vectorization/SIMD if I had to write a video codec, but I've never done that).