> why do those things have a (statistically significant?) impact in the first place?

In a word, caches. Not just the instruction / data cache, but also page faults and micro-architectural features like micro-op caches, instruction TLB entries, loop stream buffers, cache-line alignment, and aliasing in the branch predictor tables (which can also be thought of as caches).