Great, we can control for layout of code, heap, stack and other effects which mess with a performance measurement. However, why do those things have a (statistically significant?) impact in the first place? I guess that hints that with some engineering you could in principle get a speed boost by specifying the layout. "Worst case", you sample randomly and then pick the fastest arrangement, where it is statistically significant.

It could be that the problem arises when trying to measure very small speed increases (small relative differences => noise matters more). But in that case the fact that such a small speed increase is wiped out by random layout effects surely means that time would be better invested in finding a more performant layout?