The M1 has pretty high memory latency at around 100 ns [1], which is significantly higher than either AMD or Intel for typical systems. Note that physical distance between CPU and memory is rather less important for latency, as DRAM is high latency in itself, so adding a few ns at most due to wiring is not going to matter.
[1] https://www.anandtech.com/show/16252/mac-mini-apple-m1-teste...