Most modern computers have caches. On the whole they speed you up but some access patterns lead caches to slow you down. Some instruction set architectures have more registers than others.

So the fastest language also has to consider the hardware it is used on.