To be honest it does though. You could take 10 M1 chips (40+40 cores, with around 30TFLOPS of GPU) put them into a server and even at full load you would be at 150W, which is about half of the high core count Xeons. Obviously not as simple as that, but the thermal fundamentals are right.
The 40 core Xeon also costs around 10k.
There's rumors that the new iMac will have a 20 core M1 (16+4). I imagine that will be faster than even the top line $10k Xeon.
I have absolutely no doubt apple could put together a server based on the M1 which would wipe the floor with Intel if they wanted to. But I very much doubt they will since it is so far out of their core competencies these days.
I have absolutely no doubt apple could produce a ridiculously good server CPU from the M1. I doubt they will actually do it though.