I literally tried that few days a go, what you're saying is not true.

Compiling Linux:

AMD 6800HS, ~4mins.

Apple M1 Pro, Linux VM, ~10 mins.