What I've interpreted from the consensus is that Apple's M-series advantage primarily comes from the relatively large memory bandwidth due to it's unified memory architecture.

Of course, it also has a very low performance-per-watt, comparatively speaking.