I suspected that as well. Since M1 CPU and GPU shares memory, this also means you really can't do anything else while mining. You can't even scroll a webpage smoothly.