For comparison, it's also taking ~3min @ 50 iterations on my 12c Threadripper using OpenVino. It sounds like the improvements bring the M1 performance roughly in line with a GTX 1080.
So sounds plausible that the m1 can reach the same level in some use cases with the right optimizations.