Idk if it would out of the box, but it should be possible. I know that Whisper (and some variants) run on both x86 and silicon macs.
It should run with PyTorch but the performance might not be great. As a long-time Mac user myself I would love if someone would send a PR to port it to MLX :)