I recently found an article by Apple explaining how they optimized a transformer to work on Apple Neural Engine [1]. This was before their recent hardware refresh with 192 GB unified memory upgrade.
Apple could have a real chance to push their boxes as an alternative to dedicated GPUs.
1: https://machinelearning.apple.com/research/neural-engine-tra...