However, very few used it for anything, even within Apple. I feel like it was a huge wasted opportunity.
However, very few used it for anything, even within Apple. I feel like it was a huge wasted opportunity.
No, not really. Transformers were just one of the possible directions. Silicon design does not have the same time scale than software. Now, everyone is using transformers so it becomes harder to do anything else, and it’s been the case long enough that hardware had some time to align (but is still lagging). But who’s to say that a different architecture published last year won’t take the world by storm 2 years from now?
It’s easy to say it in hindsight, but transformers took a bit of effort to get where they are now.
"Neural Engine" has been a part of iPhones since 2017. So, in development since at least 2013, possibly earlier. And it targeted the rather well established, known, and widely used ML practices.
GPT-like models didn't become even remotely useful until at least 5 years later.