This will perform inference on Phi-2 or Mistral on almost any iPhone device. The first neural engine came with the A11 bionic chip which is in the iPhone 8. People can buy that phone for $150 used on Amazon.
I think that's evidence that inference on the edge is already here
> "Small" requires iPhone 14 and 15 Pros
> "Small" and "Medium" models are based on Mistral 7B 0.2
So no, also that app wouldn't work on an iPhone 8 either since it requires iOS 17 or later
For small models, today a used iphone 14 costs ~$300. Which means next year it will by down to ~$200.
I'm pretty sure by 2026, we will be able to inference "medium" models like Mistral/Llama3 8B on a <$199 device. That's not even factoring in how much better models will get by that time