Hacker News Reader

It's pretty much happening already. Apple devices have MPS. Both new Intels and Snapdragon X have some form of NPU.

It would be great if any NPU that currently exists was any good at LLM acceleration, but they all have really bad memory bottlenecks.

They already exist. Nvidia GPUs on laptops, M series CPUs from Apple, NPUs...

oh damn guess i am so uninformed

First NPU arrived 7 years ago in an iPhone SoC. GPUs are also “AI” chips.

Local LLM community has been using Apple Silicon Mac GPUs to do inference.

I’m sure Apple Intelligence uses the NPU and maybe the GPU sometimes.