The main question is did they allocate enough space and power budget to the accelerator. Or is it anemic even for basic tasks.
There is a reason, and that reason is called memory bandwidth.
ITS basically this month the And accelerators are supported by llm
For example, currently most GPUs have a max of 16Gb of NVRAM, but usually 8GB or 12GB (except for more expensive workstation cards and similar [1]). But if you can spec a system with 32GB, 64GB, 96GB, or 128GB or more system memory with sufficient bandwidth to the CPU, then the accelerator can run a broader range of bigger models locally.
Afaik Apple M2 Pro and M3 Pro/Max systems are the only ones with that capability right now.
[1]:https://www.nvidia.com/en-us/design-visualization/desktop-gr...
Should Nvidia be afraid of Apple? Is OpenAI going to ditch their A100 for Macbook Airs?
Depends on what you think about how technologies become successful.
Some people would say the foundation of nvidia's current dominance is their mass-market gaming GPUs. While every other High Performance Computing technology wanted you to call their enterprise sales team and pay a big markup, any academic or open source developer could get their hands on an nvidia GPU - and that lead to a huge academic and open source ecosystem.
In recent years, nvidia has decided if you want more than 24GB of vram, you've got to call their enterprise sales team and pay a big markup. And a lot of people working on LLMs want more than 24GB of vram.
On the other hand, nvidia do have a lot of momentum. And of course cloud options are available these days.
https://www.amd.com/en/products/accelerators/instinct/mi300....
https://www.techradar.com/computing/cpu/intel-launches-core-...
There is also some tech that lets you put some model layers on the Nvidia GPU’s local NVRAM and the rest in system RAM. It’s slower though. Some discussion of that here:
https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_mo...
https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are...
https://www.notebookcheck.net/Intel-Meteor-Lake-Analysis-Cor...