* Inference becomes cheap
- speciality accelerators hit the market and race to the bottom begins
* Training remains expensive
- This works out for Anthropic/OpenAI, they go into the business of training
* Models become rental units or purchasable assets, you run on inference hardware
- Rent or own inference hardware
* Or you pay someone to do all of the above for you, at a premium
Groq (acqui-hired by NVidia) came up with a different processor architecture: metric shit-tons of SRAM attached to a modest single core deterministic processor. No HBM needed on this card, and 32x faster inference than today's best GPUs at inference!
These LPUs are pretty useless for training though, which is useful for companies training models! Training is expensive, inference is cheap (someday, not now).
There's also a Canadian company that _literally burned the model as a silicon mask_ on a chip. It's unbelievably (1000x) fast, but not flexible of course: https://chatjimmy.ai
HBM are their own chips and dies.