851 karma · joined March 24, 2022
The largest chatgpt models are maybe 1-1.5tb in size and all of that needs to load into pooled vram. That sounds daunting, but a company like open ai has countless machines that have enough of these datacenter grade gpus with gobs of vram pooled together to run their big models.
Inference is also pretty cheap, especially when a model can comfortably fit in a pool of vram. Its not that the pool of gpus spool up each time someone sends a request, but whats more likely is that there’s a queue to f requests from someone like chatgpts 700 million users, and the multiple (I have no idea how many) pools of vram keep the models in their memory to chew through that nearly perpetual queue of requests.
If they approach things this way, and transistor progress continues linearly (relative to the last few years) maybe they can make their first devices that can meet these goals in… 2-3 years?
I’m not familiar with the facts of the case and IANAL, and its late, but how did the plaintiffs determine their books were being used for training of the llm? Was the model spitting out language that was similar or verbatim to their works?
Did I miss that the task was time sensitive?
This is assuming you don’t have one of the newer i devices that has a usb c connector, then…
I’m obviously not an expert.
Heck I’ve been able to work through the early chapters of the FastAI book using a lowly Quadro p1000
I imagine companies will have first dibs via the likes of agreements with suppliers like CDW, etc, but if Intel had enough of these battlemage dies accumulated, it could also drastically change the local ai enthusiast/hobbyist landscape; for starters this could drive down the price of workstation cards that are ideal for inference, at the very least. I’m cautiously excited.
On the AMD front (really, a sort of open compute front), Vulkan Kompute is picking up steam and it would be really cool to have a standard that mostly(?) ships with Linux, and older ports available for Freebsd, so that we can actually run free as in freedom inference locally.