Hacker News Reader

Comment by alfor | Hacker News Reader

What kind of GPU is needed to run the 65B models, can a consumer grade card do the job? Is it worth it? Or do you use cloud instances?

View on HN

You can't fit a 65B even at 4bits (~50GB) on a consumer grade GPU. With an average geek's home desktop computer running 65B inference on a CPU with 64GB of system RAM you could expect about 1 token every 2 seconds.

Is it possible to build systems with multiple GPUs to run the 65B or larger when they appear?

I’m not really sure and looking for clarification from anyone who knows. My understanding is it is possible to split the layers between the GPUs so a system with 4 high end consumer GPUs might work well.

I tried that with RTX 4090 as the primary card and 3090 as eGPU over Thunderbolt. It works, but the inference is very slow, presumably because it has to pump all that data back and forth between the two (and Thunderbolt isn't fast enough to keep up even with 3090 by itself in games). In fact, even running 30B across two GPUs in 8-bit mode like that was slower than running it on one GPU in 4-bit.

My takeaway is that if you actually want to use multiple GPUs, you need hardware that's designed to accommodate that, and most consumer-grade stuff, even high-end, is not built with two GPUs that physically large in mind.

Looking at the top end H100 80GB systems with NVLink from HPC vendors and it occurred to me we are about to swing back to massive almost mainframe like form-factor systems, giant bus, like old expandable qbus in the 80s but this time for GPUs.

What I mean is they have systems with 8x cards but given the compute requirements of these huge LLMs probably systems with 32+ all on a dedicated memory bus (NVLink) are what will be needed as weights sizes expand. This is all for inference btw, not even training, but same hold for training probably best possible interconnect between same monster systems.

I’m dreaming there might be a distributed eventually consistent partial training algorithm then that would democratize creation of these models.

In regards to smaller scale individual systems for inference, if one has resources and is fairly technical and can utilize such technology then perhaps in 5-10 years the wealthy might buy units for $50K+ that get installed in their home or something.

Really incredible developments very quickly. Apologies for the potentially inappropriately long rant to the previous comment.

The other possibility is that we'll get cards that are very specifically designed just for the LLMs, basically ditching everything that is not strictly necessary for the sake of squeezing more compute / VRAM, and perhaps optimizing around int4/int8 (the latter is apparently "good enough" for training?).

I’ve got 2 rtx 4090s on an EATX motherboard. Been using them to run the full 13B un-quantized with a good deal of success. Getting about 20 tokens/s.

What is your setup for cooling? I don't think I'd want to stick another 4090-size card in mine with just air cooling...

I have AIO liquid cooling for both cards. The radiators are annoying though, I might convert it to a custom loop if I ever add a third card.

Thanks, so to make usable you would need a GPU with at least 50GB. That probably mean A100 with 80GB at around 10K$.

Ok I understand why people use CPU and main memory.

After a quick check up you can rent a A100/80G at 1$-2$/h.

It's possible to run with a cpu and then use system memory.

Do you know what kind of speed you can expect with inference on CPU?

Yes, around 2 tokens per second on a newer CPU with fast RAM.

Reply on news.ycombinator.com