Hacker News Reader

Comment by DebtDeflation | Hacker News Reader

A 34B model is probably about the largest you can run on a consumer GPU with 24GB VRAM. 70B will require A100's or a cloud host. 13B models are everywhere already. I'm sure this was a very deliberate choice - let people play with the 13B model locally to whet their appetite and then they can pay to run the 70B model on Azure.

View on HN

I'm running a 30B model on an amd 5600x cpu at 2-3 tokens/s, which is just under a "read-aloud" pace. I'd wager that you can run a 70B model at about the same speed with a 7900x and a bit more RAM.

Or a $5000 128GB Mac Studio, that you can get for 1/2 the price of a 40GB A100 or 1/7 the price of a 80GB H100.

70B should work reasonably well on 64GB CPU RAM + any decent GPU, or maybe a 24GB GPU + 32GB CPU RAM.

do you mind teaching how to do CPU/GPU RAM math? all i know is 34B 16bit = 68GB total RAM needed (because 1B of 8bytes = 1GB definitionally), but i dont know how it splits between CPU/GPU and whether the tradeoff in tok/s is acceptable

People running LLMs on CPU are generally running them integer quantized, so they use fewer bits per parameter.

If you are doing ~4 bit quantization, a good rule of thumb is just under 1 Gigabyte per 1B parameters, plus a little room for the operating system. Longer contexts require a bit more VRAM.

For reference, 4 bit LlamaV1 33B fits snugly on a 24GB GPU with 2K context with the exLLaMA backend. But it won't do really long inputs.

Llama.cpp is pretty much the only backend that can offload to CPU efficiently. Its still quite fast and offers very flexible 3-5 bit quantization, with the leanest 3 bit quant just barely fitting LlamaV1 33b on my 6GB + 16GB laptop.

Reply on news.ycombinator.com