Hacker News Reader

Hacker News

Comment by vorticalbox | Hacker News Reader

Full thread

vorticalbox

Seems there is 7b, 13b and 70b models

https://huggingface.co/meta-llama

View on HN

"We have also trained 34B variants, which we report on in this paper but are not releasing."

"We are delaying the release of the 34B model due to a lack of time to sufficiently red team."

From the Llama 2 paper

if you red team the 13b and the 70b and they pass, what is the danger of 34B being significantly more dangerous?

edit: turns out I should RTFP. there was a ~2x spike in safety violations for 34B https://twitter.com/yacineMTB/status/1681358362057883680?s=2...

A 34B model is probably about the largest you can run on a consumer GPU with 24GB VRAM. 70B will require A100's or a cloud host. 13B models are everywhere already. I'm sure this was a very deliberate choice - let people play with the 13B model locally to whet their appetite and then they can pay to run the 70B model on Azure.

I'm running a 30B model on an amd 5600x cpu at 2-3 tokens/s, which is just under a "read-aloud" pace. I'd wager that you can run a 70B model at about the same speed with a 7900x and a bit more RAM.

Or a $5000 128GB Mac Studio, that you can get for 1/2 the price of a 40GB A100 or 1/7 the price of a 80GB H100.

70B should work reasonably well on 64GB CPU RAM + any decent GPU, or maybe a 24GB GPU + 32GB CPU RAM.

do you mind teaching how to do CPU/GPU RAM math? all i know is 34B 16bit = 68GB total RAM needed (because 1B of 8bytes = 1GB definitionally), but i dont know how it splits between CPU/GPU and whether the tradeoff in tok/s is acceptable

People running LLMs on CPU are generally running them integer quantized, so they use fewer bits per parameter.

If you are doing ~4 bit quantization, a good rule of thumb is just under 1 Gigabyte per 1B parameters, plus a little room for the operating system. Longer contexts require a bit more VRAM.

For reference, 4 bit LlamaV1 33B fits snugly on a 24GB GPU with 2K context with the exLLaMA backend. But it won't do really long inputs.

Llama.cpp is pretty much the only backend that can offload to CPU efficiently. Its still quite fast and offers very flexible 3-5 bit quantization, with the leanest 3 bit quant just barely fitting LlamaV1 33b on my 6GB + 16GB laptop.

Reply on news.ycombinator.com