Hacker News Reader

Comment by rfoo | Hacker News Reader

Congratulations on rediscovering how the original researchers run these models!

Do they do it in a way that each GPU can have smaller memory than the model weights?

I keep seeing references to these absolutely massive 48GB+ gpu memory sizes and those are extremely expensive. If more but smaller memory gpus can work that could open up a much larger range of hardware and individuals to run inference, especially if efficient load/unload of weight layers from raid pcie5 nvme devices is feasible.

Yes?

The original LLaMA-65B on fp16 weight is 130G. There are no NVIDIA GPU with 130G memory.

As for why they don't use int8/int4: would you try to optimize your code before you confirmed it can actually run?

Got it, so looking ahead to building hardware that is useful for inference would you agree with these considerations:

- max PCI lanes for more GPUs. AMD cpus and boards probably better

- PCIe 5 nvme. ideally enough to saturate bus

- max GPUs with max memory. memory more important than FLOPs probably.

for a given budget one tries to get as much as possible of above.

How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?

How much are 2x 3090 + the rig to run them? I think that’s enough to run a 4 bit quantized llama 65B

PCIe lanes: it's not that important since the data exchange between host and device during inference is pretty minimal. For model parallel / tensor parallel you want fast interconnect between GPUs, the ages old tech for this is PCIe switch, the fashion ones are NVLink, neither are consumer hardware so don't bother.

PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device.

max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff.

> How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?

$10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.

>How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?

Uh two months ago? LLaMA-13B is GPT 3.5 Turbo quality and runs locally on my Android phone's CPU at acceptable speeds.

You can even run LLaMA-65B (which far surpasses GPT 3.5) faster than GPT 3.5 Turbo with two $200 24GB Nvidia Tesla P40 cards, since in 4bit the model is only 39GB with no output quality loss. Total cost $400 plus some junker used PC with two spare PCIe 4 x16 lanes.

Is that really true! Can LLaMA-65B output code as good quality as GPT 3.5 Turbo?

Are P40s only $200!

If your information is accurate that is incredible and great tip. I’ll start on building such a rig asap.

Reply on news.ycombinator.com