Hacker News Reader

Comment by azinman2 | Hacker News Reader

We saw this before. Anyone know what the VRAM requirements are, if it’s been quantized, and if/when llama.cpp might support?

View on HN

From the paper[1]:

> We additionally scale our inference code to support million-length sequences by implementing RingAttention for decoding. Inference for such long sequences requires a minimum of v4-128 with a TPU mesh sharding of 32 tensor parallelism, and 4 sequence parallelism (ring dimension).

Each TPU-v4 has 32 GiB of HBM memory, so about 4 TiB (128 x 32GiB) of memory in fp16, without quantization.

[1]: https://arxiv.org/abs/2402.08268

The model itself is only 13.6GB though: https://huggingface.co/LargeWorldModel/LWM-Chat-1M-Jax/tree/...

Indeed. Almost all of the inference memory overhead comes from attention matrices and the KV cache.

So uh, they thought "we have this datacenter full of GPUs laying around, you know what would be best, if we made a model that needs the entire thing just for inference".

Brute forcing a quadratic complexity problem seems like wastefulness at its worst.

aren't most LLMs just a brute force probability engine?

Sounds like they're loading the internet into those vrams.

Probably never gonna run on normal boxes, it uses RingAttention which only makes sense when you have many GPUs. It's a datacenter-only model.

I’d still like to know the requirements. You can rent A100’s or have access thru work, etc.

> Scaling Inference We additionally scale our inference code to support million-length sequences by implementing RingAttention for decoding. Inference for such long sequences requires a minimum of v4-128 with a TPU mesh sharding of 32 tensor parallelism, and 4 sequence parallelism (ring dimension). We perform inference in pure single precision, where additional improvements can be made through techniques in scalability such as quantization.

It might even fit in a max-sized macbook pro.

Maybe in few years with 4bit quantisation. (around 1TiB is needed for that)

From https://arxiv.org/pdf/2402.08268.pdf

> We trained our models using TPUv4-1024, which is approximately equivalent to 450 A100s

> Inference for such long sequences requires a minimum of v4-128

So you'll need ~60 A100 for inference.

There are models with shorter attention sizes that probably are much smaller in vram needs. The model itself is only about 32g, so in 5K-M quant it wouldn’t be that bad, and the smaller attention versions might actually be workable in higher end workstations with cleverness like llama.cpp. Back of the napkin guess tells me maybe around 60g for the 32k context version

Reply on news.ycombinator.com