Hacker News Reader

I don’t know. You can run models that are in the ballpark of this on $1500 of hardware (e.g. strata on 12gb vram + 64gb ram), so I feel like unless you’re training stuff you either run smaller local models, rent your hardware, or pay for tokens.

View on HN

Kimi 3.1 is a 2.8T parameter model. You can't run it on a 12GB VRAM GPU. You need 1.4TB of VRAM just to store the parameters.

Not really. Kimi has 104B active parameters, so afaik that means you’d only need ~64gb vram to run an 8bit quantization of this, and potentially as little as 32gb to run a 4bit quantization. The rest, the MoE experts, can be in system ram… or if you’re willing to take the speed hit, offloaded to ssd. But yes you’d need a lot of system ram… like 1.5TB.

But that’s all kinda moot because I was referring to smaller open models, ones you could easily run on $1500 of hardware.

We run it on 96gb Vram

https://argonautlabs.ai

Reply on news.ycombinator.com