Hacker News Reader

Comment by Luker88 | Hacker News Reader

Currently I am running llama-cpp with `Qwen3.8-Flash-Next-UD-IQ3_XXS` on an old ryzen 8845HS with 96G of ram (and no dedicated graphics card) at 7tk/s and ~60tk/s filling, max ~120K context window.

Surprisingly useful as long as you can leave it running a couple of hours at the very least.

While huge models will still be better I think the general availability of RAM might be the downfall of AI companies.

View on HN

Remember that a hosted AI company only needs ~1000 bytes per context token per user (ie. 100mb per user for typical coding - VRAM during inference, and moved to regular RAM or SSD whilst running a tool call)

Everything else (weights) are shared amongst tens of thousands of users currently doing inference in that cluster, so even if there are terabytes of weights for the model, they aren't much on a per-user basis.

> ~1000 bytes per context token per user

Where's that figure comning from? Last time I checked (could be the 3.6 Qwen 27b) single token needed 32kb

The smallest I have seen is DeepSeek 4.1 flash at 890 bytes per token.

I suspect that commercial labs have unpublished work lowering this further. It's the obvious thing to optimise for when RAM is so expensive and you have a lot of users.

Highly recommend the RCO-GSQ quant by ITSA btw. At IQ3_XSS it is within one point of the fully unquantized model.

will try, thank you for the pointer!

Reply on news.ycombinator.com