Hacker News Reader

I ran the Strata project with this on 32gb ram 16gb AMD 7900XTX. It goes about 20tok/s off the ssd. I'm guessing 64gb would go to 40tok/s. https://github.com/Niko1221/Strata Quality wise its not fast enough for me to eval. I'll probably spend some tokens on it through openrouter. DeepSeek 4 flash though is a mess compared to GPT Sol 6.1.

Thanks (and everyone else) for the reply. I will stick with llama.cpp running llama-server Qwen3-Coder-30B-A3B-Instruct-NVFP4 for now.

Fwiw I think you can run 3.8 flash (next?) on your setup with SSD streaming.

There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).

1 - https://youtu.be/IH8XmxiwliQ?si=lJPvYzi3mHNsiw8r

32 gb of ram is gonna be your blocker for iq3, u will have to go iq2 and thus the output will suffer. At the plus side, it will spew faster than iq3.

Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.