Hacker News Reader

Comment by latentsea | Hacker News Reader

13 TPS is painfully slow?

20 t/s is about the speed you can read at, so slower than that and it always feels like it's buffering. 35 t/s is where it starts feeling decent and 50 t/s feels nice.

View on HN

I know, it is never enough, more, more, give me more. But the fact is that no one has broken though yet to offer a frontier like open weight over 13 tps in consumer hardware. It would be usable at 13tps for me, especially considering the benefits and advantages of privacy, infinite tokens, availability and no refusals/restrictions. To this day Minimax and GLM are very strong coders but have the same issue, inference speed. We need a real break through, one that thinks out of the box (Tesla turbines style perhaps).

I just use the strongest model I can run at a non-frustrating speed. That's never frontier performance on local hardware, but most of us running locally accepted that a long time ago.

Currently on R9700 + 64GB of RAM I'm running Qwen3.8-Flash-Next. It's decent.

If all you're going to do is chat with an AI, you don't need a 400B parameter MoE model.

13 TPS is unusable for agentic coding, because if decode is that slow, then prefill is absolutely abysmal, and agentic work live and die by prefill speeds.

Why does agentic work live and die by prefill speeds?

It doesn’t. Decode is by far the most dominant and expensive part of agentic tasks and once prefill hits a certain achievable speed decode dominates the time in ant agentic flows with reasoning. There is a reason input is so much cheaper :)

Isn’t input so much cheaper because it’s just less work? Prefill is usually 10-20x as fast as generation.

If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.

No commercial service has 200 t/s prefill though. It is 1000s of pps for prefill. It is largely compute bound and very very fast on modern GPUs and model architectures. That is my point. It is an order of magnitude faster than decode, which is largely bound by memory bandwidth and just has a much harder limit. Prefill tends to look like a small slice in typical compute / token generation budgets. I am not saying it isn't important (I have been working hard on my own inference engine for flash next and am intimately familiar). I am saying it does not dominate. Also compaction and reading the whole prompt tends to be one of the few times you really hit peak pps because it is cached and you can just fill your data pipes for a bit. Also Deepseek showed you can make kv cache really cheap. Decode is still decode.

And what makes decode so memory intensive and seemingly unhackable like the other steps

The math. It’s pretty brutal. You have to multiply insanely large matrices in memory. A dense model has to have and multiply across every parameter basically. MoE models work around it, sure, but you still have to hold a lot in memory and density of activated parameters matters. There is not much of a way around it yet. We are potentially getting there, but its not obvious yet.

And scaling intelligence is seemingly only achievable by making these matrices larger?

What makes an etched LLM into silicon better at big matrix multiplication

Where does quantum fit into this

> If all you're going to do is chat with an AI, you don't need a 400B parameter MoE model.

Why wouldn't I want the smartest possible model I can get? I only "chat" about smart person stuff.

I’ve been agentic coding at home with like 15 TPS on average. I don’t mind it at all. I can give my agent a big task or a list of little ones and just… do something else. Why do I need to watch it work?

Seconding this. 2-bit quant of DSv4Flash 0731 (Unsloth) running on lemonade-llamacpp on a Corsair Strix Halo. OpenCode had an issue giving me 400s constantly for the past few weeks, so I just went all-in on local dev with omp. Yes, it takes time, but also: it works quite well. I've since gotten OpenCode to behave, so I'm back to using that quite a bit, but the local machine is really good as a "smol" model in omp, for example.

It feels even slower when you consider thinking traces and tool calls which are skimmable at best.

Reply on news.ycombinator.com