If anyone needs to see what 62 tokens/second looks like: https://token.qusai.pro/?speed=62
Which seems sufficient for mortals who are not running gas town, but want a local model.
Which seems sufficient for mortals who are not running gas town, but want a local model.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
But because prefill is parallelizable, datacenters will have an edge here for a while. For your one request, a datacenter may push 1.7kW power to a single machine to give you a result in a few seconds. The rest of the time machine will stay busy with other people’s traffic so it’s not a problem. For you, unless you’re a billionaire, you won’t have 1.7kW of processing power running idle 99.9% of time.
In 1-3 years we’ll have datacenters switch to dedicated chips that are way more energy efficient, and such chips will ultimately trickle down to PCs.
BTW - Apple is already introducing LLM specific architecture. They introduced Neural Accelerator (nit to be confused with Engine) that speeds up their GPUs 4x for LLM specific tasks. M5Ultra gets 100 tflops vs nvidia Rtx6000 getting 500tflops. „Just” 5x faster, but it’s beginning to look relevant.
Still 25 slower than a server farm B200. And when you ask Chat/Claude you get the full speed of B200.
What I’m waiting for is consumer dedicated LLM boards, a similar way we had soundcards and first gen 3d accels. No GPU, no 3D, purely LLM compute. I can see that happen in 2-3 years.
https://stealthgoat.com/deepswe-flash-next.html (Total AI pile). I benched this quant for 3 days on DeepSWE in a 16GB VRAM gaming PC. It held 90 t/s on avg I think. It’s solid. The authors claims of minimal intelligence loss hold. Sorry it renders bad in mobile will fix it soon.
It feels great to have a local model ripping at these speeds. I now use Flash NeXT as an active agent managing part of the engine build process and let it do some coding on easier chores :)