While this may be true, their throughput is practically unbounded. this has been a primary challenge to using local models at work, where its normal for me to have 1-6 sessions churning away at once. So while 100t/s on a single session is very fast, theres no way you could get a cumulative ~500t/s from a consumer setup like this, do the bandwitdth bottleneck on a single card/cpu/ram/ssd (tends to be gpu ram bandwidth is the limiter in my setup, but they all have respective limiters to this kind of throughput)
I’ve had to use almost all resets that OpenAI allotted me. But I agree. I can get probably 1000t/s overall from them if I’m using subagents a lot.
I think Opus 5.5 is now fast enough with the inference stack improvements that they did. Sonnet 5.5 is around 140 tokens/s when I had measured it last time. A $20 subs on claude and using only sonnet 5.5 will last a lot
I was firmly wanting to move towards local llm, and since the Sonnet 5.5 and recent Opus speed increases, I'm putting that on hold until local inference speeds can be improved. Not sure how much juice there is to squeeze there, but I'm hopeful.
Does it matter if those 100 tokens per second are way crappier?