Openrouter on nitro gets me there. Although the TPS is 5X less still.
I have tried GPT OSS 120b on cerebras and it flies. But it's mostly wrong.
It's extremely frustrating that these fast models are very old at this point. Chat jimmy launched in february, and nothing has come of it, not even GA let alone improvements with intelligence.
AI is currently moving so fast, that might as well be a lifetime.
We get SOTA from anthropic/openai every few weeks on models that seem to focus on one useful thing (e.g. opus) and then 5 useless things like booking restuarants. We're ignoring really useful applications for some reason.
It took jev to shake things up. I remember seeing reactions that were basically "people want this? the big labs probably have some version of this lying around"... and it's like yea, maybe these labs need to fix their priorities a little bit.
Jev is super cool, and it’s super cool that it can handle novel categorization. Most organizations aren’t spending the big bucks on novel classification, though. They’re doing the same thing a billion times, which means Jev has to compete with having a custom model built and doing your own inference.
It’s very hard to build something generic like Jev, but it’s fairly manageable to build something more specific that handles a single kind of task. Especially at the scale where OpenAI would care about your attention.
I don’t really understand neural networks and I managed to train one to do hand gesture recognition on an old Leap Motion. It’s a pretty limited domain, but it trains on the CPU of my decade old laptop in under 3 minutes. The accuracy is quite good too, especially when you discount errors from the Leap (like misidentifying which finger is extended).
1. inference on arbitrary context / world knowledge with fast, structured output
2. I almost don't want to say it because I don't know if people realize this yet... but that makes traces from any kind of non-AI process or system "learnable", you can easily model it by by just asking it to predict the next outcome during learning and compare against what happened.
which also means it could also learn new strategies through rollouts in a way that generalizes across semantic space which means...
so these models are actually quite promisingly useful for existing companies because they do have general traces of what happened in their company and want to discover ways they can tweak their business processes, and then there is durable demand for models you can embed within a product or UX flow without exposing your user to slop + then train or customize and build real IP or custom capabilities
general classification itself is quite interesting IMO for applications that need to make judgment calls or select one of many UX actions, and a much better fit for this than LLMs. So if you think about it, this is the type of thing businesses might be calling a lot - every trace (eg user session, package shipment, sales motion, marketing campaign, job outreach) + proactively throughout dynamic experiences to decide whether to eg end a call or which of multiple UIs to manipulate
this is just an unfamiliar product experience to most people because it hasn't really been bootstrappable with a strong general baseline until now. But now it's posible and I think it will be very col.
I’m talking about them training their own models with a much more specific purpose.
You brought up choosing a menu option in a UI. Jev is really good at something like “find the menu option that best fits the users prompt”. That’s general, and requires a big complicated model.
Companies are usually more interested in something less variable. Ie the menu option they’re looking for is fixed. “Take an arbitrary website and find the Login button”. That’s a _way_ simpler model, and well within the abilities of many large companies to train.
You only need to pay Jev if there are so many variables the model basically has to be general.
Eg classifiers are a separate thing from LLMs and are literally only capable of structured output. It will output one of the X options it was trained to recognize.
I think some of this is that the Venn diagram of people who like AI and the people who like machine learning isn’t a circle. There’s a lot of “intelligences” out there that aren’t an LLM or decision model.
Cerebras if you're reading this, get your act together - maybe that will also help your stock price. Give consumers more options, not just to OpenAI.
https://www.cerebras.ai/press-release/cerebras-systems-annou... https://uk.finance.yahoo.com/quote/CBRS/
Thanks for your contribution to the discussion.
The fact that Cerebras, which has a willing and major customer like OpenAI begging at their heels to deploy on their hardware, now posts about a 2.8T open weight model says a lot.
[0] https://blogs.nvidia.com/blog/gpus-openai-gpt-6-astra-ultraf...
For some reason though on openrouter the tps of 6.1 sol ultrafast is usually lower than opus 5.5 fast. Is there readings off as surely cerebras hardware is faster than low user density nvidia still?
Or, from a different perspective: who is using Cerebras for what purpose?
DeepSeek V4 Flash 0731 is dirt cheap at scale, less than 10% of the cost of Qwen3.8 27B. This is because it's an MoE and it uses much less VRAM for KV cache. And it's generally a better model. V4.1 is also cheap at scale, though I've heard mixed reports on the optimized K/V caching.
You should not be running Qwen3.8 27B on an API normally. Maybe Cerberas's hardware is bad at batching or supporting more efficient K/V caching schemes, or something like that?