12,826 karma · joined September 1, 2012
Engineering at Known (https://known.com)
Co-founder and tinkerer at Supernotes (https://supernotes.app)
OpenAI is back to using Nvidia chips (with small batch sizes) for the ultrafast mode of their latest models.
That takes the speed down from from 750tok/s on Cerebras to 300tok/s, and presumably is much more expensive than the Cerebras solution, but Cerebras has supply constraints. I believe they are planning aggressive data center expansion in this quarter but much harder to scale up than Nvidia chips.
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
On the other hand, the fact that Astra is "winning" this bench is not great signal for me, since imo Astra produces basically unreadable code. Astra plays code golf. My own revealed preference is that I'd rather have the excessive comments and tests of Claude rather than the inscrutability of Astra.
Accelerating the ability for any given person to have access to some amount of these capabilities is terrifying, is it not?
The value isn't really in the I/O shape, it is in the intelligence combined with the output shape. Every extra ounce of intelligence in these models unlocks additional use-cases. But the converse is also true: a dumb decision model is going to be less useful than using a more intelligent standard LLM.
That is the appeal of Jev: for certain usage it has more intelligence than some small SOTA LLMs. It is the first decision model that actually feels intelligent (to me).
Gemini 2.5 Flash Lite is $500/Gt, Jev is $42/Gt. AKA an order of magnitude cheaper.
> BERT with more data
It is specifically not just that, in the same way that models which have been chat/task-optimized via RLHF (which made these models much more useful for a huge variety of tasks) are not just "the base transformer model with more data".