8,841 karma · joined May 4, 2016
Frontier labs have already been doing this for a while, verified in smuggled traces from OAT/Ant.
Simply a way to reduce tokens.
27 dense is far more capable than 35A3.
Less than half its price.
More than 50% discount.
Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.
But all of this is down to CUDA and/or kernel implementation issues.
When OAI released gpt-oss it was released as an mxfp4 checkpoint.
OAI, Ant, et al are also obviously employing QAT.
It's almost certainly full parameter post training of the original model weights.
I also think the Fable accusation is wrong and it was most likely Opus 4.8 which itself is likely a distillation of Fable.
There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".
It's the third service that multiplies these things that failed. Where are the tests for that?
Say of that what you will, but it's not because they want to wrest control from users.
It's because they don't want Chinese companies to do exactly what Moonshot (Kimi creators) and others have done.
It does not mean it is state-owned.
What? No it's not, and never has been.
Without even getting into the practical vs. theoretical of Fed dual mandate (funding deficits), even the most uncharitable take on modern CBs wouldn't suggest this.
This is necessarily a question that requires thought and speculation about the future.
You can't just flip a switch and do it overnight if it turns out there is demand.
And if there isn't demand, you are going to lose a lot of money.
I don't think it's at all inconceivable where people in the future are streaming high resolution multi modal personal sensor arrays to AI that is running in a data center...and ditto we are streaming ever more content back.
That's just the in vogue answer of 2026. There are undoubtedly endless innovations that I can't envision that will require ever more resources.
My point is just that every single "X ought to be good enough" has been proven false. And the only times where we see consumption really plateau are due to other reasons than desire (like cost).
Hetzner presumably have been making use of this for years already.
It's far more likely they simply began raising prices and realized the market would tolerate it.
But the point was that quality didn't magically increase.
A lot of the money that is deployed by VCs comes from pension funds and asset managers that ultimately manage money for the average Joe.
So let's say a draft model generates 5 tokens, all 5 of these can be verified in parallel with a single forward pass of the target model. The target model may only accept the first 4 tokens (or whatever) but as long as the 5 forward passes of the draft model + 1 prefill of the target model is faster than 4 forward passes of the target, you will have a speedup while maintaining the exact output distribution as the target.
Lol no they didn't. It wasn't even an acquihire. They just hired Peter.
Maybe they are paying him incredibly well, but not a billion dollars well.
Sorry, which laws stipulate these special responsibilities and principles?
These tokens are almost universally used as stop tokens which causes generation to stop and return control to the user.
If you didn't do this, the model would happily continue generating user + assistant pairs w/o any human input.
That is not a reason for snark.
As other commenters have noted, it’s well written.
This isn't true, and certainly not inherently so.
Changes to input leading to changes in output does not violate determinism.
But it's not. It might be discounted cost-wise, however it will still degrade attention and make generation slower/more computationally expensive even if you have a long prefix you can reuse during prefill.