Hacker News Reader

Comment by dmckinno | Hacker News Reader

Lots of chatter about how this model is benchmaxxed, but didn’t observe this in our testing.

Very strong open model release that is a bit off the frontier but totally amazing for most use-cases.

https://x.com/danielmckinn0n/status/2115213822957117535?a=fO...

Hypothesis for why K3.1 dropped so much in relative position to other models when AA switched to private evals? Primarily FrontierCode, CritPt V2, and the new private version of AA Omniscience where the model massively underperforms relative to its performance on public benchmarks?

I should have been a bit more precise. The model is obviously benchmaxxed compared to the western closed models, but it is not compared to the OSS competitors.

I think the AA rescoring reflects that as well. Kimi is still in the mix with the tail end of the best models and far ahead of DS, GLM, etc.

And the new AA index is obviously wonky in its own way. There is no way Haiku is outperforming Grok 4.7.

I think chatter is putting it mildly. There's plenty of evidence in this work [1] that is hard to explain. At the very least it's a failed attempt at benchmaxxing the model, but more likely is straight up training on the test set (and the attempt at hiding it failed). Something leaked pretty heavily in their setup.

And there's indications of it happily replying with fake encrypted data in oAI format here [2]. While this is a bit less conclusive than [1], it's still dubious.

[1] - https://r40.org/papers/kimi_k31_benchmark_audit.pdf

[2] - https://x.com/PeterNutter1/status/2118084568419045704

https://xnotme.com/PeterNutter1/status/2118084568419045704

I should have been a bit more precise. The model is obviously benchmaxxed compared to the western closed models, but it is not compared to the OSS competitors. I think the AA rescoring reflects that as well. Kimi is still in the mix with the tail end of the best models and far ahead of DS, GLM, etc. And the new AA index is obviously wonky in its own way. There is no way Haiku is outperforming Grok 4.7.

Reply on news.ycombinator.com