So ... why bother to cheat?
So ... why bother to cheat?
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"
For specific benchmarks, I’ve downloaded and used basically every model out there under 35b parameters - especially the tiny models: ling-3.0-tiny, spark-x2.6-4b, nanbeige, etc. etc. some of those tiny models are ranked VERY high on benchmarks like deepswe or terminalbench but are obviously incapable when you try to use them for swe or terminal based tasks.
If that’s benchmaxxing or a bad benchmark, idk, but it’s made me not trust the benchmarks much. If it benchmarks poorly, it’s probably bad. If it benchmarks well, it might be good.