Meta got caught _first_.
https://techcrunch.com/2025/01/19/ai-benchmarking-organizati...
Is anything here actually false or do you not like the conclusions that people may draw from it?
From a primary source:
"OpenAI did not use FrontierMath data to guide the development of o1 or o3, at all.... we only downloaded FrontierMath for our evals long after the training data was frozen, and only looked at o3 FrontierMath results after the final announcement checkpoint was already picked."
> If an MMLU question asks about an abstract algebra proof, is it cheating to have trained on papers about abstract algebra?
This kind of disingenuous bullshit is exactly why people call you cheaters.
> Generally, I don’t think anyone here is cheating and I think we’re relatively diligent with our evals.
You guys should follow Apple's cult guidelines: never stand out. Think different
"OpenAI did not use FrontierMath data to guide the development of o1 or o3, at all.... we only downloaded FrontierMath for our evals long after the training data was frozen, and only looked at o3 FrontierMath results after the final announcement checkpoint was already picked."
From a primary source:
"OpenAI did not use FrontierMath data to guide the development of o1 or o3, at all.... we only downloaded FrontierMath for our evals long after the training data was frozen, and only looked at o3 FrontierMath results after the final announcement checkpoint was already picked."
These ML benchmarks were never going to last very long. There is far too much pressure to game them, even unintentionally.
According to the article, Meta publicly stated, right below the benchmark comparison, that the version of Llama on LMArena was the experimental chat version:
> According to Meta’s own materials, it deployed an “experimental chat version” of Maverick to LMArena that was specifically “optimized for conversationality”
The AI benchmark in question, LMArena, compares Llama 4 experimental to closed models like ChatGPT 4o latest, and Llama performs better (https://lmarena.ai/?leaderboard).