Expect Google models to be benchmaxxed as much as possible, but don't expect commensurate performance on real-world tasks.
Expect Google models to be benchmaxxed as much as possible, but don't expect commensurate performance on real-world tasks.
This is not true. It is needless negative hyperbole that IMHO doesn't belong on HN.
If what you say is true, then the next set of benchmarks will be written to illuminate the true reflection of real-world tasks. Then those get benchmaxxed so managers can show "numbers go up."
Repeat enough times, and you get materially and genuinely improved products.
The burden is on the press, outside researchers, and general tech industry to both write and care about the metrics that matter.
In large companies (Fortune 1000 etc.), you don't have much of a choice, the culture was eroded long ago, and corporate governance by metrics is a lesser of evils. But thankfully, not all companies are there yet.