Hacker News Reader

why the stats of every AI on every release looks around the same?

Are the tests getting harder and harder so the older AIs look worst and the new ones look like they are "almost there" ?

View on HN

Yes, once benchmarks get saturated they get replaced by harder ones. You don’t see GSM8K, MMLU, or HellaSwag anymore because they’re essentially solved. It takes constant work to make benchmarks hard enough to show meaningful model performance differences but easy enough to score higher than the noise threshold.

Reply on news.ycombinator.com