Hacker News Reader

I wonder what AA's funding and affiliations are. I get the sense that they have their hand on the scale at least a bit.

Well yes, thats the entire business model of benchmarking companies. If you pay attention to AA, some models are on the leaderboard second they're released while others come days later, which to me implies that theres some level of coordination happening with them and a few labs.

https://fixupx.com/aarnogau/status/2076059071246193012

That kind of coordination doesn't bother me, personally. That's likely just early release/access to reviewers, which isn't inherently problematic.

The part that looks weird to me is some models get tested all up and down their effort level, but this one seems to just have been tested at the level where it will come out worst (I'd expect). Maybe it'll get more coverage over time?

I've also noticed the numbers (and relative rankings) change right after some model releases, which felt a little like someone didn't like who came out on top so they tweaked it.

Reply on news.ycombinator.com