Hacker News Reader

If they don't care, why are them presenting them when launching a new model, every single time?

Because people, for some stupid reasons, still read and make decisions by these public benchmarks, which makes zero sense. Create your own benchmarks and evaluations, then use those, it's pretty the only way to get any qualitative answers to how well a model will work for you and your user cases.

How many users are willing to spend the time and repeated cost (once for each new potential model+version+variant) for that - 1 in hundreds? (Edit: I was thinking about this as an individual dev. Especially for AI uses in a biz workflow, custom benchmarking makes perfect sense yes.)

Probably the same amount of users who actually care about figuring out how well a model works for their use cases, instead of what amounts of guessing.

Most companies I've consulted for when it comes to LLM usage in various ways all have their own evaluations, not sure why others would avoid them.

This is to save time, for what it's worth. Doing that testing and evaluation will take you MUCH longer than doing so automatically, for obvious reasons.

Reply on news.ycombinator.com