Hacker News Reader

Comment by siliconc0w | Hacker News Reader

What's everyone's favorite LLM leaderboard? Gemini 2 seems to be edging out 4o on chatbot arena(https://lmarena.ai/?leaderboard)

View on HN

Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage.

I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o).

4o was the worst. verbose and waste of my time.

Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues

o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now.

Gemini made a suggestion which almost got it right but it was not a full solution.

I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.

Sonnet 3.5 as of today is superior to Opus, curious if sonnet could have solved your problem

Yes. I find it a bit funny how much people care about leaderboards. I see models going up and down, winning this or that benchmark and yet, for me, Sonnet 3.5 still beats the crap out of all of them.

I too was puzzled by the response from Claude. I am using the Anthropic workbench with claude-3-5-sonnet-20241022 (latest)

But it think it has to do more with the freshness of training data.

AWS IPV6 Egress is a new technology from AWS which was introduced only recently. Previously, we had to deploy NAT gateway which supported IPV4. I am assuming claude-3-5-sonnet-20241022 (latest) was not trained on this data.

Notably, GPT-4o is a "full size" model, whereas Gemini 2 Flash is the small and efficient variant in that family as far as I understand it.

True that - and I think Gemini-Exp-1206 is Gemini 2.0 Pro in testing. I noticed how they only replaced the "experimental" moniker for one of their experimental models, and it turned into 2.0 Flash. And that still experimental model is currently leading across all categories.

https://aider.chat/docs/leaderboards/

https://livebench.ai/#/

I like that https://artificialanalysis.ai/leaderboards/models describes both quality and speed (tokens/s and first chunk s). Not sure how accurate it is; anyone know? Speed and variance of it in particular seems difficult to pin down because providers obviously vary it with load to control their costs.

AI benchmarks and leaderboards are complete nonsense though.

Find something you like, use it, be ready to look again in a month or two.

With the accelerating progress, the "be ready to look again" is becoming a full time job that we need to be able to delegate in some way, and I haven't found anything better than benchmarks, leaderboards and reviews.

EDIT: Typo

FWIW I've found the 'coding' 'category' of the leaderboard to be reasonably accurate. Claude was the best, o1-mini then was typically stronger, now the Gemini Exp 1206 is at the top.

I find myself just paying a la carte via the API rather than paying the $20/mo so I can switch between the models.

poe.com has a decent model where you buy credits and spend them talking to any LLM which makes it nice to swap between them even during the same conversation instead of paying for multiple subscriptions.

Though gpt-4o could say "David Mayer" on poe.com but not on chat.openai.com which makes me wonder if they sometimes cheat and sneak in different models.

Reply on news.ycombinator.com