Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage.

I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o).

4o was the worst. verbose and waste of my time.

Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues

o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now.

Gemini made a suggestion which almost got it right but it was not a full solution.

I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.