I too was puzzled by the response from Claude. I am using the Anthropic workbench with claude-3-5-sonnet-20241022 (latest)

But it think it has to do more with the freshness of training data.

AWS IPV6 Egress is a new technology from AWS which was introduced only recently. Previously, we had to deploy NAT gateway which supported IPV4. I am assuming claude-3-5-sonnet-20241022 (latest) was not trained on this data.

Yes. I find it a bit funny how much people care about leaderboards. I see models going up and down, winning this or that benchmark and yet, for me, Sonnet 3.5 still beats the crap out of all of them.