Hacker News Reader

Huh? What are you talking about?

https://www.anthropic.com/engineering/infrastructure-noise

Is anthropic benchmark maxxing and cheating on terminal bench too? They don't follow the strict resource "limits" either

View on HN

What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it

Yes but my point is - Resource limits are a "recommendation" and are not strictly enforced - Significantly boosting resources up to 3 did not statistically shift performance results

Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results

Reply on news.ycombinator.com