Hacker News Reader

> the worst thing you can do in QA is miss a real bug and ship broken code to prod

Maybe it varies by domain but I'm quite used to codebases where "fix every discovered bug" is basically infeasible; finding 1000 miniscule bugs isn't helpful. Finding _serious_ bugs is the goal.

View on HN

That's fair, and I think also something we've seen while creating this benchmark: it's incredibly difficult, if not impossible, to annotate every single bug in an app. This is actually sort of what we were trying to capture with severity-weighted score: the agent should get way more credit if it finds really bad bugs than if it finds a bunch of minor UI quirks, although this could definitely be improved upon when we release a v2 of the benchmark.

Reply on news.ycombinator.com