Hacker News Reader

I don't want to be too negative, but ... all this for a 0.8% improvement in SWE-bench Verified (90.2 for OpenCode vs 91)? And why is this (saturated) benchmark the one coding benchmark chosen to showcase on the homepage?

Without trying it, this seems like its probably just a massive waste of tokens.

View on HN

this might be a dumb question, but if it's saturated, then isn't 0.8% improvement a very good improvement?

I'm definitely willing to be corrected, but I think it just means that this improvement is close to pure noise -- not good or bad necessarily. Just not a good signal.

To me the really bad sign is that this is the benchmark that they selected to highlight. Why not one of the less-saturated benchmarks where this harness could (in theory) show meaningful improvement over the OpenCode baseline? Seems fishy to me.

Reply on news.ycombinator.com