It feels pretty weird that your ratings have:
gpt-5-4-high > gpt-5-4-xhigh
gpt-5-4-high > gpt-5-5-high
gpt-5-4 > gpt-5-5
gpt-5-2-high > gpt-5-2-xhigh
No other ratings I've seen show that.
gpt-5-4-high > gpt-5-4-xhigh
gpt-5-4-high > gpt-5-5-high
gpt-5-4 > gpt-5-5
gpt-5-2-high > gpt-5-2-xhigh
No other ratings I've seen show that.
We are measuring sth much closer to: when multiple agents compete on the same spec, which one produces the patch that holds up best in code review?
Most evals are static / synthetic, and for code, generally stop at tests. Test evals are weak proxies for quality since it's difficult to encode qualities like scope creep/churn, codebase fit, maintainability etc in tests. [1]
Almost every agent in a given run can pass tests at this point, but there is large separation during review.
We are not the only ones to see the reasoning inversion.: https://arxiv.org/abs/2510.11977, https://arxiv.org/abs/2502.08235, https://arxiv.org/abs/2507.14417