also feels like a good posttraining task
129 karma · joined April 24, 2025
(in a VAE-ish way, kl div on the embeddings?)
We are not the only ones to see the reasoning inversion.: https://arxiv.org/abs/2510.11977, https://arxiv.org/abs/2502.08235, https://arxiv.org/abs/2507.14417
Btw, this also helps manage scale. Eg you have 15 diffs to review. Run a few verifiers to get a short list, then review directly and apply the best.
(Which is why my prior is that third party harnesses would not perform as well. But I haven't actually measured this.)
We are measuring sth much closer to: when multiple agents compete on the same spec, which one produces the patch that holds up best in code review?
Most evals are static / synthetic, and for code, generally stop at tests. Test evals are weak proxies for quality since it's difficult to encode qualities like scope creep/churn, codebase fit, maintainability etc in tests. [1]
Almost every agent in a given run can pass tests at this point, but there is large separation during review.
So far we have been native harnessmaxxing, which simplifies things a lot.
The configuration space around open models is much larger. Eg which models, capability heterogeneity, which harness, networking, data egress / privacy, etc.
If anyone is getting very good production code out of open models, I'd love to do a user interview to better understand your setup. Email is in my bio.
If the changes needed are small, I'll apply the best implementation as a foundation and then just iterate directly.
If the changes needed are drastic, it usually signals that there was sth wrong/ambiguous/etc in the spec (or the ensemble was too weak, which is rarely the case). In cases like this, I improve the spec and then rerun.
If it's in the middle, I'll usually apply the best and write a follow on spec.
Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric).
We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels, 5.4 still edges it out on cost/perf.
We take a spec-driven approach and mostly work in TS (on product development), so if you use a more steer-y approach, or work in a different domain, YMMV.
Version control and isolation will probably stay useful, though, more for distributed development and workflow reasons than for safety.
(Note, that stops being true at higher reasoning levels, where our observed total cost goes up ~2-3x.)
To the PM behind this - developers are sensitive to this kind of thing. Just make it opt-in instead?
but i think the prior on 'this team fabricated these findings' is v low
they are definitely useful but they miss the things that are hard to encode in tests, like spec/intent alignment, scope creep, adherence to codebase patterns, team preferences (risk tolerance, etc)
and those factors are really important. which means that test-evals should be relied upon more as weak/directional priors than as definitive measures of real-world usefulness
This post measures `1x top-N` (one attempt each from N models), not `Nx top-1` (N attempts from the best-scoring model). We should make that more clear.
Part of why we chose `1x top-N` is that we expect lower error correlation compared to `Nx top-1`, which is also why the iid baseline is likely optimistic.
That said, a direct comparison (`Nx top-1` vs `1x top-N`, with the same review/compute budget) would be useful!
But that is a good point. Perhaps it should be mapped to something unidentifiable.
For comparison, there's a `review` command that launches a sandboxed agent to review a given run and rank the various implementations. We usually run 1–3 review agents, pull the top 3 diffs, and do manual review from there.
We're working on better automation for this step right now.
The question is which multi-agent architecture, hierarchical or competitive, yields the best results under some task/time/cost constraints.
In general, our sense is that competitive is better when you want breadth and uncorrelated solutions. Or when the failure modes across agents are unknown (which is always, right now, but may not be true forever).
We run big ensembles because we are doing a lot of analysis over the system etc
We still code via interactive sessions with single agents when the stakes are lower (simple things, one off scripts, etc). But for more important stuff, we generally want the highest quality solution possible.
We also use this framework for brainstorming and planning. E.g. sometimes we ask them to write design docs, then compare and contrast. Or intentionally under-specify a task, see what the agents do, and use that to refine the spec before launching the real run.
Each ran the same spec headlessly in their native harness (one shot).
Results:
Agent Cycles Time
─────────────────────────────────────────────
gpt-5-2 2,124 16m
claude-opus-4-5-20251101 4,973 1h 2m
gpt-5-1-codex-max-xhigh 5,402 34m
gpt-5-codex 5,486 7m
gpt-5-1-codex 12,453 8m
gpt-5-2-codex 12,905 6m
gpt-5-1-codex-mini 17,480 7m
claude-sonnet-4-5-20250929 21,054 10m
claude-haiku-4-5-20251001 147,734 9m
gemini-3-pro-preview 147,734 3m
gpt-5-2-codex-xhigh 147,734 25m
gpt-5-2-xhigh 147,734 34m
In this case, rebasing is nice because our changes stay in a contiguous block at the top (vs merging which would interleave them), so it's easy for me and others to see exactly where our fork diverges.