Hey thanks for sharing your work! Just wondering how many of the 1396 came from Mo, and what happens to a bug that Mo can't reproduce?
- The candidate ground truth bugs are generated using runs from all agents
- Every candidate gets reviewed by the other agents to generate a recommendation (just makes human review easier)
- A human reviewer makes the final call (no candidates are discarded just because Mo couldn't reproduce)
And in terms of grading the final runs:
- A lot of the final runs still found a small percentage of novel bugs (5-10%), because it's extremely difficult to build a comprehensive list of all bugs, even after hundreds of runs.
- Nothing that didn't match was auto-discarded (read - it wasn't rejected because Mo couldn't repro).
- All unmatched findings were run through the same human review process as above.
- All the runs were regraded after human review to get a final score with zero unmatched findings.
One interesting thing we did notice during this is that Codex and Claude tend to converge on the same findings run over run (very few initially unmatched findings on final runs), whereas Mo tends to find a lot more novel bugs on ever run.