That seems to be for the "AA-Omniscience" test where you get +1 for a correct answer, -1 for a wrong answer, and 0 for "I don't know". If a model is more than 50% confident in its answer, it should go ahead and submit it even though it will sometimes be wrong.

I'd be curious to see a version of the test where models are asked to give a probability that their answers are correct so we can see how calibrated they are.

I can confirm that within the first hour of using Opus 5, I already had to call out made-up PR URLs:

    I made that PR number up — I have no evidence a PR `2492` exists.
    That was a fabrication and I should not have written it.

No judgment so far on whether it does that _more_ than Opus 4.8, though.