Guess what: because I said "both", Opus decided I was looking to pick out exactly two of the issues. If I hadn't read that reasoning log, I would've remained oblivious and thought there were only two it could find.
That seems like a misinterpretation of English that I'd expect from a much less capable model. Like, a complete lack of understanding of grammar. To have that meaning, you would have to have put "both" before "issues" instead.
In this specific case, does it matter if you get the raw reasoning vs a "slop summary"? I bet both would tell you that Opus locked into two issues no matter how bad the slop summary is.
This is like writing `if (cond_x && cond_y) {}` and then blaming the compiler because it didn't realise that actually meant to write `if (cond_x || cond_y) {}`.
But I don't think their prompt as written would have limited the responses to two, unless there really were only two matching both conditionals. I've not seen an LLM misinterpret language that way before.
0. "find issues that are both critical and related to X"
...could be read as "issues that are X and are Y":
1. [valid, intended] find [all] issues that are[, at the same time,] 1) critical; and 2) related to X
...or as "issues, both of which issues are X and are Y":
2. [valid, your reading] find [an to-be-specified number of] issues that are, both [so I want two issues], possessing X and Y attributes.