> You can't distinguish between a machine that says "here, look at these 170 results, 10% of them are highly serious problems that you should address

The machine doesn't say that. It says "Here are 170 completely correct and verified results".

You have to check and verify all of those results yourself, and on any given day it can be anywhere from 0% to 100% incorrect.

> I assume you've come to this conclusion based on some reasoning, but you're not sharing it in this response AFAICT.

The reasoning comes from actually working with AI tools. And the reasoning can be seen in the actual comment this tgread started from: https://news.ycombinator.com/item?id=48434824