> What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking.

Well, that's example is also introducing dependence, which is a tricky thing of course whenever we talk about chance and stats.

But there's also another issue - a statement like "5% of positive published results are by chance since we have a p<=0.05 standard" treats every set of results as if p=0.05, whereas some of them are considerably lower anyway. Though the point of bad actors cherry-picking to screw up the data also comes into play here.

(And of course, fully independent things in life are much harder to find than one might think at first.)