Hacker News Reader

Comment by alisonkisk | Hacker News Reader

Modern science doesn't have peer review!

Peer review works within a community, scientists review each other's work to decide who to trust. But the public and funding agencies are not part of that review! So why to we believe or fund the results? Either we are not their peers and should not engage with them, or we should be reviewing their work as part of the publication process.

We also have "peers" in criminal prosecution. Would you allow a criminal trial where only the scientist's colleagues were in the jury?

View on HN

The problem is that laypeople or bureaucrats can't judge highly specialized technical contributions. It's already difficult to do if your subfield is a different niche.

> It's already difficult to do if your subfield is a different niche.

It's not that difficult, at least in science and science-adjacent fields. The scientific method still follows the same principles.

If you are used to reading papers, you can tell when someone has elided important detail, or not included the evidence for their claims - or, if someone [or a whole subfield] is using technical language to obfuscate their work.

There are many levels of analysis to a paper. Yours is part of it but not the full picture. One other thing is the novelty aspect. To assess that, you need to know the current state of the field. Also the best practices, known methodological pitfalls, judge what is a valid chain of reasoning and what is too speculative etc. It's far from simple and I often see outsiders online criticizing largely irrelevant things while missing core issues. There's no shortcut to expertise in a field. You need first hand experience and knowledge of the state of the art. Otherwise, if papers have to talk to laypeople or generic scientists, we'll see even more smokes and mirrors. The idea of peer review is that a work should be challenged by those best equipped to challenge it.

A journal could follow a set of statistical standards that could be examined and approved by experts outside of its narrow field.

I don't claim that paper reviews have to be or can be perfect - only that they could be held to a better standard than they are now.

We see plenty of avenues for publishing to a worse standard (self-publishing on an internet aggregator, for example). Let's see some avenues for a better standard.

Why isn't this a science in itself? We could trial some different standards and statistically compare them.

[It was not my claim that scientific papers must be understandable by laypeople, although I think this is an ideal case. If not, it should at least be understandable by people in an adjacent subfield - otherwise, it's just a small group of people talking amongst themselves]

> The scientific method still follows the same principles.

“the scientific method” is rather loosely defined and wide.

What passes as “the scientific method” in some fields would not in others.

Reply on news.ycombinator.com