Hacker News Reader

Comment by gtmitchell | Hacker News Reader

This is a systemic social problem, not a technology problem. If researchers wanted to double check every single one of their citations for subsequent retractions, they would simply have their graduate students do it.

The root of the problem is that it’s impossible to build a scientific career out of publishing negative results (assuming you can even publish them at all). What actually matters is that your original publication gets into a high tier journal and gets good press. Once that’s done, no one cares if what you published was garbage or not, because by the time anyone figures that out (if they ever do), you’ll have tenure, big fat multi-year grants, and will be out on the speaking circuit.

View on HN

It is absolutely unconscionable that "peer review" process doesn't include "review". Replicating results should be a hard requirement for publishing a conclusion. Anything short is merely a hypothesis with some suggestive evidence, a conjecture. Imagine if mathematicians cited published conjectures but called the proven results.

This is a common refrain, but if this was a requirement science would absolutely grind to a halt. There is just too much science being done. And in some fields, what scientists publish is generally sound, if incremental (and maybe of dubious importance).

I think the tradeoff we have taken is "lots of science, mostly good" vs "little science, 100% correct". Similar tradeoff is taken with software & wikipedia, for example.

Or do you apply the same rigorous standards to, say, software? Should every piece of software that is publicly distributed be thoroughly reviewed for correctness?

Hard disagree, set aside 30% of the budget from the get go by law and use that to let others you have no contact with try to replicate your wirk. I'm willing to bet, that if this were to be tried earnestly 99% of research would be compatible.

You are imagining a solution to a very hard social problem. It is trivially simple on paper, but realistically extremely difficult to implement when you think it through.

The fundamental issue: what is in it for the reviewers in this case? Scientists are people, and you can't really legislate interest in replicating other people's work that is unrelated to your own, unless you really spend money on it.

And there's some bad news on that front: https://www.science.org/content/article/final-u-s-spending-b...

I'm there with you, it's a culture and social issue. That said the argument it's not fun seems quite weak to me. Scientists have to do source citation, formatting, peer review and many other things that aren't intrinsically done for selfish hedonistic reasons.

It probably should grind to a halt because the incentives for “research” are skewed to commerce to such a degree that the structure of the investigatory mechanism is foundationally flawed

Easy to say, but good luck with that.

It's flawed for sure, but there's so much around you that comes from this system if you care to look for it.

Honestly I don't find that process hard because I will not allow myself to live in a state of cognitive dissonance - conflicts in logic MUST be resolved. If there is no epistemic resolution possible, then we have nothing to say about this question with any degree of certainty. We can literally ignore all other information that claims to resolve it because we are not smart enough to actually define the problem.

For example we can make no certain conclusions on the "Hard question of consciousness." As a result there is nothing concrete about the concept of consciousness because it's not even a well defined enough question. Anyone attempting to make a claim are making it in an epistemic vacuum. So it doesn't matter who is talking about it, the state of knowledge of the concept of "consciousness" is so lacking that it doesn't even make sense to discuss.

In my view most of the persistent problems in humanity are in this class: we haven't defined measurements for the issue in question well enough to actually be able to approach a solution.

I always keep in mind that: "I'm probably wrong about my position"

If you are seeking data to confirm your idea then you'll be wrong more often than not

If you are seeking data to falsify your idea then you'll be less wrong more often

Paper is a technology. This systemic social problem around paper has the quirks that it does because paper has limitations (you can't add highlights and margin notes after you've given it to somebody else).

If we stop replicating those limitations, it's not unreasonable to expect that the social problem will change also.

I don't think the answer is search engines in the traditional sense, but we probably do need something that knows how to search. If whatever we use to view research were to display warnings wherever a retracted citation was viewed (or whenever one the citations' citations were retracted...), similarly to how we display SSL certificate expiry warnings in a browser, I think that would create a dampening effect on the momentum that a retracted paper can have.

"Retracted" probably isn't the only color we'd want here. "Replicated" might be another. My point is that summoning up-to-date metadata on a something published last year should just be the default view mode, not something you have to ask a grad student to go do.

Knowledge graphs, not dead trees.

>If we stop replicating those limitations, it's not unreasonable to expect that the social problem will change also.

This is called technological determinism and is not an accurate heuristic for how humanity adopts tools

Humans generally adopt tools broadly after the social environment allows it, not when the tool is capable. This has been shown over and over. Never has there existed a tool whose introduction was immediately and universally adopted.

There’s almost always a period of introduction, then decades of middling adoption and refinement, then the social climate changes and production adjusts enough to adopt en masse.

This was true for every major invention and is an artifact of human social structures

I don't see how these are incompatible views. Maybe it takes a decade, maybe it takes a century, maybe it doesn't happen at all... building the alternative remains the first step. Certainly no adoption is going to happen before the medium exists. Also we wouldn't be publishing articles like this one if there wasn't already some desire to change things.

It also demonstrates that the popularity of a paper is in large part marketing. The original papers are simply better marketed than the corrections.

Reply on news.ycombinator.com