Hacker News Reader

Comment by andreilys | Hacker News Reader

Reproducibility in social psychology is less than 50% [1].

So yes I do think the majority of psych researchers are making stuff up (through p-hacking and other nefarious means) in order to get funding and tenure

1. https://replicationindex.com/2018/11/20/how-replicable-is-ps...

View on HN

A question the 50% figure raises is - what is the reproducibility rate for “real” science? This is difficult to estimate. Considering the dramatic improvements in cancer therapy, one might argue that cancer research is a “real” science, but there are reproducibility issues in that field as well. [https://www.nature.com/news/cancer-reproducibility-project-r...] And the link to the replication index site connects us to an article about John Iaonnidis, who suggests that most medical research results are wrong. (The article concludes that perhaps only 27% are false discoveries.) So what is the threshold? As others have pointed out, it’s probably more about process than reproducibility, though reproducible results are certainly preferred.

I think that would depend on how much intersection there is between the sort of scientists doing non-replicable research (mostly in academia, it seems) and the sorts of scientists developing working cancer therapies (mostly in pharma?).

I don't know if arguments about "real" science are especially useful. The nature of science has changed a lot of over time and will presumably keep changing. It takes us into questions of whether fields that can't do experiments are "real" science, for example. It also assumes replicability is the only way science can't be "real" but there are some fields (e.g. COVID research!) where papers are usually replicable but that is meaningless because the methodology is unsound in other ways, i.e. you'd just be replicating their mistakes.

The biggest problems driving non-replicable research are the fact that so much "science" is now really just data analysis, and finding some spurious correlation that just about nudges over some arbitrary and very weak threshold for significance can be considered "science", even without anyone doing any kind of double check or followup. No real world experiment needs to be done at any point. And in some fields the connection to reality has been completely lost - public health is filled with papers that make predictions about the future trends in public data, and nobody ever follows up with papers that compute the correlation between the predictions and measured values. It would be easy, but they generally don't do it.

There are several high visibility papers on lack of reproducibility in drug development by Pharma, so I don't think there is a particular bias in less reproducible research. Indeed, since life sciences research in academia is almost all funded by the NIH, with grant reviews/renewals every 3-5 years, there is probably more scrutiny of academic research than commercial.

> Reproducibility in social psychology is less than 50%.

That doesn't seem particularly terrible in comparison to scientific reproducibility in other fields. What am I missing? https://en.wikipedia.org/wiki/Replication_crisis

People usually underestimate how difficult it is to do experiments, especially when you want to get quantitatively consistent results. Physics experiments sound easier than human experiments, but if you really do it and be serious with the numbers, you will find even high school physics experiments are really hard. For example, try to confirm F=ma, or measure local gravity to the third digit.

You may be assigning more intent than is necessary to explain the (very real) reproducibility problem in psych.

Not to belabor this but to note the irony here: there is probably a similar psychological (ehem) phenomenon that happens to a psyc grad student who lets their presumptions bleed into their interpretation of a measurement.

> The first principle is that you must not fool yourself — and you are the easiest person to fool.

R. Feynman

And they are basically the only field funding replication studies. That makes them more rigorous than many alternatives.

We do have replicable, quantifiable, social psychology. Facebook and Cambridge Analytica are infamous for that. Unfortunately whenever experiments are carried out, you get end up with an uproar over issues of privacy, accountability, and informed consent.

Spying and manipulation are not necessarily reproducible hypotheses. In case of FB and CA, external conditions changed.

They could be fooling themselves too, not knowingly spreading falsehoods. A lot of of what we call p-hacking can happen by accident by somebody who isn't very good at statistics or just isn't being careful.

50% is reproducibility of experimental studies. There are fields don't even do experiment. There are fields don't even run statistics. There are fields don't even use logic.....

Reply on news.ycombinator.com