Landmark study on the replicability of psychological science overturned [pdf] (2016)
sciencedaily.com
sciencedaily.com
> As an example, Gilbert described an original study that involved showing White students at Stanford University a video of four other Stanford students discussing admissions policies at their university. Three of the discussants were White and one was Black. During the discussion, one of the White students made offensive comments about affirmative action, and the researchers found that the observers looked significantly longer at the Black student when they believed he could hear the others’ comments than when he could not.
> So how did they do the replication? With students at the University of Amsterdam!” Gilbert said. “They had Dutch students watch a video of Stanford students, speaking in English, about affirmative action policies at a university more than 5000 miles away.” In other words, unlike the participants in the original study, participants in the replication study watched students at a foreign university speaking in a foreign language about an issue of no relevance to them!
I’ve long said it’s much more useful to think of the replication crisis as a generalization crisis, and this is a good case in point. The replication (and failure to replicate in Amsterdam) doesn’t tell us the original study was wrong, but tells us that something in the context was actually part of the subject all along. In this case US/Dutch differences in current social hot topics, or in historical slavery, or maybe it only replicates for students who you believe you might run into in class. In any case, per usual, the solution is more science and more thoughtful study, not throwing in the towel and declaring the sciences conquered by “the replication crisis.”
But many disciplines, in this case psychology, did have the replication crisis coming. Cohen and Gigerenzer have warned about power for literally decades but their warnings were not heeded by a significant part of the field.
So a stark message may in fact have been necessary.
By the way, Daniel Kahneman basically wrote a "this is a problem, deal with it NOW" letter to some colleagues in 2012 [1]. Very telling and an insightful read.
* Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants.
* Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate.
* The board makes a public commitment to these five specific effects
* Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc.
* Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed).
* Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work
* Run enough subjects to guarantee power (probably more than in the original study)
* Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others.
* Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others.
I hope that psychology is a field where it is possible to fund such a protocol from grants or from internal school funds. However, that protocol seems to me to be laborious, expensive, and not at all "easy to get".
The point of the OP is that there isn't a replication crisis. Should we be assuming, for purposes of this discussion, that there is one?
That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science, like bankruptcy is an essential part of business. The reason the OSC replication failure paper was important was that it discussed widely accepted research.
> The point of the post is that the size of the replication crisis may have been exaggerated in this study.
The OP is much stronger than that. For example,
First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true.
and
“What they did,” added Gilbert, “is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. ...”
and
“The replicators realized that doing this study [using people speaking English about US social issues] in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. ...”
Also, mandatory pre-registrations provide some evidence that effects are exaggerated in other fields as well, see e.g. [1] for treatments for cardiovascular diseases.
There is a crisis on many fronts, but if you prefer we can call it "professionalization, finally".
[1] https://journals.plos.org/plosone/article?id=10.1371/journal...
If that's true, could it be a matter of funding? Social sciences receive much less, afaik. I remember going into an experimental psychology building in a major university that you've certainly heard of, though it was years ago. It was an old K-12 school, scarsely remodeled, and files were stuffed in lockers.
Right, but the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. I mentioned those particular studies because they made people sit up and go "hey, there's a bigger problem here". The OSC paper came out of this situation, it didn't start it. Other symptoms include that spoof paper where listening to newer music "made participants younger". Andrew Gelman's blog. Etc. Etc.
Is there something serious - non-partisan, sophisticated, informed - that covers this?
Gelman's blog mostly argues by dismissive tone, sadly (though people can't seem to get enough of that on the Internet!).
Absolutely and I'm sure Gelman's scholarly work is taken seriously. But as Gelman says:
Brilliance represents an upper bound on the quality of your reasoning, but there is no lower bound. The most brilliant scientist in the world can take really dumb stances. ... The best driver in the world can still decide to turn the steering wheel and crash into a tree. [0]
I don't think the blog is at the upper bound. I think many - including many smart, very human people - get taken in by path of (unfortunately) socially normative Internet rhetoric - it's easy to follow (requires conscious effort to avoid) and socially rewarding - which is heavy on contempt and outrage as the primary forces, as if they indicated truth or value, and not evidence and reason.
[0] https://statmodeling.stat.columbia.edu/2022/06/08/how-can-a-...
And the years since then haven't been kind to landmark results in psychology.
I've seen those criticisms repeated many times, but IME each time it lacks evidence. We know that is arguably the primary form of misinformation (and disinformation) - repeating false or unfounded assertions until people assume they are true - so the fact that it is accepted by some or repeated or assumed doesn't make it true. (In fact, science was created to address that exact problem.)
EDIT:
> This paper drew heavy criticism at the time: http://retractionwatch.com ...
The link is to a response by the authors of the original paper. They certainly should be heard, but they are not representative of heavy criticism in the field.
I'm sorry what? Does the author think that there are no black people in Amsterdam? Or no affirmative action? Or that Dutch students don't understand English? Race relations aren't the same in the US as in the Netherlands, but the "no relevance to them" comment is going a little far imho.
The whole point of the original study presumably being to study unconscious bias due to cultural conditioning.
A better study might have tried to replicate it using Dutch students in the video and a distinctly Romani student as the object of discrimination.
I mean... the Dutch had slaves. I'm sorry, I don't know how else to spell this out. The Dutch were slavers, just like the English. There are generations of black people living in the Netherlands for the same reason they are in the United States. Many of the slaves that came to America were brought on Dutch ships. They did outlaw it earlier, I think. But there are tons of racial issues in NL. I know I'm getting downvoted but I find this attitude that racism is somehow unique to the US to be pretty narrow to be honest.
It would have been different if they had shipped in enormous numbers of slaves to work the Polders and then had to find something for them to do once they were emancipated. This is why the US has so many lingering issues, when the slaves were freed the former masters realized very quickly that they would lose power should they organize, and thus had to immediately set up numerous systems of oppression to prevent this from happening. I'm guessing the former slaves were a fairly tiny minority of the population and thus never a danger to the ruling class so there was no need to actively oppress them.
No, but if I'm going to reproduce your recipe for muffins and announce whether it works or not, I gotta follow your recipe.
If I swap the chocolate chips for blueberries and the muscovado sugar for maple syrup and the muffins come out wrong, I can hardly claim I've proven your recipe is wrong.
I haven't bothered to follow the psychology discussion of this, but I did follow the cancer research attempt to claim a replication crisis.
The article that most people cite is an editorial where pharma researchers try to "replicate" studies by taking "landmark" studies (that were not revealed), and then do drug development off the findings of these initial papers.
Now, sure, if cancer were so simple that surprising new cell behaviors could be used to develop drugs that target lots of different cancers, that would be fantastic. But that sort of generalization should never be expected, it's just silly. If somebody finds a new cellular behavior in a handful of cell lines or animal models or tumor biopsies, there's no reason to think it will generalize.
The point of the primary literature is to document the primary evidence. It's not a textbook, with highly generalized scientific principles, it's the messy discovery phase of science where scientists are communicating findings that they can build upon to eventually build a body of work that could become a generalized principle. A single paper should never constitute the basis of an entire drug development program.
I think that a lot of the "replication" crisis is just a fundamental misunderstanding of how scientific literature works and how it's used by the intended audience.
Newton’s failure to account for near-light speed movement - and the rest of physics knowledge - doesn’t really warrant “replication crisis.” It’s just “more science” that’s refining our understanding.
My issue with replication crisis framing is that in my experience it leads people to believe the softer sciences are irredeemable. Not only is the “crisis” not limited to softer sciences, but it’s really not a crisis. As we go up the entropic chain (physics -> chem -> bio -> socio etc), we can expect that understanding all of the relevant variables will get harder and harder and harder. It doesn’t mean we shouldn’t continue trying, and it doesn’t point to another route to knowledge other than continuing to try.
But today there are a lot more studies that find issues with replicability:
- Many Labs 2: Investigating Variation in Replicability Across Samples and Settings: https://journals.sagepub.com/doi/10.1177/2515245918810225
- Reproducible brain-wide association studies require thousands of individuals: https://www.nature.com/articles/s41586-022-04492-9
- Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015: https://www.nature.com/articles/s41562-018-0399-z
- Raise standards for preclinical cancer research: https://www.nature.com/articles/483531a
- Reproducibility Project: Cancer Biology: https://www.cos.io/rpcb
- A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic: https://journals.plos.org/plosone/article?id=10.1371/journal...
Is it evidence against this study if you perform the same experiment from the Empire State Building and get a different result? Yes, because it matters that the study's conclusion wasn't specific to one building.
It's a bit of a low blow to say changing the country the study was performed in invalidates it as a reproduction, if the study itself made conclusions broader than a single country.
Especially when most of the failed "replications" were things the original study authors said up front weren't valid replications of what they were studying. It's possible to predict in advance that people from a different culture will have a different reaction than Stanford students to Stanford students talking about a hot button issue on their campus (it is also not surprising culture has more impact on psychology than physics!). And even pure ad-hoc explanations of the effect of differences in experimental design (OK, we think the problem might not be the Empire State Building, it might be that they were dropping gliders!) after an unexpected replication fail are more helpful testable hypotheses for science than writing off the original theory altogether. (Though sometimes good hypotheses include "the construction of the original experiment made this mistake, which could be tested by replicating with and without this source of sampling bias ...")
Also, a key overlooked result of the original replication paper was that the results of the original studies were reproduced, but had lower power than the original studies reported (and of course the OP calls that conclusion into question).
I recommend starting here: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
https://statmodeling.stat.columbia.edu/2021/10/12/can-the-du...
My reference to D&K here is specifically for this conclusion, from your article: "One relevant point, I guess, is that even if the observed effect is entirely a product of regression to the mean, it’s still a meaningful thing to know that people with low abilities in these settings are overestimating their <overstrike>true abilities</overstrike> how well they’ll do."
I also think that you have inaccurately characterized the article, which author specifically divorces their self from judgement of "real" and "not real" in the introduction.
Can you give specific examples? This sounds like a pretty broad generalisation.
Edit: this a fun twist on that preconceived notion. Not only is it bunk but a study that you thought shows it is bunk is also bunk! It’s bunk all the way down.
I don't think it's limited to social science; it happens with all research. I guess that we see social science targeted more because the non-HN, Internet-wide reactionary movement specifically targets social science (presumably because the reactionaries dislike the research results), and therefore the HN population has more fodder and momentum to work with - denigrating social science is more normative than denigrating natural science, there's a social tide that carries you along, which is apparently all that's needed today to make a decision.
> Edit: this a fun twist on that preconceived notion. Not only is it bunk but a study that you thought shows it is bunk is also bunk! It’s bunk all the way down.
Perhaps you know already, but that is exactly the goal of trolls and more professional propagandists: Disrupt - disrupt all institutions, sources of knowledge, authority, civil society, civil conversation, social trust, etc. It's not to persuade people of their position - like reactionaries, there is no position except 'anti-' whatever is said. Bunk all the way down is exactly what they want people to think. Look at the results with vaccines. Look at the despair people have of everything, of even the notion of progress or change, of even trying to have a civil conversation.
Ultimately, a lot of people are scoring points online calling academic research bunk because a lot of it is bunk, published for professional gain instead of pushing human knowledge further which is the entire supposed reason for the author's grant and their institution even existing. I singled out social science cause engineers/hard scientists always have thumbed their nose at social scientists, sometimes for very good reasons (they hijack the language and techniques of hard science to give themselves credibility even when its not appropriate) and sometimes for not good reasons (they can't code as well as me so they are invalid).
I don't think it's coincidence. What was a tool has become a religious practice for which the original purpose has been forgotten, and which powerful people have twisted to their own desires - more power and wealth for them by shifting it away from democratic government, the public, by making themselves arbiters of truth who judge, in their monumental, blinding genius, all of the world's scholars - all they need to do is say something disparaging. Now they disrupt everything, including democracy, truth, and freedom.
> a lot of people are scoring points online calling academic research bunk because a lot of it is bunk, published for professional gain instead of pushing human knowledge further
Isn't that just repeating Thiel's and Andressen's claims (as you describe them)? You say it like it's assumed, but other than everyone repeating it, I don't see much evidence of it. Of course it's flawed, like every human endeavor is flawed, including Thiel's and Andressen's.
Second, as ethanbond has commented, a key question in replication is how strictly one should interpret study designs and how much agreement one would expect.
I do believe though that there is a true replication crisis, not just because the situational context is different (i.e. generalization) but also because a lot of the original studies were unintentionally or intentionally wrong, had way too little power, i.e. noisy measurements, small N and publication bias.
Recent advances in the field of replicability include specification curve analyses, for example, and those have been able to show more conclusive evidence of non-replicability because the published results were unbelievably special (i.e. one out of a tiny fraction of significant results).
That seems much too obvious for all these scientists to overlook, including the OP. Do the authors of the original replication paper say that?
> Recent advances in the field of replicability include specification curve analyses, for example, and those have been able to show more conclusive evidence of non-replicability because the published results were unbelievably special (i.e. one out of a tiny fraction of significant results).
Would you have links to this research? Thanks.
Simonsohn and Simmons have a couple of great other papers as well.
Also: Sure, many of the problems in sciences are obvious, just as many of the problems in other parts of society are. That doesn't mean that it's rational for most people to care about them. Plus there are some hard problems very adjacent to easy problems, and that's an invitation to throw them all in a bucket and not care.
Examples: Low N is easy to recognize, but near impossible to change when studying rare diseases. Non-linear phenomena in linear statistical tests are easy to spot, but incompleteness of models (key valuables omitted) or specification errors in structural equation models are hard to spot (need lots of domain expertise).
[edit]: If you want to do a spec curve analysis, try this easy package https://cran.r-project.org/web//packages/specr/index.html
In there are links to other critics of it.
I'm not saying I've got proof of this or you should agree with me, it's just my anecdotal read.
Across multiple indicators of reproducibility, the Open Science Collaboration (1) (OSC2015) observed that the original result was replicated in ~40 of 100 studies sampled from three journals. Gilbert et al. (2) conclude that the reproducibility rate is, in fact, as high as could be expected, given the study methodology. We agree with them that both methodological differences between original and replication studies and statistical power affect reproducibility, but their very optimistic assessment is based on statistical misconceptions and selective interpretation of correlational data. Gilbert et al. focused on a variation of one of OSC2015’s five measures of reproducibility: how often the confidence interval (CI) of the original study contains the effect size estimate of the replication study. They misstated that the expected replication rate assuming only sampling error is 95%, which is true only if both studies estimate the same population effect size and the replication has infinite sample size (3, 4). OSC2015 replications did not have infinite sample size. In fact, the expected replication rate was 78.5% using OSC2015’s CI measure (see OSC2015’s supplementary information, pp. 56 and 76; https://osf.io/k9rnd). By this measure, the actual replication rate was only 47.4%, suggesting the influence of factors other than sampling error alone. Within another large replication study, “Many Labs” (5) (ML2014), Gilbert et al. found that 65.5% of ML2014 studies would be within the CIs of other ML2014 studies of the same phenomenon and concluded that this reflects the maximum reproducibility rate for OSC2015. Their analysis using ML2014 is misleading and does not apply to estimating reproducibility with OSC2015’s data for a number of reasons. First, Gilbert et al.’s estimates are based on pairwise comparisons between all of the replications within ML2014. As such, for roughly half of their failures to replicate, “replications” had larger effect sizes than “original studies,” whereas just 5% of OSC2015 replications had replication CIs exceeding the original study effect sizes. Second, Gilbert et al. apply the by-site variability in ML2014 to OSC2015’s findings, thereby arriving at higher estimates of reproducibility. However, ML2014’s primary finding was that by-site variability was highest for the largest (replicable) effects and lowest for the smallest (nonreplicable) effects. If ML2014’s primary finding is generalizable, then Gilbert et al.’s analysis may leverage by-site variability in ML2014’s larger effects to exaggerate the effect of by-site variability on OSC2015’s nonreproduced smaller effects, thus overestimating reproducibility. Third, Gilbert et al. use ML2014’s 85% replication rate (after aggregating across all 6344 participants) to argue that reproducibility is high when extremely high power is used. This interpretation is based on ML2014’s small, ad hoc sample of classic and new findings, as opposed to OSC2015’s effort to examine a more representative sample of studies in high-impact journals. Had Gilbert et al. selected the similar Many Labs 3 study (6) instead of ML2014, they would have arrived at a more pessimistic conclusion: a 30% overall replication success rate with a multisite, very high-powered design. That said, Gilbert et al.’s analysis demonstrates that differences between laboratories and sample populations reduce reproducibility according to the CI measure. Also, some true effects may exist even among nonsignificant replications (our additional analysis finding evidence for these effects is available at https://osf.io/smjge). True effects can fail to be detected because power calculations for replication studies are based on effect sizes in original studies. As OSC2015 demonstrates, original study effect sizes are likely inflated due to publication bias. Unfortunately, Gilbert et al.’s focus on the CI measure of reproducibility neither addresses nor can account for the facts that the OSC2015 replication effect sizes were about half the size of the original studies on average, and 83% of replications elicited smaller effect sizes than the original studies. The combined results of OSC2015’s five indicators of reproducibility suggest that, even if true, most effects are likely to be smaller than the original results suggest. Gilbert et al. attribute some of the failures to replicate to “low-fidelity protocols” with methodological differences relative to the original, for which they provide six examples. In fact, the original authors recommended or endorsed three of the six methodological differences discussed by Gilbert et al., and a fourth (the racial bias study from America replicated in Italy) was replicated successfully. Gilbert et al. also supposed that nonendorsement of protocols by the original authors was evidence of critical methodological differences. Then they showed that replications that were endorsed by the original authors were more likely to be replicated than those not endorsed (nonendorsed studies included 18 original authors not responding and 11 voicing concerns). In fact, OSC2015 tested whether rated similarity of the replication and original study was correlated with replication success and observed weak relationships across reproducibility indicators (e.g., r = 0.015 with P < 0.05 criterion; supplementary information, p. 67; https://osf.io/k9rnd). Further, there is an alternative explanation for the correlation between endorsement and replication success; authors who were less confident of their study’s robustness may have been less likely to endorse the replications. Consistent with the alternative account, prediction markets administered on OSC2015 studies showed that it is possible to predict replication failure in advance based on a brief description of the original finding (7). Finally, Gilbert et al. ignored correlational evidence in OSC2015 countering their interpretation, such as evidence that surprising or more underpowered research designs (e.g., interaction tests) were less likely to be replicated. In sum, Gilbert et al. made a causal interpretation for OSC2015’s reproducibility with selective interpretation of correlational data. A constructive step forward would be revising the previously nonendorsed protocols to see if they can achieve endorsement and then conducting replications with the updated protocols to see if reproducibility rates improve. More generally, there is no such thing as exact replication (8–10). All replications differ in innumerable ways from original studies. They are conducted in different facilities, in different weather, with different experimenters, with different computers and displays, in different languages, at different points in history, and so on. What counts as a replication involves theoretical assessments of the many differences expected to moderate a phenomenon. OSC2015 defined (direct) replication as “the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding.” When results differ, it offers an opportunity for hypothesis generation and then testing to determine why. When results do not differ, it offers some evidence that the finding is generalizable. OSC2015 provides initial, not definitive, evidence—just like the original studies it replicated.