Beneath the Replication Crisis
societyinmind.com
societyinmind.com
Imagine finishing graduate school and then suddenly learning that over half of what you learned is probably not true... but not knowing which half
this is largely due to the failure of idealism as a philosophical framework to set up any type of enduring political, economic or secular social structure. the blame lies on hegel or perhaps marx's interpretation of hegel getting stuck in praxis quagmires. which is why the turn towards asia and the new age movement which emerges from it in the early 1900s is essentially anti-science. the spiritual successor of communism is the idea of fusing the east and west traditions to form the new enlightened human, who sees no race, religion or gender but simply behaves as the universal light and creator. it has largely been a failure, as folding china into the world community has been against the pragmatic self-interest of various sensitive american industries - weapons, space, communication, high finance, which can exert sufficient power on the american leadership structures to avoid any type of idealism about world peace. there are no more enemies or allies just competitive co-morbidity, actors fraying the ropes they are tugging on until something breaks, like the ussr in the 80s.
it's only a problem if you think humans are equal. if you shift the view that humans are unequal and life is deeply unfair, then it's normal to see the next 100 years as a struggle for dominance between superstition and science, between a world led by north america or eurasia, between the new dominoes of fascist nationalism and disinterested international capitalism. it's a hot peace, which will end up with one temporary victor, possibly colonizing mars - or just landing there and coming back, before a new power emerges to challenge the old empire. i would argue that this is a bi-product of the 'correct' interpretation of hegel, the one that isn't taught directly in schools, the master and slave dialectic.
I have a historical bogeyman of my own: I think Newton--perhaps accidentally--encoded a lot more of his own non-mathematical perspective into his work on Calculus. These biases got baked into our current theory of "real numbers" (which are pretty spooky, once you get to know them, and don't strike me as befitting their name). This was done primarily because the pure mathematicians in the centuries following Newton couldn't justify his results, which was an embarrassment since his work was so useful that it obviously was true. As you say:
> people are self-interested not truth interested
So now we have this element of arbitrariness baked into our numbers, theories that underpin the sort of methods that the article refers to here:
>Faith in this methodology certainly unites a much larger number of research psychologists than does any kind of commitment to a particular theoretical framework
Somewhere between physics and psychology, an assumption that worked for Newton stopped working for us, but we didn't notice because we had only it to compare it to.
I similarly extrapolate this accusation to wider political spaces (i.e. the failure of standardized testing to make the kind of differences we wanted it to, or the propensity of our economic system to generate jobs that don't actually matter).
I see some parallels between our reactions to this piece, so I want to read Hegel and see if we're just similarly out there, or if we're out there in similar ways.
(please excuse the shitpost effects, the video is still useful)
> The replication crisis, if nothing else, has shown that productivity is not intrinsically valuable.
I think this is important to focus on - the point of universities has become to produce profit, and to give people degrees that are profitable, and to appear to be able to do those things. This has very little to do with producing research with verifiable results. It's much more to do with getting students into the funnel by making people with tenure appear as productive as possible.
It's more like overfitting the target function of publishing impactful research. A bit of p-hacking, a bit of cutting corners in experimental setup, a sloppy null hypothesis check, and you honestly believe you see an effect! Everyone is happy: you, your adviser, lab's administration, the journal where you publish the paper.
But if you carefully check for everything, then find no effect, you kill an interesting hypothesis, your paper is hard to publish, "you are not making progress", and nobody is happy.
Crooked incentives, crooked results :(
But I think the underlying reason for the push to publish things - and impactful things are easier to publish, is profit. More hireable grads, more tenured professors publishing papers.
As the Scott Alexander essay makes clear, it also affects psychiatry and it's apparently the case that many areas of medicine have this problem. Biotech studies are also hives of replication failures.
Even AI research has had replication failures and that's based on running theoretically deterministic software on theoretically deterministic machines!
Some of this is that accurately describing studies and replicating them is hard. But some of it is that academics aren't incentivised to find the truth, but rather, to make it appear that academics know a lot of things.
The major problem with current scientific practice is that good practice is actively penalised. I used to work in industry, and I was shocked at the lax standard of work, particularly with respect to accuracy and precision, of wet lab scientists in university research settings. I mentioned this to a few postdocs over the years and paraphrased was told that "if it's good enough to publish, then it's good enough for me", which if you think about it, is actually quite a low bar. Most of the people were fully aware they were doing sloppy work, but didn't care.
How can it be improved? I think there are two sides to this coin. Firstly, good practice has to be encouraged and rewarded, and sloppiness penalised. That requires a culture change in the laboratories. Too many PIs don't care about what happens in their labs so long as "good" results are being generated by their underlings. They don't look after instrument calibration and ensure that people are working to GLP standards. In industrial labs, we had to send samples off to reference labs, analyse random samples provided to us, and undergo inspections and audits to prove we were providing correct analyses. Maybe academic labs should be obligated to prove themselves as well, or lose their funding?
Secondly, a project delivering negative results should not be a career-ending move. Failing experiments does not necessarily mean one is a bad scientist. But right now, the incentives are to spin all results in a positive light, even if it means publishing bad science, because that's what it takes to keep the funding coming in. Success should be rewarded, but I think our criteria for what success is need to be recalibrated to reduce charlatans abusing the system for their own benefit. Publishing a paper isn't enough; it's got to be replicable independently.
But this does lead to the question of - why not just reduce academic funding, matched by corresponding cuts in corporation taxes? That would obviously not lead to a 1:1 transfer of research funding or anything even close to it, but if the replication crisis seems to suggest anything at all it's that there's too many scientists chasing too little real knowledge, with too few reality checks of the sort industrial labs require. If more funding was from industry, the quality of science might be higher.
In a 2017 follow-up article in Slate magazine on the "Feeling the Future" experiments, Bem is quoted as saying, “[...] If you looked at all my past experiments, they were always rhetorical devices. I gathered data to show how my point would be made. I used data as a point of persuasion, and I never really worried about, ‘Will this replicate or will this not?’”"[42] While fellow psychologist Stuart Vyse sees this statement as coming "remarkably close to an outright admission of p-hacking", he also notes that Bem "has been given substantial credit for stimulating the movement to tighten the standards for research" such as that taking place in open science.[43]
– [42]: https://redux.slate.com/cover-stories/2017/05/daryl-bem-prov... 43: https://web.archive.org/web/20180805142806/https://www.csico...
Also from one of the articles' sources:
"There is some evidence, however, for the hypothesis that people can feel the future with emotionally valenced nonerotic stimuli, with a Bayes factor of about 40. Although this value is certainly noteworthy, we believe it is orders of magnitude lower than what is required to overcome appropriate skepticism of ESP." – https://link.springer.com/article/10.3758%2Fs13423-011-0088-...
A low threshold for statistical significance won't solve all problems of p-Hacking, and the implementation of Bayesian methods doesn't seem to promising as well. Will be interesting to see how the field of psychology is going to change over time.