Hacker News Reader

I don't think it's just this field.

Statistics are routinely abused in nearly every field of science, alas.

The OP mentions findings about how forcing pre-registration on clicnicaltrials.gov seems to change results. Those are, of course, medical research studies. A discussion of those findings regarding pre-registration was on HN 10 days ago. https://news.ycombinator.com/item?id=10080680

I do not think it is conscious 'fraud' exactly. Sloppy statistics have just become routinized in standard scientific research in many fields. And yes, that sloppiness almost always errs in the direction of making it easier to show positive results, of course. But I don't think the researchers realize they are doing something wrong exactly, it's just totally routinized.

Have a look at this what this Harvard neuroscientist says about failed replication. It's pretty scary.

http://web.archive.org/web/20150115080542/http://wjh.harvard...

I find this scary because the author seems to attribute negative failure of replication to incompetence in a way that he doesn't treat the original success, and essentially ends up rejecting all failed replications - how many good results are sufficient vs how many bad results?

Is that what you meant?

Yes. It's a scary rejection of the scientific method.

How did that guy become a professor of psychology at Harvard?

(The archived link itself is dead, but he's listed as a PI here: http://www.wjh.harvard.edu/~scanlab/people.html)

If people cannot replicate your methods the fault is yours, period. The claim that it is "too hard" is bullshit. Write better methods. Science is nothing without replication. There are plenty of people writing vague descriptions of their methods to protect their status. These people are shit scientists.

Well that makes me think the fraud angle is fairly likely...

It made me think of incompetence, but I guess fraud fits too.

The rejection of scientific method was what scared me.

I don't know about neuroscience in particular, but most professions have a code of ethics that forbids you from working in an area where you're not professionally qualified. This specifically addresses the gray area between incompetence and fraud.

I know people in the life sciences who admit to frustration over the pressure to publish and the abuse of statistics.

But I don't know if statistics is the entire problem. It seems to me that a feature of good work in the "hard" sciences is that a problem will be approached from a bunch of different angles, so that the result doesn't really depend solely on statistics.

Synthetic chemists have to perform multiple orthogonal tests in order to announce a new compound, such as IR, NMR, X-Ray crystallography, and elemental analysis, on top of describing a mechanism for the compound's formation and performing reactions with the compound if possible.

Traditionally, to prove discovery of a new element, it wasn't good enough to see a blip on a mass spec. You had to do chemistry with it.

In my view a problem with data driven science is that you can do statistics without even knowing what you're actually studying. But if you don't know what you're studying, then you can't attack a problem from multiple angles because you don't even know if your tests are all measuring the same thing. You also have no tools to ask the hard questions about whether the results really make sense.

Not to try to derail the conversation, but I have noticed a very poor understanding of statistics and experimentation in all of the "soft" sciences, in particular, the social sciences.

It is not uncommon to see published and "peer reviewed" work that makes sweeping claims off a tiny amount of self-reported data with nothing else to back it up. I have seen plenty of stuff that people on the "hard" sciences side of campuses would be embarrassed to even take responsibility for ... not to say they don't try to slip one by every now and again, but it seems systemic in the "soft" sciences.

This is endemic to all the sciences, not just the social ones. Biology has just as poor (if not worse) an understanding of stats, and even worse pressures in terms of the money at stake (medicine, pharmaceuticals).

Maybe physics, and to some extent, chemistry, are immune, I'm less familiar with them.

The issue is less stats, and more the pressure to publish, file drawer effects, poor career prospects for replication work relative to original research, fishing without multiple comparisons correction, etc. etc. etc. All of which combines to result in a lot of scientists looking the other way because the rules for stats and the rules for tenure aren't in alignment.

I saw all this first-hand when I was a grad student in cognitive neuroscience. Don't for a second believe this is limited to psychology. If anything, psychology is being more honest and grappling with these issues first.

When I look at the people I know personally who study psychology, sociology and related fields and see what they learn, I don't have any problems with believing that virtually everybody in the field is incompetent.

It starts with the little things. I remember bringing up on HN the bullshit surveys most students do for their Master's thesis that are statistically worthless for the reasons said students don't understand. I've been told that hey, it's MSc, not a PhD, it doesn't have to be research-quality, etc. But hell, if people are willing to rely on bullshit methodologies for their Master's, when exactly do you think they'll learn the right way of doing things? Does the statistics god dispense additional levels of enlightenment when someone crosses the PhD threshold? I don't think so.

And honestly, it's not that much better in the hard sciences as well, on the undergraduate level. Maybe they won't let you get away with bullshit stats, but they'll happily let you get away with fabricating results and tweaking data to fit the model. Studying today is about only one thing - do everything you can to pass the exam. Research ethics? Never heard of it.

Where I study psychology, I'm noticing that the pace of learning statistics is slower compared to statistics classes in my CS program. The fact that I'm doing multiple programs is a hint in itself. For example, we receive lessons about normality assumptions but do not receive lessons about non-parametric statistics. Furthermore, whenever I gave critique about these kind of things they were always shoved away as if they were a minor inconvenience.

It doesn't require any malicious intent. An alternative possibility is that the things they were researching just weren't likely in the first place. If an effect that only has a 5% chance of being true in the first place, then even if you confirm it with a 95% confidence the chance that it is a fluke could still be as high as 50%.

This can be made worse if you test several different hypothesis, only report successful experiments, and don't replicate them.

> The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky)

This doesn't surprise me at all. Kahneman wrote a book, "Thinking, Fast and Slow", in which he spends a long time talking about how most psychologists and social scientists do not understand the basics of statistics and therefore end up with flawed conclusions.

I've always felt like Hanlon's razor needs some kind of clause to account for corporatacracy. It's difficult to figure out how to attribute enforced monopolies, which take a great deal of planning and political cunning to establish, to incompetence.