Comment by timtadh | Hacker News Reader

Comment by timtadh | Hacker News Reader

Parent Full thread

In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific method. However, such use is invalid as the scientific self-criticism

1. assumes the validity of the scientific method

2. relies on the scientific method as its critical lens

Whereas those who critique science as a whole:

1. assume that the scientific method does not work and does not arrive at "truth"

2. then use scientists being self critical to prove #1.

Such a "proof" does not work as there is its uses the assumption "the scientific method arrives at truth" to derive the contradiction "the scientific method does not arrive at truth". See for instance comment: https://news.ycombinator.com/item?id=16859200

In reality, work on reproducibility is about improving the practice of science overall. It does not in itself show that science is inherently untrustworthy. What it does show is that scientific discovery is difficult and it takes a lot of effort and new findings should be treated critically. What does critically mean in this context? It means with in the boundaries of science analyzing the theoretical basis, hypothesis, method, and experimental results for potential flaws. It does not mean to be skeptical as a default because science "doesn't work."

I agree with your overall point, but technically speaking it is logically valid to prove a hypothesis false by first assuming it and then deriving a contradiction, even when the contradiction is the negation of the original hypothesis (as it is in your example).

What you should have said is that some critics start with the premise "the scientific method does not arrive at truth", and then use other people's arguments that depend on the premise "the scientific method arrives at truth" to support their claim, which is indeed logically invalid.

>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor.

I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process.

>It does not in itself show that science is inherently untrustworthy.

I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong.

Then there are things like publication bias, that takes this to a whole new level. Things like that means that a biased body of journals can project any consensus that it favors just by selecting studies that fit its narrative. The inherent issues with statistics means that you can find studies that shows any possible outcome.

>"I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong."

Another very real issue here is that malicious use of statistics can be used to show nearly anything in ways that can be extremely difficult to detect, even when the maliciousness is hidden in plain sight. And then going a step beyond that there's plain old number fudging which is almost impossible to prove since variance works as sufficient plausible deniability. And finally there is of course plain old ineptitude. Like you mention even when trying to do things completely by the book, statistics are incredibly difficult to get right.

Something that comes to mind here is the recent MIT study stating that Uber drivers earned $3.37/hour. That study was completely broken. [1] It's debatable whether the cause was maliciousness or ineptitude, but the point is that these problems arise, with a disturbing regularity, even when the most reputable of names are attached to them.

[1] - https://qz.com/1222744/mits-uber-study-couldnt-possibly-have...

Feynmann said, "The first rule is that you must not fool yourself. And you are the easiest person to fool."

There can be malice, sure. But there can also be desire to believe. And, hey, here's a statistical analysis that shows what the investigator is already biased to believe anyway...

>I think when statistics is involved, the results are inherently untrustworthy.

Ummm...are you kidding? Statistically vetted results are inherently UNCERTAIN, but how could they possibly be inherently untrustworthy?

Even if a mechanistic effect is observed, its relationship to a particular cause or influence is only established statistically. In fact, the very observation is often performed u Dee the umbrella of statistical calibration of appropriate instruments.

  As Pearson said, "Statistics is the grammar of science".

I cannot speak for the person you are responding to, but in my [agreement] of his critique of statistics, I am implicitly speaking of social statistics. I think there is a vast difference in e.g. a statistical modeling of the behavior of electrons and e.g. the statistical modeling of some sort of human behavior.

...but that just isn't true. Statistical analysis is (among other things) a way to quantify our uncertainty. If done approriately, the statistics simply communicate the role probability played in moving from the experimental premises to the results.

  The level of uncertainty in most (perhaps all) experiments involving particles in a vacuum is far lower than experiments involving human behavior. Statistics doesn't create the uncertainty, it communicates the uncertainty.

  I can use propositional logic to arrive at the conclusion that "Unicorns are real" yet I don't encounter many people throwing logic itself under the bus because of that.

People lie while speaking English every day yet I don't encounter many people claiming that English is inherently untrustworthy.

  Statistics has been a whipping boy for a long time by people who don't know much about statistics or just fail to think clearly about what is being said.

> If done approriately..

This is the whole point. It is the difficulty in doing so, or the difficulty in deciding if this was done appropriately or not is the only thing that it untrustworthy..

>People lie while speaking English every day yet I don't encounter many people claiming that English is inherently untrustworthy.

No one expect a statement to be true simply because it was uttered in English.

I'm not entirely sure what you're trying to say, and your post formatting is not helping.

I would recommend reading the article. The executive summary summarizes things. The issue is not statistics in and of themselves, but how they are used in the overwhelming number of studies - particularly those in the social sciences or related with human physiology.

Reply on news.ycombinator.com