Hacker News Reader

Comment by gruseom | Hacker News Reader

He's writing about results that were validated through replication. That is the scientific method. If what he's saying is accurate, the problem isn't a few bad apples (that's the proverbial reflexive defense of the status quo anyway).

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain.

I have no idea if it's accurate, but Lehrer's a credible journalist, less prone to the sensational than most. If he's writing about this, it's probably because serious scientists are concerned about it.

(On another note, when I pasted the above, the following text showed up appended to my selection: Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_... How can such a venerable publication as the New Yorker resort to such tackiness?)

View on HN

The replication they are talking about in these particular pharma studies are of the nature that a pharma corp orders up twelve identical 8 person studies. Three studies come back showing their new pill helped a tiny bit, six show no the pill did nothing, and three show the patients got worse. They then cherry pick the three that showed it helped, and possibly toss in one of the ones that showed the pill did nothing just to cover up what they are doing. They then publish a paper showing that 3 out of 4 studies validated that the pills work. The other 8 studies are set on fire and never mentioned. And there is your multiple studies.

This is not some crazed conspiracy theory either, this (doing multiple very small sample size studies rather than one slightly larger study and discarding and never mentioning some of the studies with the least favorable results) is actually how it is now known to be done, and this methodology is the reason why some journals have started to say that all of the studies have to be registered in advance if they want to publish their results so that the companies can't selectively discard results like they have been doing.

A great article about how shaking our medical science is:

http://www.theatlantic.com/magazine/print/2010/11/lies-damne...

And yet, surgeons and medicines have definitely helped me....

He mentions, among other things,studies in ESP as falling prey to this mysterious "well it worked the first time" syndrome.

Now maybe the other examples are more credible, but mixing bullshit like ESP in doesn't help your case at all.

Actually, it does. The likelihood that ESP actually works is significantly less than the likelihood of "the decline effect", which is explained by a number of perfectly rational factors (touched upon in the article.)

If you want to claim that "ESP is bullshit" (as you and I both do, I imagine), you somehow need to account for those (rare) studies which oddly enough seem to support an ESP effect at a statistically significant level. "The decline effect" goes a ways toward providing this explanation.

This article is one example (pre-cognition as opposed to ESP, but in the same vein):

http://hplusmagazine.com/editors-blog/precognition-real-corn...

I think it comes down to we don't know nearly as much about our little existence as we like to think.

In other words, parapsychology is the control group for science: http://lesswrong.com/lw/1ib/parapsychology_the_control_group...

"But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain."

Good! That's what science is all about. Making theories that can later be "disproved" or "confirmed". It's when you cannot disprove something that it is no longer science.

This means nothing can ever be confirmed though -- you never know if you'll learn something later to disprove what you think you know now. Which in turn means a certain level of humility is required. To me, that's where science falls short -- anything that doesn't fit what is currently 'known' and 'proven' is not taken seriously.

>"He's writing about results that were validated through replication. That is the scientific method."

Experiments confirming phlogiston were validated through replication. The theory of phlogiston was however, incorrect. That's the problem with the scientific method.

The Theory of Phlogiston was discredited via the scientific method. It was in the act of replication that it got challenged. Phlogiston is an example of a theory that eventually got rejected as new facts were discovered and experimental design was critiqued, despite having about 100 years of apparent success.

Phlogiston theory was, however, useful to chemists. When it met its limits, and a better theory came along, they of course dumped it.

Apparently, the issue isn't scientific method per se, but of selfish interests who intentionally compromise experimental design and scientific debate for their own fame, glory, and gold.

The good news is that the these half-truths and lies will get in the way of someone else's research and agenda, and that same someone else will likely make a big stink of it.

Reply on news.ycombinator.com