The scientific study has become a flawed manual for living
theguardian.com
theguardian.com
The above sentence shows that the author -- and most readers -- miss the single most important thing about cohort studies -- which is that they can't distinguish causes from effects. Contrary to everyday understanding, the statement "women are less likely to die of heart disease if they have a partner" does not mean that choosing to have a partner results in a reduction of a woman's chance to die of heart disease, and it does not mean that seeking out a partner will reduce one's changes to die of heart disease. It only means that having a partner, and reduced chance of dying of heart disease, are correlated.
The above very common confusion is the source of any number of meaningless "breakthroughs" in science publishing, and fuels any number of public controversies with little or no scientific substance.
It certainly implies (how strongly depends on the actual data) that there is some causal link (A -> B, B -> A, or A <- C -> B), because something is reducing the entropy between the two systems. That again, though, doesn't really tell us enough to be useful.
Suppose healthy people live longer and are more attractive and Attractive people are more likely to be married. So A <- D <- C -> B.
Sometimes A between .5 and .8 -> B but A less than .5 or more than .8 -> !B. So now you can have study's that show more A -> B and others that less A -> B. (Think nutrition)
It's still true that there are potential situations I didn't touch (A <- B and B -> A, &c), but those can be reduced to multiple instances of the above.
Edited to add:
"Sometimes A between .5 and .8 -> B but A less than .5 or more than .8 -> !B. So now you can have study's that show more A -> B and others that less A -> B. (Think nutrition)"
My arrows also weren't meant to be implication, just a causal relationship. A -> B and !A -> B would be the same thing.
I should probably have been more explicit about all these points.
But in the typical case of a measured correlation, no causal relationship can be implied based on the data alone, regardless of the direction. This is especially true in examples of data mining, where many correlations are located that arise purely from chance rather than design.
A single study is generally close to worthless.
Not really -- it fails to consider the possibility that B caused A rather than the reverse, or that an unevaluated cause C caused both A and B. This is why I hate hearing the common phrase, "In a recent study, (something) has been linked to (something)". The words aren't false, but they're misleading to an untrained person.
> That again, though, doesn't really tell us enough to be useful.
That's certainly true.
Yes really! Just because things are uncertain doesn't mean they're equally uncertain as you learn new facts! "A causes B" doesn't become more likely relative to "B causes A" when you learn there is a correlation, but both become more likely relative to "there is no causal relationship between A and B whatsoever" and thus more likely overall. Probabilities sum to 100%. We learned that one possibility is less likely - that probability has to go somewhere.
But that's the foundation on which science is built -- the null hypothesis -- the default precept that a correlation has no meaning without evidence supporting one. The idea that unexplained correlations are assumed to arise from chance, not an uncovered cause.
Until there's evidence that cannot be explained any other way, two apparently connected things are presumed to be connected by chance.
> ... but both become more likely relative to "there is no causal relationship between A and B whatsoever"
You're starting out making an assumption that has no basis. In science, the default assumption about a correlation is that it has no meaning whatsoever.
> Probabilities sum to 100%.
Yes, they do, but the initial probability is 100% that the correlation arises from chance. This is why the scientists at the LHC refused to speculate about the existence of the Higgs until they had acquired a p factor of 5σ. Details:
http://blogs.scientificamerican.com/observations/2012/07/17/...
"Scientists use p-values to test the likelihood of hypotheses. In an experiment comparing some phenomenon A to phenomenon B, researchers construct two hypotheses: that “A and B are not correlated,” which is known as the null hypothesis, and that “A and B are correlated,” which is known as the research hypothesis."
"The researchers then assume the null hypothesis (because it’s the most conservative supposition, intellectually) and calculate the probability of obtaining data as extreme or more extreme than what they observed, given that there is no relationship between A and B. This calculation, which yields the p-value, can be based on any of several different statistical tests. If the p-value is low, for example 0.01, this means that there is only a small chance (one percent for p=0.01) that the data would have been observed by chance without the correlation. Usually there is a pre-established threshold in a field of study for rejecting the null hypothesis and claiming that A and B are correlated. Values of p=0.05 and p=0.01 are very common in many scientific disciplines."
If you believe that's true, I've got some wagers I'd like to make with you. You don't mind zillion-to-one odds, right?
Edited to add:
More to the point, does the universe care if you're a scientist or a gambler and change the laws of probability? Presuming the answer is no, pretend you're a gambler for a minute, and you should see what's wrong with statement "the initial probability is 100% that the correlation arises from chance": I offer you a wager, where we pick a correlation about which you have not formulated a hypothesis, and if I can show causation of any sort you give me zillions of dollars; if I can't, I give you a dollar. Your expected value, with an assumption of 100%, is that you always get the dollar. Do you really think that, over zillions of games, you'd make money?
Edited again:
The hypothetical people you're criticizing - who read this article and go out to find a man - aren't trying to "do science", they're trying to successfully wager and make the best bet they can with their health and their life. Of course, we agree that if they're making substantial changes based on this study, they're probably wagering poorly, but that's beside the point.
No--that would be silly, right? Well the logic works exactly the same even if the data set is larger than one.
There's a lot of other stuff going on in this example, but in general yes - if I discover that someone named Ralph won the lottery, it is slightly more likely that the lottery is somehow biased toward people named Ralph, because explanations of the universe where that is the case very slightly better explain my observations. It's probably still phenomenally more likely that the lottery is not biased toward people named Ralph.
Take the other extreme - you look at the 1000 most recent lottery winners, and discover that they're all named Ralph. You look at the names of people buying lottery tickets, and Ralph only shows up 1% of the time. You're not going to consider it any more likely that the next winner will be named Ralph as well?
No, that is false. It would be like flipping a coin and seeing heads, then claiming that this outcome increases the probability that heads will be the preferred outcome in future tosses, or tosses in general. Think about what you're saying.
> Take the other extreme - you look at the 1000 most recent lottery winners, and discover that they're all named Ralph. You look at the names of people buying lottery tickets, and Ralph only shows up 1% of the time. You're not going to consider it any more likely that the next winner will be named Ralph as well?
Not without an explanation that can be tested and potentially falsified -- the gold standard of science. Without an explanation, it's a coincidence.
Try to remember that science is not simply drawing conclusions on the basis of observations -- there's more to it than that. It's not science until an explanation (a theory) is proposed, is tested, and has resisted falsification.
What if I told you that someone correctly predicted the direction of the stock market for 20 months straight, all in advance? Does that prove he's a stock market genius? Here's a better explanation, one that a scientist would want to test first -- if 1,048,576 (i.e. 2^20) people try to predict the stock market's direction for 20 months straight by flipping coins, there is a better than even chance that one of them will succeed by chance alone.
Next question -- what's the chance that the one random winner will assume that his success resulted from chance, or will he start selling a worthless book called "Secrets of the Winners" on late-night TV?
There's nothing wrong with that. If you flip enough straight heads, at some point you decide there's something going on with the coin - even if you're not sure what.
"Without an explanation, it's a coincidence."
If a coincidence is sufficiently unlikely, the chance that it's not simply a coincidence becomes more likely. Just because I don't know what's stripping entropy from the system doesn't mean I can't conclude that something almost certainly is. Let's say that you and I set up an experiment where we flip coins together - each 1000 flips - and we compare our flips. We find that we've both flipped a lot of heads and a lot of tails, but with no particular observable pattern - except that we both got the same sequence. 1000 flips, and the two coins matched every time.
You don't immediately have an explanation (do you?), but you'd set about looking for one right? Why would it be crazy to start looking for some interaction between the coins before observing those 1000 flips, and make sense after? Because the existence of some causal relationship just became 1000 bits more likely.
"Try to remember that science is not simply drawing conclusions on the basis of observations -- there's more to it than that. It's not science until an explanation (a theory) is proposed, is tested, and has resisted falsification."
You keep going back to this, but my goal isn't "TO DO SCIENCE", it's to draw correct conclusions about the world so that I can make my way around it. If the particular boundaries you've drawn around Science(TM) help with that, then great. If not, then I don't have use for your "Science" - or at least I have a lot of use for things that are not Science and your condemnation of them for simply "not being science" is misleading and harmful and silly.
"What if I told you that someone correctly predicted the direction of the stock market for 20 months straight, all in advance? Does that prove he's a stock market genius? Here's a better explanation, one that a scientist would want to test first -- if 1,048,576 (i.e. 2^20) people try to predict the stock market's direction for 20 months straight by flipping coins, there is a better than even chance that one of them will succeed by chance alone."
No it doesn't "prove that he is a stock market genius" - but it is more likely that he is a stock market genius when you've observed that then when you have not, because P(20 consecutive predictions | he is a genius) > P(20 consecutive predictions | he is not a genius), and that's true regardless of how many people were picking. Given the pool you're drawing from, it remains spectacularly unlikely. You keep insisting that my "we end up thinking X is more likely than we started thinking it was" means "we think X is more likely than alternatives" in order to draw absurd conclusions. Of course that will give you absurd conclusions, it's an absurd position - but it's never been what I've said.
"Next question -- what's the chance that the one random winner will assume that his success resulted from chance, or will he start selling a worthless book called "Secrets of the Winners" on late-night TV?"
Yes, the high probability of that and the low probability of actually being a "stock market genius" means those books are likely to be worthless. That's true whether or not what I say is true.
I'm out of this argument.
But if you don't know in advance what you're looking for, you don't know how to control the data set to ensure it is representative.
I could give you a data set of 1,000 recent lottery winners, and you would find that they are all named Ralph. But you might not know that I assembled the data set by looking for lottery winners named Ralph. You'll find a super-strong correlation, but be missing the context that would disqualify it as causal. You'd only find that out if you asked me how the data set was assembled...only at this point are you starting to pursue scientific knowledge.
You can't trust human perceptions of correlations to be representative of reality. That's the reason the scientific method was developed in the first place.
Yes -- unless you're a scientist. If you're a scientist and you expect to make a contribution to the store of human knowledge, you cannot care one way or another how your experiments turn out, only that they're conducted with the maximum amount of dispassion and objectivity.
> The hypothetical people you're criticizing - who read this article and go out to find a man - aren't trying to "do science", they're trying to successfully wager ...
Yes, which is why we see so many articles like the one this thread discusses -- articles that cynically harvest human stupidity to some unworthy purpose.
That's a different thing. I wasn't offering to bet you that your hypothesis was correct, which would certainly be a problem; I was expecting you to use your understanding of the world - which should (I hope you agree) be informed by your science! - to make decisions in a non-scientific context. Scientists need to make decisions in the real world, too.
'> The hypothetical people you're criticizing - who read this article and go out to find a man - aren't trying to "do science", they're trying to successfully wager ...
;Yes, which is why we see so many articles like the one this thread discusses -- articles that cynically harvest human stupidity to some unworthy purpose.'
Trying to successfully wager, in a context where placing some bet is unavoidable (cost/benefit analysis under uncertainty), is not stupidity. So, I agree that "articles that cynically harvest human stupidity to some unworthy purpose" are depressingly common, I disagree that "people need to wager" is (or that they should instead somehow be doing science with their lives just to avoid shoddy journalism).
True, but I'm pretty sure I didn't rely on an assumption that it couldn't. Too much correlation without any thing causing that correlation is unlikely. How unlikely depends on just how much correlation. When something is way more unlikely given X than Y, that is evidence for X over Y.
We are not talking about the likelihood of an "explanation", we are talking about the likelihood of x causing y.
It's simply the basis for forming a hypothesis, which is the first step needed to even look for causation.
Most hypotheses do not pass testing. This implies that most observed correlations turn out to not be causal.
If you simply observe an unexpected correlation for the first time, no. You have no idea if it unique, remarkable, repeatable, etc.
Thus, most nutritional, health, and economics 'science' is trash.
Unfortunately, it also means it's very hard to prove things are safe or unsafe. Phracking, pesticides, GMO, ... all get a free ride for too long, IMHO, because they can say 'well nobody has proved they're unsafe'.
Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is.
> Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is.
There's a logical error in this approach. If one study has some probability p to be systematically flawed, can an evaluation of that study's outcome be modified by the knowledge that ten other studies, each with the same probability of flaw, exist? Well, no, not really -- they must all be assumed to be flawed, regardless of how many studies exist. If this were not the case, we could cancel out the effect of the systematic flaws in each study by performing a lot of flawed studies. (When this idea is put in practice, often it results in the round-filing of all the studies that failed to meet the researchers' expectations.)
Notice that I'm addressing the question of how we might evaluate the outcome of one study, given the existence of other similar studies. A meta-analysis of all the studies at once uses different rules.
No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist. I'm not saying that doesn't happen -- it does -- I'm only saying it contradicts scientific best practice.
> Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed.
Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.
> That doesn't mean "assume it's flawed" is correct advice.
In science? Of course it's correct advice. If studies are flawed, and if we don't make the default scientific assumption that they're broken and meaningless, we may not exercise the effort required to sift the few properly conducted studies from the chaff.
1) That's replication, and 2) that's not what I was suggesting. So I don't understand your "no" at all.
To clarify a little:
First, by "similar studies" I mean "studies drawn from the same population of studies", not "studies that are looking at the same thing in the same way".
Second, I said that the study has a similar chance of being flawed as those other studies, not that more studies means less chance of being wrong (which would be true to that degree the chances of error are uncorrelated, but wasn't at all what I was saying).
'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.'
So wait... first, peer review is not a part of science?
Second, a paper having passed peer review tells us nothing about whether it's flawed? Seriously? So if I pick two papers at random, one from "the set of papers that have not been peer reviewed" and one from "the set of papers that have been peer reviewed in a reputable journal and not found wanting", they are equally likely to be erroneous?
> So wait... first, peer review is not a part of science?
It's part of how science is practiced, but it is not part of science's fundamentals -- the latter honors only direct evidence -- see below.
> Second, a paper having passed peer review tells us nothing about whether it's flawed?
Well, since I never said that, I don't need to defend it. Instead, I will refer you to the motto of the Royal Society, science's oldest professional society:
http://en.wikipedia.org/wiki/Nullius_in_verba
Quote: "Nullius in verba (Latin for "on the word of no one" or "Take nobody's word for it") is the motto of the Royal Society. John Evelyn and other Royal Society fellows chose the motto soon after the founding of the Society.[1] The current Royal Society website explains the motto thus: 'It is an expression of the determination of Fellows to withstand the domination of authority and to verify all statements by an appeal to facts determined by experiment.'[2]"
In the context of fields where junk studies are the norm, then it is trash, and actually worse than nothing, because it's a symptom of a widespread misguidance of the scientific approach.
Homeopathy has probably plenty of such studies, and you know... it's homeopathy.
Let's test this idea with a thought experiment. Let's say I'm a doctor and I've created a revolutionary cure for the common cold. My cure is to shake a dried gourd over the cold sufferer until he gets better. The cure might take a week, but it always works. My method is repeatable and perfectly reliable, and I've published my cure in a refereed scientific journal (there are now any number of phony refereed scientific journals). And, because (in this thought experiment) science can get along without defining theories, I'm under no obligation to try to explain my cure, or consider alternative explanations for my breakthrough — I only have to describe it.
Because I've cured the common cold, and because I've met all the requirements that non-scientists recognize for science, I deserve a Nobel Prize. Yes or no?
The answer is obviously no, and to establish the reliability of dried gourds as a cure for the common cold, we would need a double-blind experiment -- both the experimenter and the subject would need to be blinded about which treatment is being applied.
This is disingenuous. The primary reason your example was flawed is the lack of a control group, and your phrasing in establishing that it is flawed more or less points that out. Using that to condemn single-blind studies is inane - if it were double blind but there was no control, it would give the same result.
As one paper humorously pointed out, there's no double-blind random crossover trial to see if parachutes are effective.
More seriously, astronomy deals with a bunch of singular events. How do we do double-blind random crossover trials of a comet crashing into Jupiter (like Shoemaker–Levy 9)?
Closer to home, and closer to the point, detecting rare but deadly drug side-effect (and especially drug combination effects) cannot be done blind, since the drugs are on the market and people know what they are taking.
"As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."
[1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/?utm_sour...
Fracking: short for "hydraulic fracturing", the fracturing of rock via pressurized liquid.
What do you base that opinion on? Have you actually looked for any conclusive studies on GMO, and found them wanting?
There's also a curious study comparing ancient wheat with modern wheat (I realize that may be not technically GMO) in, you guessed it, a blind crossover trial. http://www.ncbi.nlm.nih.gov/pubmed/24521561?dopt=Citation
Even that sentence needs work: They can't distinguish between correlations and causes. And having spent too much time with the medical community lately, I'm not sure they're completely straight on the concepts either.
Now I'm lost. As far as I understand, those studies only find correlations. What am I missing?
Fair enough, that's correct and important. My original didn't encompass the issue of tertiary causes, i.e. causes not accounted for in the correlation.
Why can't I find a list of some of the largest and most robust systematic reviews and meta anlyses ever done?
The Cochrane Collaboration is the closest thing I know of:
http://www.cochrane.org/ http://en.wikipedia.org/wiki/Cochrane_Collaboration