Comment by mattkrause | Hacker News Reader

Comment by mattkrause | Hacker News Reader

Parent Full thread

Okay, why do you think the null is crappy?

In the absence of any information whatsoever, the idea that an analysis pipeline produces false positives at or below its nominal rate seems pretty reasonable.

But, they have some prior information.

Let's look at the E1 paradigm (2 sec on, 6 sec off). In the NeuroImage Paper (Figure 1A, 2A), the FWER on voxel tests is statistically indistinguishable from 5%. In other words, it's appropriately sized. They replicate this result in the rightmost panel of the PNAS paper, where it's also within the 95% CI around 5%.

Now, for the cluster inference, look at the left-most panel of PNAS Figure 1A. The E1 paradigm is the green bar. Using the defaults for FSL (left panel) and SPM (middle panel), the FWER is about 30% and 25%, respectively. That is not good.

I agree that the Block designs look awful in the NeuroImage paper, which makes it hard to say whether this phenomena makes it worse in the PNAS data. It's unfortunate that these numbers are going to be in the press release (70% is much sexier than 30%), but 6x the nominal rate is still bad.

>"Okay, why do you think the null is crappy?"

Because their goal is to determine if some sort of treatment has an effect. If the null is false for other reasons, then statistical significance can't be used to support the existence of a treatment effect. So these would be pointless, pedantic calculations.

@nonbel: Are you saying that an fMRI, when taken of a subject at rest, twice - then what the data here show - is that this is likely to be interpreted as a subject in two different states? And the researchers are ignoring the fact, rather insisting that we should be able to tell that these two states are the same (and then, perhaps tell them from other states)?

I see a few ways this could come about: perhaps the way we record and model activity doesn't conform to the distribution we assume (I'm not sure if they assume a normal distribution here - or if that even makes sense given the nature of the data) -- or perhaps the issue is with taking 3d/4d data and "turning it into" an easy-to-model statistical model (like the normal distribution)?

At any rate, it does seem that they're saying we can't tell that one individual at rest, measured twice, is in the same (rest) state both times? Hence, they're null hypothesis is bunk?

>"Are you saying that an fMRI, when taken of a subject at rest, twice - then what the data here show - is that this is likely to be interpreted as a subject in two different states?"

Yes, that is what they seem to be saying. I didn't read the code, or even the paper very closely. However, from what I quoted, they seem to be saying there is some assumption about autocorrelation that introduces what they call "false positives".

I am saying they have mischaracterized the problem. These are true positives.

Reply on news.ycombinator.com