Neuroimaging results altered by varying analysis pipelines
nature.com
nature.com
For the most part it wasn't a radical finding, but really good to quantify and see the effects. It also demonstrated the importance of including and documenting the parameters of the analysis pipeline.
Something else to keep in mind, especially when thinking about cognitive neuroscience (where scanning is an important tool), is that analysis are not done in isolation. Every experiment is motivated by behavioural results, neurobiological results or both. The goal of fMRI is to gain insight into the activity of the brain, but its also not very precise (given the scale of the brain).
Basically this data is analyzed in the context of what has been shown before, and how it matches our current hypothesis. This doesn't mean such variation is acceptable (see first part of my comment), but it does mean that this doesn't cause all results on fMRI to be invalid.
This paper published in 2016 suggests that commonly used statistical packages for analysis of fMRI data can have a false discovery rate of up to 70%: https://www.pnas.org/content/113/28/7900
More fun, this poster presents the results of fMRI in dead salmon given an open ended mentalising task: https://www.psychology.mcmaster.ca/bennett/psy710/readings/B...
> Task: The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing.
This is just so, so good. Thank you to this salmon for participating in science.
So, while the dead salmon experiment was eye opening to some when the neuroimaging field was very new, it is no longer relevant now.
Neuroimaging techniques are constantly improving, both from better analytical techniques and knowledge about the factors that can compromise the veracity of analyses, but they, including the present paper, show that neuroimgaging is bunk science or useless, which is most certainly incorrect.
It was my first time hearing about this, and it struck me as the type of funny anecdote that could be slipped into a talk/lecture (I find it takes quite a bit of finesse to keep non-technical people engaged when talking about statistics and data science, so humor and relatable examples always help). So, it’s good to know the broader context.
The takeaway from Salmon experiment IMO is not specific to fMRI: you cannot trust a single technique or field "golden" standard (whether flawed or not), and should rather pursue accuracy across different pre- post-processing methods.
That's not realistically doable single-handily (at least in my experience), but in the current age it should become the norm to rely on publically available pipelines, probably with some sort of configuration files or even GUIs. I personally worked with fMRI only sparingly, so cannot comment on that in the context of the posted article.
Put another way. If your effect of interest disappears with slight changes in your pipeline then the effect probably isn't that robust to begin with. That doesn't mean it isn't a true effect, but on average it probably isn't.
> This finding highlights the potential consequences of a lack of standardized pipelines for processing complex data.
Lack of a standardized pipeline is not a problem in my mind. Replication of findings on multiple pipelines is more convincing that results are not computational artifacts or analysis quirks. Replication using independently acquired data is of course even more convincing, but that's can be expensive and difficult in this field.
If your effect of interest disappears with slight changes to your pipeline (perturbation), then maybe the effect is weak or chaotic or unstable, but also maybe there is a systemic bias in your pipeline!
And even when you use the same software package, there are various things you might do differently, parameters you might set differently; even the order of certain operations maybe switched around. A classic example of that has to do with slice timing correction. Since fMRI slices are acquired at different times, a typical thing to do would be to interpolate the slices so that you can treat them as if they were acquired all at the same time. However, motion correction between volumes, in which subject motion is partially corrected by jiggling around each image until they maximally overlap each other, can be done before or after slice timing correction; but the consequences of that is that motion artifacts can propagate through volumes in different ways depending on the order you do it in; but neither order is definitely better, because it matters how much motion is a problem in the scans, and how long it takes to acquire a volume (e.g. tr 2 seconds), and so on.
This means that all the TED talks saying ‘we know this because fmri’ should be assumed incorrect until proven otherwise with multiple double blind studies (which is too expensive in the current era to actually reproduce all of these studies).
To be fair though, you shouldn't believe any science in most disciplines unless its been replicated by multiple labs/authors.
From my limited understanding, replication does not have much prestige and can be difficult to get published. Is this a problem, and have research universities found a solution?
And because of the bias towards novelty, replications are not regarded as worth publishing and so these results remain unchallenged.
It's a hard problem, and has been known to be a hard problem for at least the last forty years (within psychology, at least) and yet very little has been done.
Brian Nosek and the OSF are doing good work here though.
I know very little about the brain, except what I did in my undergrad (which wasn't a lot), but all the statistics in papers looked really, really dodgy to me.
This finding highlights the potential consequences of a lack of standardized pipelines for processing complex data.
Couldn't it at least well be concluded to 'highlight the potential consequences should standardized pipelines for processing complex data be introduced' ?
If they had all been doing the same thing, aren't odds the results would be just as fishy but it would have been even harder to notice? Seems to me the way you know something really is there is generally that it keeps showing up in analysis with a multitude of different appropriate methods.
(Also, not sure what the HN norms are, but to avoiding self-plagarization, note this is cross-posted from my twitter account.)