many options: frightening results

If you need a good scare this Halloween season, I suggest reading On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments.

This is not an MVPA paper, but I have no doubt its conclusions are just as relevant to MVPA. Joshua Carp took one fMRI dataset and constructed  no less than 34,560 significance maps (i.e. mass-univariate group statistical maps), deriving from 6,912 analysis pipelines (e.g. smoothing or not, slice-time correction or not) and various statistical choices (e.g. FDR or RFT corrections).

The scary part is that these various analysis choices are all reasonable and in use, but produced different results. To quote from his conclusion, "While some research outcomes were relatively stable across analysis pipelines, others varied widely from one pipeline to another. Given the extent of this variability, a motivated researcher determined to find significant activation in practically any brain region will very likely succeed–as will another researcher determined to find null results in the same region."

For just one highlight, Figure 4 shows the peak voxels identified in the 6,912 pipelines, color-coded by the number of pipelines in which each was peak. The color bar maxes at 526: no voxel was peak in more than 526 of the 6,912 maps. But the peaks are not distributed randomly: they're grouped in anatomically sensible ways (which is good).

This particular map reinforces my bias towards ROI-based analyses: should we really be interpreting tiny blobs or coordinate locations when they can be so susceptible to being shifted by reasonable analysis choices?

I am reminded of Simmons et. al's recommendations for describing results. We must be more disciplined and stringent about the sensitivity of our results to somewhat arbitrary choices, and more forgiving to less-than-perfect results when reviewing.

I certainly don't think that these results indicate that we should all give up, abandoning all fMRI analysis. But we should be even more skeptical about our results. Do they only appear in one 'magic' pipeline? Or do they more-or-less hold over perturbations in thresholds and processing?

Carp, J. (2012). On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments Frontiers in Neuroscience, 6 DOI: 10.3389/fnins.2012.00149

Ah, I see that Neuroskeptic also commented on this paper.