Monday, September 17, 2012

permutation testing: opening thoughts

This is the first of what will probably be a continuing series of musing, meandering posts about permutation testing for group fMRI MVPA.

I'm a big fan of permutation testing, especially for MVPA: our questions, analyses, sample sizes, distributions, etc. are so far from what was envisioned for parametric stats that using a "standard" parametric test "out of the box" gives me pause (or, in practice, lots of assumption-checking). I like that with permutation tests it's possible to design a test that gets at exactly the hypothesis we're asking, while keeping the relevant variance structures (e.g. how many runs, type of classifier, number of examples in each cross-validation fold) intact.

But, the exact way two different people design a permutation test for any particular question will probably be different. And this gets worse in group analyses.

I'll be writing a series of posts illustrating some of the common ways permutation tests are done, both single subject and group analyses. My intent is that these posts will highlight and explain how to carry out what I think are the most useful and common methods, but also share my perception of where some of the open questions lie. As always, please feel free to comment, especially pointing out where your practice/observation/experience/conclusions differ from mine.

1 comment:

  1. Are there (preferably published) studies that use the permutation test for ROI based stimulation classification group analysis (not mapping)? I would be especially interested in a comparison to t-test results. One of the examples of Golland2003 shows what to do on group-level, but in that case we are talking about a diagnosis task with patients as examples not voxels so subject classification instead of stimulus classification... Yang2012 is mentioning two studies, that use the permutation test for stimulation classification but there it is the searchlight maps that were assessed by nonparametric statistical
    tests... From your statement 'And this gets worse in group analyses.' I infer, that there are at least tentative applications of the permutation test at group-level!? Thanks