Very interesting and helpful summary of how the general idea of permutation statistics work! It seems like the Blair & Karniski, (1993) tMAX correction seems to be used an awful lot in neurophysiological data. I think I would recommend really lloking at the Maris and Oostenveld (2007) paper regarding permutation statistics of EEG data.
Dear Jeanette, thank you so much for these great videos. Could you please provide a bit of explanation for permutation for correlation ? or any tips to move forward ? thanks
Maybe this blog post will have some useful links: www.ohbmbrainmappingblog.com/blog/a-brief-overview-of-permutation-testing-with-examples Maybe this paper (I haven't read it, but it seems general and Anderson Winkler's papers are generally great and useful) pubmed.ncbi.nlm.nih.gov/24530839/
@@mumfordbrainstats Dear Jeanette, thanks for the videos. Based on these papers, it seems that PALM is the recommended option for testing correlations at the group level. Yet, is it still possible to use randomise? and If it is, how?
@@cesarcaballero7617 ranomise and PALM likely have huge overlap. PALM is great if you have special data structures or mutiple modalities. For just one set of data with independent subjects, randomise is good. Some settings where you have related subjects can still be dealt with in randomise, I believe.
You permute the data and compute the t-statistics in the usual way. Is that your question? Then you compare your unpermuted t-stat to the permuted t-stat distribution to get a p-value
Thanks! I was just confused about the t-value because I thought you'd compare it to a t-distribution and then use that value, but you can just calculate the t-value like normal, I didn't think about that :D
Hi! Thanks for the video, super useful! Could you explain why you are shuffling the labels and not the observation data? And is the fact that you are shuffling the labels why you get only 20 possible permutations instead of 6! if you had shuffled the observations? Thanks a lot!
The order of one of the vectors. So, for example, if you have cor(X,Y), then in each step permute the elements of X (let's call this permuted version of X X_p) and then recompute cor(X_p,Y). Of course this would also work by permuting Y instead of X, since the idea is to break the connection between X and Y.
What about small sample sizes? Do I need a certain number of observations to justify permutation tests or will just the power decrease? Thank you for the approachable video!
The number of possible unique permutations is limited by small sample sizes. For example, if you only have 5subjects and you're running a 1-sample t-test, permutation are done by sign flipping and only 2^5 = 32 flips are possible, so the smallest p-value possible is 1/32 = .03125. I think randomise tells you how many unique permutations are possible? It has been a while since I've run it.
OK, so the number of possible permutations is my bottleneck and when I can compute with all of them I'll get an exact p value, when I need to sample from them an approximated p value. That seems to make sense. Again, thank you very much. Any chances of further videos on permutation tests (like chi squared, more groups, time series?) from you?
A great video! Thank you! Can I ask whether using one-tailed t-test, FDR-corrected can replace a permutation test with N=32 for analysing EEG data? In other words, can using a t-test while conducting univariate and multivariate EEG analysis methods be appropriate?
The permutation test (just a vanilla permutation test) doesn't control for multiple comparisons. Only when you combine with the max T-stat to build your null distribution. This is a family-wise error correction. To reiterate. A permutation test alone just addresses any divergence from the normality assumption. Combine with max t and you now have a multiple comparison correction that doesn't assume normality. FDR is a different type of multiple comparison correction and I'm not completely familiar with EEG data, but with fMRI data there are voxelwise FDR corrections and cluster-based FDR corrections and there are reasons why the cluster-based make more sense (do we want a "false discovery" to be on a voxel level or cluster level? Cluster level makes more sense). I'm unsure if there's the same issue with EEG data. Either way FDR correction is an alternative way to go.
What would of been good is if you gave more explanation of how you calculated the mean difference statistics, instead of skipping over that important detail
Sorry for any confusion, are you referring to the cartoon example? You simply calculate the standard t-statistic using the swapped labels. Hope that helps!
Oh GOD! I love your voice. Thanks for the explanation
Very interesting and helpful summary of how the general idea of permutation statistics work! It seems like the Blair & Karniski, (1993) tMAX correction seems to be used an awful lot in neurophysiological data. I think I would recommend really lloking at the Maris and Oostenveld (2007) paper regarding permutation statistics of EEG data.
Dear Jeanette, thank you so much for these great videos. Could you please provide a bit of explanation for permutation for correlation ? or any tips to move forward ? thanks
Maybe this blog post will have some useful links: www.ohbmbrainmappingblog.com/blog/a-brief-overview-of-permutation-testing-with-examples
Maybe this paper (I haven't read it, but it seems general and Anderson Winkler's papers are generally great and useful) pubmed.ncbi.nlm.nih.gov/24530839/
@@mumfordbrainstats Dear Jeanette, thanks for the videos. Based on these papers, it seems that PALM is the recommended option for testing correlations at the group level. Yet, is it still possible to use randomise? and If it is, how?
@@cesarcaballero7617 ranomise and PALM likely have huge overlap. PALM is great if you have special data structures or mutiple modalities. For just one set of data with independent subjects, randomise is good. Some settings where you have related subjects can still be dealt with in randomise, I believe.
jk it's in the next vid
How do I get from the Permutations to the t-statistics? That's the only step that is still missing in my understanding of this test.
You permute the data and compute the t-statistics in the usual way. Is that your question? Then you compare your unpermuted t-stat to the permuted t-stat distribution to get a p-value
Thanks! I was just confused about the t-value because I thought you'd compare it to a t-distribution and then use that value, but you can just calculate the t-value like normal, I didn't think about that :D
Hi! Thanks for the video, super useful! Could you explain why you are shuffling the labels and not the observation data? And is the fact that you are shuffling the labels why you get only 20 possible permutations instead of 6! if you had shuffled the observations? Thanks a lot!
can I get the answer for what's permuted in the case of correlation...very much a cliffhanger
The order of one of the vectors. So, for example, if you have cor(X,Y), then in each step permute the elements of X (let's call this permuted version of X X_p) and then recompute cor(X_p,Y).
Of course this would also work by permuting Y instead of X, since the idea is to break the connection between X and Y.
What about small sample sizes? Do I need a certain number of observations to justify permutation tests or will just the power decrease?
Thank you for the approachable video!
The number of possible unique permutations is limited by small sample sizes. For example, if you only have 5subjects and you're running a 1-sample t-test, permutation are done by sign flipping and only 2^5 = 32 flips are possible, so the smallest p-value possible is 1/32 = .03125. I think randomise tells you how many unique permutations are possible? It has been a while since I've run it.
So you'd hope to have 5000 iterations or so....maybe 1000 would do in a pinch.
OK, so the number of possible permutations is my bottleneck and when I can compute with all of them I'll get an exact p value, when I need to sample from them an approximated p value.
That seems to make sense. Again, thank you very much. Any chances of further videos on permutation tests (like chi squared, more groups, time series?) from you?
A great video! Thank you! Can I ask whether using one-tailed t-test, FDR-corrected can replace a permutation test with N=32 for analysing EEG data? In other words, can using a t-test while conducting univariate and multivariate EEG analysis methods be appropriate?
The permutation test (just a vanilla permutation test) doesn't control for multiple comparisons. Only when you combine with the max T-stat to build your null distribution. This is a family-wise error correction. To reiterate. A permutation test alone just addresses any divergence from the normality assumption. Combine with max t and you now have a multiple comparison correction that doesn't assume normality. FDR is a different type of multiple comparison correction and I'm not completely familiar with EEG data, but with fMRI data there are voxelwise FDR corrections and cluster-based FDR corrections and there are reasons why the cluster-based make more sense (do we want a "false discovery" to be on a voxel level or cluster level? Cluster level makes more sense). I'm unsure if there's the same issue with EEG data. Either way FDR correction is an alternative way to go.
What would of been good is if you gave more explanation of how you calculated the mean difference statistics, instead of skipping over that important detail
Sorry for any confusion, are you referring to the cartoon example? You simply calculate the standard t-statistic using the swapped labels. Hope that helps!