Check Exchangeability Before a Permutation Test

State what is allowed to move

A permutation test compares an observed statistic with statistics calculated under allowed rearrangements. Its validity depends on why those rearrangements represent the null hypothesis or randomization design. “Nonparametric” does not mean that every dataset can be shuffled freely.

The SciPy permutation-test reference distinguishes independent-sample, paired-sample and pairing permutations. Choose the structure that matches the question. Shuffling individual observations across clusters or breaking matched pairs can create a reference distribution for a design you did not conduct.

Map the design to the algorithm

Record the randomized or exchangeable unit, any blocks or strata, the test statistic and the alternative direction. For a randomized experiment, preserve the actual allocation restrictions. For observational comparisons, explain the assumptions that would make labels exchangeable under the proposed null.

Consider whether unequal distribution shapes or variances undermine a simple label-permutation interpretation. Testing equality of full distributions is not identical to testing only equal means. A carefully chosen studentized statistic may address a different inferential target, but that choice needs a justified method rather than an informal claim of assumption-free analysis.

Count a hypothetical permutation space

Suppose six independent observations were assigned to two groups of three with no additional allocation restrictions. There are 6 choose 3 = 20 distinct group assignments. Enumerating all 20 produces an exact reference distribution for the specified assignment mechanism and statistic.

If the same six observations instead form three matched pairs, unrestricted reassignment into groups of three ignores the pairs. A within-pair swap scheme has 2³ = 8 possible swap patterns. The different counts reflect different designs; selecting the larger permutation space is not automatically more accurate.

Preserve computation and interpretation

State whether all allowed permutations were enumerated or a random subset was sampled. For Monte Carlo permutations, record the generator, number of draws and p-value convention, including handling of ties and the observed arrangement. A reported zero from a finite random sample should trigger inspection of the calculation.

Use a tiny synthetic dataset to enumerate the allowed rearrangements manually and compare the software result. Then retain the statistic and permutation settings with the analysis. The final interpretation should name the tested null and the design restrictions, avoiding the broad claim that a permutation p-value is valid simply because it does not rely on a normal approximation.

Source verification: 6 November 2026. Examples are hypothetical.