Concept Architecture
Concept
Theoretically, the Permutation Test is a non-parametric hypothesis test that evaluates whether an observed test statistic could reasonably have arisen under the null hypothesis by considering all possible rearrangements of the observed data. It is founded on randomisation theory and exact statistical inference, requiring minimal distributional assumptions. The test exists to provide valid significance testing when conventional parametric assumptions such as normality may not be satisfied.
Mathematically, the permutation test constructs the null distribution of a chosen test statistic by repeatedly permuting group labels while keeping the observed data values fixed. The p-value is estimated as the proportion of permutations producing a test statistic at least as extreme as the observed statistic. When all possible permutations are evaluated, the test provides an exact significance level; otherwise, Monte Carlo sampling provides an approximation.
In practice, permutation tests are implemented by calculating the observed test statistic, generating a large number of random permutations and comparing the observed value with the empirical permutation distribution. In health economics, permutation tests are applied to compare costs, quality-adjusted life-years, utilities and other outcomes when sample sizes are small or distributional assumptions are uncertain.
Purpose
Used to perform distribution-free hypothesis testing by comparing an observed test statistic with its permutation distribution, providing robust statistical inference when parametric assumptions are inappropriate.
Mathematical Formulae
Primary Formula
p = (1/B) ? ? I(T? � T???)
where:
p = estimated permutation p-value
B = number of permutations
T??? = observed test statistic
T? = test statistic from permutation i
I(�) = indicator function
Supporting Formulae
T = X?? ? X??
Number of possible permutations:
N = n! / (n?!n?!)
where:
n = total sample size
n?, n? = group sample sizes
Related Mathematical Methods
- Randomisation Test
- Monte Carlo Simulation
- Bootstrap
- Exact Test
- Non-Parametric Estimation
- Hypothesis Testing
Example
A study compares mean healthcare costs for two treatment groups containing 20 patients each. The observed difference in mean costs is �1,250. Group labels are randomly permuted 10,000 times, producing an empirical null distribution. In 184 permutations, the simulated difference equals or exceeds the observed value.
The estimated p-value is:
p = 184 � 10,000 = 0.0184
This provides evidence that the observed cost difference is unlikely to have occurred by chance under the null hypothesis.
Excel Implementation
| Function | Example Formula | Health Economics Application |
|---|---|---|
| RAND | =RAND() | Generate random values for permuting observations. |
| SORTBY | =SORTBY(A2:B41,C2:C41) | Randomly reorder treatment labels. |
| AVERAGE | =AVERAGE(D2:D21)-AVERAGE(D22:D41) | Calculate the permutation test statistic. |
| COUNTIF | =COUNTIF(E2:E10001,">="&E1)/10000 | Estimate the permutation p-value. |
VBA (Optional)
Automate large-scale permutation testing, generate empirical null distributions and calculate exact or Monte Carlo p-values.
Sources
- Good P. Permutation, Parametric and Bootstrap Tests of Hypotheses.
- Ernst MD. Permutation Methods: A Basis for Exact Inference. Statistical Science. 2004.
- Pesarin F, Salmaso L. Permutation Tests for Complex Data.
- Efron B, Tibshirani RJ. An Introduction to the Bootstrap.
- Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation.
- NICE. Health Technology Evaluation Manual.
Related Concepts (3)
Library
Publications
1
NICE DSU Technical Support Document 21: Flexible methods for survival analysis — Rutherford, Lambert, Sweeting, Pennington, Crowther, Abrams & Latimer, TSD 21 ed., 2020 (NICE Decision Support Unit (University of Sheffield))
Guidance extending standard survival analysis to flexible parametric methods — spline-based models, fractional polynomials, mixture and cure models, and relative-survival approaches — for capturing complex hazard functions in economic evaluation.
Frequently Asked Questions (6)
What is a permutation test?
A non-parametric hypothesis test determining significance by repeatedly rearranging observed data labels to build an empirical distribution under the null hypothesis.
Source: Fisher 1935
Why does a permutation test make few distributional assumptions?
A permutation test builds its reference distribution from the data themselves rather than from an assumed theoretical distribution. Under the hypothesis of no effect, the group labels are exchangeable, so repeatedly shuffling them and recomputing the statistic shows what values would arise by chance alone. The observed statistic is then compared against this shuffled distribution to obtain a probability. Because it relies on rearrangement rather than a formula, it needs few assumptions about how the data are distributed. Good (2005) describes the method.
Source: Good 2005
How does a permutation test work?
A permutation test works by computing a test statistic for the observed data, then repeatedly rearranging, or permuting, the group labels or values under the null hypothesis of no effect, recomputing the statistic for each permutation to build its distribution under the null. The proportion of permuted statistics as extreme as or more extreme than the observed gives the p-value. This uses the data themselves to generate the null distribution, without assuming a particular parametric form for the statistic.
Source: Fisher 1935
Why use a permutation test?
A permutation test is used because it makes minimal assumptions about the distribution of the data, deriving the null distribution empirically by rearranging the data, so it is valid without relying on parametric assumptions that may not hold. It is useful for small samples, complex statistics, or situations where the distribution of the test statistic is unknown or hard to derive analytically. This robustness and generality make permutation tests valuable where standard parametric tests' assumptions are questionable.
Source: Neyman & Pearson 1933
What is the logic of a permutation test?
The logic of a permutation test is that, under the null hypothesis of no difference between groups, the group labels are exchangeable, so any rearrangement of them is equally likely to have produced the data. By permuting the labels and recomputing the statistic, the test generates the distribution of the statistic expected under the null. If the observed statistic falls in the extreme tail of this distribution, it is unlikely under the null, providing evidence against it. This exchangeability under the null underlies the test's validity.
Source: Fisher 1935
What are the limitations of permutation tests?
Permutation tests can be computationally intensive, since building the null distribution may require many permutations, though this is less of a barrier with modern computing, and for large samples the number of possible permutations is enormous, so a random sample of them is used. They rely on the exchangeability of the data under the null, which must hold for validity, and they test a specific null hypothesis. These limitations mean permutation tests are applied where their assumptions hold and computation is feasible.
Source: Fisher 1935
Trust Record
Verified by Dr Darrin Baines
British health economist
Professional identity: darrinbaines.org
Verification date: 23 Oct 2025
Content version: 1.0.0
Canonical Identity
- Persistent URI
- https://healtheconomics.wiki/concept/permutation-test
- Term code
- HE-EM-SM-064
Stable URI · Machine-readable · Resolvable · CC BY 4.0