Statistical Significance

A judgement that an observed result is unlikely to have occurred by chance under the null hypothesis, based on a p-value below a threshold.

Source: Fisher 1925

Statistical significance establishes only that an observed result would be unlikely to arise by chance if the null hypothesis of no effect were true, judged by a p-value falling below a chosen threshold. It says the result is hard to dismiss as a fluke, but it does not say the effect is large or clinically important, since a huge study can render a trivial difference significant. Nor does it prove the effect is real. Unlikeliness under chance, rather than importance, is what it captures. Kirkwood and Sterne (2003) describe this.

Source: Kirkwood & Sterne 2003

Statistical significance is determined by computing the p-value, the probability of a result as extreme as the one observed if the null hypothesis were true, and comparing it with the significance level: if the p-value is below the threshold, the result is declared statistically significant. So statistical significance is determined by whether the p-value falls below the chosen significance level, which provides the decision rule, and this is equivalent to the test statistic falling in the rejection region, meaning a significant result is one where the data are sufficiently improbable under the null hypothesis to reject it at the set level.

Source: Fisher 1925

Statistical significance does not mean an effect is important or large, since it depends on the sample size as well as the effect size, so a large study can find a trivial effect statistically significant, and a small study can miss a meaningful one. So statistical significance is distinct from practical importance, which is why a significant result should not be assumed to be clinically or practically meaningful, and effect sizes and confidence intervals are considered alongside significance, since statistical significance indicates only that an effect is unlikely to be due to chance, not that it is large enough to matter.

Source: Fisher 1925

The limitations of statistical significance include its dependence on sample size, so significance does not indicate importance; the arbitrariness of the threshold, creating a false dichotomy between significant and non-significant; frequent misinterpretation, such as treating a non-significant result as proof of no effect; and the neglect of effect size. So statistical significance is used with awareness of these limitations, which is why it is interpreted alongside effect sizes and confidence intervals rather than as a standalone verdict, since over-reliance on significance can mislead, and a fuller account of results considers the magnitude and precision of effects, not merely whether they cross the significance threshold.

Source: Fisher 1925

Statistical significance should be interpreted as indicating that an observed result is unlikely under the null hypothesis, providing evidence against it, but not as a measure of the size or importance of an effect or the probability that the null is true. So statistical significance should be interpreted together with effect sizes and confidence intervals, which convey the magnitude and precision, since significance alone tells only whether an effect is detectable, not whether it matters, and interpreting it in isolation, or treating the threshold as a sharp divide, can mislead, which is why a careful interpretation considers the evidence, the effect size, and the context together.

Source: Fisher 1925