Significance Level

Significance Level: Concept Architecture

Orientation and learning roadmap

This page follows the decision rule from a prespecified hypothesis and test statistic to a conclusion about statistical evidence. It then examines error control, interpretation, and why a significance threshold alone cannot settle a health economic decision.

Hypothesis test and error rate

State the null hypothesis, the alternative, the test statistic, and whether the test is one-sided or two-sided before choosing the decision threshold. The significance level is a property of the testing procedure under the specified null model, rather than a measured probability that the null hypothesis is true.

For a rejection region $R$ and a simple null hypothesis $H_0$, the false-rejection probability is:

$$P_{H_0}(T\in R)=\alpha$$

for a test calibrated at level $\alpha$; for a composite null, a level-$\alpha$ test controls the supremum of that probability across the null parameter values at no more than $\alpha$. Under the model and sampling assumptions, repeated use of a valid test with $\alpha=0.05$ would falsely reject a true null in at most 5% of such repetitions. That statement does not give a 5% probability that this study's rejected null is actually true.

From p-value to conclusion

The p-value measures how incompatible the observed result, or one more extreme by the test's ordering, is with the specified null model. A conventional decision rule rejects the null when the valid p-value is at or below the prespecified significance level, with exact boundary conventions stated when they matter.

For example, if a two-sided comparison has $p=0.03$ and the prespecified $\alpha=0.05$, it meets that test's statistical-significance criterion. If $p=0.07$, it does not meet the criterion; this is not proof of no effect, and the estimated effect and its uncertainty still matter. Reporting the estimate and a compatible interval is more informative than reporting only whether a threshold was crossed.

Choosing the level and managing error

The level is usually set in advance in view of the consequences of a false positive, study purpose, multiplicity, and applicable study protocol or regulatory requirements. A smaller significance level can reduce false-positive rejections under the null, but with sample size held fixed it often reduces power to detect a specified real effect.

When many hypotheses, outcomes, subgroups, or interim looks are tested, a collection of unadjusted 5% tests does not generally control the probability of at least one false rejection at 5%. Prespecified multiplicity procedures may target family-wise error or a different error criterion, depending on the question. Changing the threshold after seeing the results undermines its stated repeated-sampling guarantee.

Distinctions that prevent misinterpretation

The significance level is a chosen rule, while the p-value is computed from the observed data and the null model. Statistical significance is neither the magnitude of a clinical effect nor the probability that an intervention is cost-effective.

The numerical link between a two-sided test at 5% and a corresponding 95% confidence interval excluding the null depends on using compatible procedures and assumptions. It should not be imposed on arbitrary adjusted intervals, one-sided tests, or different estimands.

Health economic decision use

A trial's significance level may govern tests of clinical outcomes that supply an economic model, but economic decisions compare expected costs and health consequences under uncertainty. A non-significant treatment-effect test does not justify setting the effect to zero; nor does a significant result prove that the intervention provides sufficient value for its cost.

Economic evaluation should propagate uncertainty in relevant inputs, examine decision uncertainty, and report the size and direction of costs and health outcomes. The choice of comparator, perspective, opportunity cost, and consequences of a wrong decision remain essential even when a study uses a conventional hypothesis-testing threshold.

Evidence base

The repeated-sampling definition and test mechanics are set out in the NIST statistical testing handbook. The American Statistical Association statement on p-values cautions against equating significance with effect size or importance, while NICE guidance on economic evaluation discusses incorporating uncertainty into cost-effectiveness estimates.