Statistical Significance

Statistical Significance: Concept Architecture

Orientation and learning roadmap

Statistical significance is a decision rule within hypothesis testing, not a measure of effect size, practical importance, truth, or reproducibility. The sections below connect null hypotheses, test statistics, p-values, significance levels, confidence intervals, power, and multiplicity, then show how statistical evidence should be interpreted alongside clinical and economic consequences.

What a significance conclusion actually says

A statistical test begins with a null hypothesis and a probability model describing how data would behave under specified assumptions. A result is called statistically significant when its p-value is at or below a significance level chosen before examining the result. The conclusion is therefore conditional on the hypothesis, test, model, data quality, and analysis plan.

• Statistical significance indicates that the observed data are relatively incompatible with the tested null model at the chosen threshold.
• Statistical significance does not give the probability that the null hypothesis is true.
• Statistical significance does not give the probability that the result occurred “by chance.”
• Statistical significance does not measure the magnitude or importance of an effect.
• Statistical significance does not guarantee replication in a new study.
• Statistical significance does not correct bias, confounding, selective reporting, measurement error, or model misspecification.

The binary label compresses continuous evidence into two categories. Results just above and just below a threshold are usually much more alike than the labels “non-significant” and “significant” suggest.

The null hypothesis and alternative hypothesis

The null hypothesis (H_0) specifies the value or model being tested, often no difference or no association. The alternative hypothesis (H_1) describes departures considered by the test. A two-sided alternative allows departures in either direction, while a one-sided alternative addresses a prespecified direction and should not be chosen after seeing the data.

For a treatment-effect parameter (\theta), a common two-sided test is:

[ H_0:\theta=0 \qquad\text{versus}\qquad H_1:\theta\neq0 ]

The null value need not be zero. Equivalence and non-inferiority questions test hypotheses involving prespecified margins, and their logic differs from a conventional superiority test.

How a test statistic standardizes the result

A test statistic compares the estimated effect with its null value relative to the estimate's sampling uncertainty. Under a large-sample normal approximation, a standardized statistic can be written as the estimated departure from the null divided by its standard error. Larger absolute values indicate greater incompatibility with the null model when the assumptions hold.

[ z=\frac{\hat\theta-\theta_0}{SE(\hat\theta)} ]

The reference distribution depends on the test. Some procedures use normal, t, chi-squared, F, permutation, randomization, exact, or bootstrap distributions. Applying the wrong reference distribution can make the reported p-value invalid even when the arithmetic is correct.

What the p-value means

The p-value is calculated under the null hypothesis and the other assumptions in the statistical model. For a two-sided test, it represents the probability of obtaining a test statistic at least as incompatible with the null as the observed statistic, in either direction, if the null model generated the data. It is a property of the data and analysis under that model, not a posterior probability assigned to the hypothesis.

Using a standard-normal reference distribution with cumulative distribution function (\Phi):

[ p=2\left[1-\Phi\left(|z_{obs}|\right)\right] ]

A small p-value can arise from a large effect, a precise estimate, a large sample, or a combination of these. A large p-value can arise because the null is compatible with the data, but it can also reflect imprecision, low power, noisy measurement, poor adherence, or an inappropriate model.

The significance level controls a long-run error rate

The significance level (\alpha) is the prespecified probability of rejecting the null hypothesis when it is true under repeated valid use of the testing procedure. A conventional value such as 0.05 is a design choice, not a natural boundary separating real from unreal effects. The threshold should reflect the testing context, consequences, multiplicity, and analysis plan.

The rule for a conventional p-value test is:

[ \text{Reject }H_0\text{ when }p\leq\alpha ]

The statement that (\alpha=0.05) controls a 5% Type I error rate applies across repeated studies satisfying the test assumptions when the null is true. It does not mean that 5% of statistically significant results in a particular literature are false, because that proportion also depends on the hypotheses tested, power, bias, selection, and prior plausibility.

Type I error, Type II error, and power

Hypothesis testing allows two kinds of decision error. A Type I error rejects a true null hypothesis, while a Type II error fails to reject a false null hypothesis for a specified alternative. Power is the probability that the test rejects the null when the specified alternative is true.

Power depends on the true effect size, sample size, outcome variability, allocation, missing data, adherence, significance level, and test. It is not a fixed property of a study independent of the alternative effect considered.

Failure to reach significance is not evidence of no effect

A non-significant result means the analysis did not cross the chosen rejection threshold. It does not prove that the null is true or that the treatments are equivalent. The confidence interval may still include effects that are clinically important in either direction.

Claims of equivalence or non-inferiority require designs and analyses built around prespecified margins. A superiority study with (p>0.05) cannot be converted into an equivalence conclusion merely because it failed to detect a difference. The relevant interval must exclude effects outside the justified equivalence or non-inferiority range.

Confidence intervals show magnitude and precision

A confidence interval presents a range of parameter values that remain compatible with the data under the procedure and its assumptions. For a symmetric large-sample 95% interval, the estimate is extended approximately 1.96 standard errors in each direction. The interval communicates effect magnitude and precision more directly than a significance label.

[ 95%\ CI=\hat\theta\pm1.96\times SE(\hat\theta) ]

For matching two-sided procedures, a 95% confidence interval that excludes the null value corresponds to a p-value below 0.05. This duality can fail when the test and interval use different methods, adjustments, models, or confidence levels, so apparent discrepancies should be investigated rather than edited away.

Worked example with a QALY difference

Suppose a study estimates that a new intervention produces 0.030 additional quality-adjusted life-years per patient compared with usual care, with a standard error of 0.014. A two-sided large-sample test assesses the null hypothesis of no QALY difference. The example illustrates statistical interpretation and does not by itself establish cost effectiveness.

The standardized statistic is:

[ z=\frac{0.030-0}{0.014}\approx2.143 ]

The corresponding two-sided normal-approximation p-value is approximately 0.032. The 95% confidence interval is:

[ 0.030\pm1.96(0.014) =0.030\pm0.02744 =(0.00256,\ 0.05744) ]

The estimated QALY gain is statistically significant at (\alpha=0.05) because the p-value is below 0.05 and the matching interval excludes zero. The interval is more informative than the label because it shows that effects from a very small gain to a materially larger gain remain compatible with the data and model.

Statistical significance is not clinical importance

Clinical importance concerns whether the magnitude of an effect matters to patients, clinicians, or population health. A very small effect can be statistically significant in a large or precise study, while an important effect can be non-significant in a small or noisy study. Interpretation should compare the entire confidence interval with a clinically justified threshold or range.

A minimally important difference should be defined using substantive evidence rather than selected to match the result. If the interval spans effects below and above that threshold, the evidence is uncertain about clinical importance even when the null value has been excluded.

Statistical significance is not economic value

Health-economic decisions concern expected costs, health outcomes, opportunity costs, uncertainty, and the decision maker's objective. Separate significance tests for costs and effects do not determine whether an intervention is cost effective. The joint distribution of incremental costs and outcomes matters.

Using the worked QALY estimate, suppose incremental cost is $4,000 and the decision threshold is$100,000 per QALY. Incremental net monetary benefit is:

[ INMB=\lambda\Delta E-\Delta C ]

[ INMB=(100{,}000)(0.030)-4{,}000=-1{,}000 ]

The illustrative intervention has a statistically significant estimated QALY gain but a negative point estimate of incremental net monetary benefit at the stated threshold. This is not a contradiction: the significance test and the economic decision rule answer different questions. A complete analysis would estimate expected net benefit and its joint uncertainty rather than decide from either point estimate alone.

Sample size changes the chance of significance

Standard errors generally decrease as information increases, so the same estimated effect can be statistically significant in a large study and non-significant in a small study. This dependence is one reason that p-values cannot measure effect importance. Sample-size planning should target a meaningful estimand, plausible effect, desired precision or power, and anticipated missing data rather than merely seek a threshold-crossing result.

For many simple estimators, standard error decreases approximately with the square root of sample size:

[ SE\propto\frac{1}{\sqrt n} ]

Doubling the sample size therefore does not usually halve the standard error; approximately four times the sample size is needed for that change, all else equal. Complex designs, clustering, unequal allocation, repeated measures, censoring, and survey weights alter the information calculation.

Multiple testing raises false-positive risk

When many hypotheses are tested, the probability of at least one small p-value under true null hypotheses can become much larger than the nominal level. Multiplicity can arise from multiple outcomes, time points, subgroups, treatment arms, model specifications, interim analyses, or repeated looks at the data. Selective reporting compounds the problem by hiding the number of opportunities to find a favorable result.

If (m) independent null hypotheses are each tested at level (\alpha), the probability of at least one Type I error is:

[ 1-(1-\alpha)^m ]

For 20 independent tests at (\alpha=0.05), this probability is:

[ 1-(0.95)^{20}\approx0.642 ]

Prespecification and a clear outcome hierarchy are the first safeguards. Family-wise error procedures, false-discovery-rate procedures, hierarchical testing, gatekeeping, or model-based approaches may be appropriate depending on the inferential objective and dependence among tests.

Subgroup significance comparisons are often wrong

Evidence that an effect is statistically significant in one subgroup and non-significant in another does not show that the subgroup effects differ. The correct question is whether the difference between subgroup effects is supported, usually through an interaction test or an appropriate hierarchical model. Low power and multiple subgroup analyses make isolated findings especially unstable.

Subgroups should be prespecified, clinically credible, limited in number, and interpreted with interaction estimates and confidence intervals. Post hoc categories created after inspecting the data require explicit labeling and cautious interpretation.

Model assumptions determine whether the test is valid

Every p-value is conditional on assumptions about the study design, outcome distribution, dependence, variance, missing data, censoring, model specification, and analysis process. Randomization supports particular tests only when assignment and analysis are handled correctly. Large samples do not eliminate bias from confounding, poor measurement, selection, or informative missingness.

Important checks include:

• whether observations are independent or clustered;
• whether standard errors reflect repeated measures, matching, sites, or survey design;
• whether the outcome model and link function are appropriate;
• whether variance assumptions and small-sample corrections are justified;
• whether censoring and missing-data assumptions are credible;
• whether the analysis follows the randomization or sampling design;
• whether transformations, exclusions, covariates, and stopping rules were prespecified;
• whether numerical convergence and test reference distributions are valid.

A precisely calculated p-value from a misspecified analysis can be precisely misleading.

P-hacking, selective reporting, and publication bias

Repeatedly changing outcomes, exclusions, covariates, subgroups, transformations, or stopping points until a threshold is crossed invalidates the nominal interpretation of the p-value. Selective publication of significant results then distorts the visible evidence base. Transparency about the full analysis process is essential.

Prospective registration, protocol publication, prespecified analysis plans, complete outcome reporting, reproducible code, and disclosure of exploratory analyses help protect inference. Exploratory work is legitimate when labeled honestly and followed by independent confirmation rather than presented as a prespecified test.

Thresholds should not replace evidence appraisal

A fixed threshold offers a consistent decision rule for a specified testing procedure, but it should not divide results into “real” and “not real.” A p-value of 0.049 and one of 0.051 provide nearly the same degree of compatibility information under otherwise identical analyses. Study quality, prior evidence, effect magnitude, precision, multiplicity, and consequences remain relevant on both sides of the threshold.

P-values should usually be reported as actual values to sensible precision rather than only as (p<0.05) or “NS.” Very small values can be reported with a justified bound when software precision or journal conventions require it. Excessive digits should not imply accuracy beyond the design and data.

Bayesian evidence answers a different probability question

Bayesian methods combine a likelihood with a prior distribution to produce a posterior distribution for parameters or hypotheses. A posterior probability can address questions such as the probability that an effect exceeds a decision-relevant threshold, but its interpretation depends on the prior, likelihood, and model. It is not obtained by subtracting a p-value from one.

Frequentist and Bayesian approaches can both be rigorous when their assumptions and targets are clear. Replacing a p-value with a Bayesian quantity does not remove the need to assess bias, design quality, multiplicity, model fit, and decision consequences.

A disciplined testing and reporting workflow

Sound significance testing begins before data analysis. The hypothesis, estimand, direction, threshold, multiplicity strategy, and statistical model should be linked to the scientific and decision question. Reporting should preserve the effect estimate and uncertainty rather than foreground a binary label.

1. Define the estimand. State the population, treatment or exposure, comparator, outcome, time point, and summary measure.
2. Specify the hypotheses. Record the null value, alternative, direction, and any non-inferiority or equivalence margin.
3. Choose the error criterion. Prespecify (\alpha), desired power or precision, and the family of tests requiring multiplicity control.
4. Design the study. Align randomization, sampling, clustering, follow-up, outcome measurement, and sample size with the estimand.
5. Prespecify the analysis. Define exclusions, covariates, transformations, missing-data methods, interim analyses, and sensitivity checks.
6. Verify data and code. Check outcome derivation, group coding, units, denominators, test direction, standard errors, and software convergence.
7. Estimate the effect. Report the point estimate in interpretable units with a confidence interval or other uncertainty summary.
8. Calculate the test result. Use the correct reference distribution and apply all planned multiplicity adjustments.
9. Evaluate assumptions. Examine model diagnostics, missingness, influential data, design deviations, and robustness.
10. Interpret in context. Consider magnitude, precision, clinical importance, economic value, prior evidence, and potential bias.
11. Report completely. Present all prespecified outcomes and distinguish confirmatory from exploratory analyses.

Excel implementation for a simple z-test

An auditable worksheet can reproduce a simple large-sample test when the estimate, null value, standard error, tail choice, and assumptions are explicit. It should not be used to replace specialized methods for clustered, censored, longitudinal, non-normal, or multiply tested data. The worksheet must preserve the sign of the effect even though a two-sided p-value uses the absolute test statistic.

If B2 contains the estimate, B3 the null value, and B4 the standard error:

=(B2-B3)/B4

calculates the z-statistic, and:

=2*(1-NORM.S.DIST(ABS(B5),TRUE))

calculates the two-sided p-value. A 95% confidence interval can be calculated with =B2-1.96*B4 and =B2+1.96*B4. The worksheet should flag nonpositive standard errors and should not allow the tail direction or significance level to change after the result is known.

Validation checks before interpretation

Validation should confirm both calculation and meaning. Many errors arise not in the final formula but in the outcome definition, contrast direction, standard error, analysis population, or number of tests attempted. Independent reproduction is especially important when a threshold affects publication, regulation, or policy.

• Confirm that the tested hypothesis matches the stated research question and estimand.
• Confirm that the null value, contrast direction, tail choice, and significance level were prespecified.
• Recalculate the test statistic, p-value, and confidence interval from stored estimates and standard errors.
• Confirm that clustering, stratification, weighting, repeated measures, censoring, and missingness are handled appropriately.
• Reconcile the analysis population and outcome count with the study flow and source data.
• Verify all multiplicity adjustments and the full number of outcomes, subgroups, time points, and analyses examined.
• Compare the result with effect-size thresholds for clinical and economic importance.
• Inspect sensitivity analyses and alternative defensible models rather than reporting only the most favorable specification.
• Report deviations, exclusions, interim looks, and exploratory analyses transparently.

Common errors and safeguards

The most damaging error is to treat a threshold as a substitute for reasoning. Statistical significance can contribute to evidence appraisal when the test is well designed and correctly interpreted, but it cannot validate the study or decide whether an effect matters. Effect estimates, intervals, design quality, and context must remain primary.

• Do not interpret (p=0.03) as a 3% probability that the null hypothesis is true.
• Do not interpret (1-p) as the probability that the alternative hypothesis is true.
• Do not interpret a non-significant result as proof of no difference.
• Do not infer a subgroup difference from significance in one subgroup and non-significance in another.
• Do not equate statistical significance with clinical importance, cost effectiveness, or policy relevance.
• Do not compare p-values as though the smaller value necessarily represents the larger effect.
• Do not change a two-sided test to one-sided after seeing the result.
• Do not ignore multiplicity created by multiple outcomes, time points, subgroups, or model choices.
• Do not report only threshold categories when exact p-values, effect estimates, and uncertainty can be shown.
• Do not claim that randomization, large sample size, or sophisticated software protects against every source of bias.

Interpreting statistical evidence responsibly

Statistical significance is one output of a prespecified testing procedure. Its meaning is narrow: the observed result crossed a chosen incompatibility threshold under the null model and analysis assumptions. It should be reported as part of a larger evidence account rather than as a verdict on whether a finding is true or important.

For health-economic decisions, effect magnitude, joint cost-and-outcome uncertainty, opportunity cost, external validity, heterogeneity, and consequences are more directly relevant than a standalone significance label. When uncertainty remains decision-relevant, the appropriate response may be sensitivity analysis, evidence synthesis, further research, or explicit decision modelling rather than stronger wording around a p-value.