P-Value

P-Value

A p-value quantifies how incompatible an observed test statistic is with a specified null hypothesis under the assumptions of a chosen statistical model. It is calculated from a hypothetical distribution of results if that null model and its assumptions hold. Its interpretation depends on the exact test, its direction, and the analysis that was planned and performed.

Definition and reference distribution

Let $T(X)$ be a test statistic calculated from data $X$, and let $t_{\mathrm{obs}}$ be its observed value. For a continuous upper-tail test with a fully specified null model $H_0$, the p-value is

$$p=\Pr_{H_0}\{T(X)\geq t_{\mathrm{obs}}\}.$$

The probability includes outcomes at least as extreme in the direction specified by the test, conditional on the null model; it is not the probability that the null hypothesis is true after seeing the data. A two-sided test defines extremeness in both directions according to its reference distribution and testing procedure. For a symmetric normal statistic $Z$ with observed value $z_{\mathrm{obs}}$, a conventional two-sided p-value is $2[1-\Phi(\vert z_{\mathrm{obs}}\vert)]$, where $\Phi$ is the standard normal cumulative distribution function. Other tests and discrete distributions require their own tail definitions, and a simplistic doubling rule is not universal.

When the null is composite or nuisance parameters are estimated, the valid reference distribution and p-value may require an appropriate test statistic, conditioning, asymptotic approximation, permutation scheme, simulation, or conservative supremum over null values. Model assumptions include features such as randomisation, exchangeability, independence, measurement and the specified sampling process. A numerically precise p-value cannot repair biased data or an invalid analysis.

Worked example

Suppose a prespecified two-sided large-sample test of a mean difference uses the null $H_0:\delta=0$. An estimated difference of 4 outcome units has standard error 2 units, giving $z_{\mathrm{obs}}=4/2=2.00$. Under a standard normal null reference, $\Phi(2.00)\approx0.97725$, so

$$p=2[1-\Phi(2.00)]\approx2(0.02275)=0.0455.$$

If the test and its assumptions are valid, data generating a statistic with magnitude at least 2.00 would occur about 4.55% of the time under this null model. This statement concerns hypothetical repeated data under $H_0$; it does not say there is a 4.55% chance of no effect, a 95.45% chance of an effect, or a 4.55% chance of a false positive in this particular result.

Under the same normal approximation, a 95% confidence interval is $4\pm1.96(2)$, or approximately $[0.08,7.92]$ units. It narrowly excludes zero, consistent with a two-sided $p<0.05$ using the matching test. Its width also shows how imprecisely the effect is estimated: effects near zero and materially larger effects remain compatible with the interval's underlying procedure. The interval is not a 95% posterior probability interval for the fixed effect. Rounding the effect, standard error, interval or p-value at different stages can create apparent discrepancies; use full precision for calculation.

At a prespecified $\alpha=0.05$, this test crosses the conventional rejection threshold. That threshold is a decision rule set before examining the data, not a scientific discontinuity between $p=0.049$ and $p=0.051$. Report the numerical p-value with the effect estimate and interval; do not replace them with a “significant” label alone.

What p-values do and do not establish

A small p-value indicates tension between the data and the entire tested null model, potentially including its background assumptions. It does not by itself identify whether the problem is the point-null effect, the study design, a model assumption or data quality. A large p-value indicates insufficient incompatibility to reject that null by the chosen procedure; it does not establish equivalence, absence of an effect or evidence that a clinically important effect has been ruled out.

Sample size matters: the same estimated effect can yield a smaller p-value with better precision, while a small study may fail to rule out substantial effects. Statistical significance is not the magnitude, importance, causal validity, clinical value or cost-effectiveness of a result. A confidence interval and a context-specific meaningful-effect threshold help readers assess what the data can and cannot exclude. Equivalence and non-inferiority questions need appropriate prespecified margins and tests; non-significance on an ordinary superiority test is not an equivalence demonstration.

For a genuinely prespecified level-$\alpha$ test under a true null and valid assumptions, the probability of a rejection over repeated studies is controlled at no more than $\alpha$ (with equality for an exact continuous test). That long-run operating characteristic is not the probability that this study's rejected null is true or that a claim is false. A posterior probability for hypotheses requires a different framework and prior assumptions.

Analysis choices that change interpretation

The p-value depends on the outcome, comparator, null, sidedness, time point, covariate adjustment, missing-data method and other analytic choices. Prespecification and transparent reporting matter because choosing among many analyses after seeing the data can make an unadjusted p-value appear more compelling than the full search warrants. Multiple outcomes, repeated looks, subgroups and model selection can require multiplicity control or a clearly exploratory interpretation; a nominal p-value does not automatically control the familywise error rate.

Optional stopping can invalidate a fixed-sample reference distribution unless the sequential design accounts for interim analyses. For clustered, correlated or repeated observations, use a variance estimator or model appropriate to the dependence; treating correlated observations as independent may make the standard error too small. Missing outcomes, nonadherence, confounding in observational research and selective publication can bias inference in ways a low p-value does not diagnose.

A difference between “significant” and “not significant” subgroup results does not establish that treatment effects differ between subgroups. Test an interaction directly with an appropriate estimate and interval, while considering power and multiplicity. Likewise, two separate p-values do not by themselves test the difference between two estimates.

Use in health economics

Economic decisions require both the size and uncertainty of differences in health and resources, alongside the decision threshold, affected population, equity and budget implications. A statistically significant difference in cost or QALYs need not imply that an intervention is cost-effective, and non-significance of a component need not make its expected value irrelevant. Analyse their joint distribution where possible; patient-level costs are often skewed and costs and effects may be correlated.

For two strategies, incremental net monetary benefit at willingness to pay $\lambda$ is $\mathrm{INMB}(\lambda)=\lambda\Delta Q-\Delta C$. A study might test $H_0:\mathrm{INMB}(\lambda)=0$ at a stated threshold, but its p-value addresses that specified statistical null and design. A health system still needs the estimated net benefit, uncertainty, decision context and opportunity costs. In a probabilistic model, the probability a strategy has the highest net benefit is a different quantity from a frequentist p-value; neither should be silently substituted for the other.

Reporting and critical appraisal

State the hypothesis, statistic or model, sidedness, analysis population, sample size, assumptions, prespecified $\alpha$, and how multiplicity, missing data and interim analyses were handled. Present effect estimates with units and uncertainty intervals, preferably for all prespecified outcomes, along with numerical p-values when informative. Avoid reporting only values below a cutoff or using $p=0$ for a very small nonzero value; report a suitable bound, such as $p<0.001$, if precision is limited.

The American Statistical Association statement on p-values explains that p-values do not measure the probability a hypothesis is true or the size or importance of an effect. Its task force statement on statistical significance and replicability also recognizes appropriately used p-values as one part of sound inference. CONSORT 2025 emphasizes reporting trial effect sizes and their precision, while the CONSORT explanation provides context for analyses and presentation. Interpret these principles with the actual study protocol and decision question.