Null Hypothesis

Null Hypothesis in Trials and Economic Evaluation

A null hypothesis is the precise statement about a population quantity that a significance test takes as true while it judges how unusual the observed data would be. In health economics it is most often a claim of zero difference between two treatments in incremental net benefit or one of its parts, although it can equally fix a non-zero value or a margin. A test can only reject or fail to reject the statement written down, so the choice of null decides what a significant or non-significant result means. Many misreadings of trial evidence in health technology assessment start from losing track of that statement, and the p-value in hypothesis testing is computed as if that null were true.

Fisher's exact null and the Neyman and Pearson framework

Ronald Fisher introduced the term in The Design of Experiments in 1935, through an experiment with a lady who claimed she could tell whether milk or tea had been added first to a cup. His test separated results showing a significant discrepancy from the hypothesis that her answers were in no way influenced by the order of pouring from results showing none. He called that hypothesis the null hypothesis and wrote that it is never proved or established, but is possibly disproved, in the course of experimentation. He also required it to be exact and free from vagueness, because it must supply the distribution on which the test of significance is based, so the inexact claim that she had some power of discrimination could not serve as a null. His calculation counted the possible arrangements of cups, the logic that survives in the permutation test.

Neyman and Pearson, writing in 1933, set the hypothesis under test, $H_0$, against a set of alternatives. They observed that rejecting $H_0$ risks rejecting it when it is true, while accepting it risks accepting it when some alternative is really true. The type I error, the type II error and the statistical power to detect a stated alternative are each defined relative to the null, so changing the null changes what each of them measures.

Writing the null as a restriction on a parameter

A null hypothesis restricts an unknown population parameter. A point null fixes the parameter at one value, and a composite null allows a range of values.

$$H_0: \theta = \theta_0 \;\; \text{against} \;\; H_1: \theta \neq \theta_0, \qquad H_0: \theta \le \theta_0 \;\; \text{against} \;\; H_1: \theta > \theta_0$$

where $\theta$ is the population parameter of interest, such as the difference in mean cost per patient between two treatments, $\theta_0$ is the value fixed by the null (zero for a test of no difference), and $H_1$ is the alternative hypothesis that the test favours when $H_0$ is rejected.

A large-sample test then compares the estimate with the null value on the scale of its sampling error.

$$Z = \frac{\hat{\theta} - \theta_0}{\text{SE}(\hat{\theta})}$$

where $\hat{\theta}$ is the estimate from the sample, $\text{SE}(\hat{\theta})$ is its standard error and $Z$ is referred to the standard normal distribution, which holds approximately in large samples if $H_0$ is true. The null enters through $\theta_0$ and through the distribution assumed for $Z$, which is why Fisher insisted that it be exact. A one-sided composite null is tested at its boundary, the value inside $H_0$ closest to the alternative, and the same statistic underlies the Wald test of a regression coefficient.

Choosing the parameter and scale for cost data

Patient-level costs are usually right-skewed, which tempts analysts to transform them or to use rank tests. Thompson and Barber argue that the arithmetic mean is the informative measure for cost data in pragmatic trials, because multiplied by the number of patients it gives the total budget needed. They point out that a t test on untransformed costs compares arithmetic means, a t test on log costs compares geometric means, and the Mann-Whitney U test is an overall comparison of distributions in shape and location, although it is often read as a comparison of medians. Each method tests a different null, and only the first is a null about the quantity on which a budget depends.

Their low back pain example shows the stakes. Mean costs over 12 months were GBP 360 with an exercise programme and GBP 508 with usual general practitioner care. A t-test-based analysis of the GBP 148 difference gave a 95% confidence interval of minus GBP 146 to GBP 442 and a P value of 0.32, while a Mann-Whitney U test on the same data gave P = 0.02. The second result rejects a null of identical distributions, not a null of equal mean cost, and Thompson and Barber judged that relying on it would have been extremely misleading. Where normality is doubtful they point to the non-parametric bootstrap, which keeps the null on the arithmetic mean without assuming a particular distribution.

Nulls that move the burden of proof

In a superiority trial the null is no difference, so the new treatment has to earn a rejection. Non-inferiority and equivalence trials turn the null round. ICH E9 describes the non-inferiority analysis as a one-sided test of the null that the treatment difference, investigational product minus control, equals the lower equivalence margin, and the equivalence analysis as two one-sided tests of the composite null that the difference lies outside the margins.

The null then asserts an important difference, and only data inconsistent with it allow a claim of similarity. ICH E9 states that concluding equivalence or non-inferiority from a non-significant test of the null of no difference is inappropriate. Margins are discussed under non-inferiority trial and equivalence trial.

What failing to reject the null shows

Rejecting a null at the chosen significance level means the data would be unusual if the null and the rest of the model held. Greenland and colleagues stress that the P value tests the whole model, including assumptions about how the analysis was conducted, rather than the targeted null alone. Altman and Bland wrote that a trial without a significant difference has usually shown only an absence of evidence of a difference, not that there is no difference. In their example a trial of 100 patients, against a reported requirement of 1,800, concluded that two treatments were equally effective although the 95% confidence interval included differences in cure rates of up to 20 percentage points.

Greenland and colleagues add that a P value above 0.05 means only that the null is one among many hypotheses with P above 0.05, so unless the estimate equals the null value exactly, reporting no association is a mistake. The confidence interval shows which of those other hypotheses remain compatible with the data. The American Statistical Association statement of 2016 adds that P values do not measure the probability that the studied hypothesis is true.

Worked example with one trial and four null hypotheses

All numbers in this example are illustrative. A trial compares a new outpatient regimen with standard care and estimates an incremental gain of 0.010 QALYs per patient (standard error 0.015) and an incremental cost of minus GBP 400 per patient (standard error GBP 250). The cost and QALY estimates are treated as independent, and all tests use the normal approximation.

1. Superiority null on QALYs. The null is $\Delta E = 0$. The statistic is 0.010 / 0.015 = 0.667, and the two-sided P value is 0.50, so the null is not rejected. Since 1.96 × 0.015 = 0.0294, the 95% confidence interval is 0.010 plus or minus 0.0294, or minus 0.019 to 0.039 QALYs. The data allow a small loss or a gain near 0.04 QALYs, so reporting no difference would be the error Altman and Bland describe.

2. Non-inferiority null on QALYs. With an illustrative margin of 0.03 QALYs, the null is $\Delta E \le -0.03$, tested at its boundary. The statistic is (0.010 + 0.03) / 0.015, which is 0.040 / 0.015 = 2.667, with a one-sided P value of 0.0038, below a one-sided level of 0.025. The null of an important loss is rejected, and the lower confidence limit of minus 0.019 lying above minus 0.03 gives the same verdict. The same data now support a claim that the regimen is not materially worse.

3. Null of equal mean cost. The null is $\Delta C = 0$. The statistic is minus 400 / 250 = minus 1.6, with a two-sided P value of 0.11, so the null is not rejected. The 95% confidence interval is minus 400 plus or minus 490, or minus GBP 890 to GBP 90.

4. Null of zero incremental net monetary benefit. At a value of GBP 25,000 per QALY the estimate is

$$\text{INMB} = \lambda \Delta E - \Delta C = 25000 \times 0.010 - (-400) = 650$$

where $\lambda$ is the value of a QALY in GBP, $\Delta E$ is the incremental QALYs, $\Delta C$ is the incremental cost and INMB is the incremental net benefit in GBP per patient. Its standard error is

$$\text{SE}(\text{INMB}) = \sqrt{\lambda^2 \, \text{SE}(\Delta E)^2 + \text{SE}(\Delta C)^2} = \sqrt{375^2 + 250^2} = 450.7$$

where $\lambda \, \text{SE}(\Delta E)$ = 25,000 × 0.015 = 375 and the covariance term is absent because the estimates are treated as independent. The statistic is 650 / 450.7 = 1.442, with a two-sided P value of 0.15, and the 95% interval is minus GBP 233 to GBP 1,533. At GBP 35,000 per QALY, INMB is 350 + 400 = GBP 750 with a standard error of 581.5, giving a statistic of 1.290 and P = 0.20. Neither null is rejected.

5. Interpretation. Three of the four nulls stand, yet the expected incremental net benefit is positive at both values. Expressed as net health benefit, the gain is 0.010 + 400 / 25,000 = 0.026 QALYs per patient at GBP 25,000 and 0.010 + 400 / 35,000 = 0.021 QALYs at GBP 35,000. On a decision-theoretic reading the regimen would be preferred on current evidence, and the width of the net benefit interval bears on whether further research is worth funding. A real analysis would also estimate the covariance between costs and effects, for example by bootstrapping patient-level pairs.

Null hypotheses and reimbursement decisions

Claxton argued that rules of inference are arbitrary and entirely irrelevant to the decisions that clinical and economic evaluations claim to inform. In his view decisions should rest only on mean net benefit, whether or not differences are statistically significant, and failing to decide in this way imposes costs measured in resources or health benefits forgone. The distribution of net benefit matters only for deciding whether more information is required, the question value of information analysis addresses, and on that argument a null of zero incremental cost or QALYs should not act as a gate for including an effect in a model.

The NICE manual (PMG36) follows expected values rather than tests. Section 4.2.16 requires ICERs to be the ratio of expected additional total cost to expected additional QALYs and asks for net health benefits at values of GBP 25,000 and GBP 35,000 per QALY gained. Section 4.7.12 says the committee's preferred cost-effectiveness estimate should come from a probabilistic analysis when possible unless the model is linear. Neither section frames adoption as the rejection of a null, and tests of nulls, for instance on a covariate in a cost regression, shape the inputs to a model rather than the decision.

Common misreadings of the null hypothesis

Most errors come from treating the null as something a test can confirm, or from forgetting which null was tested. Four of them recur in trial reports and submissions.

1. Accepting the null after a non-significant test. Fisher held that the null is never proved, and a wide interval around zero shows imprecision rather than equivalence.
2. Testing a null that answers a different question. Rank tests and tests on log costs address nulls other than equal arithmetic mean cost.
3. Testing a null that is true by design. The CONSORT 2010 explanation notes that tests of baseline differences in a randomised trial ask whether differences could have arisen by chance when any differences are already known to be caused by chance, and calls such testing superfluous.
4. Letting separate cost and QALY tests decide adoption. Claxton's argument places the decision on expected net benefit.

Sources

• Altman DG, Bland JM. Statistics notes: Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. https://doi.org/10.1136/bmj.311.7003.485
• American Statistical Association. American Statistical Association releases statement on statistical significance and p-values. Press release, 7 March 2016. https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364. https://doi.org/10.1016/S0167-6296(98)00039-3
• Fisher RA. The Design of Experiments. Edinburgh: Oliver and Boyd; 1935. Section 8, The null hypothesis. https://archive.org/details/in.ernet.dli.2015.502684
• Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350. https://doi.org/10.1007/s10654-016-0149-3
• International Conference on Harmonisation. ICH E9 Statistical principles for clinical trials. Step 4, 5 February 1998. Section 3.3.2. https://database.ich.org/sites/default/files/E9_Guideline.pdf
• Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869. https://doi.org/10.1136/bmj.c869
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Sections 4.2.16 and 4.7.12. https://www.nice.org.uk/process/pmg36
• Neyman J, Pearson ES. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A. 1933;231:289-337. https://doi.org/10.1098/rsta.1933.0009
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200. https://doi.org/10.1136/bmj.320.7243.1197