Type II Error

Type II Error in Trials and Economic Evaluation

A type II error happens when a real difference between treatments exists but the trial or analysis fails to detect it, so the result is reported as no significant difference. In health economics the missed difference may be a gain in survival, a gain in quality-adjusted life years or a cost saving, and a decision maker who reads the null result as proof of no effect can turn down a technology that is worth adopting. The long-run probability of this error for a given design and a stated true effect is called beta, and its complement is the statistical power of the study. A target value of beta is set at the design stage, together with the significance level, when the sample size is chosen. How much a type II error matters depends on what is lost when a real effect goes undetected, which is why economic evaluation treats it as a question about consequences as well as error rates.

Two errors in the Neyman and Pearson framework

Jerzy Neyman and Egon Pearson set out the logic in 1933. Any rule that decides whether to reject a null hypothesis can go wrong in two ways, by rejecting it when it is true or by accepting it when it is false, which the paper describes as errors of the first kind and errors of the second type. They observed that the two sources of error can rarely be removed completely, and that in some cases it is more important to avoid the first and in others the second, depending on the consequences of each.

The two probabilities are defined under different states of the world. Alpha is a single number because the null hypothesis fixes the effect at zero, while beta has no single value until an alternative effect is stated, since a large true effect is rarely missed and a small one often is. In hypothesis testing for a trial, the stated alternative is usually the smallest difference worth detecting, often a minimal clinically important difference or a cost difference large enough to change a decision.

How beta is calculated for a planned comparison

For a two-arm comparison of means analysed with a two-sided z test, beta depends on the size of the true difference relative to its standard error. The same logic applies to proportions, event rates and net benefit once the right standard error is used.

$$\beta \approx \Phi\left( z_{1-\alpha/2} - \frac{|\Delta|}{SE} \right), \qquad SE = \sigma \sqrt{\frac{2}{n}}$$

where $\beta$ is the probability of a type II error, $\Phi$ is the standard normal distribution function, $z_{1-\alpha/2}$ is the critical value for a two-sided test at significance level $\alpha$ (1.960 when $\alpha$ is 0.05), $\Delta$ is the true difference in means between the arms, $SE$ is the standard error of the estimated difference, $\sigma$ is the common standard deviation of the outcome and $n$ is the number of patients in each arm. The approximation ignores the tiny chance of a significant result in the wrong direction.

The expression shows the four levers. Beta falls as the true difference grows, as the sample grows, as outcome variability shrinks and as alpha is relaxed. The beta level chosen in a protocol is the target value of this quantity at the design effect, and a sample size calculation solves the same equation for $n$.

The ICH E9 guideline on statistical principles for clinical trials asks that the protocol state the probability of erroneously failing to reject the null hypothesis alongside the type I error and the difference to be detected. It records that the probability of type II error is conventionally set at 10% to 20%, that it is in the sponsor's interest to keep it as low as feasible, especially for trials that are difficult or impossible to repeat, and that other values may be acceptable or even preferable in some cases. Against a 5% type I error, the usual convention therefore tolerates a type II error rate two to four times as large.

Worked example with an underpowered cost comparison

All numbers in this example are illustrative. A new outpatient pathway is expected to reduce annual NHS costs per patient by GBP 600 with no difference in health outcomes, and the standard deviation of annual cost is GBP 1,500. A trial with 40 patients per arm is proposed, analysed with a two-sided test at 5%.

1. Standard error. With 40 per arm, the standard error of the cost difference is 1,500 × 0.2236 = GBP 335.4, since the square root of 2/40 is 0.2236.

2. Standardised effect. The true saving is 600 / 335.4 = 1.789 standard errors.

3. Beta. The shortfall below the critical value is 1.960 minus 1.789, or 0.171, and the standard normal probability below 0.171 is 0.568. The trial has a 56.8% chance of missing a real GBP 600 saving, so its power is 43.2%.

4. Relaxing alpha. At a two-sided alpha of 0.10 the critical value is 1.645, and 1.645 minus 1.789 is minus 0.144, giving beta of 0.443. Doubling the allowance for false positives lowers beta by only 0.125.

5. Enlarging the trial. With 100 per arm the standard error is 1,500 × 0.1414 = GBP 212.1, the saving is 600 / 212.13 = 2.828 standard errors, and 1.960 minus 2.828 is minus 0.868, giving beta of 0.193 and power of 80.7%.

6. What the error costs. Suppose 20,000 patients would use the pathway over its lifetime, so a real saving is worth 20,000 × 600 = GBP 12 million. If the pathway is adopted only when the saving is statistically significant, the expected loss from type II errors, given that the saving is real, is 0.568 × 12 million = GBP 6.8 million with 40 per arm and 0.193 × 12 million = GBP 2.3 million with 100 per arm.

7. Deciding on the mean instead. If the pathway is adopted whenever the observed mean saving is positive, the wrong decision follows only when the observed difference is zero or worse. That probability is the standard normal probability below minus 1.789, or 0.037, so the expected loss falls to 0.037 × 12 million = GBP 0.44 million with 40 per arm.

The example shows that a trial can carry a high probability of a type II error while still informing the decision well. The inferential error rate and the decision error rate diverge once the decision rule stops depending on significance.

Why economic evaluation weighs the error by its consequences

Neyman and Pearson left the balance between the two errors to the investigator. Health economists have argued that the balance should come from the costs and benefits at stake. Claxton argued in 1999 that rules of inference are arbitrary and irrelevant to the decisions that economic evaluations inform, that decisions should rest on mean net benefit whether or not differences are statistically significant, and that failing to decide this way imposes costs measurable in resources or health forgone. The spread of net benefit then matters only for whether more information is needed.

Claxton and Posnett had earlier noted that the traditional approach to trial design ignores the marginal costs and benefits of sample information, which can produce arbitrary sample sizes. Their decision-analytic alternative takes explicit account of the costs of sampling and the benefits of sample information, an approach developed further in value of information analysis and the expected value of sample information. In that framing a fixed beta of 0.2 is replaced by a comparison of what a larger trial costs with what it is expected to save in wrong decisions.

The asymmetry also changes between outcomes. ICH E9, in its section on the statistical evaluation of safety, notes that when hypothesis tests are used for safety, adjustments for multiplicity to quantify the type I error are appropriate but the type II error is usually of more concern. The NICE technology appraisal and highly specialised technologies guidance manual (PMG36, section 6.2.32) asks committees considering uncertainty to take into account the likelihood of decision error and its consequences for patients and the NHS.

Type II errors in trials and economic endpoints

Surveys of published negative trials have long found many too small to detect plausible effects, and the negative study page reviews that evidence. Economic endpoints raise the risk further. Briggs and Gray noted calls from many commentators for trials collecting economic data to have sample sizes based on the requirements of the economic analysis as well as the clinical one. Cost data can be highly variable, so a trial sized for a clinical endpoint can carry a large beta for the cost difference or the incremental net benefit, as the worked example shows.

Reading a non-significant result

A non-significant result is where a type II error may have occurred, and its interpretation rests on the estimate and its confidence interval rather than on the p-value alone. Greenland and colleagues describe a related misreading. Beta is a pre-study frequency over repeated uses of a design under the alternative used to compute power, so a power of 90% does not mean that a particular accepted null has a 10% chance of being wrong. A claim that a cheaper technology is as good as the standard needs a non-inferiority trial or an equivalence trial with a prespecified margin.

Boundaries with nearby ideas

Several terms sit close to the type II error and are easily run together. Each answers a different question about the same study or decision.

1. Beta and power. The type II error is the event of missing a real effect, beta is its probability for a stated effect, and power is one minus beta.
2. Type I error. The opposite mistake, a false positive, whose probability alpha is fixed by the significance level.
3. Diagnostic false negative. A negative test result in a single patient who has the condition, a property of a diagnostic test and not of a hypothesis test.
4. Decision error. Adopting or rejecting a technology wrongly, which in economic evaluation depends on the decision rule as well as on beta, as step 7 of the example shows.

Common errors with type II errors

Most mistakes come from reading the absence of significance as a finding in its own right. They appear in trial reports, appraisal submissions and model inputs alike.

1. Treating non-significance as proof of no effect. A trial with a large beta can easily miss a real difference.
2. Quoting beta without the effect it refers to. Beta of 0.2 for a large effect says little about the chance of missing a smaller one.
3. Assuming power for the clinical endpoint covers costs. Cost differences can need much larger samples than the clinical endpoint.
4. Fixing beta by convention alone. The consequences of a missed effect, and the value of more information, can justify a different balance.

Sources

• Briggs AH, Gray AM. Power and sample size calculations for stochastic cost-effectiveness analysis. Medical Decision Making. 1998;18(2 Suppl):S81-S92. https://doi.org/10.1177/0272989X98018002S10
• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364. https://doi.org/10.1016/S0167-6296(98)00039-3
• Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health Economics. 1996;5(6):513-524. https://pubmed.ncbi.nlm.nih.gov/9003938/
• Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31:337-350. https://doi.org/10.1007/s10654-016-0149-3
• International Conference on Harmonisation. ICH E9 Statistical principles for clinical trials. Step 4 version, 5 February 1998. Sections 3.5 and 6.4. https://database.ich.org/sites/default/files/E9_Guideline.pdf
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Section 6.2.32. https://www.nice.org.uk/process/pmg36/chapter/committee-recommendations
• Neyman J, Pearson ES. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A. 1933;231:289-337. https://doi.org/10.1098/rsta.1933.0009