Negative Study

Negative Study in Clinical Trials and Health Technology Assessment

A negative study, also called a negative trial or null result, is one whose primary analysis does not show the effect it was designed to detect, usually because the comparison is not statistically significant. Its opposite, a positive study, is one whose primary result is statistically significant or judged important in favour of the intervention. The label describes the finding, not the quality of the study, and most negative trials leave the size of the effect uncertain. In health technology assessment the reading of a null trial decides whether a committee rejects a technology, accepts it as no different from its comparator, or asks for more research.

Positive, negative and inconclusive results

A Cochrane methodology review of publication bias classified trial findings as positive when the investigators reported them as statistically significant (P < 0.05), striking or important, or showing a positive direction of effect. Findings were negative when they were not statistically significant, were perceived as unimportant, or showed a negative or null direction.

Altman and Bland argued in the BMJ in 1995 that calling such trials negative wrongly implies that the study has shown there is no difference, when usually all it has shown is an absence of evidence of a difference, and they advised against the term. A useful working distinction is between an inconclusive negative study, whose interval includes both no effect and an effect large enough to matter, and an informative null result, whose interval rules out any effect large enough to matter. Only the second supports a claim that the treatment makes no important difference.

Why a trial can miss a real effect

A trial's statistical power is the probability, before the data are collected, that it will reject the null hypothesis if a stated alternative is true, and its complement is the probability of a type II error. Power rises with sample size, event frequency and the effect assumed in the sample size calculation, so a small trial of a modest effect can easily be negative when the treatment works.

Surveys show this has been common. Freiman and colleagues re-examined 71 negative trials in 1978. Sixty-seven had more than a 10% risk of missing a true 25% improvement, and in 57 the 90% confidence interval was compatible with a 25% improvement. Moher, Dulberg and Wells reviewed 383 randomised trials in JAMA, the Lancet and the New England Journal of Medicine in four sample years from 1975 to 1990, of which 27% (102) were negative. Among 70 negative two-group parallel trials, only 16% had 80% power to detect a 25% relative difference and only 36% a 50% difference, and only 20 of the 102 reports discussed whether the observed difference was clinically important.

Altman and Bland describe a variceal bleeding trial of 100 patients whose own calculation suggested 1,800 were needed. Its authors concluded that the two treatments were equally effective, although the 95% interval included differences in cure rates of up to 20 percentage points.

Poor conduct also pushes results towards the null. The ICH E9 guideline notes that equivalence and non-inferiority trials are not conservative, because many flaws in design or conduct tend to bias results towards a conclusion of equivalence. Non-adherence, crossover and losses to follow-up dilute the contrast between arms in the same way, so a carelessly run superiority trial is more likely to be negative for reasons unrelated to the treatment.

Reading a null result through its confidence interval

A p-value above 0.05 says only that the data would not be unusual if there were no effect. Greenland and colleagues list as a common misinterpretation the belief that such a result shows absence of an effect, because many other hypotheses, including important effects, may be just as compatible with the data.

Chapter 15 of the Cochrane Handbook calls the confusion of no evidence of an effect with evidence of no effect a common mistake, and states that when a confidence interval is too wide it is wrong to claim the intervention has no effect or is no different from its comparator.

The practical test compares the interval with the smallest effect that would change practice, often the minimal clinically important difference. If the whole interval lies inside the range of unimportant effects, the study is an informative null, and if it reaches an important benefit or harm, the study is inconclusive however large its p-value. Power calculated afterwards from the observed effect adds nothing, because Greenland and colleagues note it is a direct transformation of the p-value.

Non-inferiority and equivalence need their own design

A negative superiority trial is sometimes offered as proof that a cheaper treatment is as good as the standard. The ICH E9 guideline states that concluding equivalence or non-inferiority from a non-significant test of no difference is inappropriate. A non-inferiority trial or equivalence trial instead fixes a margin in advance, is sized to exclude differences beyond it, and succeeds when the confidence interval lies inside the margin.

The NICE technology appraisal manual (PMG36, section 6.2.9) says that, for cost-comparison analysis, a non-inferiority or equivalence study with appropriate margins should ideally be presented to support a conclusion that health benefits are similar. A failed superiority trial does not meet that standard on its own.

Worked example with an inconclusive trial and an informative null

All numbers in this example are illustrative. A new treatment aims to prevent a complication affecting 20% of patients on standard care, and clinicians regard an absolute reduction of 5 percentage points as the smallest worthwhile effect. Trial A randomises 250 patients to each arm and trial B 2,000 to each arm.

$$RD = p_T - p_C, \qquad SE = \sqrt{\frac{p_T(1-p_T)}{n_T} + \frac{p_C(1-p_C)}{n_C}}$$

where $RD$ is the risk difference, $p_T$ and $p_C$ are the event risks with the new treatment and standard care, $n_T$ and $n_C$ are the numbers randomised and $SE$ is the standard error. The 95% interval is the risk difference plus or minus 1.96 standard errors.

Both trials are negative. Trial A is inconclusive, because its interval runs from 1.9 points of harm to 11.5 points of benefit, and about 903 patients per arm would have been needed for 80% power. Trial B is an informative null, because its whole interval lies within 5 points either side of zero.

Suppose the new treatment costs GBP 400 more per patient, each complication costs GBP 6,000 and loses 0.5 QALYs, and the threshold is GBP 25,000 per QALY.

$$INB = \lambda \times ARR \times q_E - (c_D - ARR \times c_E)$$

where $INB$ is the incremental net benefit in GBP per patient, $\lambda$ is the threshold, $ARR$ is the absolute risk reduction (0.048 in trial A), $q_E$ is the QALY loss per complication, $c_D$ is the extra treatment cost and $c_E$ is the cost of a complication. Trial A gives 600 minus 112, or GBP 488. Treating the risk reduction as normal with standard error 0.034, the standard deviation of net benefit is 18,500 times 0.034, about GBP 629, and the probability that the treatment is cost-effective is about 78%.

$$EVPI = \mu\,\Phi\!\left(\frac{\mu}{\sigma}\right) + \sigma\,\phi\!\left(\frac{\mu}{\sigma}\right) - \max(\mu, 0)$$

where $EVPI$ is the expected value of perfect information per patient, $\mu$ is the expected net benefit (GBP 488), $\sigma$ is its standard deviation (GBP 629), and $\Phi$ and $\phi$ are the standard normal distribution and density functions. With $\mu / \sigma$ of 0.776 the first two terms are GBP 381 and GBP 186, so the EVPI is about GBP 79 per patient, or GBP 1.6 million across an assumed 20,000 future patients. The non-significant trial favours adoption on expected value, but there is a 22% chance that adoption is wrong, and GBP 1.6 million is the most any further trial could be worth.

Negative trials in meta-analysis and appraisal

Several inconclusive trials can add up to a clear answer. Altman and Bland cite an overview of fibrinolytic treatment after myocardial infarction that found a 22% reduction in mortality, although only five of the 24 trials had been significant alone. A meta-analysis uses each null trial's estimate and precision instead of counting it as a vote against.

Claxton argued in 1999 that rules of inference are irrelevant to the decisions economic evaluations inform. Decisions should rest on mean net benefit whether or not differences are statistically significant, and the distribution of net benefit matters only for deciding whether more information is needed. Value of information methods, including the expected value of sample information for a proposed trial, answer that second question.

The NICE manual (PMG36) ties uncertainty to the decision and to further research. Section 6.2.32 asks committees to consider the likelihood of decision error and its consequences for patients and the NHS. Under section 6.4.6 a medicine with plausible potential to be cost-effective but with evidence currently too uncertain can be recommended with managed access, a form of managed entry agreement, when new evidence is expected within up to 5 years. Under section 6.4.14, when evidence is absent, weak or uncertain, a committee may recommend that the technology is used while more research is done, after weighing the value of further evidence, and section 6.4.15 says such a recommendation is not established NHS practice.

Publication of negative results

Negative trials are less likely to reach the evidence base. A 2009 Cochrane methodology review by Hopewell and colleagues found that trials with positive findings were more likely to be published than those with negative or null findings (odds ratio 3.90, 95% interval 2.68 to 5.68), so that if 41% of negative trials are published, about 73% of positive trials would be. Two included studies found positive trials published after four to five years and negative trials after six to eight, the delay described as time-lag bias.

Pooled estimates and model inputs drawn only from published trials then overstate benefit. The ICMJE recommendations state that authors should make public, and editors should not exclude, studies with findings that are not statistically significant or are inconclusive, since they may help answer questions through meta-analysis. The ICMJE also asks journals to require registration in a public trials registry at or before first patient consent, to prevent selective publication and reporting.

Misreadings of negative studies

Most errors come from treating a failure to reject the null hypothesis as a finding in its own right. The following are common in submissions and models.

1. No significant difference read as no difference. An inconclusive trial says little about whether the treatment works.
2. Equivalence claimed from a failed superiority trial. Similar benefit needs a prespecified margin and a trial designed to test it.
3. Post hoc power used as an explanation. Power from the observed effect only restates the p-value.
4. A null effect set to zero in the model. The estimate and its uncertainty belong in the probabilistic sensitivity analysis.
5. Vote counting across trials. Several small negative trials can show a clear effect when pooled.

Sources

• Altman DG, Bland JM. Statistics notes: Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. https://doi.org/10.1136/bmj.311.7003.485
• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364. https://doi.org/10.1016/s0167-6296(98)00039-3
• Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 "negative" trials. New England Journal of Medicine. 1978;299(13):690-694. https://doi.org/10.1056/NEJM197809282991304
• Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31:337-350. https://doi.org/10.1007/s10654-016-0149-3
• Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews. 2009;(1):MR000006. https://doi.org/10.1002/14651858.MR000006.pub3
• International Conference on Harmonisation. ICH E9 Statistical principles for clinical trials. Step 4 version, 5 February 1998. Section 3.3.2. https://database.ich.org/sites/default/files/E9_Guideline.pdf
• International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Sections II.C and III.L. Accessed 14 October 2026. https://www.icmje.org/recommendations/browse/roles-and-responsibilities/responsibilities-in-the-submission-and-peer-peview-process.html and https://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html
• Moher D, Dulberg CS, Wells GA. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA. 1994;272(2):122-124. https://pubmed.ncbi.nlm.nih.gov/8015121/
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 2022, last updated 13 October 2026. Sections 6.2.9, 6.2.32, 6.4.6, 6.4.14 and 6.4.15. https://www.nice.org.uk/process/pmg36/chapter/committee-recommendations
• Schünemann HJ, Vist GE, Higgins JPT, et al. Chapter 15: Interpreting results and drawing conclusions (last updated August 2023). In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Section 15.6.4. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-15