Time-Lag Bias

Time-Lag Bias in Early Evidence and Health Technology Assessment

Time-lag bias arises when how quickly a study's results appear depends on what they show, so trials with favourable or statistically significant findings reach journals sooner, by about a year on average in the Cochrane Handbook's summary, than trials with null or unfavourable ones. Any evidence base assembled early in a technology's life therefore over-represents positive results, and a pooled estimate made at that point tends to overstate the benefit even if the missing trials are published later. This matters most in health technology assessment, where appraisals of new medicines are often made at or near launch from the first few trials available. The error is temporary in principle, but reimbursement decisions, prices and model inputs fixed during the lag can outlast it.

Whether results appear and when they appear

The Cochrane Handbook lists time-lag bias among the non-reporting biases and defines it as the rapid or delayed publication of research findings depending on the nature and direction of the results. Its close relative, publication bias, concerns whether findings are published at all. Time-lag bias concerns when they are published, so it shapes the evidence that exists at a given date rather than the evidence that will exist in the end.

Trials with striking results may be stopped early, written up faster and accepted sooner, while null trials wait or pass through several journals. For a systematic review or meta-analysis, the effect is the same whichever step causes it, because only results available at the search date can be pooled.

How large the delay has been

Cohort studies that follow trials through to publication give direct evidence. The figures below use different settings and starting points, so they show the direction and rough size of the lag rather than one value.

Ioannidis found that most of the difference arose after follow-up, in slower submission and acceptance of negative trials. He also observed that trials terminated early had significant results, whereas long-protracted trials often had low event rates and failed to reach statistical significance. The Cochrane Handbook summarises the evidence as trials favouring the experimental intervention appearing in journals about one year earlier on average.

The 2024 update of the Cochrane methodology review, by Showell and colleagues, included 204 reports tracking 165,135 trials. Only 53% of the trials had been published in full. Trials with positive results were more likely to be published (odds ratio 2.69) and were published faster (adjusted hazard ratio 1.92), taking 2 years on average against 2.6 years for negative or null trials. The authors concluded that review authors should consider time-lag bias, especially when updating a review.

Why the first trials tend to show the largest effects

Other mechanisms also make early evidence look better than later evidence. Bassler and colleagues compared 91 randomised controlled trials stopped early for benefit with 424 similar trials that were not. The pooled ratio of relative risks was 0.71, large differences occurred when truncated trials had fewer than 500 events, and in 62% of the questions the trials that ran to completion failed to show significant benefit. Hopewell and colleagues describe time-lag bias as trials with striking results being stopped earlier than planned, published quicker or both, so truncation and time lag act together.

Pereira, Horwitz and Ioannidis examined 85,002 forest plots in the Cochrane Database of Systematic Reviews. Where the first published trial showed a significant and very large effect (odds ratio of 5 or more), 90% of those effects became smaller once other trials were added, and the median odds ratio fell from 11.88 to 4.20. These studies do not separate time lag from small-study effects and chance, but show how far early estimates can move.

Consequences for early appraisal, living reviews and value of information

An appraisal at launch usually rests on a handful of trials. The Handbook advises that in a fast-moving research area the available studies may be subject to time-lag bias, and that when only a few early studies are available it may be reasonable to assume that results from further studies are missing. It also reports that phase III trials are more likely to be published than phase II trials (odds ratio 2.0), so missing evidence is a greater worry for an intervention that has not yet reached phase III.

In a cost-effectiveness analysis, an inflated relative effect raises the QALYs gained, lowers the ICER and may lift the technology under the cost-effectiveness threshold. The reverse case is possible when early publication favours the comparator, but the cohort studies above measured the lag for results favouring the experimental arm.

Living systematic reviews, which Elliott and colleagues describe as up-to-date online summaries updated as new research becomes available, correct the estimate as delayed trials arrive. Their early versions carry the bias, and the authors warn that early effect estimates can be unstable and that naive repeated testing inflates false positive findings. In a cumulative meta-analysis ordered by publication date, a pooled effect that drifts towards the null as later trials are added is consistent with time-lag bias, although small-study effects, changes in populations or comparators, and chance can produce the same drift.

Value of information analysis takes the current distribution of the effect as its starting point. If that distribution is centred on an optimistic early estimate, the probability that the technology is cost-effective is overstated and the expected value of further research can look low, so a decision maker may adopt without asking for more evidence when more is already on its way.

Detecting and reducing time-lag bias in reviews and submissions

Time-lag bias leaves little statistical trace in the trials already published, so the main defences lie in how evidence is searched for and when. The Handbook notes that trials registers identify initiated, ongoing and completed studies whether or not they are published, and that ClinicalTrials.gov has accepted summary results since 2008. If only an inception cohort of prospectively registered trials is eligible, no study is missed by design. Stern and Simes recommended prospective registration and, for reviews, restricting selection to trials started before a common date.

The NICE technology appraisal and highly specialised technologies guidance manual (PMG36) asks that attempts be made to identify evidence not in the public domain, including unpublished trial data, data in abstract form only and incomplete data, and that such evidence be appraised, adjusted for bias and tested in sensitivity analysis. It also invites a chart of expected interim and final data readouts from ongoing studies, which shows which trials have yet to report.

The Handbook's ROB-ME tool assesses risk of bias due to missing evidence in a meta-analysis through eight signalling questions. Funnel plot tests offer little help at launch, because the Handbook advises using them only when at least 10 studies are pooled. Analysts can still run a scenario analysis that adds registered but unreported trials with null effects, or that uses a less favourable relative effect, to show how much the decision depends on early evidence.

Worked example with five trials and a two-year lag

The numbers in this example are illustrative. Five trials each randomise 400 patients to a new drug and 400 to standard care, and the outcome is a serious event with a 20% risk (80 events) on standard care in every trial. Trials A, B and C record 56, 58 and 54 events on the new drug, each a statistically significant reduction at the 5% level. Trials D and E record 76 and 72 events, which are not significant. At the time of appraisal only A, B and C are published, and D and E appear two years later.

Because every arm has 400 patients, the Mantel-Haenszel pooled relative risk equals total events on the new drug divided by total events on standard care. A simple model per patient treated then gives the following.

$$\Delta C = c_{D} - ARR \times c_{E}, \qquad \Delta E = ARR \times q_{E}, \qquad NHB = \Delta E - \frac{\Delta C}{\lambda}$$

where $\Delta C$ is the incremental cost in GBP, $\Delta E$ is the incremental QALYs, $c_D$ is the extra cost of the new drug (GBP 1,500), $ARR$ is the pooled absolute risk reduction, $c_E$ is the cost of treating an event (GBP 5,000), $q_E$ is the QALY loss from an event (1.0), $NHB$ is the net health benefit in QALYs and $\lambda$ is an illustrative threshold of GBP 25,000 per QALY.

At appraisal the net health benefit is 0.06 minus 1,200 divided by 25,000, or 0.012 QALYs per patient, and the drug appears good value. Once the two null trials appear it is 0.042 minus 1,290 divided by 25,000, or minus 0.0096, and the drug displaces more health than it adds. At GBP 35,000 per QALY, the upper value PMG36 uses for net health benefit, the later figure is 0.042 minus 1,290 divided by 35,000, or 0.0051, so the case moves from clearly cost-effective to borderline. No trial was suppressed or misreported, yet the launch decision rested on an estimate the full evidence does not support.

Boundaries with neighbouring biases

Publication bias is the permanent counterpart, in which unfavourable results never appear. A lag that outlasts the decision has the same effect on it, and the 2024 Cochrane update found nearly half of trials unpublished, so the two often operate together. Outcome reporting bias works within a published study, where some outcomes are reported and others are not.

Early stopping for benefit is a separate mechanism that inflates the estimate from a single trial, while time-lag bias changes which trials are available. Detection bias and other risk-of-bias domains concern how each study was run, so a review free of them can still pool a time-lagged set of trials.

Misreadings of early evidence and publication timing

A search that finds every published trial is not complete at launch. Missing trials may be registered and finished, and a register search shows how many have yet to report.

A large effect from the first one or two trials is not reassurance. The Pereira and Bassler findings show that very large early effects and truncated trials tend to overstate benefit, and the certainty of evidence given to a small early body of trials should reflect that.

A symmetrical funnel plot or a non-significant asymmetry test does not rule out time-lag bias when fewer than 10 studies are available, because the tests have low power. Hopewell and colleagues grouped results significantly favouring the control arm with null results, and both appeared later, so the lag has mainly favoured the experimental intervention.

Sources

• Bassler D, Briel M, Montori VM, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010;303(12):1180-1187. https://doi.org/10.1001/jama.2010.310
• Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies (last updated August 2022). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07
• Elliott JH, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Medicine. 2014;11(2):e1001603. https://doi.org/10.1371/journal.pmed.1001603
• Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews. 2007;(2):MR000011. https://doi.org/10.1002/14651858.MR000011.pub2
• Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998;279(4):281-286. https://doi.org/10.1001/jama.279.4.281
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Page MJ, Higgins JPT, Sterne JAC. Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis (last updated August 2024). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-13
• Pereira TV, Horwitz RI, Ioannidis JPA. Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012;308(16):1676-1684. https://doi.org/10.1001/jama.2012.13444
• Showell MG, Cole S, Clarke MJ, DeVito NJ, Farquhar C, Jordan V. Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews. 2024;(11):MR000011. https://doi.org/10.1002/14651858.MR000011.pub3
• Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315(7109):640-645. https://doi.org/10.1136/bmj.315.7109.640