Language Bias

Language Bias in Evidence Reviews for HTA

Investigators may send favourable trials to English-language journals and others to national journals, and a review that searches English-language sources only inherits that selection. The Cochrane Handbook lists this language bias among the non-reporting biases, alongside publication, time-lag, citation and location bias. It matters in health economics because the relative effects, utility values and costs in a cost-effectiveness analysis are drawn from systematic reviews, and a review that misses results in other languages passes its error into the ICER. Empirical studies show that English-only restriction seldom changes conclusions for conventional medicine but can for complementary medicine.

How trial results steer the language of publication

The bias begins with investigators rather than reviewers. The Handbook describes researchers in non-English-speaking countries publishing some work in local journals that may not be indexed in the major biomedical databases. A reviewer who then searches only English-language sources, or excludes other languages at screening, turns that selective dissemination into a biased pooled estimate.

The classic evidence is a 1997 study by Egger and colleagues, which analysed 40 pairs of trial reports matched for first author and time of publication, one in German and one in English. Only 35% of the German reports, against 62% of the English reports, found a statistically significant difference in the main endpoint, and the only characteristic that predicted publication in English was a significant result (odds ratio 3.75, 95% CI 1.25 to 11.3).

The pattern has recurred. Among drug trials sponsored from China and registered in the Chinese Clinical Trial Registry, positive trials were 3.92 times more likely than negative ones to be published in English, and they were also more often indexed in English-language databases, which the authors called indexing bias.

Why the direction of the bias varies

If authors send positive trials to English-language journals, an English-only review should overstate benefit, but later meta-epidemiological work found a less tidy picture. Jüni and colleagues compared 485 English-language and 115 non-English trials within 50 meta-analyses based on searches without language limits. The non-English trials were smaller (median 88 against 116 participants), more often statistically significant (41.7% against 31.3%) and tended to be of lower methodological quality, and their treatment effects were on average 16% more beneficial.

A study of 5,659 randomised trials in 698 Cochrane meta-analyses found the same direction. Trials published in a language other than English, 7.4% of journal articles, showed larger treatment effects than those published in English (ratio of odds ratios 0.86, 95% CI 0.78 to 0.95), which the authors suggested may relate to the higher risk of bias observed in those trials.

Language bias and study quality can therefore pull in opposite directions. Excluding non-English trials may remove less favourable results that authors kept out of English journals, or smaller and weaker trials whose inflated effects would otherwise bias the pooled estimate. In the Jüni sample, exclusion changed the pooled estimate by less than 5% in 29 of 50 meta-analyses, showed more benefit in 5 and less benefit in 16.

What English-only restriction has done to review results

Methods studies that re-analysed reviews with and without other languages agree that restriction rarely alters conclusions in conventional medicine. The exceptions cluster in particular fields.

In the Nussbaumer-Streit study, statistical significance changed for 2 of the 40 outcomes, but the reviewers would have drawn the same conclusion with less certainty, and most excluded studies were small. The Handbook adds psychiatry, rheumatology and orthopaedics to complementary medicine as fields where studies in other languages have influenced review findings.

Search and eligibility choices that create or prevent it

Language bias enters an evidence synthesis through two decisions, which sources to search and which reports to admit. The Cochrane Handbook asks that searches not be restricted by language, partly because language fields in records can be missing or wrong, and notes the argument that any justified restriction belongs in the eligibility criteria. Removing language limits in English-language databases is, it adds, no substitute for searching non-English journals and databases.

Where a team cannot translate a report, the Handbook recommends listing it under studies awaiting classification, not excluded studies, so readers see what may be missing. It also gives a test for when missing studies are likely. If the intervention is often delivered in countries that speak another language, as with traditional Chinese medicine, eligible studies may well have been missed by a search of English-language databases, a judgement that feeds the ROB-ME tool for risk of bias due to missing evidence.

Consequences for HTA and cost-effectiveness models

NICE's manual (PMG36) states that NICE considers evidence from non-UK sources and that a systematic review aims to minimise the risk of biased selection of studies. It asks that evidence on health effects come from a systematic review and that EQ-5D data from the literature, and resource use and cost data, be identified systematically. A language restriction in any of these reviews can shift a model input.

The effect on relative efficacy is the most direct. A pooled effect that is too favourable lowers the incremental cost per QALY, and because the direction of language bias varies, an English-only review cannot be assumed to be conservative.

Cost and utility evidence raise a different problem. Unit costs and health state utility values belong to a health system and population, and the Handbook chapter on economic evidence recommends reporting the country, currency and price year behind monetised estimates and leaving users to judge transferability to their own setting. An English-only search can under-represent cost and utility studies from health systems that publish in other languages, which matters most for decisions in those countries. A scenario analysis with and without non-English studies shows whether a decision depends on them.

Worked example with an English-only and a full search

The numbers in this example are illustrative. Four trials compare a new treatment with standard care for preventing hospital admission over one year. Two published in English report absolute risk reductions of 0.06 and 0.04, and two published in Spanish and Japanese report 0.01 each. Every trial has a standard error of 0.02, so only the English trials are statistically significant.

The trials are pooled with a fixed-effect model.

$$\hat{\theta} = \frac{\sum_i w_i \hat{\theta}_i}{\sum_i w_i}, \qquad w_i = \frac{1}{SE_i^{2}}$$

where $\hat{\theta}_i$ is the absolute risk reduction in trial $i$, $SE_i$ is its standard error, $w_i$ is its weight (2,500 for every trial here) and $\hat{\theta}$ is the pooled estimate. With equal weights the pooled value is the simple mean, 0.05 for the English trials and 0.03 for all four.

A per-patient model then uses these quantities.

$$\Delta C = c_T - ARR \times c_A, \qquad \Delta E = ARR \times q_A, \qquad ICER = \frac{\Delta C}{\Delta E}$$

where $\Delta C$ is the incremental cost in GBP, $\Delta E$ is the incremental QALYs, $c_T$ is the extra cost of the new treatment (GBP 400), $ARR$ is the pooled absolute risk reduction, $c_A$ is the cost of an admission (GBP 5,000) and $q_A$ is the QALY loss from an admission (0.2).

At an illustrative cost-effectiveness threshold of GBP 20,000 per QALY, the net health benefit is 0.01 minus 150 divided by 20,000, or 0.0025 QALYs per patient, with the English-only review and 0.006 minus 250 divided by 20,000, or minus 0.0065, with all four trials. Two missing trials turn a technology that displaces more health than it adds into one that appears good value. Had the non-English trials reported the larger effects, the English-only review would have understated value instead.

Boundaries with neighbouring reporting biases

Publication bias concerns whether a study is published at all, whereas language bias concerns where a published study appears. Location bias refers to journals with different ease of access or levels of indexing, time-lag bias to when results appear and citation bias to how often they are cited. All of these act on which studies reach a review, unlike biases inside a study such as detection bias.

Funnel plots display, and the Egger test detects, small-study effects of any cause, so asymmetry alone cannot be attributed to language bias.

Misreadings of the evidence on language restriction

An English-only review is not automatically safe because methods studies found few changed conclusions. Those results are averages over conventional medicine, and Hartling and colleagues, who found non-English studies changed results in only four of 129 child-relevant reviews, noted they may matter where there are few relevant studies or questionable vested interests in the published literature.

Including every language does not remove bias by itself either. Non-English trials in some samples were smaller and of lower quality, so inclusion needs a risk-of-bias assessment that separates language from quality. Nor does language bias always favour the new treatment.

Sources

• Aluko P, Graybill E, Craig D, et al. Chapter 20: Economic evidence (last updated August 2023). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-20
• Boutron I, Page MJ, Higgins JPT, et al. Chapter 7: Considering bias and conflicts of interest among the included studies (last updated August 2022). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07
• Dechartres A, Atal I, Riveros C, et al. Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study. Annals of Internal Medicine. 2018;169(6):385-393. https://doi.org/10.7326/M18-1517
• Egger M, Zellweger-Zähner T, Schneider M, et al. Language bias in randomised controlled trials published in English and German. Lancet. 1997;350(9074):326-329. https://doi.org/10.1016/S0140-6736(97)02419-7
• Hartling L, Featherstone R, Nuspl M, et al. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology. 2017;17:64. https://doi.org/10.1186/s12874-017-0347-z
• Jia Y, Huang D, Wen J, et al. Assessment of language and indexing biases among Chinese-sponsored randomized clinical trials. JAMA Network Open. 2020;3(5):e205894. https://doi.org/10.1001/jamanetworkopen.2020.5894
• Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. International Journal of Epidemiology. 2002;31(1):115-123. https://doi.org/10.1093/ije/31.1.115
• Lefebvre C, Glanville J, Briscoe S, et al. Chapter 4: Searching for and selecting studies (last updated March 2025). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5.1. Cochrane; 2025. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04
• Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technology Assessment. 2003;7(41):1-90. https://doi.org/10.3310/hta7410
• Morrison A, Polisena J, Husereau D, et al. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. International Journal of Technology Assessment in Health Care. 2012;28(2):138-144. https://doi.org/10.1017/S0266462312000086
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Nussbaumer-Streit B, Klerings I, Dobrescu AI, et al. Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study. Journal of Clinical Epidemiology. 2020;118:42-54. https://doi.org/10.1016/j.jclinepi.2019.10.011
• Page MJ, Higgins JPT, Sterne JAC. Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis (last updated August 2024). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-13
• Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. Journal of Clinical Epidemiology. 2005;58(8):769-776. https://doi.org/10.1016/j.jclinepi.2004.08.021