Performance Bias

Performance Bias in Trials and Cost-Effectiveness Evidence

Performance bias arises after randomisation, when the arms of a trial come to differ in the care they receive or in how participants behave, for reasons other than the intervention being tested. Typical causes are extra monitoring in one arm, unequal co-interventions and comparator patients seeking the new treatment, often because participants, carers or clinicians know the allocation. It matters in health economics because the relative treatment effect from trials is the main clinical input to a cost-effectiveness analysis, so a distorted effect moves costs, QALYs and the ICER together. The current Cochrane tool, RoB 2, assesses it as bias due to deviations from intended interventions, and whether a result is at risk depends on which treatment effect the analysis is trying to estimate.

From blinding of participants to deviations from intended interventions

The 2011 Cochrane risk-of-bias tool grouped its items into six types of bias. Performance bias, defined as bias due to knowledge of the allocated interventions by participants and personnel during the study, was addressed by blinding of participants and personnel, while blinding of outcome assessment was a separate item for detection bias.

RoB 2 (Sterne and colleagues, 2019) dropped the old labels because they were used inconsistently and were not known by many people outside Cochrane. Its second domain, bias due to deviations from intended interventions, covers extra interventions inconsistent with the protocol, failure to implement the protocol interventions as intended, and non-adherence by participants. The Cochrane Handbook notes that such biases are sometimes referred to as performance biases, and the MetaBLIND study described them as systematic differences between arms in the care provided or in participants' behaviour.

Mechanisms that open a gap between the arms

The intended interventions are those in the trial protocol, which may plan for change, such as second-line treatment on progression. Bias comes from unintended changes that arise because people know they are in a trial and which arm they are in.

In an unblinded trial, participants may feel unlucky to be assigned to the comparator and seek the experimental intervention or other treatments that improve their prognosis. The Handbook adds that monitoring patients on a novel intervention more often than those on standard care raises the risk of bias, unless the extra monitoring is an intended part of the intervention.

In a review of 12 trials that randomised patients to blinded and non-blinded sub-studies, lack of patient blinding was associated with more control group attrition (risk ratio 1.79) and more control group co-intervention (1.55, 95% confidence interval 0.99 to 2.43). Risk in this domain can differ between outcomes in the same trial, because knowing the allocation may change behaviour such as clinic visits without affecting physiology or mortality.

Which treatment effect the analysis is after

RoB 2 asks reviewers to state the effect of interest, either the effect of assignment to an intervention (the intention-to-treat effect) or the effect of adhering to it as the protocol specifies (the per-protocol effect). The Handbook judges the effect of assignment most appropriate for a health policy question, such as whether to prescribe a new drug in a health system, and the effect of adhering for an individual patient's choice.

For the effect of assignment, only deviations that are inconsistent with the protocol, arise from the trial context and affect the outcome count as bias, and the analysis should follow intention-to-treat principles. For the effect of adhering, adherence, implementation and balance of non-protocol interventions all matter. The Handbook classes as-treated and naive per-protocol analyses as inappropriate for the effect of assignment and as problematic for the effect of adhering, because prognostic factors may influence who adheres. It notes that heavy switching to the other arm may reduce the observed difference and so bias the estimated effect of adhering towards the null.

How RoB 2 reaches a judgement in this domain

Reviewers answer signalling questions, and an algorithm maps the answers to low risk, some concerns or high risk. The questions for the effect of assignment are summarised below.

Sterne and colleagues note that open trials can be at low risk of bias if no deviations arose because of the trial context.

Blinding, allocation concealment and what each prevents

Blinding of participants, carers and people delivering the interventions should, if successful, prevent knowledge of the assignment from influencing contamination (one arm's intervention reaching the other arm), switches to non-protocol interventions or non-adherence. It can be impossible, as when surgery is compared with non-surgical treatment, and NICE's manual for technology evaluations (PMG36) notes that blinding is often not possible in trials of specific medical devices or diagnostics.

Attempted blinding may still fail, because drug side effects can reveal the allocation unless the comparator has similar side effects or an active placebo is used. The Handbook states that deducing the allocation does not in itself lead to a risk of bias.

Allocation concealment is a different safeguard. It stops anyone knowing the next assignment before enrolment, so it protects the randomisation process, and the Handbook states that it can always be implemented, whereas blinding prevents bias after assignment and cannot always be implemented.

How large the bias has been in practice

Meta-epidemiological studies estimate average bias. Wood and colleagues, pooling 1,346 trials, found that lack of blinding exaggerated effects in trials with subjective outcomes (ratio of odds ratios 0.75) but showed little evidence of bias with objective outcomes (1.01). Savović and colleagues, using 1,973 trials, linked lack of or unclear double blinding to an average exaggeration of 13% (ratio of odds ratios 0.87), driven mainly by subjective outcomes. Both studies mixed performance and detection bias.

Studies that separate the two disagree. In the 12 trials with blinded and non-blinded sub-studies, non-blinded patients exaggerated effects on patient-reported outcomes by an average of 0.56 standard deviations, mostly in acupuncture trials. The MetaBLIND study of 1,153 trials found no evidence of an average difference for blinding of patients (ratio of odds ratios 0.91, 95% credible interval 0.61 to 1.34, for patient-reported outcomes) or of healthcare providers (1.01 for provider decision outcomes such as readmission), and its authors said blinding should remain a methodological safeguard.

Consequences for economic evaluation and HTA

A health technology assessment usually takes its relative effect from randomised controlled trials, pooled in a meta-analysis or network meta-analysis. PMG36 states that blinding, when applied, prevents knowledge of treatment allocation from influencing behaviours, and counts it in a trial's internal validity. It asks for details of how bias was assessed and addressed for each model input source, and for bias adjustments to be considered in a network meta-analysis when included studies raise quality concerns.

The direction of the error depends on the mechanism. Extra attention or co-interventions in the experimental arm favour the new technology and lower its ICER. Comparator patients obtaining the new treatment, or effective co-interventions in the control arm, dilute the difference and raise the ICER. In trial-based costing, unequal monitoring also generates resource use that would not occur in routine care. Health-related quality of life is patient-reported, so QALY gains from open trials need the scrutiny given to subjective outcomes. Wood and colleagues recommended reporting meta-analyses restricted to trials at low risk of bias, and a scenario analysis of that kind shows how far the decision depends on them.

Worked example with comparator patients obtaining the new treatment

The numbers in this example are illustrative. Without deviations, the one-year risk of hospital admission would be 25% with standard care and 15% with a new treatment, a relative risk of 0.60. In an open trial, 30% of the standard care arm obtain the new treatment outside the protocol because they know their allocation, and it works for them as it does in the new arm. The observed control risk becomes 0.7 times 0.25 plus 0.3 times 0.15, or 0.22, and the observed relative risk is 0.15 divided by 0.22, or 0.68.

A model applies the relative risk to a baseline risk of 0.25 and costs the treatment only in the new arm, as in the decision problem.

$$ARR = p_0 (1 - RR), \qquad \Delta C = c_T - ARR \times c_A, \qquad \Delta E = ARR \times q_A$$

where $ARR$ is the absolute risk reduction in admissions, $p_0$ is the baseline risk (0.25), $RR$ is the relative risk, $\Delta C$ is the incremental cost per patient in GBP, $c_T$ is the treatment cost (GBP 800), $c_A$ is the cost of an admission (GBP 4,000), $\Delta E$ is the incremental QALYs per patient and $q_A$ is the QALY loss from an admission (0.25).

The second column uses unrounded values. At an illustrative cost-effectiveness threshold of GBP 20,000 per QALY, the net health benefit, incremental QALYs minus incremental cost divided by 20,000, falls from 0.005 to minus 0.0042 QALYs per patient. Deviations in the comparator arm make a cost-effective technology appear to displace more health than it adds.

Boundaries with neighbouring biases

Detection bias concerns how outcomes are measured, so RoB 2 assesses blinding of outcome assessors in a separate domain, and an open trial can still have blinded outcome assessment. Differential dropout is attrition, assessed in the missing outcome data domain. Selection bias at enrolment and confounding concern who ends up in each group, whereas performance bias concerns what happens to groups that were comparable at randomisation.

Treatment switching that the protocol allows, such as crossover to the new treatment on progression, raises a different problem, since the effect of assignment then includes the switch and may be a poor fit for a decision problem in which the comparator arm would not receive the technology. PMG36 allows switching-adjusted estimates when intention-to-treat analysis is considered inappropriate, warning that censoring or excluding switchers is very susceptible to selection bias.

Common misreadings of blinding and deviations

An open-label trial is not automatically at high risk in this domain, and a double-blind label does not settle the question either, since the Handbook notes that the term makes it hard to know who was blinded. RoB 2 judges whether deviations arose from the trial context and affected the outcome, result by result, so all-cause mortality may be at low risk where a symptom score in the same trial is not.

A per-protocol analysis does not remove performance bias, since valid estimation of the effect of adhering needs methods that rest on strong assumptions.

Null average results from MetaBLIND do not show that a particular trial is free of bias. Wood and colleagues found that, except for all-cause mortality, the size of bias varied between meta-analyses, and the worked example shows that a modest deviation in one arm can change a reimbursement decision.

Sources

• Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. https://doi.org/10.1136/bmj.d5928
• Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial (last updated October 2019). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
• Hróbjartsson A, Emanuelsson F, Skou Thomsen AS, et al. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. International Journal of Epidemiology. 2014;43(4):1272-1283. https://doi.org/10.1093/ije/dyu115
• Moustgaard H, Clayton GL, Jones HE, et al. Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study. BMJ. 2020;368:l6802. https://doi.org/10.1136/bmj.l6802
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Savović J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine. 2012;157(6):429-438. https://doi.org/10.7326/0003-4819-157-6-201209180-00537
• Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. https://doi.org/10.1136/bmj.l4898
• Wood L, Egger M, Gluud LL, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008;336(7644):601-605. https://doi.org/10.1136/bmj.39465.451748.AD