Simulated Treatment Comparison

Simulated Treatment Comparison in Population-Adjusted Indirect Comparisons

A simulated treatment comparison (STC) estimates how a company's treatment would have fared among the patients of a competitor's trial. A regression model fitted to the company's own trial data is applied to the competitor's published patient characteristics, and the prediction is set against the competitor's results. Health technology assessment bodies meet STC when no head-to-head trial exists and the two trials enrolled different kinds of patients. Within the family of indirect treatment comparison methods, STC is the outcome regression approach and matching-adjusted indirect comparison (MAIC) its reweighting counterpart. The estimate applies to the competitor trial's population, and its credibility rests on which effect modifiers enter the model and on the scale of the comparison.

Why separate trials need population adjustment

A standard indirect comparison of B and C through a common comparator A combines the effect of B against A from one trial with that of C against A from another. NICE Decision Support Unit Technical Support Document (TSD) 18 names balance between the trials in effect-modifying variables as the key assumption of these standard methods, and STC relaxes it by modelling those variables. An effect modifier alters the effect of treatment, a status specific to a given scale. A prognostic variable affects the outcome whichever treatment is given, and in a comparison anchored on the shared arm its imbalance is taken care of by randomisation within each trial.

TSD 18 warns that in networks of one or two trials per treatment, as in many submissions to NICE, indirect comparisons are highly vulnerable to bias from imbalanced effect modifiers. STC was designed for a company holding individual patient data (IPD) from its own trial of A and B but only published summaries from a competitor's trial of A and C. Caro and Ishak described STC in 2010 as a way of simulating the missing arms of a trial, and TSD 18 reports that all applications published by 2016 follow the later formulation of Ishak and colleagues, which predicts into the competitor trial's population.

Fitting the outcome model and predicting in the competitor population

STC applies classic outcome regression to the indirect comparison problem. The first step fits an outcome model to the IPD in the AB trial.

$$g\left(\mu_{t(AB)}(X)\right) = \beta_0 + \beta_1^{T} X + \left(\beta_B + \beta_2^{T} X^{EM}\right) I(t = B)$$

where $\mu_{t(AB)}(X)$ is the expected outcome for a patient on treatment $t$ with covariates $X$, $g$ is a link function such as the logit, $\beta_0$ is an intercept, $\beta_1$ holds coefficients for prognostic variables, $\beta_B$ is the effect of B relative to A at $X = 0$, $X^{EM}$ is the subvector of effect modifiers, $\beta_2$ holds their interaction coefficients and $I(t = B)$ equals one for patients on B and zero otherwise.

The model then predicts mean outcomes on A and B in the AC population. Plugging in the published covariate means is simplest, but TSD 18 reports that Ishak and colleagues regard this plug-in estimator as systematically biased whenever $g$ is not the identity function, because the mean outcome depends on the full distribution of the covariates and not just their mean. They suggest drawing samples from the AC trial's covariate distribution and averaging the predictions, and TSD 18 explains that this numerical step is why the method is called simulated.

$$\hat{Y}_{t(AC)} = \frac{1}{M} \sum_{m=1}^{M} g^{-1}\left(\hat\beta_0 + \hat\beta_1^{T} x_m + \left(\hat\beta_B + \hat\beta_2^{T} x_m^{EM}\right) I(t = B)\right)$$

where $\hat{Y}_{t(AC)}$ is the predicted mean outcome on treatment $t$ in the AC population, $x_m$ is the $m$th of $M$ simulated covariate vectors drawn to resemble the AC trial, $x_m^{EM}$ holds its effect modifier values, $g^{-1}$ is the inverse link and hats mark AB trial estimates.

As trials usually publish only marginal summaries, TSD 18 reports that Ishak and colleagues assume that covariate correlations in the AC population equal those in the AB population. TSD 18 cautions that this simulation step estimates the expected effect for a randomly selected AC patient rather than an average over the AC covariate distribution, which leads to an underestimate of the precision of the final comparison.

Anchored and unanchored forms of STC

An anchored STC keeps the common comparator. TSD 18 Recommendation 5 asks for the comparison on the usual linear predictor scale, giving the anchored estimator below.

$$\hat{d}_{BC(AC)} = \left[ g(\bar{Y}_{C(AC)}) - g(\bar{Y}_{A(AC)}) \right] - \left[ g(\hat{Y}_{B(AC)}) - g(\hat{Y}_{A(AC)}) \right]$$

where $\hat{d}_{BC(AC)}$ is the relative effect of C against B in the AC population, $\bar{Y}_{A(AC)}$ and $\bar{Y}_{C(AC)}$ are the published outcomes of the AC trial, and $\hat{Y}_{A(AC)}$ and $\hat{Y}_{B(AC)}$ are the STC predictions.

An unanchored STC keeps only the C and B terms, so it relies on the outcome model predicting absolute response. TSD 18 says this assumes that all effect modifiers and prognostic factors are accounted for, which is largely considered impossible to meet, and its Recommendation 1 permits an unanchored comparison only when no connected randomised network exists or single-arm studies are involved.

Choosing the covariates and the scale

For an STC the case for adjustment that TSD 18 Recommendation 2 asks for can be read from the outcome model. With interaction term $\beta_2$ and a difference $u$ between the AC and AB trial means of that effect modifier, the potential bias reduction against a standard indirect comparison is $\beta_2 u$.

Recommendation 4 treats STC differently from MAIC. An anchored STC should adjust for all effect modifiers in imbalance and may add other variables that improve model fit, for example by AIC, which can gain precision without reducing bias further. An unanchored STC must adjust for all effect modifiers and prognostic variables.

TSD 18 reports that the STC literature typically forms the comparison on the natural outcome scale, such as a risk difference. When the outcome model uses a non-identity link, as a logistic regression does, the model assumes additivity on the log odds scale while the comparison assumes it on another, so prognostic variables no longer cancel and any misspecification of them biases the estimate. The one published STC found by the 2016 review, in multiple sclerosis, paired a linear probability model with a log relative risk comparison, which TSD 18 says breaks the anchoring.

Conditional and marginal treatment effects

With an identity link and no products of covariates, the plug-in and averaged predictions give the same mean. With a non-identity link they differ, and for an odds ratio or hazard ratio conditional and marginal effects also differ, as these measures are non-collapsible. Remiro-Azócar and colleagues argue that conventional outcome regression targets a conditional effect that is incompatible with the marginal effect published for the AC trial, and they average over the AC population by parametric G-computation to recover a compatible marginal effect.

Phillippo and colleagues noted in their 2020 simulation study that the typical STC is flawed for non-collapsible measures, since it combines an adjusted estimate with an unadjusted one from the AC trial, although their simulations showed no noticeable bias when the assumptions held, while other simulation studies had found bias from this issue. A submission therefore needs to state which estimand its STC targets.

Worked example with prior therapy as an effect modifier

All numbers in this example are illustrative and match the example on the MAIC page. A company trial randomises patients to A (placebo) and B, and 30% in each arm have had prior therapy. Response without prior therapy is 30% on A and 50% on B, and with prior therapy 20% on A and 25% on B. The competitor trial of C against A reports that 50% of its patients had prior therapy, with response of 25% on A and 40% on C.

1. Fit the outcome model. A logistic model with prior therapy as prognostic factor and effect modifier fits all four groups, with $\hat\beta_0 = -0.8473$, the logit of 0.30, $\hat\beta_1 = -0.5390$, $\hat\beta_B = 0.8473$ and $\hat\beta_2 = -0.5596$. The odds ratio for B against A is $e^{0.8473} = 2.33$ without prior therapy and $e^{\hat\beta_B + \hat\beta_2} = e^{0.2877} = 1.33$ with it.

2. Check the case for adjustment. The imbalance is $u = 0.5 - 0.3 = 0.2$, so the potential bias reduction on the log odds scale is $\hat\beta_2 u = -0.5596 \times 0.2 = -0.112$, an odds ratio for B about 11% smaller in the AC population.

3. Predict with plug-in means. Substituting the AC mean of 0.5 gives a log odds ratio for B against A of $\hat\beta_B + 0.5 \hat\beta_2 = 0.5675$, an odds ratio of 1.76.

4. Predict by averaging. With one binary covariate the AC population is known exactly, so no random draws are needed. The averaged predictions are 0.5 × 0.30 + 0.5 × 0.20 = 0.25 on A and 0.5 × 0.50 + 0.5 × 0.25 = 0.375 on B, an odds ratio of $(0.375/0.625)/(0.25/0.75) = 1.80$.

5. Form the anchored comparison. The competitor trial's odds ratio for C against A is $(0.40/0.60)/(0.25/0.75) = 2.00$. Subtracting log odds ratios is the same as dividing odds ratios.

$$OR_{CB}^{STC} = \frac{OR_{CA}}{OR_{BA}^{STC}} = \frac{2.00}{1.80} = 1.11$$

where $OR_{CB}^{STC}$ is the odds ratio for C against B in the AC population, $OR_{CA}$ the published one for C against A and $OR_{BA}^{STC}$ the averaged STC one. The plug-in value gives 2.00/1.764 = 1.13 instead. Unadjusted, the company trial responds at 0.27 on A and 0.425 on B, an odds ratio of $(0.425/0.575)/(0.27/0.73) = 2.00$, so the standard anchored comparison finds no difference.

6. Interpret the result. STC finds odds of response about 11% higher on C than on B in the AC population, because B works less well after prior therapy and that trial enrolled more such patients. The averaged answer equals the MAIC answer because the model is saturated, and its gap from the plug-in 1.13 is the conditional and marginal mismatch.

Uncertainty, overlap and reporting

TSD 18 notes that treating the predicted AC outcomes as fixed and known ignores the uncertainty from covariate imbalance and from estimating the outcome model, whereas carrying the prediction error through captures it. Bootstrapping the analysis is one way of doing this, and the replicates can feed probabilistic sensitivity analysis.

Regression can predict for patients unlike any in the AB trial, but Phillippo and colleagues note that such extrapolation assumes predictions stay valid beyond the range of the IPD, and in their simulations STC was biased without overlap when a linear model was fitted to a non-linear relationship. TSD 18 asks STC reports to consider how much extrapolation is required and to present the standard indirect comparison beside an anchored STC.

Evidence on performance and decision use

In the 2020 simulation study by Phillippo and colleagues, STC and multilevel network meta-regression (ML-NMR) eliminated bias when the requisite assumptions were met, while MAIC performed poorly in nearly all scenarios. All methods incurred bias when an effect modifier was missing, and the authors call ML-NMR attractive as it gives estimates in any target population.

STC as proposed gives estimates only for the AC trial population, while TSD 18 expects a NICE decision population to be a UK cohort or registry that may differ from both trials. The NICE manual, PMG36 (section 3.4.14), says population adjustment in connected networks can be considered when effect modifiers may be imbalanced, needs IPD from at least one trial, and should be reported with its limitations and, if possible, the likely size of any systematic bias. In a cost-effectiveness analysis, STC usually supplies the relative effect driving the health economic model, and where another adjustment method could change the decision the choice is a source of structural uncertainty.

Frequent misreadings of STC results

Most errors read more into the regression than it holds. Four recur.

• Reading "simulated" as invented evidence. The simulation only averages model predictions over AC-like covariate values.
• Treating the model as checked for the AC population. TSD 18 notes that checks such as residuals apply in the AB trial, while the assumptions for predicting in the AC population are hard to test.
• Taking the plug-in prediction as the AC average. With a non-identity link, TSD 18 reports Ishak and colleagues noting that outcomes predicted at the AC covariate means are a biased estimate of the AC trial's mean outcome.
• Assuming adjustment removes all bias. Only measured covariates enter the model, so unmeasured confounding remains.

Sources

• Caro JJ, Ishak KJ. No head-to-head trial? Simulate the missing arms. PharmacoEconomics. 2010;28(10):957-967. https://doi.org/10.2165/11537420-000000000-00000
• Ishak KJ, Proskorovsky I, Benedict A. Simulation and matching-based approaches for indirect comparison of treatments. PharmacoEconomics. 2015;33(6):537-549. https://doi.org/10.1007/s40273-015-0271-1
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Section 3.4.14. https://www.nice.org.uk/process/pmg36
• Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. NICE DSU Technical Support Document 18: Methods for population-adjusted indirect comparisons in submissions to NICE. Decision Support Unit, ScHARR, University of Sheffield; December 2016. https://research-information.bris.ac.uk/ws/files/94868463/Population_adjustment_TSD_FINAL.pdf
• Phillippo DM, Dias S, Ades AE, Welton NJ. Assessing the performance of population adjustment methods for anchored indirect comparisons: a simulation study. Statistics in Medicine. 2020;39(30):4885-4911. https://doi.org/10.1002/sim.8759
• Remiro-Azócar A, Heath A, Baio G. Parametric G-computation for compatible indirect treatment comparisons with limited individual patient data. Research Synthesis Methods. 2022;13(6):716-744. https://doi.org/10.1002/jrsm.1565