Average Treatment Effect

Average Treatment Effect in Potential Outcomes and Cost-Effectiveness Decisions

Every incremental cost and incremental QALY in a cost-effectiveness model is, in principle, an average treatment effect. In the potential-outcomes definition, the average treatment effect (ATE) is distinct from the average effect on the treated (ATT) and the conditional average effect in a subgroup (CATE). Cost-effectiveness analysis needs the mean incremental cost and effect, which trials and observational data identify in different ways, as set out in the NICE Decision Support Unit's technical support document on non-randomised data (TSD 17). An illustrative example shows how a favourable average can hide a subgroup where the treatment is not cost-effective.

Potential outcomes and three averages

The potential-outcomes framework, formalised by Rubin for randomised and non-randomised studies, describes each patient by two outcomes, one with the treatment and one with the comparator. Only one is ever observed, so an individual's causal effect cannot be measured, but averages over a population can be identified under stated conditions. The ATE is a precise form of the treatment effect and a central estimand of causal inference.

$$ATE = E[Y(1) - Y(0)]$$

where $Y(1)$ is the potential outcome under the treatment, $Y(0)$ is the potential outcome under the comparator, and $E[\cdot]$ is the expectation over the population of interest. The other two averages condition on who is described, as follows.

$$ATT = E[Y(1) - Y(0) \mid T = 1], \quad CATE(x) = E[Y(1) - Y(0) \mid X = x]$$

where $T = 1$ marks patients who actually received the treatment and $X = x$ marks patients with given baseline characteristics, such as a risk group. The population average is the mean of the subgroup averages, weighted by subgroup size, as follows.

$$ATE = \sum_{x} P(X = x) \, CATE(x)$$

where $P(X = x)$ is the share of the population with characteristics $x$. TSD 17 describes the ATT as relevant when interest lies only in those treated, for example when patients unlikely to benefit are almost never treated. It notes that when everyone has the same effect the ATE, ATT and local average treatment effect are identical, and that selection into treatment driven by the sources of heterogeneity makes them differ.

Why cost-effectiveness analysis needs the population mean

Cost-effectiveness analysis informs what a health system should fund for a population, so its inputs are expectations over that population. The NICE methods manual (PMG36) states that ICERs must be the ratio of expected additional total cost to expected additional QALYs. In potential-outcomes terms both parts are average treatment effects, as follows.

$$ICER = \frac{\Delta C}{\Delta Q} = \frac{E[C(1) - C(0)]}{E[Q(1) - Q(0)]}$$

where $C(1)$ and $C(0)$ are a patient's potential costs with and without the treatment, $Q(1)$ and $Q(0)$ are the corresponding QALYs, $\Delta C$ is the incremental cost and $\Delta Q$ is the incremental effectiveness. Because expectation is linear, the mean of each patient's net benefit gain equals the value built from the two averages, as follows.

$$INMB = \lambda \Delta Q - \Delta C$$

where $INMB$ is the incremental net monetary benefit per patient and $\lambda$ is the value placed on one QALY. The ICER, by contrast, is a ratio of averages, not an average of individual ratios.

Two rules follow. Thompson and Barber pointed out that the budget for treating all patients is the arithmetic mean cost multiplied by the number treated, so the arithmetic mean, not the median or geometric mean, is the informative summary of skewed costs. Claxton argued that adoption decisions should rest on mean net benefit irrespective of statistical significance. A choice made on the ATE maximises expected net benefit only if the treatment would be offered to the whole population the ATE describes and targeting is not possible; for a narrower decision the CATE of that group is the relevant quantity.

What randomisation identifies

Randomisation makes treatment assignment independent of the potential outcomes, so the arms are comparable on average in measured and unmeasured characteristics. The difference in arm means then estimates the ATE for the enrolled population, and TSD 17 describes the ATE as the effect a randomised controlled trial would identify. This holds only with full adherence. With non-adherence or crossover, the comparison by randomised arm estimates the effect of assignment, the intention-to-treat effect, as Hernán and Hernández-Díaz note, and instrumental variable methods that use randomisation as the instrument estimate a local average effect among compliers, an approach DiazOrdaz and colleagues extend to cost-effectiveness analysis.

The phrase "for the enrolled population" carries weight. If trial participants differ from the population covered by a reimbursement decision in characteristics that change the effect, such as baseline risk, the two averages can differ. The NICE manual allows for this, stating that relative effects from randomised trials may be applied to baseline risk data for the populations or subgroups of interest.

Estimation from observational and real-world data

Without a suitable trial, the average is estimated from observational studies and other real-world data, where treatment often depends on prognosis. TSD 17, by Faria and colleagues, states that the effect typically of interest in NICE technology appraisals is the ATE, and notes that the ATE needs more demanding assumptions than the alternatives.

Methods assuming selection on observables rest on two conditions. Ignorability (unconfoundedness) requires that, given a properly chosen set of observed covariates, the potential outcomes are independent of treatment assignment. Overlap requires that patients with any combination of covariates could be in either group; Rosenbaum and Rubin called the two together strong ignorability, and weaker versions suffice for the ATT. TSD 17 states that ignorability cannot be tested and must be justified from literature or expert opinion, that variables affected by treatment should not be conditioned on, and that, assuming ignorability holds, lack of overlap is probably the biggest issue an analyst will face.

The inverse probability weighting estimator reweights each arm to resemble the whole population, as follows.

$$\widehat{ATE}_{IPW} = \frac{1}{n} \sum_{i=1}^{n} \frac{T_i Y_i}{e(X_i)} - \frac{1}{n} \sum_{i=1}^{n} \frac{(1 - T_i) Y_i}{1 - e(X_i)}$$

where $n$ is the number of patients, $T_i$ is 1 if patient $i$ was treated and 0 otherwise, $Y_i$ is the observed outcome, and $e(X_i)$ is the propensity score given covariates $X_i$. In an economic evaluation the same weights apply to costs and QALYs. TSD 17's final recommendations ask analysts to state the parameter of interest (ATE, ATT or other), justify the base-case method and run pre-planned sensitivity analyses with different methods. Kreif and colleagues show why. In an observational cost-effectiveness analysis of drotrecogin alfa for severe sepsis, the probability of cost-effectiveness in the high-risk subgroup at £20,000 per QALY ranged from 30% with inverse probability weighting to 90% with propensity score matching and genetic matching.

Worked example of confounding by indication in two risk groups

The figures are illustrative. Of 1,000 patients, 600 are low risk and 400 high risk. Clinicians treat 25% of low-risk patients (150) and 87.5% of high-risk patients (350), so the treated group is 30% low risk and the untreated group 90% low risk.

1. True average. Weighting each subgroup by its population share gives the following.

$$\Delta Q = 0.6 \times 0.10 + 0.4 \times 0.40 = 0.22$$

where $\Delta Q$ is the ATE on QALYs. The same weights give an ATE on cost of 0.6 × £3,000 + 0.4 × £4,000 = £3,400.

2. Naive comparison. Treated patients average 0.3 × 6.10 + 0.7 × 4.40 = 4.91 QALYs and untreated patients 0.9 × 6.00 + 0.1 × 4.00 = 5.80, a difference of minus 0.89. Mean costs are £7,800 against £2,300. Because sicker patients are treated, the treatment looks more costly and less effective.

3. Inverse probability weighting. Treated patients are weighted by one over their probability of treatment (4 for low risk, about 1.143 for high risk) and untreated patients by one over their probability of no treatment (about 1.333 and 8). Each weighted arm then holds the equivalent of 600 low-risk and 400 high-risk patients, as follows.

$$\widehat{ATE}_{IPW} = \frac{600 \times 6.10 + 400 \times 4.40}{1000} - \frac{600 \times 6.00 + 400 \times 4.00}{1000} = 5.42 - 5.20 = 0.22$$

where the numerators are the weighted QALY totals and 1000 is the weighted size of each arm. Costs give £6,600 minus £3,200 = £3,400, matching step 1 because the propensity scores are known exactly.

4. Effect on the treated. With the treated group's mix, the ATT is 0.31 QALYs and £3,700, about £11,935 per QALY. The population ICER is as follows.

$$ICER = \frac{3400}{0.22} \approx 15455$$

where the result is the cost in pounds per QALY gained if everyone were treated. The ATT looks better because it describes a group weighted towards high-risk patients.

5. Decision. At an illustrative £25,000 per QALY, the lower end of the £25,000 to £35,000 range in the NICE manual, net benefit per patient is 25,000 × 0.22 minus 3,400 = £2,100, so treating everyone appears worthwhile. By subgroup it is 25,000 × 0.10 minus 3,000 = minus £500 for low risk and 25,000 × 0.40 minus 4,000 = £6,000 for high risk. Treating only high-risk patients yields 0.4 × £6,000 = £2,400 per member of the population, £300 more than treating everyone.

From the population average to subgroup decisions

The ATE is necessary for a population decision but not always sufficient. When effects vary, a decision maker able to target treatment gains by acting on subgroup averages, which links the ATE to heterogeneous treatment effects and subgroup analysis.

The NICE manual (PMG36, section 4.9) asks for clinical- and cost-effectiveness estimates for each relevant subgroup, preferably based on plausible mechanisms and, where possible, identified at scoping. It treats post hoc data dredging sceptically, asks for the scale of effect modification to be specified and for the precision of subgroup estimates to be reflected in the uncertainty analysis. It also notes that the overall net treatment effect depends on baseline risk as well as relative effect, as Sculpher argues when he extends sources of heterogeneity beyond relative treatment effects to baseline event rates.

Coyle, Buxton and O'Brien framed targeting as stratified cost-effectiveness analysis. Espinoza and colleagues separate the static value of acting on known sources of heterogeneity from the dynamic value of research to resolve subgroup uncertainty, linking subgroup decisions to value of information. The £300 per patient in step 5 is a static value.

ATE beside nearby estimands

Related quantities answer different questions and are often named loosely in reports. Matching the estimand to the decision is the first check on any estimate.

Where the average misleads

A common error is to read the ATE as each patient's gain. An average of 0.22 QALYs fits gains of 0.10 and 0.40 in two groups, or gains for some and losses for others.

A second error is a mismatch between estimand and decision. A matched analysis often reports the ATT and a trial gives the ATE for its participants, and either may be reported as if it covered the decision population. TSD 17 adds that an ATE from a sample restricted to similar patients will not necessarily hold for the original population. Observational estimates also remain exposed to unmeasured confounding, and TSD 17 suggests that counter-intuitive results, such as comorbidities appearing protective, may signal omitted variables. Subgroup averages are less precise than the population average, so targeting built on unplanned comparisons can lose more than it gains.

Sources

• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364.
• Coyle D, Buxton MJ, O'Brien BJ. Stratified cost-effectiveness analysis: a framework for establishing efficient limited use criteria. Health Economics. 2003;12(5):421-427.
• Espinoza MA, Manca A, Claxton K, Sculpher MJ. The value of heterogeneity for cost-effectiveness subgroup analysis: conceptual framework and application. Medical Decision Making. 2014;34(8):951-964.
• DiazOrdaz K, Franchini AJ, Grieve R. Methods for estimating complier average causal effects for cost-effectiveness analysis. Journal of the Royal Statistical Society: Series A. 2018;181(1):277-297.
• Faria R, Hernández Alava M, Manca A, Wailoo AJ. NICE DSU Technical Support Document 17: The use of observational data to inform estimates of treatment effectiveness in technology appraisal: methods for comparative individual patient data. Sheffield: Decision Support Unit, ScHARR; 2015. Sections 2.1 to 2.4 and 5.2.
• Hernán MA, Hernández-Díaz S. Beyond the intention-to-treat in comparative effectiveness research. Clinical Trials. 2012;9(1):48-55.
• Hernán MA, Robins JM. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC; 2020.
• Kreif N, Grieve R, Radice R, Sadique Z, Ramsahai R, Sekhon JS. Methods for estimating subgroup effects in cost-effectiveness analyses that use observational data. Medical Decision Making. 2012;32(6):750-763.
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 4.2, 4.6, 4.7 and 4.9.
• Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41-55.
• Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology. 1974;66(5):688-701.
• Sculpher M. Subgroups and heterogeneity in cost-effectiveness analysis. PharmacoEconomics. 2008;26(9):799-806.
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200.