Propensity Score

Propensity Score in Observational Comparisons for Health Economic Evaluation

A propensity score gives the chance, judged from what was recorded before treatment started, that a patient ended up on the treatment under study. Health economists meet it when incremental costs and QALYs have to come from registries or claims databases instead of a randomised controlled trial, because the score compresses many baseline differences between treated and untreated patients into a single number that can be matched, stratified or weighted on. Balance on that one number brings balance on every covariate that went into it, which makes the score a common basis for adjusted comparisons in real-world evidence offered to health technology assessment. The score works only with variables that were measured, so an adjusted cost-effectiveness analysis built on it still depends on the claim that no important confounder was left out.

The balancing property behind the score

Rosenbaum and Rubin defined the score in 1983 as the conditional probability of assignment to a treatment given a vector of observed covariates. Their central result is that adjustment for this one scalar is sufficient to remove bias due to all the observed covariates.

$$e(X) = \Pr(T = 1 \mid X)$$

where $e(X)$ is the propensity score, $T$ equals 1 for a patient who received the treatment of interest and 0 for the comparator, and $X$ is the vector of baseline covariates measured before treatment.

Austin calls it a balancing score, since patients who share a score have the same distribution of observed covariates whether treated or not. In a randomised trial the true score is known and fixed by the design, for example 0.5 for every patient under equal allocation. In an observational study it is unknown and must be estimated.

Two conditions that make the score sufficient

Balance on measured covariates yields an unbiased treatment effect under what Rosenbaum and Rubin called strongly ignorable treatment assignment. Austin sets out its two parts as follows.

$$\{Y(1), Y(0)\} \perp T \mid X \quad \text{and} \quad 0 < e(X) < 1$$

where $Y(1)$ and $Y(0)$ are the potential outcomes, such as cost or QALYs, that a patient would have with and without treatment, $\perp$ means statistical independence, and the second condition gives every patient a nonzero chance of either option.

The first condition is the no unmeasured confounders assumption. NICE Decision Support Unit Technical Support Document 17 (TSD 17) states that it cannot be tested and should be justified with reference to the expert literature and opinion. The second condition is overlap. TSD 17 explains that the average treatment effect cannot be estimated for people whose characteristics place them always in one group, while the effect in the treated needs only a weaker form of overlap.

Estimating the score from baseline data

The score is most often estimated by logistic regression of treatment received on baseline characteristics. Each patient's fitted probability is the estimated score used in every later step.

$$\log\left(\frac{e(X)}{1 - e(X)}\right) = \beta_0 + \beta_1 X_1 + \dots + \beta_k X_k$$

where $\beta_0$ is the intercept, $X_1$ to $X_k$ are the $k$ baseline covariates, and $\beta_1$ to $\beta_k$ are their coefficients on the log odds scale.

The model is judged by the balance it produces. Austin describes an iterative approach that adds covariates, interactions or nonlinear terms until residual differences are acceptably small, without being guided by the significance of the coefficients, and notes that the c-statistic gives no information on whether the model is correctly specified.

Covariate choice is a causal judgement. The NICE real-world evidence framework (ECD9) warns against adjusting for variables on the causal pathway, colliders or instruments, and against choosing covariates by statistical significance. It also notes that the score is developed without reference to outcome data, which can reduce the risk of selective reporting when combined with strong research governance.

Four ways the score enters an analysis

Austin groups the uses of the score into four methods. Each answers a different question.

Propensity score matching has its own page. For stratification, Austin reports Rosenbaum and Rubin's finding that stratifying on the quintiles of the score removes approximately 90% of the bias due to measured confounders when estimating a linear treatment effect.

Weighting creates a synthetic sample in which measured covariates are independent of treatment. Austin and Stuart give weights for the average treatment effect (ATE) and the average treatment effect in the treated (ATT).

$$w_i^{ATE} = \frac{T_i}{e_i} + \frac{1 - T_i}{1 - e_i} \qquad w_i^{ATT} = T_i + \frac{e_i (1 - T_i)}{1 - e_i}$$

where $w_i$ is the weight for patient $i$, $T_i$ is 1 if patient $i$ was treated and 0 otherwise, and $e_i$ is that patient's estimated score.

Treated patients with very small scores, or untreated patients with scores close to one, receive very large weights. Austin and Stuart describe stabilised weights as one remedy and truncating weights at percentiles such as the 1st and 99th as another.

$$w_i^{S} = \frac{T_i \Pr(T = 1)}{e_i} + \frac{(1 - T_i) \Pr(T = 0)}{1 - e_i}$$

where $\Pr(T = 1)$ and $\Pr(T = 0)$ are the overall proportions treated and untreated.

Checking balance and overlap before reading the outcomes

Balance is checked on covariates after matching or weighting. The usual measure is the standardised mean difference, here for a binary covariate as Austin and Stuart give it, without their factor of 100.

$$d = \frac{\hat{p}_1 - \hat{p}_0}{\sqrt{\left[\hat{p}_1 (1 - \hat{p}_1) + \hat{p}_0 (1 - \hat{p}_0)\right] / 2}}$$

where $d$ is the standardised difference, and $\hat{p}_1$ and $\hat{p}_0$ are the prevalences among treated and untreated patients, weighted after weighting.

Austin reports that a difference below 0.1 has been taken to indicate negligible imbalance, and he discourages significance tests for balance. Austin and Stuart report advice that a mean stabilised weight far from one, or very extreme weights, can signal a positivity problem or a misspecified model.

Worked example of weights in a remote monitoring evaluation

All numbers in this example are illustrative. A health board compares a remote monitoring service for heart failure with usual care in 1,000 registry patients, 430 of whom used the service. Baseline severity is the only confounder. Of 500 low-severity patients 100 used the service, of 300 medium-severity patients 150 did, and of 200 high-severity patients 180 did.

1. Estimate the scores. The scores are 100/500 = 0.2, 150/300 = 0.5 and 180/200 = 0.9. A logistic model with severity indicators reproduces them exactly.

2. Measure the imbalance. High severity affects 180/430 = 0.4186 of users and 20/570 = 0.0351 of non-users.

$$d = \frac{0.4186 - 0.0351}{\sqrt{(0.4186 \times 0.5814 + 0.0351 \times 0.9649) / 2}} = \frac{0.3835}{0.3723} = 1.03$$

where 0.4186 and 0.0351 are the two prevalences, and 1.03 is far above the 0.1 guide.

3. Apply ATE weights. Users get 1/0.2 = 5, 1/0.5 = 2 and 1/0.9 = 1.111, and non-users 1/0.8 = 1.25, 2 and 1/0.1 = 10. Each weighted arm then holds 500 low, 300 medium and 200 high-severity patients, for example 100 × 5 = 500 and 20 × 10 = 200, so high severity is 0.2 in both arms and the difference is zero.

4. Inspect the weights. Stabilised weights multiply by 0.43 for users and 0.57 for non-users, giving a mean of 1.0 but a maximum of 0.57 × 10 = 5.7. The 20 high-severity non-users are 2% of the sample yet stand for 20% of the weighted usual care arm.

5. Compare estimands. Suppose the service changes 12-month costs by GBP 900, GBP 800 and GBP 500, and QALYs by 0.01, 0.02 and 0.05, in the three groups. With ATE weights the incremental cost is 0.5 × 900 + 0.3 × 800 + 0.2 × 500 = GBP 790 and the QALY gain 0.5 × 0.01 + 0.3 × 0.02 + 0.2 × 0.05 = 0.021. ATT weights of 0.25, 1 and 9 reshape non-users to the users' mix. Over the 430 users the cost total is 100 × 900 + 150 × 800 + 180 × 500 = 300,000 and the QALY total 100 × 0.01 + 150 × 0.02 + 180 × 0.05 = 13.

$$\text{ICER}_{ATE} = \frac{790}{0.021} = 37{,}619 \qquad \text{ICER}_{ATT} = \frac{300{,}000}{13} = 23{,}077$$

where each ICER is in GBP per QALY gained, first for the registry and then for users.

6. Trim the extreme stratum. Dropping patients with scores of 0.9 leaves 800 patients, with incremental cost (500 × 900 + 300 × 800) / 800 = GBP 862.50, QALY gain (500 × 0.01 + 300 × 0.02) / 800 = 0.01375 and an ICER of 862.50 / 0.01375 = GBP 62,727 per QALY.

7. Interpretation. Against an illustrative threshold of GBP 30,000 per QALY, the service looks cost effective for current users but not for the whole registry population or the trimmed one. Each figure answers a different question about a different population, so the estimand has to match the decision before the outcomes are examined.

How HTA bodies treat propensity score analyses

The NICE manual (PMG36, section 3.3.9) notes that non-randomised studies tend to be at high risk of bias because the factors influencing treatment assignment may predict outcomes, and points to TSD 17 for methods addressing confounding with individual patient data. ECD9 states that where there are many potential confounders, multivariable regression and propensity score (or disease risk score) methods are preferred. For time-varying confounders it prefers g-methods such as marginal structural models with weighting.

TSD 17 asks for alternative methods to be presented as sensitivity analysis and warns that trimming to the region of common support redefines the group for whom the effect is estimated. Kreif and colleagues, whose drotrecogin alfa results the average treatment effect page reports, concluded that each method can give unbiased cost-effectiveness estimates if the score is correctly specified and the weights are stable.

The score beside neighbouring methods

Regression adjustment models the outcome, while propensity methods model treatment assignment. Austin notes that both rest on the no unmeasured confounding assumption, so switching between them leaves hidden bias untouched. TSD 17 states that none of these selection on observables methods can identify the effect when selection depends on unobservables, which is where instrumental variable analysis and difference-in-differences come in, with assumptions of their own. The causal inference page compares these designs in more depth.

Common misreadings of propensity score results

Several errors come from treating the score as a fix in itself instead of a device for balancing measured covariates. The sources above warn against each of them.

• Good balance proves no confounding. Balance is shown only for covariates in the model, and unmeasured differences are untouched.
• Weighting everyone by the inverse of the score. Austin and Stuart found studies that used the reciprocal of the score instead of the reciprocal of the probability of the treatment actually received.
• Judging the model by its c-statistic. TSD 17 treats scores of zero or one, which a near-perfect model of treatment produces, as a sign that overlap has failed.
• Treating weighted samples as ordinary data. Austin notes that variance estimation must allow for the weighting.

Sources

• Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research. 2011;46(3):399-424. https://pmc.ncbi.nlm.nih.gov/articles/PMC3144483/
• Austin PC, Stuart EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine. 2015;34(28):3661-3679. https://pmc.ncbi.nlm.nih.gov/articles/PMC4626409/
• Faria R, Hernandez Alava M, Manca A, Wailoo AJ. NICE DSU Technical Support Document 17: The use of observational data to inform estimates of treatment effectiveness in technology appraisal: methods for comparative individual patient data. Decision Support Unit, ScHARR; May 2015. https://doi.org/10.15131/shef.data.33847855
• Kreif N, Grieve R, Radice R, Sadique Z, Ramsahai R, Sekhon JS. Methods for estimating subgroup effects in cost-effectiveness analyses that use observational data. Medical Decision Making. 2012;32(6):750-763. https://doi.org/10.1177/0272989X12448929
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Section 3.3.9. https://www.nice.org.uk/process/pmg36/chapter/evidence-2
• National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). Published 23 June 2022. Methods for real-world studies of comparative effects. https://www.nice.org.uk/corporate/ecd9/chapter/methods-for-real-world-studies-of-comparative-effects
• Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41-55. https://doi.org/10.1093/biomet/70.1.41