Selection Model

Selection Model for Missing Outcomes in Trials

A selection model describes how a trial outcome, such as each participant's QALYs over 12 months, would be distributed if everyone had been followed up, and adds a second equation for the chance that each value was recorded. Because that second equation may depend on the outcome itself, the model can represent data that are missing not at random (MNAR), where people in poorer health are less likely to return a questionnaire than their recorded data would predict. In trial-based cost-effectiveness analysis it is mainly a sensitivity analysis, testing whether incremental QALYs and the ICER survive nonresponse that depends on unseen health. The best-known econometric form, Heckman's sample selection model, is also called the Heckman correction. The term has nothing to do with model selection, which means choosing between candidate statistical models.

Two equations for one incomplete outcome

A selection model splits the joint distribution of the outcome and the response indicator into a model for the outcome in the whole sample and a model for response given the outcome. The second part is where any assumption about why data are missing is written down.

$$f(Y, R \mid X) = f(Y \mid X)\, P(R \mid Y, X)$$

where $Y$ is the outcome (for example total QALYs or NHS costs), $R$ is the response indicator ($R = 1$ if $Y$ is observed and $R = 0$ if not), $X$ holds fully observed covariates such as trial arm and baseline utility, $f(Y \mid X)$ is the outcome model the economic analysis wants, and $P(R \mid Y, X)$ is the selection or response model.

If the response model depends only on observed quantities, the data are missing at random (MAR) and, under the conditions set out in Little and Rubin's textbook, the response model can be set aside when the outcome model is estimated by likelihood methods. The National Research Council (NRC) panel on missing data in clinical trials describes selection models as modelling the probability of nonresponse as a function of the outcome, and notes a one-to-one relationship between specifying a selection model and specifying a pattern mixture model.

A response model with one sensitivity parameter

The NRC panel's simplest selection model assumes that the log odds of nonresponse change by a fixed amount for each unit difference in the outcome. The analyst sets the amount and repeats the analysis across plausible values.

$$\operatorname{logit} P(R = 0 \mid Y = y) = h + \alpha y$$

where $h$ is an intercept estimated from the data, $\alpha$ is the change in the log odds of nonresponse per unit increase in $y$, and $y$ is the outcome value, observed or not. Setting $\alpha = 0$ gives MAR. The panel stresses that each value of $\alpha$ corresponds to a different unverifiable assumption, and that the observed data have no information about it.

Applying Bayes' theorem shows what a chosen $\alpha$ implies about the people who are missing.

$$f(y \mid R = 0) = \frac{f(y \mid R = 1)\, e^{\alpha y}}{E\left[e^{\alpha Y} \mid R = 1\right]}$$

where $f(y \mid R = 0)$ and $f(y \mid R = 1)$ are the outcome distributions among nonrespondents and respondents, and the denominator, the mean of $e^{\alpha Y}$ among respondents, makes the result a proper distribution. A negative $\alpha$ shifts the nonrespondents towards low values. The panel adds that two further assumptions cannot be checked, that the log odds are linear in $y$ and that nonrespondents have outcomes in the same range as respondents.

Once $\alpha$ is fixed, the full-sample mean can be estimated by inverse probability weighting of the respondents.

$$\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} \frac{R_i Y_i}{\pi(Y_i)}, \qquad \pi(y) = \frac{1}{1 + e^{h + \alpha y}}$$

where $n$ is the number randomised to the arm, $R_i$ and $Y_i$ are the response indicator and outcome for participant $i$, and $\pi(y)$ is the probability of being observed at outcome value $y$. The intercept $h$ cannot come from an ordinary logistic regression of $R$ on $Y$, because $Y$ is missing whenever $R = 0$. It is chosen so that the inverse probability weights, summed over respondents, equal $n$.

Heckman's sample selection model

Heckman's 1979 paper in Econometrica treated the bias from a nonrandomly selected sample as an ordinary specification error, a form of omitted variables bias, and proposed a simple consistent two-stage estimator. In its standard form a latent selection equation decides whether the outcome is seen.

$$S_i^{*} = Z_i \gamma + u_i, \qquad S_i = 1 \text{ if } S_i^{*} > 0, \qquad Y_i = X_i \beta + \varepsilon_i \text{ observed only if } S_i = 1$$

where $S_i^{*}$ is a latent propensity to be observed, $Z_i$ are selection variables with coefficients $\gamma$, $X_i$ are the outcome covariates with coefficients $\beta$, and $u_i$ and $\varepsilon_i$ are bivariate normal errors with correlation $\rho$, the variance of $u_i$ fixed at 1 and the standard deviation of $\varepsilon_i$ equal to $\sigma$.

$$E[Y_i \mid X_i, S_i = 1] = X_i \beta + \rho \sigma \lambda(Z_i \gamma), \qquad \lambda(c) = \frac{\phi(c)}{\Phi(c)}$$

where $\lambda$ is the inverse Mills ratio, $\phi$ is the standard normal density and $\Phi$ is the standard normal distribution function. A regression of the observed outcomes on $X_i$ alone omits the $\lambda$ term, which is the selection bias, and there is no bias when $\rho = 0$. In the two-step procedure in William Greene's lecture notes, a probit model for $S_i$ gives an estimated $\lambda$ for each person, the observed $Y_i$ are regressed on $X_i$ and $\lambda$, and the standard errors are corrected for the estimated regressor. Full information maximum likelihood estimation of both equations is the efficient alternative.

An exclusion restriction is a variable that enters $Z_i$ but not $X_i$, shifting the chance of being observed without affecting the outcome, a role close to that of an instrumental variable. Gomes, Kenward, Grieve and Carpenter, writing in Statistics in Medicine in 2020, note that the Heckman model needs correct assumptions about the joint distribution of missingness and outcome and a valid exclusion restriction. Motivated by the REFLUX study of laparoscopic surgery for gastro-oesophageal reflux disease, where half the sample had missing outcome data, they found a full-likelihood selection model less sensitive to the exclusion restriction than Heckman-type models.

Use in trial-based cost-effectiveness analysis

Faria, Gomes, Epstein and White, in their 2014 PharmacoEconomics guide, frame sensitivity analysis to the MAR assumption as asking whether the resource allocation decision changes if the data are MNAR, a form of structural uncertainty. They name selection models and pattern mixture approaches as the two main methods, and give a typical selection model assumption, that people in worse health may be more likely to have missing QALYs. A weighting approach approximates the selection model. Multiple imputation is done under MAR and the estimates are combined by a weighted version of Rubin's rules that gives more weight to imputations compatible with the proposed MNAR mechanism. The guide warns that this tends to fail for large departures from MAR, because a few imputations are over-weighted.

Leurent and colleagues' 2018 tutorial in PharmacoEconomics adds that selection model results can be very sensitive to their untestable assumptions, a particular problem in cost-effectiveness analysis because cost and effectiveness endpoints tend to be difficult to parametrise. The usual sensitivity parameter, an odds ratio for how a unit change in the outcome affects the chance of being observed, is also hard to elicit and to explain. Their example statement is that for each decrease of 0.1 in quality of life, the chance of being missing doubles. For repeated measurements the NRC panel models the odds of dropping out between visits given the recorded history and the possibly missing future outcomes.

Worked example with QALYs missing in both arms

All numbers here are illustrative. A two-arm trial randomises 100 people to each arm and records 12-month QALYs. To keep the arithmetic visible, every respondent has one of three values, 0.5, 0.7 or 0.9.

1. MAR base case. With $\alpha = 0$ the missing people resemble respondents, so the arm means stay at 0.675 and 0.6556. The incremental QALYs are 0.675 minus 0.655556, or 0.019444. With an illustrative incremental cost of GBP 400, the ICER is 400 / 0.019444 = GBP 20,571 per QALY.

2. Set the sensitivity parameter. Suppose the odds of nonresponse double for each 0.1 lower QALY value, in both arms. Then $\alpha = -10 \ln 2 = -6.931$ per QALY, and $e^{\alpha y} = 2^{-10y}$, which is 1/32 = 0.03125 at 0.5, 1/128 = 0.0078125 at 0.7 and 1/512 = 0.001953 at 0.9.

3. Tilt the new treatment arm. The weights are respondent counts times $e^{\alpha y}$, so 30 × 0.03125 = 0.9375, 30 × 0.0078125 = 0.2344 and 20 × 0.001953 = 0.0391, summing to 1.2109. The 20 missing people are shared in these proportions, giving 20 × 0.9375 / 1.2109 = 15.484 at 0.5, 3.871 at 0.7 and 0.645 at 0.9.

4. Means in the new treatment arm. The implied nonrespondent mean is (15.484 × 0.5 + 3.871 × 0.7 + 0.645 × 0.9) / 20 = 11.032 / 20 = 0.5516. The full-arm mean is (54 + 11.032) / 100 = 0.65032.

5. Usual care arm. The weights are 40 × 0.03125 = 1.25, 0.2344 and 0.0391, summing to 1.5234. The 10 missing people are shared as 8.205, 1.538 and 0.256, with a mean of (4.103 + 1.077 + 0.231) / 10 = 0.5410. The full-arm mean is (59 + 5.410) / 100 = 0.64410.

6. Decision result. Subtracting 0.64410 from 0.65032 leaves incremental QALYs of 0.00622, and the ICER rises to 400 / 0.00622 = GBP 64,309 per QALY. The new arm moves more because twice as many of its participants are missing.

Repeating steps 2 to 6 for milder assumptions gives a tipping-point table. At an illustrative threshold of GBP 30,000 per QALY the conclusion reverses between an odds ratio of 1.25 and 1.5 per 0.1 QALY.

The NRC panel suggests plotting the nonrespondents' implied mean against $\alpha$, because odds for a continuous outcome are hard to judge, and the table shows those means. The panel also notes that a different $\alpha$ can be chosen for each arm.

Where the selection model sits among nearby methods

The pattern mixture model places the assumption on the difference in outcome between respondents and nonrespondents. The NRC panel credits it with transparent sensitivity parameters and model checking that does not depend on the assumed mechanism, while the selection approach can easily accommodate auxiliary variables of any type.

Zero costs raise a different problem from missing costs. Mihaylova and colleagues, reviewing methods for healthcare cost data, excluded selection models from their review because zeros are genuine observations rather than unobserved values. The wider choice of method under MAR is covered on the missing data page.

Limitations and common misreadings

The commonest error is to read a fitted selection model as evidence about the missing data mechanism. Kenward, applying the Diggle and Kenward selection model for nonrandom dropout to a simple example, showed that conclusions about the dropout mechanism depended on untestable distributional assumptions, and that two outlying observations were the source of the apparent evidence for nonrandom dropout. A significant outcome coefficient in the response equation can therefore be an artefact of the assumed distribution, and one fitted $\alpha$ is no substitute for a range of values.

The NRC panel lists two practical weaknesses of the weighting form, that inverse weighting can give large standard errors and that the observed-data model has to be rechecked for every value of the sensitivity parameter. For the Heckman form, Gomes and colleagues found that the relative performance of methods differs with the relevance and strength of the exclusion restriction, so an exclusion variable chosen for convenience can mislead.

Sources

• Faria R, Gomes M, Epstein D, White IR. A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials. PharmacoEconomics. 2014;32(12):1157-1170. https://doi.org/10.1007/s40273-014-0193-3
• Gomes M, Kenward MG, Grieve R, Carpenter J. Estimating treatment effects under untestable assumptions with nonignorable missing data. Statistics in Medicine. 2020;39(11):1658-1674. https://doi.org/10.1002/sim.8504
• Greene WH. Econometrics I, Part 19, Sample Selection. Lecture notes, NYU Stern School of Business; 2017. https://pages.stern.nyu.edu/~wgreene/Econometrics/Econometrics-I-19.pdf
• Heckman JJ. Sample selection bias as a specification error. Econometrica. 1979;47(1):153-162. https://doi.org/10.2307/1912352
• Kenward MG. Selection models for repeated measurements with non-random dropout: an illustration of sensitivity. Statistics in Medicine. 1998;17(23):2723-2732. https://pubmed.ncbi.nlm.nih.gov/9881418/
• Leurent B, Gomes M, Faria R, Morris S, Grieve R, Carpenter JR. Sensitivity analysis for not-at-random missing data in trial-based cost-effectiveness analysis: a tutorial. PharmacoEconomics. 2018;36(8):889-901. https://doi.org/10.1007/s40273-018-0650-5
• Little RJA, Rubin DB. Statistical Analysis with Missing Data. 3rd ed. Wiley; 2019.
• Mihaylova B, Briggs A, O'Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Economics. 2011;20(8):897-916. https://doi.org/10.1002/hec.1653
• National Research Council. The Prevention and Treatment of Missing Data in Clinical Trials. Chapter 5, Principles and methods of sensitivity analyses. Washington, DC: National Academies Press; 2010. https://doi.org/10.17226/12955