Tipping-point shift for QALYs lost to follow-up at a cost-effectiveness threshold

Signature

H_d = Delta_C / lambda; delta_star = (H_d - Delta_E_obs) / (p_T - r * p_C)

Function

Delta-adjusted analysis of QALYs lost to follow-up with its tipping-point shift

Computational function

• Computational function for delta-adjusted QALY gain, ICER and tipping-point shift over a grid for loss to follow-up

  Takes each arm's QALYs after imputation under missing at random, with a flag for the participants lost to follow-up, the incremental cost, the threshold, the shift ratio r and a grid of shifts, adds each shift to the imputed QALYs of those lost (r times the shift in usual care), and returns the delta-adjusted QALY gain, ICER and incremental net health benefit at every shift, with the tipping point found where the net health benefit changes sign by the interpolation of HE-FM-OWSA-002. This is the scan that Leurent and colleagues and Faria and colleagues describe, run on one completed dataset; with several imputed datasets the scan is run on each and the gains are averaged by Rubin's rules. The arm means follow HE-FM-LOCF-001 and HE-FM-LOCF-002 as read in HE-FN-LTFU-001, the ratio is HE-FM-ICER-001 and the net health benefit is HE-FM-CET-002. For an additive shift on QALYs alone the net health benefit is linear in the shift, so the grid returns the closed form of HE-FM-LTFU-001, which the Math / Code line computes from arm summaries. The grid is needed when the conclusion is judged on a quantity that is not linear in the shift, such as the probability of cost-effectiveness from bootstrap samples, or when several parameters are varied together.

  Inputs and outputs: q is each participant's QALYs after imputation, in QALYs; arm is 1 for new treatment and 0 for usual care; lost is 1 for a participant lost to follow-up whose QALYs were imputed and 0 otherwise; dC is the incremental cost per participant in currency; lambda is the threshold in currency per QALY; r is the shift ratio, 0 for the new treatment arm only and 1 for both arms; grid is the shifts to try, in QALYs. In the Math / Code line E_obs_T and E_obs_C are the arm means of those observed, p_T and p_C the proportions lost, delta one shift, Delta_C the incremental cost, E_T and E_C the delta-adjusted arm means, Delta_E the gain, ICER the ratio, INHB the incremental net health benefit in QALYs and delta_star the tipping-point shift. The R and Python versions return the grid with the net health benefit at each shift and the tipping point, or NA and None when the sign does not change on the grid.

  Assumption: The shift is additive on the imputed QALYs of those lost and costs are not shifted (HE-AS-LTFU-001, HE-AS-LTFU-002). The interpolation is exact when the net health benefit is linear in the shift and an approximation otherwise (HE-FM-OWSA-002). The grid runs from zero towards worse outcomes, so it finds a tipping point only when the base case is cost-effective; for a base case that is not, the grid runs towards positive shifts.

  Worked example (Article's step 1, shift of -0.10 in both arms): The arm means fall to 0.76 and 0.67, the gain is 0.09, the ICER is GBP 16,667 per QALY and the net health benefit at GBP 20,000 per QALY is 0.015 QALYs, as in the article; the common-shift tipping point is -0.25. E_obs_T = 0.78; E_obs_C = 0.68; p_T = 0.2; p_C = 0.1; r = 1; delta = -0.1; Delta_C = 1500; lambda = 20000; E_T = 0.76; E_C = 0.67; Delta_E = 0.09; ICER = 16666.67; INHB = 0.015; delta_star = -0.25

  Worked example (Article's step 2, shift of -0.10 in the new treatment arm only): The means are 0.76 and 0.68, the gain is 0.08 and the ICER is GBP 18,750 per QALY, as in the article, with a tipping point of -0.125. E_obs_T = 0.78; E_obs_C = 0.68; p_T = 0.2; p_C = 0.1; r = 0; delta = -0.1; Delta_C = 1500; lambda = 20000; E_T = 0.76; E_C = 0.68; Delta_E = 0.08; ICER = 18750; INHB = 0.005; delta_star = -0.125

  Worked example (At the tipping point): A shift of -0.125 in the new treatment arm only gives a gain of 0.075, an ICER equal to the threshold and a net health benefit of zero, a limiting case. E_obs_T = 0.78; E_obs_C = 0.68; p_T = 0.2; p_C = 0.1; r = 0; delta = -0.125; Delta_C = 1500; lambda = 20000; E_T = 0.755; E_C = 0.68; Delta_E = 0.075; ICER = 20000; INHB = 0; delta_star = -0.125

  Worked example (Participant-level table, for the R and Python code): An illustrative new treatment arm of five with QALYs 0.70, 0.80, 0.84 and 0.78 observed and one participant lost and imputed at 0.78, and a usual care arm of ten with 0.60, 0.65, 0.70, 0.72, 0.66, 0.70, 0.68, 0.71 and 0.70 observed and one imputed at 0.68, reproduce the article's means and shares lost. ltfu_scan(q, arm, lost, 1500, 20000) with the grid 0 to -0.30 in steps of 0.02 returns a tipping point of -0.125, and with r = 1 it returns -0.25. The scalar form is the Math / Code line and HE-EX-LTFU-001.

  Excel: With the inputs in cells named ObsMeanNew, ObsMeanUsual, LostShareNew, LostShareUsual, ShiftRatio, DeltaCost and ThresholdQALY, put the shifts 0, -0.02 and so on to -0.30 in A2:A17 (named GridShift), enter =(ObsMeanNew+LostShareNew*A2)-(ObsMeanUsual+ShiftRatio*LostShareUsual*A2) in B2 for the gain, =DeltaCost/B2 in C2 for the ICER and =B2-DeltaCost/ThresholdQALY in D2 for the net health benefit, and fill down to row 17 (name C2:C17 GridICER and D2:D17 GridINHB). A cell named GridRowBelow holds =MATCH(TRUE,INDEX(GridINHB<0,0),0), the first grid row with a negative net health benefit, and a cell named GridTipping holds =INDEX(GridShift,GridRowBelow-1)-INDEX(GridINHB,GridRowBelow-1)*(INDEX(GridShift,GridRowBelow)-INDEX(GridShift,GridRowBelow-1))/(INDEX(GridINHB,GridRowBelow)-INDEX(GridINHB,GridRowBelow-1)).

  R: ltfu_scan <- function(q, arm, lost, dC, lambda, r = 0, grid = seq(0, -0.3, by = -0.02)) { inhb <- sapply(grid, function(d) { qa <- q+d*lost*(arm == 1)+r*d*lost*(arm == 0); mean(qa[arm == 1])-mean(qa[arm == 0])-dC/lambda }); k <- which(diff(sign(inhb)) != 0)[1]; tip <- if (is.na(k)) NA else grid[k]-inhb[k]*(grid[k+1]-grid[k])/(inhb[k+1]-inhb[k]); list(table = data.frame(delta = grid, INHB = inhb, ICER = dC/(inhb+dC/lambda)), tipping = tip) } Run on each imputed dataset when there are several and average the gains.

  Python: def ltfu_scan(q, arm, lost, dC, lam, r=0.0, grid=None): grid = grid or [-0.02*i for i in range(16)]; f = lambda d: (lambda qa: sum(x for x, a in zip(qa, arm) if a == 1)/arm.count(1)-sum(x for x, a in zip(qa, arm) if a == 0)/arm.count(0)-dC/lam)([x+d*l*(a == 1)+r*d*l*(a == 0) for x, a, l in zip(q, arm, lost)]); v = [f(d) for d in grid]; k = next((i for i in range(len(v)-1) if (v[i] > 0) != (v[i+1] > 0)), None); return list(zip(grid, v)), (None if k is None else grid[k]-v[k]*(grid[k+1]-grid[k])/(v[k+1]-v[k])) Plain Python with no imports.

  Test (Grid tipping point equals the closed form): With the article's inputs the value in GridTipping equals TippingShift of HE-IM-LTFU-001 within 1E-9. Expected result TRUE. FALSE shows a grid that adds the shift to every participant in the new treatment arm rather than to those lost, which returns -0.025 instead of -0.125. Excel check =ABS(GridTipping-TippingShift)<1E-9

  Test (Article's step 2 ICER on the grid): With the article's inputs and ShiftRatio 0, the sixth grid row, a shift of -0.10, holds an ICER of GBP 18,750 per QALY. Expected result TRUE. FALSE shows ShiftRatio left at 1, which gives GBP 16,667 per QALY, the article's step 1. Excel check =ABS(INDEX(GridICER,6)-18750)<0.01

  Common error (Observed means used when the imputation moved the base case): If the participant lost from the new treatment arm in the table above is imputed at 0.70 under missing at random, for example because of a worse baseline, the base-case gain is 0.084 and the tipping point is -0.045. Feeding the observed-mean gain of 0.10 into HE-FM-LTFU-001 returns -0.125 and overstates the shift the conclusion can absorb by a factor of almost three (computed here for illustration).

  Source: Leurent B, Gomes M, Faria R, Morris S, Grieve R, Carpenter JR. Sensitivity analysis for not-at-random missing data in trial-based cost-effectiveness analysis: a tutorial. PharmacoEconomics. 2018;36(8):889-901. Section 2.3, on adding delta to or multiplying the values imputed under missing at random and combining with Rubin's rules, and section 4.3, on the tipping-point approach. Faria R, Gomes M, Epstein D, White IR. PharmacoEconomics. 2014;32(12):1157-1170. Section 5, on repeating the shifted analysis over a range of values and reporting the value of the sensitivity parameter required to change the results.

  E_T = E_obs_T + p_T * delta; E_C = E_obs_C + r * p_C * delta; Delta_E = E_T - E_C; ICER = Delta_C / Delta_E; INHB = Delta_E - Delta_C / lambda; delta_star = (Delta_C / lambda - (E_obs_T - E_obs_C)) / (p_T - r * p_C)

Try this function

Implementations

• Excel

  Excel tipping-point shift for loss to follow-up from arm summaries

  With the incremental cost in a cell named DeltaCost, the threshold in ThresholdQALY, the base-case QALY gain in GainObserved, the proportions lost in LostShareNew and LostShareUsual and the shift ratio r in ShiftRatio, this formula in a cell named TippingShift returns the tipping-point shift, or #N/A when the denominator is not above zero and no tipping point exists.

  =IF(LostShareNew-ShiftRatio*LostShareUsual<=0,NA(),(DeltaCost/ThresholdQALY-GainObserved)/(LostShareNew-ShiftRatio*LostShareUsual))

Assumptions

• Additive shift on imputed QALYs of those lost to follow-up

  The shift is an offset added to the QALYs of those lost, the delta parameter of the pattern-mixture approach that Leurent and colleagues describe, applied after imputation under missing at random. Because a fraction p of an arm is shifted by delta, the arm mean moves by p times delta in every imputed dataset, and Rubin's rules average the point estimates, so the pooled QALY gain moves by the same amount (derived). With covariates in the imputation model the base-case gain is the multiple imputation estimate, which can differ from the difference in observed means; the article's example takes the two to be equal.

• Incremental cost unchanged by the loss to follow-up shift

  Only QALYs are shifted and Delta_C and lambda are fixed, as in the article's example, where costs come from complete linked hospital records. When missing costs are also shifted, as in the extension that Leurent and colleagues report in section 4.1 of their tutorial, the incremental cost moves with the assumption and the tipping point is found by repeating the analysis over a grid of shifts on the incremental net monetary benefit (HE-FM-NMB-002), as HE-CF-LTFU-001 does for QALYs.

• Positive denominator and the tie at the loss to follow-up tipping point

  The denominator, p_T minus r times p_C, is above zero. That holds for a shift confined to the new treatment arm whenever anyone in that arm is lost, and for a common shift only when the new treatment arm loses the larger share. With a common shift and equal shares lost the QALY gain does not move and there is no tipping point, and with a smaller share lost from the new treatment arm a common worsening raises the gain (derived). At delta equal to delta_star the delta-adjusted ICER equals lambda and the incremental net health benefit of HE-FM-CET-002 is zero. The package reports that boundary value as the tipping point and does not classify the tie as acceptance or rejection.

Worked examples

• Loss to follow-up tipping point with the shift in the new treatment arm only

  The article's illustrative trial has an incremental cost of GBP 1,500, observed mean QALYs of 0.78 and 0.68, and 20% lost from the new treatment arm against 10% from usual care. At an illustrative GBP 20,000 per QALY the gain must stay above 0.075, and with the shift in the new treatment arm only the tipping point is -0.125, as in the article. The conclusion holds unless those lost from the new treatment arm are worse off than those observed by more than 0.125 QALYs.

  Delta_C = 1500; lambda = 20000; H_d = 0.075; Delta_E_obs = 0.1; p_T = 0.2; p_C = 0.1; r = 0; delta_star = -0.125

• Loss to follow-up tipping point with the same shift in both arms

  With the same illustrative inputs and the same shift in both arms, the gain moves by only 0.1 times the shift, the difference between the 20% and 10% lost, so the tipping point is -0.25 (computed here for illustration). This matches step 1 of the article, where a common shift of -0.10 lowers the gain by only 0.01.

  Delta_C = 1500; lambda = 20000; H_d = 0.075; Delta_E_obs = 0.1; p_T = 0.2; p_C = 0.1; r = 1; delta_star = -0.25

• Loss to follow-up tipping point at a threshold equal to the observed ICER

  When the threshold is set at the observed ICER of GBP 15,000 per QALY, the gain needed equals the observed gain of 0.10 and the tipping point is zero, a limiting case in which any worsening for those lost from the new treatment arm changes the conclusion (illustrative).

  Delta_C = 1500; lambda = 15000; H_d = 0.1; Delta_E_obs = 0.1; p_T = 0.2; p_C = 0.1; r = 0; delta_star = 0

Common errors

• QALY gain shortfall reported as the tipping-point shift for those lost to follow-up

  In the article's example the gain must fall from 0.10 to 0.075 for the ICER to reach GBP 20,000 per QALY, a shortfall of 0.025, but only the 20% of the new treatment arm lost to follow-up are shifted, so those lost must be worse off by 0.025 / 0.2 = 0.125 QALYs. Reporting 0.025 as the tipping point understates the shift the conclusion withstands by a factor of 1 / p_T, five here. The shift is a difference in mean QALYs for the people lost, while the shortfall is a difference in the arm mean over everyone randomised.

• Common-shift tipping point quoted for loss to follow-up confined to one arm

  With 20% lost from the new treatment arm and 10% from usual care, a shift common to both arms moves the gain by only 0.1 times the shift, so its tipping point is -0.25, twice as far from zero as the -0.125 for a shift confined to the new treatment arm. Quoting -0.25 when those who leave the new treatment arm may be worse off because of side effects overstates how far the conclusion can be trusted. With equal shares lost a common shift has no tipping point, which says nothing about a shift in one arm. Leurent and colleagues found that departures assumed the same in both arms had little effect on incremental QALYs, which they expect when the missing data pattern is broadly similar across arms, so that the bias applies roughly equally to each arm and cancels out in the comparison, while arm-specific departures changed the conclusion.

• Agreement of complete case and multiple imputation read as robustness to loss to follow-up

  Running complete case analysis as the base case and multiple imputation as a sensitivity analysis, or the reverse, and finding the same conclusion does not show that the result holds under loss to follow-up that is missing not at random. Leurent, Gomes and Carpenter call this misleading because both methods rely on missingness being independent of the unobserved data. In the article's example, where the missing QALYs are taken to resemble the observed ones, both give a gain of 0.10, while the conclusion at GBP 20,000 per QALY changes once those lost from the new treatment arm are worse off by more than 0.125 QALYs.

Sources

• Leurent and colleagues tutorial on delta and tipping-point analysis for missing trial data

  Leurent B, Gomes M, Faria R, Morris S, Grieve R, Carpenter JR. Sensitivity analysis for not-at-random missing data in trial-based cost-effectiveness analysis: a tutorial. PharmacoEconomics. 2018;36(8):889-901. Section 2.3, Sensitivity Analysis with Pattern-Mixture Models, describes an offset parameter delta as the average difference between the missing and observed values, Y_miss = Y_obs + delta, and the steps of imputing under missing at random, modifying the imputed values by adding delta or multiplying by a scale parameter, and combining the analyses with Rubin's rules. Section 4.3, Choosing the MNAR Parameters, describes the tipping-point or threshold analysis as changing the MNAR parameter until a different conclusion is reached, such as being or not being cost-effective. Section 4.1, Missing Cost, extends the shifts to missing costs. The Results of the illustrative application state that departures assumed the same in both arms had little effect on incremental QALYs, which will usually be the case when the missing data pattern is broadly similar across arms, as the bias applies roughly equally to each arm and cancels out in the comparison. The closed form for the tipping-point shift is derived from these statements, not printed in the paper.

  View source →

• Faria and colleagues guide on shifting imputed costs and QALYs by a sensitivity parameter

  Faria R, Gomes M, Epstein D, White IR. A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials. PharmacoEconomics. 2014;32(12):1157-1170. Section 5, Stage 3, Sensitivity Analysis to the Missing at Random (MAR) Assumption, states that under the pattern mixture approach data are first imputed under MAR, the imputed values are shifted by a sensitivity parameter, the analysis is repeated for a range of plausible values, and either a range of results or the value of the sensitivity parameter required to change the results is reported. Section 6.3 applies the approach to the REFLUX study and finds that the probability of cost-effectiveness changes considerably only when the QALYs of those with missing data in the surgery arm alone are reduced.

  View source →

• Leurent, Gomes and Carpenter review on sensitivity analyses for missing data in trial-based CEA

  Leurent B, Gomes M, Carpenter JR. Missing data in trial-based cost-effectiveness analysis: an incomplete journey. Health Economics. 2018;27(6):1024-1040. Section 4.3, Recommendations, states that many reviewed studies used complete case analysis for the primary analysis and multiple imputation as a sensitivity analysis, or the reverse, and concluded that the results were robust to missing data, and calls this misleading because both methods rely on the assumption that the missingness is independent of the unobserved data. Section 4.1, Summary of findings, reports that only two studies (4%) assessed whether conclusions were sensitive to departures from missing at random.

  View source →

Canonical Identity