Regression-based EVPPI estimator from fitted incremental net benefits

Regresses the incremental net benefit of each option against a baseline on the parameters of interest across the N runs of one probabilistic sample and treats the fitted values as the conditional expected net benefits, so no inner loop or extra model runs are needed. The estimate is the mean over runs of the largest fitted value, with zero for the baseline, less the largest mean fitted value. This is Equation 9 of Strong, Oakley and Brennan written for three options as incremental net benefits, which they note leaves the result unchanged and needs one regression fewer. Using the fitted values in both terms exploits the positive correlation between them and improves precision. Strong, Oakley and Brennan fit a generalised additive model or a Gaussian process, and Algorithm 4 of the second ISPOR report sets out the same scheme.

Signature

EVPPI_R = sum_(n=1)^N [max(0, g_2n, g_3n)] / N - max(0, sum_(n=1)^N [g_2n] / N, sum_(n=1)^N [g_3n] / N)
Inputs
InputsDefinitionUnit
g_2nFitted value at run n from the regression of option 2's net benefit less the baseline's on theta_i, listed over the N runscurrency per person
g_3nFitted value at run n from the regression of option 3's net benefit less the baseline's on theta_i, listed over the N runscurrency per person
NNumber of runs in the probabilistic sample, each with sampled values of all parameters and the net benefit of every optiona whole number
Output
EVPPI_RRegression-based estimate of the per-person expected value of perfect information about theta_icurrency per person, such as GBP

Function

Estimation of the expected value of partial perfect information from a probabilistic analysis

Maps a decision model, or the probabilistic sample it has produced, and a chosen group of parameters of interest theta_i to an estimate of the expected value of partial perfect information (EVPPI) per person. Each estimator approximates the expected net benefit of every option given theta_i, takes the best option for each value of theta_i, averages, and subtracts the value of the best option on current information. The definition and its order of expectation and maximisation, with a discrete worked case, are on the Value of Information page (HE-FM-VOI-003), the per-person EVPI that bounds every EVPPI from the same sample is HE-FM-VOI-001, and population scaling is HE-FM-VOI-002. This package adds the estimators set out by the ISPOR Value of Information Task Force, with checks that each estimate is at least zero and no larger than the EVPI. Notation follows the Expected Value of Partial Perfect Information article.

Computational function

  • Computational function for regression-based EVPPI and its EVPI bound from a probabilistic sample table

    Takes a probabilistic sample as an analyst holds it, a table of N runs with the sampled values of the parameters of interest and the net benefit of each of D options, fits a least squares regression of each option's net benefit less the baseline's on the chosen parameters, and returns the EVPPI from the fitted values, the EVPI from the raw net benefits and a flag that the estimate lies within its bounds. It chains HE-FM-EVPP-003 with HE-FM-VOI-001 for any number of options, so the inputs differ from the formula's variables, which start from fitted values for three options. A linear model in indicator, polynomial or product terms stands in here for the generalised additive model of Strong, Oakley and Brennan. The Math / Code line takes two options and four runs, with the fitted values from the regression entered as inputs, and the R and Python versions fit the regression for any number of runs and options.

    Inputs and outputs: X is the N by p table of sampled values of the parameters of interest, one row per run, with any interaction terms added as further columns. NB is the N by D table of net benefits from the same runs, column 1 the baseline option, in currency per person. EVPPI is the regression-based estimate, EVPI the per-person EVPI from the raw net benefits, both in currency per person, and ok is 1 when EVPPI is at least zero and no more than EVPI plus a tolerance. In the Math / Code line g_1 to g_4 are the fitted incremental net benefits of the new option at the four runs and b_1 to b_4 the raw incremental net benefits, in currency per person.

    Assumption: The runs are equally weighted draws from the joint distribution of all parameters. The regression is flexible enough to give the conditional mean of each incremental net benefit given the parameters of interest, so interactions among them are entered as columns of X.

    Worked example (Article's treatment effect A from the four equally likely combinations): With an indicator of high A as the only column of X and standard care as the baseline, the fitted values are the conditional means GBP 1,500 and GBP -1,100 and the estimate is the article's GBP 550, with an EVPI of GBP 600. X = [[1],[1],[0],[0]]; NB = [[0,2000],[0,1000],[0,200],[0,-2400]]; EVPPI = 550; EVPI = 600; ok = 1 For the Math / Code line g_1 = 1500; g_2 = 1500; g_3 = -1100; g_4 = -1100; b_1 = 2000; b_2 = 1000; b_3 = 200; b_4 = -2400; EVPPI = 550; EVPI = 600; ok = 1

    Worked example (Article's utility B): An indicator of high B returns the article's GBP 350. X = [[1],[0],[1],[0]]; NB = [[0,2000],[0,1000],[0,200],[0,-2400]]; EVPPI = 350; EVPI = 600; ok = 1 For the Math / Code line g_1 = 1100; g_2 = -700; g_3 = 1100; g_4 = -700; b_1 = 2000; b_2 = 1000; b_3 = 200; b_4 = -2400; EVPPI = 350; EVPI = 600; ok = 1

    Worked example (A and B together with their interaction): Columns for A, B and their product fit every output exactly, so the estimate equals the EVPI, as the article states for the pair. X = [[1,1,1],[1,0,0],[0,1,0],[0,0,0]]; NB = [[0,2000],[0,1000],[0,200],[0,-2400]]; EVPPI = 600; EVPI = 600; ok = 1 For the Math / Code line g_1 = 2000; g_2 = 1000; g_3 = 200; g_4 = -2400; b_1 = 2000; b_2 = 1000; b_3 = 200; b_4 = -2400; EVPPI = 600; EVPI = 600; ok = 1

    Excel: =LET(g,TREND(RawInc2,ThetaI),AVERAGE(IF(g>0,g,0))-MAX(AVERAGE(g),0)) Returns the estimate for two options from the raw incremental net benefits in RawInc2 and the regressor columns in ThetaI. =AVERAGE(IF(RawInc2>0,RawInc2,0))-MAX(AVERAGE(RawInc2),0) returns the EVPI from the same runs. LET needs Excel 365 or Excel 2021.

    R: evppi_reg <- function(X, NB, tol = 1e-6) { X <- as.data.frame(X); B <- NB-NB[, 1]; G <- sapply(1:ncol(B), function(d) { y <- B[, d]; if (all(y == 0)) y else fitted(lm(y ~ ., data = X)) }); evppi <- mean(apply(G, 1, max))-max(colMeans(G)); evpi <- mean(apply(B, 1, max))-max(colMeans(B)); c(EVPPI = evppi, EVPI = evpi, ok = as.numeric(evppi >= -tol && evppi <= evpi + tol)) } Base R only. For a generalised additive model, the lm call is replaced by a gam call from the mgcv package.

    Python: import numpy as np then def evppi_reg(X, NB, tol=1e-6): B = NB-NB[:, [0]]; Z = np.column_stack([np.ones(len(B)), X]); G = Z @ np.linalg.lstsq(Z, B, rcond=None)[0]; evppi = G.max(axis=1).mean()-G.mean(axis=0).max(); evpi = B.max(axis=1).mean()-B.mean(axis=0).max(); return evppi, evpi, int(-tol <= evppi <= evpi+tol) NumPy arrays in, a tuple out in the order of the outputs.

    Test (Parameter that never changes the decision): Adding the article's adverse-event cost, which moves incremental net benefit by GBP 100 up or down, to the four combinations gives eight equally likely runs. A regression on an indicator of the adverse-event cost value alone returns an EVPPI of zero and an EVPI of GBP 600, as in the article. The expected result is TRUE. Excel check =LET(g,TREND({2100;1900;1100;900;300;100;-2300;-2500},{1;0;1;0;1;0;1;0}),ROUND(AVERAGE(IF(g>0,g,0))-MAX(AVERAGE(g),0),6)=0)

    Test (Article's EVPPI for A from TREND): The four combinations with an indicator of high A return GBP 550. The expected result is TRUE. Excel check =LET(g,TREND({2000;1000;200;-2400},{1;1;0;0}),ROUND(AVERAGE(IF(g>0,g,0))-MAX(AVERAGE(g),0),6)=550)

    Common error (Interaction left out of the regression): Columns for A and B without their product fit GBP 2,400, 600, -200 and -2,000 and return GBP 550 for the pair in place of GBP 600, the EVPI. Strong, Oakley and Brennan use a tensor product term te(x5,x14) when inputs interact.

    Source: Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample, a nonparametric regression approach. Medical Decision Making. 2014;34(3) 311-326. Equations 8 and 9, the invariance to incremental net benefits and Box 1. Rothery C, Strong M, Koffijberg HE, et al. Value of information analytical methods. Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(3) 277-286. Algorithm 4.

    EVPPI = (max(g_1, 0) + max(g_2, 0) + max(g_3, 0) + max(g_4, 0)) / 4 - max((g_1 + g_2 + g_3 + g_4) / 4, 0); EVPI = (max(b_1, 0) + max(b_2, 0) + max(b_3, 0) + max(b_4, 0)) / 4 - max((b_1 + b_2 + b_3 + b_4) / 4, 0); ok = (EVPPI >= 0) and (EVPPI <= EVPI)

Implementations

  • Excel

    Regression-based EVPPI from columns of fitted values

    With the fitted incremental net benefits of options 2 and 3 against option 1 in ranges named FitInc2 and FitInc3, the first formula returns the estimate in Excel 365. A linear least squares regression on the columns of theta_i in a range named ThetaI gives the fitted values of the raw incremental net benefits in RawInc2 with TREND, as in the second formula, entered as a spill range. A flexible regression such as a generalised additive model is fitted outside Excel and its fitted values pasted in.

    =LET(m,BYROW(HSTACK(FitInc2,FitInc3),LAMBDA(x,MAX(0,x))),AVERAGE(m)-MAX(0,AVERAGE(FitInc2),AVERAGE(FitInc3))); =TREND(RawInc2,ThetaI)

Assumptions

  • Regression captures the conditional mean of net benefit given theta_i

    The fitted value at each run estimates the expectation of incremental net benefit over theta_c given theta_i at that run. The regression must therefore be flexible enough for the shape of that relationship, including interactions within theta_i, and Strong, Oakley and Brennan inspect residual plots for each fit.

  • One probabilistic sample from the joint distribution for regression EVPPI

    The N runs are draws from the joint distribution of all parameters, correlated or not, with the net benefit of every option computed at each run, and the same runs give the EVPI used as the upper bound. Strong, Oakley and Brennan note that only the joint distribution is required, even when inputs are correlated.

Worked examples

  • Illustrative three-option regression EVPPI from four fitted runs

    All values are illustrative. Over four equally weighted runs the fitted incremental net benefits are GBP 900, 300, -200 and -600 for option 2 and GBP 400, 700, 100 and -900 for option 3. The largest of zero and the two fitted values averages GBP 425, the best mean is option 2's GBP 100, and the estimate is GBP 325.

    N = 4; g_2n = [900,300,-200,-600]; g_3n = [400,700,100,-900]; EVPPI_R = 325

  • Fitted values equal to the model outputs return the EVPI

    Limiting case, illustrative. If the regression reproduces every model output, as a regression on all parameters can, the fitted values are the raw incremental net benefits, here GBP 1,300, 500, -600 and -800 for option 2 and GBP 100, 1,100, 300 and -1,200 for option 3, with the same means as in the first example. The estimate is then the EVPI from these runs, GBP 575.

    N = 4; g_2n = [1300,500,-600,-800]; g_3n = [100,1100,300,-1200]; EVPPI_R = 575

  • Parameters of interest unrelated to net benefit give zero regression EVPPI

    Limiting case. When the regression finds no relationship, every fitted value equals the mean of its option, GBP 100 for option 2 and GBP 75 for option 3, the same option is best at every run and the estimate is zero.

    N = 4; g_2n = [100,100,100,100]; g_3n = [75,75,75,75]; EVPPI_R = 0

Common errors

  • Additive regression that omits an interaction between parameters of interest

    In the article's example the incremental net benefits of GBP 2,000, 1,000, 200 and -2,400 depend on A and B jointly. A least squares regression on indicators of A and B without an interaction term fits GBP 2,400, 600, -200 and -2,000 and returns GBP 550 for the pair, although the EVPPI for A and B together equals the EVPI of GBP 600 (computed here for illustration). Strong, Oakley and Brennan replace the additive formula INB ~ s(x5)+s(x14) with the tensor product INB ~ te(x5,x14) when the parameters interact.

  • Generalised additive model with many interacting parameters of interest

    Strong, Oakley and Brennan warn that with more than about six parameters of interest that interact in the model, the number of GAM parameters is likely to exceed the number of runs, so the method fails. They recommend Gaussian process regression in that case.

  • Regressing on every model parameter for a group EVPPI

    Including parameters outside theta_i as regressors lets the fit follow the model outputs more closely, so the estimate moves towards the EVPI. In the illustrative table a fit that reproduces every output returns GBP 575, the EVPI, in place of GBP 325 for theta_i.

Sources

  • Nonparametric regression estimator of partial EVPI

    Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample, a nonparametric regression approach. Medical Decision Making. 2014;34(3) 311-326. Methods section, Equation 8, which writes each net benefit as an unknown function g of the parameters of interest plus a mean-zero error, Equation 9, the estimator from the fitted values, the note that fitted values in both terms exploit the positive correlation between the terms, the invariance to incremental net benefits relative to a baseline option, Box 1 with the mgcv code INB ~ s(x5)+s(x14) and te(x5,x14) for interacting inputs, and the Limitations paragraph on more than about six interacting parameters.

    View source →

  • ISPOR regression-based single-loop scheme for EVPPI

    Rothery C, Strong M, Koffijberg HE, Basu A, Ghabri S, Knies S, Murray JF, Sanders Schmidler GD, Steuten L, Fenwick E. Value of information analytical methods. Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(3) 277-286. Section on EVPPI computation and Algorithm 4, which generates the probabilistic sample, regresses the utility of each decision option on the target parameters, calculates the fitted values and follows the EVPI steps, and notes that the method needs only the single set of samples from the probabilistic analysis.

    View source →

Canonical Identity

Persistent URI
https://healtheconomics.wiki/formula/regression-based-evppi-estimator-from-fitted-incremental-net-benefits

Stable URI · Machine-readable · Resolvable · CC BY 4.0