Signature
ibar_1 = (INB_11 + INB_12) / J; ibar_2 = (INB_21 + INB_22) / J; EVPPI_2L = (max(ibar_1, 0) + max(ibar_2, 0)) / K - max((ibar_1 + ibar_2) / K, 0)
| Inputs | Definition | Unit |
|---|---|---|
INB_11 | Net benefit of the new option less that of standard care from the model run at outer draw 1 of theta_i and inner draw 1 of theta_c | currency per person |
INB_12 | Net benefit of the new option less that of standard care at outer draw 1 of theta_i and inner draw 2 of theta_c | currency per person |
J | Number of values of theta_c drawn in the inner loop for each outer draw, 2 in the calculator form, so the model is run K times J times | a whole number |
INB_21 | Net benefit of the new option less that of standard care at outer draw 2 of theta_i and inner draw 1 of theta_c | currency per person |
INB_22 | Net benefit of the new option less that of standard care at outer draw 2 of theta_i and inner draw 2 of theta_c | currency per person |
K | Number of values of theta_i drawn in the outer loop, 2 in the calculator form | a whole number |
ibar_1 | Mean of the inner-loop incremental net benefits at outer draw 1, an estimate of the expected incremental net benefit given theta_i at that draw | currency per person |
|---|---|---|
ibar_2 | Mean of the inner-loop incremental net benefits at outer draw 2 | currency per person |
EVPPI_2L | Nested Monte Carlo estimate of the per-person expected value of perfect information about theta_i | currency per person, such as GBP |
Function
Estimation of the expected value of partial perfect information from a probabilistic analysis
Maps a decision model, or the probabilistic sample it has produced, and a chosen group of parameters of interest theta_i to an estimate of the expected value of partial perfect information (EVPPI) per person. Each estimator approximates the expected net benefit of every option given theta_i, takes the best option for each value of theta_i, averages, and subtracts the value of the best option on current information. The definition and its order of expectation and maximisation, with a discrete worked case, are on the Value of Information page (HE-FM-VOI-003), the per-person EVPI that bounds every EVPPI from the same sample is HE-FM-VOI-001, and population scaling is HE-FM-VOI-002. This package adds the estimators set out by the ISPOR Value of Information Task Force, with checks that each estimate is at least zero and no larger than the EVPI. Notation follows the Expected Value of Partial Perfect Information article.
Try this function
Implementations
Excel
Nested EVPPI from a grid of inner-loop results
With the incremental net benefits in a range named INBGrid, one row for each outer draw and one column for each inner draw, the first formula returns the estimate in Excel 365, where BYROW and LAMBDA are available. In older versions a column named InnerMean holding the AVERAGE of each row feeds the second formula. Both use the mean of the inner means for the current decision, as in step 7 of Algorithm 2.
=LET(m,BYROW(INBGrid,LAMBDA(x,AVERAGE(x))),AVERAGE(IF(m>0,m,0))-MAX(AVERAGE(m),0)); =SUMPRODUCT((InnerMean>0)*InnerMean)/COUNT(InnerMean)-MAX(AVERAGE(InnerMean),0)
Assumptions
Inner draws from the conditional distribution of the remaining parameters
For each outer draw the J values of theta_c come from their distribution conditional on theta_i at that draw, as step 2 of Algorithm 2 in the second ISPOR report requires. Only when theta_i and theta_c are independent may the inner draws come from the marginal distribution of theta_c. Strong, Oakley and Brennan note that sampling the conditional distribution can require Markov chain Monte Carlo when parameters are correlated, which adds substantially to the computation.
Inner and outer loop sizes chosen for bias and precision of nested EVPPI
The maximum of a Monte Carlo mean is biased upwards, so a small inner loop J biases the estimate, and good practice recommendation 5 of the second ISPOR report asks for inner and outer loop sizes chosen to give acceptable bias and precision. Brennan and colleagues found in their case studies that fewer samples on the outer level and more on the inner level could be efficient.
Incremental net benefit at a stated threshold for nested EVPPI
Each INB_kj is the net monetary benefit of the new option less that of standard care at one stated cost-effectiveness threshold, computed from the same parameter values for both options. EVPPI changes with the threshold, so the loops are rerun for every threshold reported and the results are not rescaled.
Worked examples
Article's EVPPI for the treatment effect with exact inner means
All values are illustrative and come from the article's example. The two equally likely values of the treatment effect A are the outer draws and the two equally likely values of the utility B the inner draws, so the inner means are GBP 1,500 when A is high and GBP -1,100 when A is low. Standard care is chosen when A is low, the average of the kept values is GBP 750, and subtracting the current-decision value of GBP 200 leaves GBP 550, the article's EVPPI for A.
K = 2; J = 2; INB_11 = 2000; INB_12 = 1000; INB_21 = 200; INB_22 = -2400; ibar_1 = 1500; ibar_2 = -1100; EVPPI_2L = 550
Article's EVPPI for the utility with exact inner means
With the utility B in the outer loop and the treatment effect A in the inner loop, the inner means are GBP 1,100 when B is high and GBP -700 when B is low, and the estimate is GBP 350, the article's EVPPI for B. The two values sum to GBP 900, more than the EVPI of GBP 600, because EVPPIs are not additive.
K = 2; J = 2; INB_11 = 2000; INB_12 = 200; INB_21 = 1000; INB_22 = -2400; ibar_1 = 1100; ibar_2 = -700; EVPPI_2L = 350
Nested estimator with every parameter of interest returns the EVPI
Limiting case. When the outer draws carry every uncertain parameter, nothing is left to vary in the inner loop, so the inner runs at each outer draw are identical and each inner mean is a single model run. With two equally likely runs of GBP 1,500 and GBP -1,100 the estimate is GBP 750 less GBP 200, which gives GBP 550, the per-person EVPI of the same two runs (HE-FM-VOI-001). Validation Test HE-VT-EVPP-003 checks the same identity on the article's four combinations, where it gives the EVPI of GBP 600.
K = 2; J = 2; INB_11 = 1500; INB_12 = 1500; INB_21 = -1100; INB_22 = -1100; ibar_1 = 1500; ibar_2 = -1100; EVPPI_2L = 550
Common errors
Inner loop too small for the nested EVPPI estimator
In the first case study of Strong, Oakley and Brennan, 1,000 outer by 10 inner runs gave GBP 351.92 for their parameter set 1 against a reference value of GBP 247.95, with an estimated upward bias of GBP 130.69. In the article's example a single inner run for each draw of A returns the EVPI of GBP 600 in place of the EVPPI for A of GBP 550. An estimate above the EVPI is a sign of Monte Carlo error, not a finding.
Inner draws from the marginal distribution when parameters are correlated
When theta_c is correlated with theta_i, drawing the inner values from the marginal distribution of theta_c ignores what each value of theta_i implies for them, so the inner means and the estimate are wrong. Strong, Oakley and Brennan describe this conditional sampling as a limitation of the two-level method, and the regression estimator HE-FM-EVPP-003 needs only the joint sample.
Sources
Nested Monte Carlo sampling algorithm for partial EVPI
Brennan A, Kharroubi S, O'Hagan A, Chilcott J. Calculating partial expected value of perfect information via Monte Carlo sampling algorithms. Medical Decision Making. 2007;27(4) 448-470. Abstract, which describes the generalised Monte Carlo algorithm with an outer loop sampling the parameters of interest and, conditional on these, an inner loop sampling the remaining uncertain parameters, states that maxima of Monte Carlo estimates of expectations are biased upwards so that small samples bias the estimate, and reports that fewer samples on the outer level and more on the inner level could be efficient.
ISPOR double-loop algorithm for EVPPI
Rothery C, Strong M, Koffijberg HE, Basu A, Ghabri S, Knies S, Murray JF, Sanders Schmidler GD, Steuten L, Fenwick E. Value of information analytical methods. Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(3) 277-286. Section on EVPPI computation, Equation 13 with K outer samples of theta_i and J inner samples of theta_c from the conditional distribution, Algorithm 2 steps 1 to 11, and good practice recommendation 5 on choosing inner and outer loop sizes for acceptable bias and precision.
Two-level partial EVPI estimator and its bias in a case study
Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample, a nonparametric regression approach. Medical Decision Making. 2014;34(3) 311-326. Methods section, the nested 2-level estimator with K outer and J inner simulations and the upward bias induced by the maximisation step, and Table 2 of case study 1, where 1,000 outer by 10 inner runs gave 351.92 (SE 25.20, upward bias 130.69) against the reference 247.95 for parameter set 1, in pounds.
Canonical Identity
Stable URI · Machine-readable · Resolvable · CC BY 4.0