ENBS Calculation

ENBS Calculation for a Proposed Study

Calculating the expected net benefit of sampling (ENBS) turns a value of information analysis into a figure a research funder can act on, by valuing one candidate study design in money or health and subtracting everything the study would cost. The calculation runs in four linked steps, namely per-person EVSI for each design, scaling to the people who could benefit once the study reports, costing the study, and searching across designs for the largest result. A slip in any step, such as forgetting the delay before results arrive, can change which sample size looks best. The concept and its decision rule sit on the expected net benefit of sampling page, while most practical errors arise in the calculation itself.

The full expression behind an ENBS figure

The ISPOR value of information task force describes ENBS as the difference between population EVSI for a specific study design and the expected total cost of that study. For a design indexed by $n$ it can be written as follows.

$$\text{ENBS}(n) = \text{EVSI}(n) \sum_{t=\tau(n)}^{T} \frac{I_t}{(1+d)^t}-\left[ C_F + c_V N(n) + L(n) \right]$$

where $\text{EVSI}(n)$ is the per-person expected value of sample information of design $n$, $I_t$ is the number of people facing the decision in period $t$, $d$ is the discount rate per period, $T$ is the last period in which the evidence is still useful, $\tau(n)$ is the period in which the study reports, $C_F$ is the fixed cost of the study, $c_V$ is the variable cost per participant, $N(n)$ is the number of participants and $L(n)$ is the expected net benefit forgone by participants allocated to the option expected to be inferior.

Report 2 of the task force (Rothery and colleagues, 2020) sums discounted incidence from period 0 to $T$. Starting the sum at $\tau(n)$ applies the adjustment for the time a study takes to finish, which Report 1 (Fenwick and colleagues, 2020) describes as usual. Every term is in net monetary benefit or net health benefit at a stated cost-effectiveness threshold, so the calculation is repeated for each threshold of interest.

Step 1 per-person EVSI for each candidate design

Rothery and colleagues note that EVSI has an analytic solution when incremental net benefit between two options is normally distributed and the study gives a known reduction in its variance, the case used in the worked example below. For a sample size search Kunst and colleagues (2020) recommend Gaussian approximation or moment matching, which compute EVSI across sample sizes at about the cost of one estimate.

Step 2 the beneficiary population and its timing

Fenwick and colleagues describe the beneficiary population as the prevalent and incident cohorts over the period in which the decision stays relevant, shaped by influences such as patent expiry, the launch of an in-class substitute and expected price changes. They report that published studies tend to use horizons of 1 year and of 10 to 20 years without clear justification, and recommend testing alternative horizons in scenario analyses because results can be highly sensitive to the choice.

Eckermann and Willan (2008) show that the beneficiary population should be reduced by those treated outside as well as within a trial until its evidence is available, including time for accrual, follow-up and analysis, which lowers EVSI and favours smaller trials where trials are worthwhile. The same discounted sum, started at period 0, turns per-person EVPI into population EVPI, a ceiling for every design.

Step 3 the cost of the study

Fenwick and colleagues list the expected cost of a design as fixed costs such as set-up and salaries, a variable cost per participant, and an opportunity cost for participants who receive the technology expected to be inferior while the study runs. The total is commonly taken from a societal perspective, though the sponsor's perspective may also be used.

The opportunity cost term depends on access while the study runs. If the technology is approved alongside the research, only participants allocated to the comparator forgo its expected incremental net benefit. If access is restricted to people in the study, everyone treated outside it meanwhile also forgoes that benefit when current evidence favours the technology, which can change the efficient design.

Step 4 searching across designs

ENBS is calculated for every candidate design, and Fenwick and colleagues identify the most efficient as the one with the greatest ENBS. Willan and Pinto (2005) note that conventional sample size calculation uses type I and type II error rates that ignore the cost of either error, and show how to find the sample size that maximises the difference between the value of the information and the cost of the trial.

$$n^* = \arg\max_{n \in \mathcal{N}} \text{ENBS}(n)$$

where $n^*$ is the optimal sample size or efficient design and $\mathcal{N}$ is the set of feasible designs considered. Fenwick and colleagues recommend presenting EVSI and ENBS for each design over a range of thresholds.

Worked example with a normal prior and four trial sizes

All numbers in this example are illustrative. A new treatment is compared with standard care at a threshold of GBP 25,000 per QALY. Treating per-patient incremental net benefit as the quantity the trial measures, current evidence puts it on a normal distribution with mean $m$ of GBP 400 and standard deviation $s_0$ of GBP 1,600, judged to hold as much information as a trial of 300 patients per arm, so the prior effective sample size $n_0$ is 300. The mean is positive, so the treatment is adopted on current evidence.

1. Per-person EVPI. For two options with normally distributed incremental net benefit, the expected opportunity loss has a closed form based on the unit normal loss integral.

$$\text{EVPI} = s_0 \left[ \varphi(z_0)-z_0 \{ 1-\Phi(z_0) \} \right], \quad z_0 = \frac{|m|}{s_0}$$

where $\varphi$ and $\Phi$ are the standard normal density and distribution functions and $z_0$ is the standardised distance of the prior mean from zero. Here $z_0$ is 400 / 1,600 = 0.25, $\varphi(0.25)$ is 0.386668 and one minus $\Phi(0.25)$ is 0.401294. Since 0.25 × 0.401294 = 0.100324, the bracket is 0.386668 minus 0.100324, or 0.286344, and per-person EVPI is 1,600 × 0.286344 = GBP 458.15.

2. Per-person EVSI. Jalal and Alarid-Escudero (2018) show that, for a normal prior and normal data, the variance of the preposterior distribution of the posterior mean is the fraction $n/(n_0+n)$ of the prior variance. EVSI then takes the form of EVPI with a smaller standard deviation.

$$s_n = s_0 \sqrt{\frac{n}{n_0+n}}, \qquad \text{EVSI}(n) = s_n \left[ \varphi(z_n)-z_n \{ 1-\Phi(z_n) \} \right], \quad z_n = \frac{|m|}{s_n}$$

where $n$ is the number of patients per arm, $s_n$ is the standard deviation of the expected posterior mean of incremental net benefit and $z_n$ is its standardised distance from zero. For 100 per arm the variance fraction is 100 / 400 = 0.25, so $s_n$ is 1,600 × 0.5 = 800 and $z_n$ is 0.5. With $\varphi(0.5)$ of 0.352065 and one minus $\Phi(0.5)$ of 0.308538, the product 0.5 × 0.308538 = 0.154269 leaves a bracket of 0.352065 minus 0.154269, or 0.197796, and EVSI is 800 × 0.197796 = GBP 158.24 per patient.

EVSI rises with sample size and stays below the EVPI of GBP 458.15, as it must.

3. Beneficiary population. 3,000 new patients a year face the decision, the evidence is useful for ten years (periods 0 to 9) and the discount rate is 3.5%. The ten discount factors sum to 8.6077, giving 25,823.1 discounted patients and a population EVPI of 458.15 × 25,823.1, about GBP 11.83 million. The trials with 100 and 250 per arm report in period 2, the 500 trial in period 3 and the 1,000 trial in period 4, so their discount sums are 6.6415, 5.7080 and 4.8060 and their beneficiary populations are 19,924.5, 17,124.0 and 14,418.1 discounted patients.

4. Study costs. Each trial costs GBP 400,000 fixed plus GBP 1,200 per participant, and a trial with $n$ per arm enrols twice $n$ participants. The treatment is approved alongside the trial, so only the $n$ patients randomised to standard care forgo the expected incremental net benefit of GBP 400 each. For 250 per arm the cost is 400,000 + 1,200 × 500 + 250 × 400 = GBP 1,100,000.

5. ENBS for each design. Population EVSI is per-person EVSI times the delayed beneficiary population, for example 259.60 × 19,924.5, about GBP 5,172,000 for 250 per arm. Figures are rounded to the nearest GBP 1,000.

All four designs are worthwhile and 250 per arm is the efficient size, with ENBS of 5,172,000 minus 1,100,000, or GBP 4,072,000. Every population EVSI lies below the population EVPI of GBP 11.83 million.

6. What the delay adjustment changed. Applying the full 25,823.1 discounted patients to every design gives ENBS of about GBP 3,406,000, 5,604,000, 6,713,000 and 6,675,000 for the four sizes, so the 500 trial would wrongly appear best. The larger trial gains information but reports a year later, and patients treated meanwhile cannot benefit, the effect Eckermann and Willan describe.

7. Interpretation. On these assumptions a trial of 250 per arm is expected to repay its cost by about GBP 4.1 million. The ranking is conditional on the threshold, the prior, the horizon and the access arrangement, which scenario analyses would vary before a funding decision.

Checks before an ENBS result is reported

Errors in ENBS work often arise outside the EVSI algorithm, in the steps that scale and cost it. Each check can be run on the spreadsheet or script that holds the calculation.

• Using the undelayed population. Step 6 shows that ignoring the delay can favour a larger design than is efficient.
• Accepting an EVSI above EVPI. Per-person EVSI cannot exceed per-person EVPI, so a larger value points to simulation error or a coding mistake.

Sources

• Eckermann S, Willan AR. Time and expected value of sample information wait for no patient. Value in Health. 2008;11(3):522-526. https://doi.org/10.1111/j.1524-4733.2007.00296.x
• Fenwick E, Steuten L, Knies S, et al. Value of information analysis for research decisions: an introduction. Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(2):139-150. https://doi.org/10.1016/j.jval.2020.01.001
• Jalal H, Alarid-Escudero F. A Gaussian approximation approach for value of information analysis. Medical Decision Making. 2018;38(2):174-188. https://doi.org/10.1177/0272989X17715627
• Kunst N, Wilson ECF, Glynn D, et al. Computing the expected value of sample information efficiently: practical guidance and recommendations for four model-based methods. Value in Health. 2020;23(6):734-742. https://doi.org/10.1016/j.jval.2020.02.010
• Rothery C, Strong M, Koffijberg HE, et al. Value of information analytical methods. Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(3):277-286. https://doi.org/10.1016/j.jval.2020.01.004
• Willan AR, Pinto EM. The value of information and optimal clinical trial design. Statistics in Medicine. 2005;24(12):1791-1806. https://doi.org/10.1002/sim.2069