ENBS of one study design from its population EVSI and the study cost

Signature

ENBS = EVSI_pop - C_n

Function

Expected net benefit of sampling as population sample information less the cost of a study

Computational function

• Computational function for the efficient sample size over a grid of study designs by ENBS

  Takes the inputs for a grid of candidate designs, the sample size and per-person EVSI of each, the people facing the decision each year, the number of years, a discount rate, the study cost inputs and, for only in research, the number of people treated outside each design while it runs. For one design it returns the discount factor, the population EVSI, the expected study cost and the ENBS, and the code below repeats this over the grid to return the largest ENBS and the efficient sample size, or no design when none has a positive ENBS. It chains HE-FM-EVSI-004, HE-FM-ENBS-002 and HE-FM-ENBS-003, so its inputs differ from the formulas' variables. Per-person EVSI for each sample size comes from the methods on the EVSI page.

  Inputs and outputs: EVSI is the per-person EVSI of the design, in currency per person. N is the number of people a year whose treatment follows the result, Y the number of years counted from year 0 and r the annual discount rate as a decimal above zero, as in HE-FM-EVSI-004. C_F is the fixed cost and c_V the cost per participant, in currency. n is the sample size. s_inf is the share of participants on the arm expected to be inferior. INB is the expected incremental net benefit per person of the new treatment on current information. N_w is the number of people treated outside the design while it runs under only in research, zero under approval with research. D is the discount factor, EVSI_pop the population EVSI, C_n the expected study cost and ENBS the expected net benefit of sampling of the design. Over the grid the code returns ENBS_max, the largest ENBS, and n_star, the efficient sample size, only when ENBS_max is above zero.

  Assumption: One beneficiary population applies to every design, the article's simplification. Where larger designs report later, N and Y are entered separately for each design or N_w carries the difference. Per-person EVSI rises with sample size but stays below per-person EVPI while costs grow with each participant, so ENBS typically rises to a peak and then falls, but the code evaluates the whole grid and does not rely on that shape.

  Worked example (Article's 600-patient trial with approval with research): The article's discounted population of 8,000 is entered as N of 8,000 for one year, which gives a factor of 1, and the ENBS is GBP 940,000. The 100, 300 and 1,000-patient designs give GBP 390,000, GBP 770,000 and GBP 500,000, so every design is worthwhile and 600 is the efficient size, as in the article's table. EVSI = 300; N = 8000; Y = 1; r = 0.035; C_F = 200000; c_V = 2000; n = 600; s_inf = 0.5; INB = 200; N_w = 0; D = 1; EVSI_pop = 2400000; C_n = 1460000; ENBS = 940000

  Worked example (Article's 300-patient trial under only in research): With 1,000 people waiting, each forgoing GBP 200, the ENBS is GBP 570,000, against GBP 540,000 for the 600-patient trial with 2,000 waiting, so the smaller trial becomes efficient, as in the article. EVSI = 200; N = 8000; Y = 1; r = 0.035; C_F = 200000; c_V = 2000; n = 300; s_inf = 0.5; INB = 200; N_w = 1000; D = 1; EVSI_pop = 1600000; C_n = 830000; ENBS = 570000

  Worked example (600-patient trial with 2,000 people a year for five years at 3.5 per cent): The factor is about 4.6731 and the ENBS about GBP 1,343,848. The 100, 300 and 1,000-patient designs give about GBP 524,616, GBP 1,039,232 and GBP 971,155, so the 600-patient trial stays efficient (computed here for illustration). EVSI = 300; N = 2000; Y = 5; r = 0.035; C_F = 200000; c_V = 2000; n = 600; s_inf = 0.5; INB = 200; N_w = 0; D = 4.673079; EVSI_pop = 2803847.53; C_n = 1460000; ENBS = 1343847.53

  Worked example (No design worthwhile for a discounted population of 1,000): The 100-patient trial has an ENBS of minus GBP 310,000 and the 300, 600 and 1,000-patient designs minus GBP 630,000, minus GBP 1,160,000 and minus GBP 1,950,000, so no sample size is returned and, in the words of Fenwick and colleagues, the current evidence should be considered sufficient for decision making (illustrative population). EVSI = 100; N = 1000; Y = 1; r = 0.035; C_F = 200000; c_V = 2000; n = 100; s_inf = 0.5; INB = 200; N_w = 0; D = 1; EVSI_pop = 100000; C_n = 410000; ENBS = -310000

  Excel: =IF(MAX(ENBSRange)>0,INDEX(SampleSize,MATCH(MAX(ENBSRange),ENBSRange,0)),"none") Returns the efficient sample size, where SampleSize holds the candidate sizes in column A and ENBSRange the ENBS of each design, filled down from =B2*AnnualPopulation*(1+DiscountRate)*(1-(1+DiscountRate)^(-Years))/DiscountRate-(FixedCost+CostPerParticipant*A2+InferiorShare*A2*ABS(ExpectedINB))-C2*MAX(ExpectedINB,0) with the per-person EVSI of each design in column B and the number waiting in column C. Both formulas work in any current version of Excel.

  R: enbs_grid <- function(n, evsi, N, Y, r, cf, cv, s_inf, inb, wait = 0) { D <- (1+r)*(1-(1+r)^(-Y))/r; e <- evsi*N*D-(cf+cv*n+s_inf*n*abs(inb))-wait*max(inb, 0); k <- which.max(e); list(D = D, ENBS = e, n_star = if (e[k] > 0) n[k] else NA, ENBS_max = e[k]) } Returns a list, with wait a single value or one value for each design.

  Python: def enbs_grid(n, evsi, N, Y, r, cf, cv, s_inf, inb, wait=None): wait = wait or [0]*len(n); D = (1+r)*(1-(1+r)**(-Y))/r; e = [v*N*D-(cf+cv*m+s_inf*m*abs(inb))-w*max(inb, 0) for v, m, w in zip(evsi, n, wait)]; k = e.index(max(e)); return D, e, (n[k] if e[k] > 0 else None), e[k] Plain Python with no imports, returning the discount factor, the list of ENBS values, the efficient sample size or None, and the largest ENBS.

  Test (Efficient design has the largest ENBS on the grid): With the ENBS of each design in ENBSRange and the returned size in OptimalSize, the ENBS of the returned design equals the largest value. The expected result is TRUE. Excel check =MAX(ENBSRange)=INDEX(ENBSRange,MATCH(OptimalSize,SampleSize,0))

  Test (Largest ENBS within the population EVPI ceiling): No design can be worth more than population EVPI less the cost of the cheapest design, GBP 3,200,000 less GBP 410,000 in the article, with population EVPI from HE-FM-VOI-002 in PopulationEVPI and the cost of each design in StudyCostRange. The expected result is TRUE. Excel check =MAX(ENBSRange)<=PopulationEVPI-MIN(StudyCostRange)

  Common error (Choosing the design with the largest population EVSI): The article's 1,000-patient trial has the largest population EVSI, GBP 2,800,000, but an ENBS of GBP 500,000, which is GBP 440,000 below the 600-patient design. The efficient design is the one with the largest ENBS, which nets the cost against the value.

  Source: Rothery C, Strong M, Koffijberg HE, et al. Value of information analytical methods. Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(3) 277-286. Section on the expected net benefit of sampling, where the optimal design is the one that maximises the ENBS, for choices such as sample size and length of follow-up. Fenwick E, Steuten L, Knies S, et al. Value of information analysis for research decisions, an introduction. Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(2) 139-150. Good practice recommendation 9 and the text before it, where EVSI and ENBS are calculated for a range of designs and, when no design has a positive ENBS, the current evidence is considered sufficient for decision making. Willan AR, Pinto EM. The value of information and optimal clinical trial design. Statistics in Medicine. 2005;24(12) 1791-1806. Abstract, checked on PubMed (15806619), which shows how to find the sample size that maximises the difference between the cost of the trial and the value of the information. Conti S, Claxton K. Dimensions of design space, a decision-theoretic approach to optimal research design. Medical Decision Making. 2009;29(6) 643-660. Abstract, checked on PubMed (19605884), where sample sizes and allocation between treatments are chosen to maximise ENBS.

  D = (1 + r) * (1 - (1 + r)^(-Y)) / r; EVSI_pop = EVSI * N * D; C_n = C_F + c_V * n + s_inf * n * abs(INB); ENBS = EVSI_pop - C_n - N_w * max(INB, 0)

Try this function

Implementations

• Excel

  ENBS of one design from population EVSI in Excel

  With the design's population EVSI from HE-FM-EVSI-004 in PopulationEVSI and its expected total cost from HE-FM-ENBS-002 in StudyCost, the cell subtracts the cost from the population value.

  =PopulationEVSI-StudyCost

Assumptions

• Study result informs only the decision being modelled

  Rothery and colleagues state that ENBS of zero or more is a necessary condition for conducting a study only under the assumption that the study is relevant to this decision alone and has no wider value. A study that would also inform other decisions or jurisdictions can be worth more than its ENBS for this decision.

• Beneficiary population counted once the study reports

  The population behind EVSI_pop, N people a year for Y years in HE-FM-EVSI-004, counts only the people whose treatment follows the result, adjusted for the time the study takes to finish, as Fenwick and colleagues recommend. Eckermann and Willan show that time for accrual, follow-up and analysis lowers EVSI and the expected net gain at any trial size. The article holds the discounted population at 8,000 for every design, a simplification since larger trials take longer.

• EVSI and study cost on one net benefit scale at a stated threshold

  EVSI_pop and C_n are both in net monetary benefit or both in net health benefit, valued at the same cost-effectiveness threshold. Rothery and colleagues ask for value of information results at explicit thresholds or as a function of the threshold, so ENBS is reported in the same way.

Worked examples

• ENBS of the article's 600-patient trial with a discounted population of 8,000

  In the article's illustrative example the 600-patient trial has a per-person EVSI of GBP 300 and a discounted population of 8,000, so its population EVSI is GBP 2,400,000 (HE-FM-EVSI-004). The study costs GBP 1,460,000, so ENBS is GBP 940,000, the largest of the article's four designs.

  EVSI_pop = 2400000; C_n = 1460000; ENBS = 940000

• ENBS of a 600-patient trial with 2,000 people a year for five years at 3.5 per cent

  With an illustrative 2,000 people a year facing the decision for five years and later cohorts discounted at 3.5% a year, HE-FM-EVSI-004 gives a discounted population of about 9,346 and a population EVSI of about GBP 2,803,848 for the per-person EVSI of GBP 300. Against the same study cost of GBP 1,460,000, ENBS is about GBP 1,343,848 (computed here for illustration).

  EVSI_pop = 2803847.53; C_n = 1460000; ENBS = 1343847.53

• Negative ENBS of a 100-patient trial for a small population

  If only 1,000 discounted people would follow the result, the article's 100-patient trial, with a per-person EVSI of GBP 100, has a population EVSI of GBP 100,000 against a cost of GBP 410,000, so ENBS is minus GBP 310,000 and the design is not worthwhile (illustrative population). The negative value applies to this design only.

  EVSI_pop = 100000; C_n = 410000; ENBS = -310000

Common errors

• Study cost set against per-person EVSI instead of population EVSI

  Comparing the per-person EVSI of GBP 300 for the article's 600-patient trial with its cost of GBP 2,000 per participant suggests the study cannot pay its way, yet its ENBS is GBP 940,000. The cost belongs against the population value, after scaling and discounting.

• Future cohorts left undiscounted in ENBS

  Leaving the 2,000 people a year in years 1 to 4 undiscounted when forming EVSI_pop counts 10,000 people instead of about 9,346 and raises the illustrative ENBS of the 600-patient trial to GBP 1,540,000 instead of about GBP 1,343,848 (computed here for illustration). Rothery and colleagues discount the value of research over the time horizon, as HE-FM-EVSI-004 does.

• People treated before the study reports counted as beneficiaries

  If 2,000 of the article's 8,000 people would be treated before the 600-patient trial reports, only 6,000 can benefit and ENBS falls to GBP 340,000 instead of GBP 940,000 (computed here for illustration). Eckermann and Willan reduce the number who benefit by those treated inside and outside the trial until its evidence is available.

Sources

• ISPOR task force definition of ENBS and the population sum

  Rothery C, Strong M, Koffijberg HE, et al. Value of information analytical methods. Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(3) 277-286. Section on the expected net benefit of sampling, which defines ENBS as the difference between population EVSI and the cost of the data collection, a measure of the net value of any particular study. Section on reporting of results, equation 15, where the per-person value is multiplied by the sum over periods 0 to T of I_t divided by (1 + d)^t, with I_t the incidence in period t, T the time horizon and d the discount rate for a single period.

  View source →

• ISPOR task force rule that a positive ENBS shows a study is worthwhile

  Fenwick E, Steuten L, Knies S, et al. Value of information analysis for research decisions, an introduction. Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(2) 139-150. Sections on estimating the value of specific research and identifying the appropriate research design, with good practice recommendations 8 and 9. The difference between population EVSI for a specific study and its expected total cost is the ENBS, a positive ENBS is a sufficient condition for the worth of a specific study, and the most efficient design is the one with the greatest ENBS.

  View source →

• Decision-analytic trial design with the costs and benefits of sample information

  Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health Economics. 1996;5(6) 513-524. Abstract, checked on PubMed (9003938), which presents a decision-analytic approach to trial design that takes explicit account of the costs of sampling and the benefits of sample information, and a set of screens or hurdles for research proposals.

  View source →

Canonical Identity