Total Effect Index

Total Effect Index in Health Economic Sensitivity Analysis

The total effect index, also called the total-order or total Sobol index, measures the share of variance in a decision model result, such as incremental net monetary benefit, that would be left on average if every uncertain parameter except one were known exactly. A value near zero means the result barely depends on that parameter, even counting its combinations with others. Health economic modellers use the index mainly to decide which parameters can be held at a point value in a large probabilistic model without losing meaningful output uncertainty. It is read alongside the first-order index, and the gap between the two shows how much of a parameter's influence works through interactions. The wider family of variance-based measures is set out on the Sobol indices page.

Two equivalent forms of the total effect index

The index is defined for a scalar model output with finite, positive variance, and its reading as a sum of variance terms assumes inputs that are mutually independent under their assigned distributions. Saltelli and colleagues, in their 2010 paper on estimators for the total sensitivity index, write it in two forms that give the same number.

$$S_{T_i} = \frac{\mathrm{E}_{\mathbf{X}_{\sim i}}\left[\operatorname{Var}_{X_i}(Y \mid \mathbf{X}_{\sim i})\right]}{\operatorname{Var}(Y)} = 1 - \frac{\operatorname{Var}_{\mathbf{X}_{\sim i}}\left[\mathrm{E}_{X_i}(Y \mid \mathbf{X}_{\sim i})\right]}{\operatorname{Var}(Y)}$$

where $Y$ is the model output, $X_i$ is the input of interest, $\mathbf{X}_{\sim i}$ is the set of all other inputs, $\operatorname{Var}_{X_i}(Y \mid \mathbf{X}_{\sim i})$ is the output variance left when every other input is held at given values and only $X_i$ varies, and $\operatorname{Var}(Y)$ is the unconditional output variance across the full probabilistic sensitivity analysis distribution.

The first form is the expected residual variance once everything except $X_i$ has been learnt. The second subtracts from one the first-order effect of the whole group $\mathbf{X}_{\sim i}$, so what remains is the share of every variance term that contains input $i$, whether alone or in an interaction of any order. Saltelli and colleagues state that the total sensitivity indices were introduced by Homma and Saltelli, and Oakley and O'Hagan describe the measure as first proposed by Homma and Saltelli in 1996.

Several bounds follow under independence. Oakley and O'Hagan show that $S_{T_i} \ge S_i$ and that the first-order indices sum to at most one while the total indices sum to at least one, with equality only when all interactions are zero. The total indices overlap, because a pairwise interaction between two parameters is counted in the total for each of them. For a purely additive model the first-order and total indices coincide.

Why the total index decides which parameters can be fixed

Sobol's 2001 paper names three uses of global sensitivity indices, which are ranking variables, fixing non-influential variables and deleting high-order terms from the model. Ranking by first-order indices alone is, in Sobol's words, insufficient when the first-order indices sum to much less than one, because a parameter can matter mostly through combinations with others. For fixing, Sobol gives a theorem in which the total index of a set of inputs is a lower bound on the error of freezing them.

$$\delta_i(x_i^0) = \frac{\mathrm{E}\left\{\left[f(\mathbf{X}) - f(x_i^0, \mathbf{X}_{\sim i})\right]^2\right\}}{\operatorname{Var}(Y)} \ge S_{T_i}$$

where $f$ is the model, $x_i^0$ is any fixed value chosen for input $i$, $f(x_i^0, \mathbf{X}_{\sim i})$ is the simplified model with that input frozen, and $\delta_i(x_i^0)$ is the mean squared approximation error scaled by the output variance. Sobol states the result for any subset of inputs.

The inequality means no choice of fixed value can make the approximation error smaller than the total index, so a large total index rules out fixing. Sobol adds that if the fixed value is drawn at random from the input's distribution, the probability that the error is below three times the total index exceeds one half. He also notes that when both indices for an input are zero the model does not depend on that input at all. A first-order index of zero on its own does not give that assurance, because the input may still act through interactions.

Oakley and O'Hagan draw a contrast that matters for research planning. The first-order variance is the expected reduction in output variance from learning the true value of one input, so it points to where new research could narrow uncertainty most. They state that they do not believe there is any comparable interpretation of the total index in terms of guiding research effort. A report that ranks research priorities by total index alone is answering the fixing question instead.

Estimating total effects from model runs

Few health economic models allow the conditional variances to be worked out by hand, so the index is estimated from repeated model runs. The standard approach draws two independent sample matrices, $\mathbf{A}$ and $\mathbf{B}$, each with $N$ rows and one column per uncertain input, and builds for each input $i$ a matrix $\mathbf{A}_B^{(i)}$ that copies $\mathbf{A}$ except for column $i$, which is taken from $\mathbf{B}$. Each pair of matching rows then differs only in input $i$. Saltelli and colleagues compared the available estimators and recommend the one due to Jansen.

$$\widehat{S}_{T_i} = \frac{\dfrac{1}{2N}\sum_{j=1}^{N}\left[f(\mathbf{A})_j - f(\mathbf{A}_B^{(i)})_j\right]^2}{\widehat{\operatorname{Var}}(Y)}$$

where $N$ is the number of base sample rows, $f(\mathbf{A})_j$ is the model output for row $j$ of matrix $\mathbf{A}$, $f(\mathbf{A}_B^{(i)})_j$ is the output for the same row with input $i$ replaced, and $\widehat{\operatorname{Var}}(Y)$ is the estimated output variance.

The cost of this design for both first-order and total indices is $N(k+2)$ model evaluations, where $k$ is the number of inputs, and Saltelli and colleagues describe $N$ as a sufficiently large number of 500 or higher. Total indices alone need $N(k+1)$ runs using $\mathbf{A}$ and the $k$ matrices $\mathbf{A}_B^{(i)}$. A model with 30 uncertain parameters and $N$ of 1,000 would therefore need 31,000 runs for the totals, a real obstacle for a slow patient-level simulation. Their recommended set-up combines the Jansen estimator, quasi-random points from a Sobol sequence arranged as $\mathbf{A}$ and $\mathbf{A}_B^{(i)}$, and a radial design, and they suggest bootstrapping the blocks of runs to estimate the Monte Carlo error in each index.

Because the estimator needs paired runs that differ in one input at a time, an ordinary probabilistic sensitivity analysis sample supplies at most the base matrices, and the swapped-column runs have to be added. Oakley and O'Hagan describe a Bayesian Gaussian process approach that needs far fewer model runs than standard Monte Carlo methods, one route when each run is expensive. Saltelli and colleagues note that a winding stairs trajectory with one resample point is the same as the trajectory of the Morris method, a semi-quantitative screening method.

Worked example with a response-dependent QALY gain

All numbers in this example are illustrative. A new treatment changes incremental net monetary benefit through the probability of response $p$, the QALY gain per responder $q$, the incremental treatment cost $c$ and an adverse event cost $a$, with an illustrative willingness to pay of GBP 25,000 per QALY.

$$Y = \lambda p q - c - a$$

where $Y$ is incremental net monetary benefit per patient in GBP, $\lambda$ is the willingness to pay per QALY, and the four inputs are independent. Here $p$ has mean 0.30 and standard deviation 0.15, $q$ has mean 0.60 and standard deviation 0.18, $c$ has mean GBP 2,500 and standard deviation GBP 800, and $a$ has mean GBP 400 and standard deviation GBP 150. A beta distribution for $p$ and a gamma distribution for $q$ would fit these moments, and only the means and variances are needed below.

1. Expected value. The mean is 25,000 × 0.30 × 0.60 minus 2,500 minus 400, which is 4,500 minus 2,900, or GBP 1,600.

2. Variance components. With $\lambda^2$ equal to 625,000,000, the main effect of $p$ is $\lambda^2 \operatorname{Var}(p) \mathrm{E}(q)^2$, or 625,000,000 × 0.0225 × 0.36 = 5,062,500. The main effect of $q$ is $\lambda^2 \operatorname{Var}(q) \mathrm{E}(p)^2$, or 625,000,000 × 0.0324 × 0.09 = 1,822,500. The interaction of $p$ and $q$ is $\lambda^2 \operatorname{Var}(p) \operatorname{Var}(q)$, or 625,000,000 × 0.0225 × 0.0324 = 455,625. The additive costs contribute their own variances, 640,000 for $c$ and 22,500 for $a$.

3. Total variance. Adding the five terms gives 5,062,500 + 1,822,500 + 455,625 + 640,000 + 22,500 = 8,003,125, a standard deviation of about GBP 2,829.

4. Indices. Each index divides the relevant components by 8,003,125.

The first-order indices sum to 0.9431 and the total indices to 1.0569, because the interaction share of 0.0569 appears in the totals for both $p$ and $q$. For $q$, the gap of 0.0569 is about a fifth of its total of 0.2847, since the QALY gain only pays off in patients who respond.

5. Fixing a parameter. Freezing $a$ at its mean leaves a scaled error of 22,500 / 8,003,125 = 0.0028, equal to its total index, so the adverse event cost can be fixed with almost no loss. Freezing $a$ one standard deviation high at GBP 550 doubles the error to 45,000 / 8,003,125 = 0.0056, which is consistent with Sobol's bound. Freezing $p$ at its mean leaves $\lambda^2 \operatorname{Var}(p) \mathrm{E}(q^2)$, or 625,000,000 × 0.0225 × 0.3924 = 5,518,125, which is 0.6895 of the variance, so $p$ must stay probabilistic.

6. Dependence on the threshold. At an illustrative GBP 35,000 per QALY the output variance rises to 15,050,125 and the total index for $c$ falls to 640,000 / 15,050,125 = 0.0425. The indices describe this output at this willingness to pay, and a second threshold can change the ranking of cost and outcome parameters.

Reading the index in cost-effectiveness decisions

A total effect index describes variance, and variance in net benefit is not the same as decision uncertainty. A parameter can carry a large total index while every plausible value leaves the same option preferred, and a parameter with a modest index can sit close to the point where the decision switches. Oakley and O'Hagan note that where a model is used for decision-making, sensitivity can instead be measured by the expected value of information defined through the relevant loss function, and they cite a cost-effectiveness example. In health technology assessment that role is filled by expected value of partial perfect information, which values resolving a parameter by the change in the decision it would bring.

Total effect indices tell an analyst which parameters can be fixed to simplify a model or reduce run time, while value of information analysis tells a decision maker which uncertainty is worth paying to resolve. Both are global, which separates them from a one-way sensitivity analysis or a tornado diagram, where each parameter moves across a chosen range with the others held at base case values and interactions stay hidden.

Assumptions that change the result

The index is conditional on the distributions placed on the inputs. A parameter given a narrow distribution will show a small total index even if the model is very responsive to it, so the index ranks parameters under the stated parameter uncertainty and is not an intrinsic property of the parameter. The choice of output matters as well, since indices for total cost, total QALYs and incremental net monetary benefit at a stated threshold can rank the same parameters differently.

Independence is often hard to defend in health economic models, where baseline risks, treatment effects and utilities can be correlated. Saltelli and colleagues state that their decomposition relations hold for independent factors and that the dependent case is not treated in their paper. Oakley and O'Hagan explain that without independence the variance attributable to a set of inputs can still be defined, but the contributions no longer partition the total. Sobol defines first-order and total indices for any subset of inputs, so correlated parameters, such as the coefficients of a fitted survival curve, can be treated as one group with a single total index, provided the groups are independent of one another. Indices computed within one model structure also say nothing about structural uncertainty, which needs separate scenarios.

Misreadings of total effect results

Most errors with the index come from treating a variance share as a statement about importance in a wider sense. Four common ones follow.

1. Summing totals to 100%. Total indices overlap, so a column of totals adding to more than one is expected when interactions exist and does not indicate an error.
2. Fixing on the first-order index. A small first-order index can hide a large interaction, and only a small total index supports holding a parameter at a point value.
3. Reading a small total index as clinical irrelevance. The index reflects the assumed distributions and one output, and a parameter can matter for another output, threshold or subgroup.
4. Ignoring estimation noise. Sample-based estimates carry Monte Carlo error, and small totals near zero should be reported with an interval before a parameter is fixed.

Sources

• Oakley JE, O'Hagan A. Probabilistic sensitivity analysis of complex models: a Bayesian approach. Journal of the Royal Statistical Society Series B. 2004;66(3):751-769. https://doi.org/10.1111/j.1467-9868.2004.05304.x
• Saltelli A, Annoni P, Azzini I, Campolongo F, Ratto M, Tarantola S. Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Computer Physics Communications. 2010;181(2):259-270. https://doi.org/10.1016/j.cpc.2009.09.018
• Sobol IM. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and Computers in Simulation. 2001;55(1-3):271-280. https://doi.org/10.1016/S0378-4754(00)00270-6