Calibration

Calibration of Health Economic Model Parameters to Observed Data

In health economic modelling, calibration is how analysts set parameters that no study measures directly, such as the rate at which preclinical cancer progresses, by running the model at candidate values and keeping those whose outputs reproduce observed data such as registry incidence or screening prevalence. Screening and disease progression models often rest on such parameters, and the values chosen can move an incremental cost-effectiveness ratio. Calibration of prediction models and calibration of experts are different senses of the word.

Why decision models need calibrated parameters

Some inputs come straight from evidence, such as a test's sensitivity from a meta-analysis. Menzies and colleagues contrast these with evidence on modelled outcomes, such as the number of people starting treatment, which depends on many parameters at once; calibration is how that outcome evidence enters the model. Stout and colleagues describe the classic case in cancer screening, where predicted screening effectiveness depends on unobservable natural history parameters. Of 154 cancer simulation articles in their review, 131 may have used calibration.

Stout and colleagues distinguish calibration, often called model fitting, from direct estimation, in which parameters are estimated outside the model and the fit of model output plays no part. Some models use a hybrid of the two.

Seven decisions in every calibration

Vanni and colleagues divide the calibration of disease models in economic evaluation into seven steps. Each is a choice that can change the result.

1. Which parameters to vary.
2. Which calibration targets to use.
3. Which goodness-of-fit measure to use.
4. Which parameter search strategy to use.
5. What makes a parameter set's fit acceptable (convergence criteria).
6. When to stop searching (the stopping rule).
7. How to integrate calibration results with the economic parameters.

Menzies and colleagues note that unit costs and utility weights, which typically have no mathematical relationship with calibrated outcomes, can stay outside the calibration.

Calibration targets and goodness-of-fit measures

A calibration target is observed data the model tries to reproduce. Stout and colleagues stress that target data are, by definition, not used as direct model inputs, and that a target may be a single statistic, such as mean age at diagnosis, or a series, such as an age-specific incidence curve. A goodness of fit measure turns the distance between outputs and targets into one score, often a weighted sum of squares of the form below.

$$G(\theta) = \sum_{k=1}^{K} \left( \frac{m_k(\theta) - y_k}{s_k} \right)^2$$

where $G(\theta)$ is the score for parameter set $\theta$, $K$ is the number of targets, $m_k(\theta)$ is the model output matched to target $k$, $y_k$ is the observed value and $s_k$ is its standard error, so precise targets carry more weight. A likelihood measure instead gives the probability of the observed data under the model. For prevalence surveys with $x_k$ positives among $n_k$ people, the binomial log-likelihood is as follows.

$$\ell(\theta) = \sum_{k=1}^{K} \left[ x_k \ln p_k(\theta) + (n_k - x_k) \ln\{1 - p_k(\theta)\} \right]$$

where $\ell(\theta)$ is the binomial log-likelihood up to a constant and $p_k(\theta)$ is the modelled prevalence for survey $k$.

Scoring choices matter. Karnon and Vanni found that a chi-squared measure separated the accuracy of parameter sets far more than a likelihood measure. Enns and colleagues showed that two weighted-sum schemes on a simple Markov model picked best-fitting sets whose results barely overlapped on the cost-effectiveness plane, with median ICERs of USD 79,300 and USD 139,700 per QALY gained; they proposed keeping the Pareto frontier, the sets that no other set matches or beats on every target at once.

Searching the parameter space

A search algorithm decides which parameter sets to evaluate. Stout and colleagues found informal trial and error, grid search, random sampling and directed searches such as the Nelder-Mead method, which are forms of numerical optimisation that move iteratively toward better fit.

In a Markov model of human papillomavirus and cervical disease, Taylor and colleagues measured fit as mean percentage deviation from targets. Nelder-Mead reached 7% with less analyst time than manual calibration (10%), while random search reached 39%, judged unacceptable. Kong and colleagues calibrated 28 natural history parameters of the Lung Cancer Policy Model to 11 targets, and simulated annealing found a lower score than a genetic algorithm. For stochastic microsimulation, where Monte Carlo error makes each score noisy, Menzies and colleagues suggest simulated annealing or calibrating a fast emulator fitted to model runs.

Two rules close the search. Acceptance criteria set the fit a parameter set must reach to be kept, for example a likelihood ratio below a set threshold, and the stopping rule says when searching ends, such as convergence of a directed search.

Bayesian calibration, a distribution instead of a point

Bayesian calibration estimates the joint distribution of the calibrated parameters rather than one best value. Prior evidence and calibration data are combined through Bayes' theorem. Menzies and colleagues write the posterior as follows.

$$p(\theta \mid Y) \propto p(\theta)\, p(Y \mid M(\theta))$$

where $\theta$ is the parameter set, $Y$ is the calibration data, $p(\theta)$ is the prior distribution, $M(\theta)$ is the model output for $\theta$ and $p(Y \mid M(\theta))$ is the likelihood. The output is a sample from the posterior distribution, then run through the model.

Sampling importance resampling draws sets from the prior and resamples them with likelihoods as weights. In the Menzies tutorial, 100,000 draws yielded only 797 unique sets and an effective sample size of 88, while incremental-mixture importance sampling was 150 to 200 times more efficient. Rutter and colleagues developed a Markov chain Monte Carlo method for a colorectal cancer microsimulation, and found some parameters were identified mainly through their priors. Whyte, Walsh and Chilcott used the Metropolis-Hastings algorithm with English bowel screening data and ran the posterior sets in probabilistic sensitivity analysis. Where the likelihood is intractable, approximate Bayesian computation uses summary statistics from simulated data instead. In the Menzies example, calibration moved the ICER from USD 1,300 to USD 947 per life-year saved and substantially reduced uncertainty.

Identifiability when several answers fit equally well

A calibration is identifiable when the model and targets single out one best-fitting parameter set. Alarid-Escudero and colleagues calibrated a three-state Markov model of cancer relative survival. With relative survival as the only target, two different sets gave similar maximum likelihood values but predicted life expectancy gains from treatment of 0.67 and 0.31 years. Adding the ratio between the two non-death states over time as a second target made the problem identifiable, with a collinearity index of 3.5. Their remedies are more targets or constraints and priors that restrict the parameter space, and, where nonidentifiability remains, reporting the range of equally good-fitting sets.

From calibrated parameters to the cost-effectiveness result

Stout and colleagues found that, with few exceptions, cancer modellers used one best-fitting set rather than several sets capturing parameter uncertainty. Karnon and Vanni describe probabilistic calibration, which integrates easily with probabilistic sensitivity analysis, and Vanni and colleagues note that calibration can represent correlations between parameters more accurately. Accepted sets are therefore sampled whole, keeping the correlations the targets induce.

Worked example with one target, then two

The numbers are illustrative. A cancer natural history model has two unknown parameters, the onset rate of preclinical cancer and its mean sojourn time. Assuming a steady state, no other-cause death and a perfectly sensitive screen, the relations are as follows.

$$I = \lambda, \qquad P = \lambda D$$

where $I$ is clinical incidence and $P$ is preclinical prevalence at a first screen, both per 100,000, $\lambda$ is the onset rate per 100,000 person-years and $D$ is the mean sojourn time in years. Walter and Day showed that screen prevalence and incidence together inform sojourn time, which the example mirrors. Targets are registry incidence of 100 (standard error 5) and first-screen prevalence of 300 (standard error 30), scored with $G(\theta)$.

1. Incidence target only. Sets 1, 2 and 3 all score 0, because incidence depends on $\lambda$ alone, so $D$ is not identified.

2. Why it matters. Screening 100,000 people once at GBP 50 each costs GBP 5,000,000. Set 1 predicts 200 cancers found, GBP 25,000 per cancer detected; set 3 predicts 500, GBP 10,000. Equally good fits differ 2.5-fold.

3. Adding the prevalence target. Set 1 scores $((200 - 300)/30)^2 = 11.11$, set 3 scores $((500 - 300)/30)^2 = 44.44$ and set 4 scores $((110 - 100)/5)^2 + ((297 - 300)/30)^2 = 4.01$. Set 2 fits best.

4. A Bayesian version. With $\lambda = 100$, equal prior weight on $D$ of 2.5, 3.0 and 3.5, and a normal likelihood $\exp(-G/2)$, prevalences of 250, 300 and 350 give $G$ of 2.78, 0 and 2.78, likelihoods of 0.2494, 1 and 0.2494, and posterior weights of 0.166, 0.667 and 0.166 after dividing by 1.4988. Costs per cancer detected are GBP 20,000, 16,667 and 14,286, so the posterior mean is GBP 16,825, against GBP 16,667 at the best fit.

5. Interpretation. The second target, not a better search, resolved the sojourn time. Because cost per cancer detected is non-linear in $D$, carrying the posterior forward shifts the expected result as well as showing its uncertainty.

Reporting a calibration

Published reporting has lagged practice. Of the 131 articles in Stout's review, 87 (66%) documented or referenced their calibration methods and 47 of those 87 described a goodness-of-fit metric, while search algorithms, acceptance criteria and stopping rules were seldom well described. Their Calibration Reporting Checklist covers targets, goodness-of-fit metrics, search algorithms, acceptance criteria and stopping rules, and the ISPOR-SMDM report on model transparency and validation by Eddy and colleagues names the same five items.

Calibration, validation and other meanings of the word

Calibration and model validation answer different questions. Eddy and colleagues class a validation as independent only if no information from the data source was used to build the model, and describe dependent validations as closely related to calibration. Fit to calibration targets is therefore not independent validation. They add that a model ready for external validation should not be refitted to each data source.

Two other senses are common. In prediction modelling, calibration means agreement between predicted probabilities and observed event frequencies, often shown in a calibration plot. In structured expert elicitation, calibration of experts scores experts on questions with known answers and weights their judgements accordingly.

Limitations and common misreadings

Calibration improves consistency with data but cannot certify a model, and fitted values hold only for the population and period of the targets. Four pitfalls recur.

• A close fit does not show the structure is right; parameters can absorb structural error, a matter for structural uncertainty analysis.
• Menzies and colleagues warn against tweaking priors, likelihood or model only to improve fit; perfect fit to many outputs is unrealistic, and chasing it causes overfitting and overconfident predictions.
• A single best-fitting set treats calibrated values as known and can hide nonidentifiability.
• Target weights are analyst choices that can change the ICER.

Sources

• Alarid-Escudero F, MacLehose RF, Peralta Y, et al. Nonidentifiability in model calibration and implications for medical decision making. Medical Decision Making. 2018;38(7):810-821.
• Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health. 2012;15(6):843-850.
• Enns EA, Cipriano LE, Simons CT, et al. Identifying best-fitting inputs in health-economic model calibration: a Pareto frontier approach. Medical Decision Making. 2015;35(2):170-182.
• Karnon J, Vanni T. Calibrating models in economic evaluation: a comparison of alternative measures of goodness of fit, parameter search strategies and convergence criteria. PharmacoEconomics. 2011;29(1):51-62.
• Kong CY, McMahon PM, Gazelle GS. Calibration of disease simulation model using an engineering approach. Value in Health. 2009;12(4):521-529.
• Menzies NA, Soeteman DI, Pandya A, et al. Bayesian methods for calibrating health policy models: a tutorial. PharmacoEconomics. 2017;35(6):613-624.
• Rutter CM, Miglioretti DL, Savarino JE. Bayesian calibration of microsimulation models. Journal of the American Statistical Association. 2009;104(488):1338-1350.
• Stout NK, Knudsen AB, Kong CY, et al. Calibration methods used in cancer simulation models and suggested reporting guidelines. PharmacoEconomics. 2009;27(7):533-545.
• Taylor DC, Pawar V, Kruzikas D, et al. Methods of model calibration: observations from a mathematical model of cervical cancer. PharmacoEconomics. 2010;28(11):995-1000.
• Vanni T, Karnon J, Madan J, et al. Calibrating models in economic evaluation: a seven-step approach. PharmacoEconomics. 2011;29(1):35-49.
• Walter SD, Day NE. Estimation of the duration of a pre-clinical disease state using screening data. American Journal of Epidemiology. 1983;118(6):865-886.
• Whyte S, Walsh C, Chilcott J. Bayesian calibration of a natural history model with application to a population model for colorectal cancer. Medical Decision Making. 2011;31(4):625-641.