Signature
L = w_1 * abs(g_1 - T_1) + w_2 * abs(g_2 - T_2)
| Inputs | Definition | Unit |
|---|---|---|
w_1 | Relative importance of target 1 in the loss | none |
g_1 | Output of the model run for target 1, such as ten-year cumulative incidence | per cent |
T_1 | Empirical value the model should reproduce for target 1 | per cent |
w_2 | Relative importance of target 2 in the loss; 0 for a single target | none |
g_2 | Output of the model run for target 2 | per cent |
T_2 | Empirical value the model should reproduce for target 2 | per cent |
L | Weighted sum of distances between model predictions and targets; lower is a better fit | units of the targets |
|---|
Function
Bayesian optimisation of a model calibration loss with a Gaussian process surrogate
Searches a bounded parameter region for the values that minimise a goodness-of-fit loss between model outputs and calibration targets when each model run is expensive. A Gaussian process fitted to the losses already observed gives a normal posterior for the loss at any untried value, and an acquisition function, here expected improvement, chooses the next run. The notation follows the Bayesian Optimisation article and its one-parameter incidence calibration.
Try this function
Implementations
Excel
Two-target calibration loss from named cells
With Weight1, Pred1, Target1, Weight2, Pred2 and Target2 named, the formula returns the loss, held in CalibLoss.
=Weight1*ABS(Pred1-Target1)+Weight2*ABS(Pred2-Target2)
Assumptions
Targets on comparable scales or weighted to be so
Absolute differences are added across targets, so targets in different units or of different precision need weights that make their distances comparable.
Each loss evaluation is a full model run
The loss is a black-box function of the parameters; the search sees only the values returned by the runs made.
Worked examples
Ten-year incidence of 9.56 per cent against a 30 per cent target
With annual probability 0.01, ten-year incidence is 1 minus 0.99 to the power 10, about 9.5618 per cent, a loss of about 20.4382 points against the 30 per cent target (20.44 in the article).
w_1 = 1; g_1 = 9.5618; T_1 = 30; w_2 = 0; g_2 = 0; T_2 = 0; L = 20.4382
Ten-year incidence at an annual probability of 0.04
At 0.04 the model predicts about 33.5167 per cent, a loss of about 3.5167 points (3.52 in the article).
w_1 = 1; g_1 = 33.5167; T_1 = 30; w_2 = 0; g_2 = 0; T_2 = 0; L = 3.5167
Incidence and prevalence targets with a weight of 0.5 on prevalence
Adding a prevalence target of 10 per cent, predicted at 12 per cent with weight 0.5, gives 3.52 plus 1, a loss of 4.52 (computed here for illustration).
w_1 = 1; g_1 = 33.52; T_1 = 30; w_2 = 0.5; g_2 = 12; T_2 = 10; L = 4.52
Common errors
Setting target weights without checking which target dominates
Kong and colleagues combined 11 targets into one goodness-of-fit score by a weighted sum reflecting user-defined importance; the weights decide which targets win when they conflict, so results should be checked under alternative weights.
Treating the best-fitting parameter set as the full calibration result
Menzies and colleagues note that a single best-fitting set can be an important first step, but estimates of uncertainty around modelled results need a distribution of parameter values, as a probabilistic analysis requires.
Assuming the best fit is unique
Alarid-Escudero and colleagues found two equally good-fitting parameter sets for a cancer model that gave treatment gains of 0.67 and 0.31 years; an optimiser that stops at one gives no warning of the other.
Sources
Weighted-sum goodness-of-fit score for multiple calibration targets
Kong CY, McMahon PM, Gazelle GS. Calibration of disease simulation model using an engineering approach. Value in Health. 2009;12(4):521-529. doi:10.1111/j.1524-4733.2008.00484.x (abstract read). Abstract: 11 targets derived from clinical and epidemiologic data were combined into a total goodness-of-fit score by a weighted-sum approach, accounting for the user-defined relative importance of the calibration targets, and 28 natural history parameters were searched to minimise it.
Single best-fitting parameter set as a first step before uncertainty analysis
Menzies NA, Soeteman DI, Pandya A, Kim JJ. Bayesian methods for calibrating health policy models: a tutorial. PharmacoEconomics. 2017;35(6):613-624. doi:10.1007/s40273-017-0494-4. Section 3.5.1: to produce estimates of uncertainty around modelled results the calibration will need to produce a distribution of values for model parameters; however, obtaining a single best fitting parameter set can be an important first step.
Equally good-fitting parameter sets with different treatment effects
Alarid-Escudero F, MacLehose RF, Peralta Y, Kuntz KM, Enns EA. Nonidentifiability in model calibration and implications for medical decision making. Medical Decision Making. 2018;38(7):810-821. doi:10.1177/0272989X18792283 (abstract read). Abstract: different, equally good-fitting parameter sets produce different estimates of the treatment effectiveness (0.67 against 0.31 years), which could influence the optimal decision.
Black-box derivative-free setting of Bayesian optimisation
Frazier PI. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811 [stat.ML], version 1, 8 July 2018. Section 1: the objective is expensive to evaluate, has no known structure and gives no derivatives; Bayesian optimisation is designed for black-box derivative-free global optimisation, typically with 20 or fewer inputs and a few hundred evaluations.
Canonical Identity
Stable URI · Machine-readable · Resolvable · CC BY 4.0