Bayesian Information Criterion

Bayesian Information Criterion and Survival Curve Choice

The Bayesian information criterion (BIC), also called the Schwarz criterion, is one of the statistics that NICE DSU Technical Support Document (TSD) 14 recommends presenting, beside the Akaike information criterion, when a health technology assessment compares parametric survival curves before extending trial data over a lifetime horizon. What sets it apart is its origin and its penalty. It approximates a Bayes factor, and it charges each extra parameter the natural logarithm of the sample size rather than a fixed amount. Using it well means knowing how that penalty compares with AIC's, which sample size to use when most patients are censored and how Kass and Raftery's scale grades BIC differences. In an illustrative four-curve example the choice of sample size reorders the candidates, and even then the ranking cannot choose the extrapolation.

From a Bayes factor to a penalised likelihood

Schwarz derived the criterion in 1978 by treating the choice among models of different dimension as a Bayesian decision and keeping the leading terms of its large-sample expansion. Because those terms do not depend on the prior distribution, he argued that they give a valid criterion beyond the Bayesian setting. His rule was to choose the model for which the maximised log-likelihood minus half the number of parameters times the log of the sample size is largest. Multiplying that quantity by minus 2 gives the usual form.

$$BIC = -2 \ln \hat{L} + k \ln n$$

where $\hat{L}$ is the maximised likelihood of the model, obtained by maximum likelihood estimation, $k$ is the number of estimated parameters and $n$ is the sample size. Lower values are preferred, and only differences between models fitted to the same data carry meaning. Burnham and Anderson point out that the name is a misnomer, since the criterion has no connection with information theory.

The Bayesian reading comes from the Bayes factor, the ratio of the probabilities of the observed data under two competing models, each averaged over its own prior distribution; by Bayes' theorem, it is the factor that converts prior odds on the two models into posterior odds. Kass and Raftery show that the Schwarz criterion is a rough approximation to the logarithm of this ratio, so that for models 1 and 2 the relationship is approximately as follows.

$$2 \ln B_{12} \approx BIC_2 - BIC_1$$

where $B_{12}$ is the Bayes factor in favour of model 1 over model 2, and $BIC_1$ and $BIC_2$ are the criterion values of the two models. The approximation works on the logarithmic scale. Kass and Raftery show that its relative error as an estimate of the log Bayes factor tends to zero as the sample grows, whereas the exponentiated criterion, taken as an estimate of the Bayes factor itself, keeps an error of constant order even in very large samples. BIC differences therefore indicate the strength of evidence without giving the Bayes factor exactly. It is closer when the prior on the extra parameters carries about as much information as a single observation, the so-called unit-information prior.

How the k ln n penalty compares with AIC's 2k

AIC charges 2 per parameter whatever the sample size, while BIC charges $\ln n$. Because $\ln n$ exceeds 2 once $n$ is greater than $e^2 \approx 7.39$, BIC penalises extra parameters more heavily in any realistic data set, and the gap widens as the sample grows, so that the BIC charge is 3.91 at 50 patients, 5.70 at 300 and 6.91 at 1,000. A model with one extra parameter is therefore preferred by AIC if its maximised log-likelihood rises by more than 1, but by BIC only if it rises by more than $\tfrac{1}{2} \ln n$, which is 2.85 at 300 patients.

TSD 14 makes the same point for survival models. Both criteria weigh improved fit against extra parameters, and BIC penalises the additional parameters more highly. Burnham and Anderson explain the difference in purpose. As data accumulate with the candidate set held fixed, BIC settles with increasing certainty on the simplest candidate that loses no more information than any other, which they term quasi-true, whereas the model AIC aims for depends on how much data there is. At the sample sizes found in practice, they caution, BIC's choice may be underfitted.

Which n to use for censored survival data

The penalty depends on $n$, and censoring makes its value a genuine choice. Kass and Raftery explain that $n$ should be the rate at which the Hessian of the log-likelihood grows, which makes it the number of data values contributing to the summation that forms the Hessian, and so the observed information matrix; in survival analysis they note that Raftery and colleagues took $n$ to be the number of uncensored observations, that is, deaths. Volinsky and Raftery developed the argument for censored survival analysis and proposed a version based on events.

$$BIC_d = -2 \ln \hat{L} + k \ln d$$

where $d$ is the number of uncensored events and the other terms are as defined above. In a simple censored data model this revision gave a better approximation to the exact Bayes factor, and in an analysis of stroke risk in the Cardiovascular Health Study it improved predictive performance.

The choice matters for parametric survival models fitted to immature trial data, where $d$ is much smaller than the number of patients, so the penalty is smaller and BIC moves closer to AIC. Because every parameter count is multiplied by the same logarithm, switching from patients to events leaves the differences between models with the same number of parameters unchanged and alters only comparisons between models of different size. When arms are fitted separately, each arm has its own count, and the analysis should state which count it used and apply it to every model in the comparison.

Reading BIC differences on the Kass and Raftery scale

Since a BIC difference approximates twice the natural log of a Bayes factor, it can be read on the scale Kass and Raftery proposed for that quantity. They adapted Jeffreys' categories; the table applies their version to the difference between a model and the best-scoring model. Burnham and Anderson's scale for AIC differences is a separate guide, described on the AIC page.

Kass and Raftery describe these categories as a rough descriptive statement about standards of evidence rather than a calibration, and accept that the reading may depend on context. The differences also convert into approximate posterior probabilities for each model, calculated as follows.

$$p_i = \frac{\exp(-\Delta BIC_i / 2)}{\sum_{r=1}^{R} \exp(-\Delta BIC_r / 2)}$$

where $p_i$ is the approximate posterior probability of model $i$, $\Delta BIC_i$ is its BIC minus the smallest BIC in the set, $\Delta BIC_r$ is the same difference for model $r$, and $R$ is the number of candidate models. Burnham and Anderson note that these probabilities assume every model has the same prior probability, and that each is best read as the probability that a model is the quasi-true model in the set, not that it is the true process. Because the probabilities sum to 1 across the candidates, they shift whenever a curve is added to or dropped from the set. Burnham and Anderson argue that model selection is best viewed as a way of computing such model weights, often as a step towards model averaging.

Worked example with patients or deaths as the count

The figures below are illustrative. One trial arm of 400 patients has been followed for three years and 120 patients have died, so 280 observations (70%) are censored. Four distributions, from the one-parameter exponential to the three-parameter generalised gamma, are fitted to the overall survival data, and BIC is calculated twice, once with $n$ as the 400 patients and with $d$ as the 120 deaths.

1. Calculate BIC with the patient count. The penalty is $\ln 400 = 5.9915$ per parameter. For the Weibull model the calculation is as follows.

$$BIC_{W} = 1048.0 + 2 \times 5.9915 = 1048.0 + 11.983 = 1059.98$$

where $BIC_{W}$ is the Weibull BIC, 1048.0 is minus twice its maximised log-likelihood of -524.0, and 2 is its number of parameters. The generalised gamma fits better by 2.7 log-likelihood units, but its third parameter costs 5.99, so it ranks second, 0.59 behind.

2. Repeat with the event count. The penalty falls to $\ln 120 = 4.7875$ per parameter. For the generalised gamma the calculation is as follows.

$$BIC_{GG} = 1042.6 + 3 \times 4.7875 = 1042.6 + 14.3625 = 1056.96$$

where $BIC_{GG}$ is the generalised gamma BIC and 1042.6 is minus twice its maximised log-likelihood of -521.3. The better fit now outweighs the smaller penalty, and the generalised gamma moves 0.61 ahead of the Weibull. The log-normal stays exactly 3.00 behind the Weibull under both counts (1060.57 minus 1057.57), because the two models have the same number of parameters.

3. Read the differences. Whichever count is used, the Weibull and the generalised gamma are less than 2 apart, which on the Kass and Raftery scale is not worth more than a bare mention. The log-normal is 3.00 behind the best model under the patient count and 3.61 under the event count, positive evidence against it either way, and the exponential, 7.83 behind under the event count, faces strong evidence, an approximate Bayes factor of $\exp(7.83/2) \approx 50$.

4. Convert to posterior probabilities. Under the event count the numerators $\exp(-\Delta BIC_i/2)$, computed from the unrounded BIC values, sum as follows.

$$1 + 0.7362 + 0.1643 + 0.0200 = 1.9205$$

where the terms are, in order, the generalised gamma, Weibull, log-normal and exponential models. Dividing each by 1.9205 gives 0.52 for the generalised gamma and 0.38 for the Weibull, so the data give similar support to the two leading curves.

5. Compare with other tools. The generalised gamma includes the Weibull as a special case, so the likelihood ratio test can compare them. The statistic is twice the log-likelihood gain, 5.4 on one degree of freedom, a p-value of about 0.02. AIC also prefers the generalised gamma, by 3.4 units (1048.6 against 1052.0). A conventional test calls the extra parameter significant while BIC, whichever count is used, finds the evidence too weak to mention, which shows how conservative the $\ln n$ penalty is.

6. Interpret. The data within follow-up cannot separate the Weibull from the generalised gamma, and the choice of count decides which ranks first. With 70% of observations censored, TSD 14 expects external data, clinical plausibility and expert judgement to decide. The Weibull hazard can only rise or fall monotonically, while the generalised gamma is more flexible, so their extrapolated tails can differ widely; the more plausible becomes the base case and the other belongs in scenario analysis.

Why a low BIC cannot choose the extrapolation

BIC describes fit within follow-up only. TSD 14 states that AIC and BIC address internal validity, the fit to the trial data, but not external validity, the plausibility of the extrapolated portion that drives mean survival and so the life years and QALYs in a partitioned survival model. The Bayesian reading does not reach beyond follow-up either, since Kass and Raftery show that a Bayes factor measures how well each model predicted the data actually observed. The NICE manual (PMG36, section 4.6.20) asks that the limitations of such measures of relative fit be documented when they are used to compare extrapolation models. The AIC page summarises the wider DSU guidance and a TSD 21 simulation in which the two lowest-AIC curves extrapolated worst.

Misreadings specific to BIC

Like any likelihood-based comparison, BIC requires the models to be fitted to the same patients, endpoint and data cut. Its Bayesian reading adds conditions of its own, and most misreadings come from ignoring them.

A BIC difference only approximates a Bayes factor, so treating 2.1 and 1.9 as different verdicts reads more precision into the scale than it holds. Posterior probabilities depend on the candidate set and on equal priors, so they cannot be compared across analyses with different candidates. A probability near 1 means that one model stands out on the observed data, not that it will extrapolate well, and a near-tie between curves with different tails signals structural uncertainty.

BIC beside significance tests and full Bayesian comparison

Each tool for comparing survival models answers a different question. The likelihood ratio test applies only to nested models and treats the extra parameter as a hypothesis to be tested at a chosen significance level; TSD 14 notes that this comparison had been used erroneously in past NICE appraisals for distributions that are not nested.

A full Bayesian analysis can compute Bayes factors directly with explicit priors rather than through the large-sample shortcut that BIC provides. In each case goodness of fit within the trial is one input to model selection, and the extrapolation still has to be justified on other grounds.

Sources

• Burnham KP, Anderson DR. Multimodel inference: understanding AIC and BIC in model selection. Sociological Methods & Research. 2004;33(2):261-304. Sections 1 and 3. https://doi.org/10.1177/0049124104268644
• Kass RE, Raftery AE. Bayes factors. Journal of the American Statistical Association. 1995;90(430):773-795. Sections 3.1, 3.2 and 4.1.3. https://doi.org/10.1080/01621459.1995.10476572
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2011, last updated March 2013. Sections 2.2, 2.6, 3.3, 3.5, 5 and 6.1. https://doi.org/10.15131/shef.data.33745843
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Section 4.6.20.
• Rutherford MJ, Lambert PC, Sweeting MJ, Pennington B, Crowther MJ, Abrams KR, Latimer NR. NICE DSU Technical Support Document 21: Flexible methods for survival analysis. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2020 (updated March 2022). Section 6.3. https://doi.org/10.15131/shef.data.33849118
• Schwarz G. Estimating the dimension of a model. Annals of Statistics. 1978;6(2):461-464. https://doi.org/10.1214/aos/1176344136
• Volinsky CT, Raftery AE. Bayesian information criterion for censored survival models. Biometrics. 2000;56(1):256-262. https://doi.org/10.1111/j.0006-341X.2000.00256.x