Akaike Information Criterion

Akaike Information Criterion for Choosing Survival Curves in HTA

The Akaike information criterion (AIC) is one of the fit statistics that NICE Decision Support Unit guidance expects when a submission to a NICE health technology assessment justifies the parametric curve used to extend trial survival data over a lifetime horizon. Because it compares maximised likelihoods rather than testing one model against another, it can be used for model selection among survival distributions that are not nested within one another. Its formula is simple and NICE guidance expects it to be reported alongside the Bayesian Information Criterion (BIC), and an illustrative six-model example shows how the two are used. The curve with the lowest AIC can still give an implausible estimate of mean survival, the figure that drives life years, QALYs and the cost-effectiveness result.

Information loss as the basis for ranking models

Akaike set out the criterion in 1973 and 1974, presenting it in the 1974 paper as a way to identify a statistical model without the ambiguities of conventional hypothesis testing. Burnham and Anderson describe it as an estimate of relative expected Kullback-Leibler information, the information lost when a fitted model is used to approximate the process that generated the data, and it is defined as follows.

$$AIC = -2 \ln \hat{L} + 2k$$

where $\hat{L}$ is the maximised likelihood of the model, obtained by maximum likelihood estimation, and $k$ is the number of estimated parameters. A more complex model is preferred only if each extra parameter raises the maximised log-likelihood by more than one unit; lower values are better, and the penalty guards against overfitting.

AIC values contain an arbitrary constant, so a single value means nothing on its own; only differences between models fitted to the same observations can be read. The candidate set also need not contain a true model, which suits survival modelling, where no standard distribution describes a disease process exactly.

The survival likelihood behind each value

In an economic evaluation the likelihood is usually that of trial time-to-event data, in which many patients are still event-free when follow-up ends. A censored observation contributes only the information that the event had not happened by the censoring time, as the log-likelihood of a parametric survival model shows.

$$\ln L(\theta) = \sum_{i=1}^{n} \left[ \delta_i \ln h(t_i \mid \theta) + \ln S(t_i \mid \theta) \right]$$

where $\theta$ is the vector of model parameters, $n$ is the number of patients, $t_i$ is the event or censoring time for patient $i$, $\delta_i$ equals 1 for an observed event and 0 for a censored observation, $h(t \mid \theta)$ is the hazard function and $S(t \mid \theta)$ is the survival function. Every term refers to a time within follow-up, so the likelihood carries no direct information about the period that extrapolation has to fill.

The parameter count differs across distributions. NICE DSU TSD 14 gives the exponential model one parameter, the Weibull, Gompertz, log-logistic and log-normal models two, and the generalised gamma three. In models built from a restricted cubic spline, the count rises with the number of knots.

Reading AIC differences and Akaike weights

Because only differences carry meaning, results are reported relative to the best-scoring model. Burnham and Anderson rescale each value accordingly. The rescaled value is defined as follows.

$$\Delta_i = AIC_i - AIC_{min}$$

where $AIC_i$ is the value for model $i$ and $AIC_{min}$ is the smallest value in the set. On their guide, models with a difference of 2 or less have substantial support, those between 4 and 7 considerably less, and those above 10 almost none. The differences convert into Akaike weights as follows.

$$w_i = \frac{\exp(-\Delta_i / 2)}{\sum_{r=1}^{R} \exp(-\Delta_r / 2)}$$

where $w_i$ is the weight for model $i$, $R$ is the number of candidate models and $\Delta_r$ is the difference for model $r$. The weights sum to 1 and are read as the weight of evidence that a model is the best approximating model in the set, given the data. Adding or removing a candidate changes every weight.

AIC and BIC set different penalties

TSD 14 recommends presenting AIC and BIC tests, or other suitable tests of internal validity, when survival models are compared for a NICE technology appraisal. BIC, derived by Schwarz, has a penalty that grows with the sample size. It is defined as follows.

$$BIC = -2 \ln \hat{L} + k \ln n$$

where $n$ is the sample size and $\hat{L}$ and $k$ are as defined for AIC. AIC charges 2 per parameter whatever the sample size, while BIC charges $\ln n$. With 300 patients the BIC penalty is about 5.70 per parameter, so BIC leans towards simpler distributions; TSD 14 notes that additional parameters are penalised more highly by BIC than by AIC.

The two criteria also aim at different targets. AIC seeks the model with the smallest expected Kullback-Leibler information loss for the sample size at hand, whereas, as the sample grows, BIC converges on the most parsimonious model closest to the truth within the set, which Burnham and Anderson call a quasi-true model; they also note that BIC can be derived without assuming that the true model is in the set. BIC differences convert into approximate posterior model probabilities when every model has an equal prior probability. For censored data the $n$ in the penalty is itself a choice, and Volinsky and Raftery proposed the number of uncensored events rather than the number of patients, which lowers the penalty when censoring is heavy.

What NICE DSU guidance says about AIC

The NICE Decision Support Unit's technical support documents shape how survival models are justified in NICE appraisals, although each states that NICE is not bound by it. Two deal with AIC directly.

TSD 14, written by Latimer in 2011 and last updated in March 2013, covers extrapolation with patient-level trial data. It explains that the negative 2 log likelihood suits only nested models, that many candidate distributions are not nested, and that the measure had been used erroneously in past NICE appraisals to compare them, whereas AIC and BIC can compare models that are not nested. It recommends that the six standard distributions above all be considered, with fit assessed through log-cumulative hazard plots, AIC and BIC tests (or other suitable tests of internal validity), and clinical plausibility based on expert judgement, external data or biological reasoning. In its selection algorithm, AIC and BIC can settle the choice with the hazard plots only when the data are very close to complete; with substantial censoring, external data, clinical plausibility and expert judgement are needed.

TSD 21, by Rutherford and colleagues, published in 2020 and updated in March 2022, covers flexible methods such as spline-based, mixture and cure models. It treats AIC and BIC as an informal guide to the number of spline knots and states that they give little information about how well a model extrapolates, so they should not be the sole basis for model selection when the aim is to extrapolate, for any parametric model. For a cure model, it warns that the criteria weight fit early in follow-up, where events are dense, and give very little weight to the tail where the cure fraction is estimated.

The NICE manual (PMG36) carries the same message. Section 4.6.20 asks that the limitations of statistical measures of fit to the trial data be documented when they are used to assess alternative extrapolation models, and section 4.6.25 asks that the clinical plausibility of the underlying hazard functions be assessed routinely.

Why the best fit within the trial can extrapolate badly

The gap between fit and plausibility comes from what the likelihood measures. Most events in a typical trial occur early, so the comparison is dominated by the first part of the curve, while mean survival, and with it the life years and QALYs in a partitioned survival model, depends heavily on the tail beyond the trial. TSD 14 frames this as internal validity, fit to the observed data, against external validity, the plausibility of the extrapolated portion, and notes that AIC and BIC address only the first.

An example in TSD 21 shows how far apart the two can be. The authors fitted survival models to one simulated dataset with three years of follow-up, in which the true disease-specific hazard was a decreasing Weibull hazard with unobserved frailty and other-cause mortality rose with age. Within follow-up most models agreed with the Kaplan-Meier curve; the log-normal had the lowest AIC and the Weibull the highest. Extrapolated to 40 years, the two models with the lowest AIC gave the poorest extrapolations and the Weibull did best, although still a long way from the truth; because the true disease-specific hazard was itself Weibull, its relative success says little about other datasets. Every fitted hazard was falling at the end of follow-up, whereas the true all-cause hazard later turned upwards. Plotted against expected mortality based on population rates, none of the fitted models was suitable for estimating overall mean survival, and the authors concluded that all should be rejected for that purpose, then showed that building background mortality into the model may help.

Worked example with six candidate curves for one trial arm

The figures below are illustrative. One trial arm of 300 patients has been followed for three years. Six distributions are fitted to its overall survival data, and AIC and BIC use the 300 patients as the sample size.

1. Calculate AIC. For the log-normal model the calculation is as follows.

$$AIC_{LN} = 2 \times 2 - 2 \times (-603.2) = 4 + 1206.4 = 1210.4$$

where $AIC_{LN}$ is the AIC of the log-normal model, with two parameters and a maximised log-likelihood of -603.2. The three-parameter generalised gamma fits better by 0.4 log-likelihood units but pays 2 more in penalty, so it ranks second at 1211.6.

2. Rank and weight. The generalised gamma and log-logistic are within 2 units of the log-normal and the exponential has almost no support. The weight numerators sum to the following.

$$1 + 0.5488 + 0.4966 + 0.1496 + 0.0608 + 0.0003 = 2.2561$$

where each term is $\exp(-\Delta_i/2)$ for, in order, the log-normal, generalised gamma, log-logistic, Weibull, Gompertz and exponential models. The log-normal weight is $w_{LN} = 1 / 2.2561 = 0.44$.

3. Repeat with BIC. The penalty becomes $\ln 300 = 5.7038$ per parameter, so the two-parameter models keep their AIC gaps, while the generalised gamma moves from 1.2 to 4.90 units behind the log-normal.

4. Check plausibility. The log-normal hazard rises to a peak and then falls, as does the log-logistic hazard when its shape parameter exceeds 1, and TSD 14 notes that both can produce long tails. Suppose clinical experts expect the hazard to rise as the cohort ages, and extrapolated hazard plots show both curves falling below general population mortality from national life tables. The Weibull (3.8 AIC units behind) and the Gompertz (5.6) then become the candidates, provided their fitted hazards rise and stay above general population mortality. Their weaker fit within the trial is itself evidence to weigh against the clinical view, so the other plausible curves belong in scenario analysis; the exponential is set aside.

5. Interpret. AIC narrowed the field but did not choose the base case. Had the log-normal been chosen on AIC alone, a falling hazard that the external evidence contradicts would have been carried into the extrapolated period, and any mean survival estimate built on it would inherit that error.

Conditions for a fair AIC comparison

AIC comparisons hold only under conditions that analyses do not always state. The models must be fitted to the same patients and endpoint at the same data cut. Burnham and Anderson treat a well-chosen set of models specified before fitting as a precondition of the method. When arms are fitted separately, the rankings may disagree; TSD 14 advises the same type of distribution in both arms unless there is strong evidence that an alternative is more plausible. For small arms, Burnham and Anderson call the use of AIC where its small-sample version, AICc, is needed a pervasive mistake, and recommend AICc unless the ratio of sample size to parameters exceeds about 40. A near-tie signals structural uncertainty, which belongs in scenario analysis rather than being settled by the smallest number. The most consequential misreading is to treat the lowest AIC, or the largest Akaike weight, as evidence that a curve extrapolates plausibly, when both describe fit to the observed data only.

How AIC relates to neighbouring tools

Each method for comparing survival models answers a different question. For nested models, such as a Weibull against the exponential, which TSD 14 describes as a special case of the Weibull, the likelihood ratio test is the significance-testing alternative to AIC, since it asks whether the extra parameter improves fit by more than chance would, rather than ranking models by expected information loss. Visual comparison with the Kaplan-Meier curve checks goodness of fit, but TSD 14 advises against relying on visual inspection alone.

Sources

• Akaike H. Information theory and an extension of the maximum likelihood principle. In: Petrov BN, Csaki F, eds. Second International Symposium on Information Theory. Budapest: Akademiai Kiado; 1973:267-281.
• Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control. 1974;19(6):716-723. https://doi.org/10.1109/TAC.1974.1100705
• Burnham KP, Anderson DR. Multimodel inference: understanding AIC and BIC in model selection. Sociological Methods & Research. 2004;33(2):261-304. https://doi.org/10.1177/0049124104268644
• Collett D. Modelling Survival Data in Medical Research. 3rd ed. Boca Raton: CRC Press; 2015.
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2011, last updated March 2013. Sections 2, 3.3, 3.5, 5 and 6. https://doi.org/10.15131/shef.data.33745843
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 4.6.20 and 4.6.25.
• Rutherford MJ, Lambert PC, Sweeting MJ, Pennington B, Crowther MJ, Abrams KR, Latimer NR. NICE DSU Technical Support Document 21: Flexible methods for survival analysis. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2020 (updated March 2022). Sections 3.2.2, 3.2.3, 3.6.3, 6.3 and 6.4. https://doi.org/10.15131/shef.data.33849118
• Schwarz G. Estimating the dimension of a model. Annals of Statistics. 1978;6(2):461-464. https://doi.org/10.1214/aos/1176344136
• Volinsky CT, Raftery AE. Bayesian information criterion for censored survival models. Biometrics. 2000;56(1):256-262. https://doi.org/10.1111/j.0006-341X.2000.00256.x