BIC with the number of uncensored events as the penalty count

Signature

BIC_d = -2 * lnL + k * log(d)

Function

Schwarz criterion for ranking survival curves and approximating Bayes factors

Computational function

• Computational function: BIC, BIC differences and posterior model probabilities for candidate survival curves

  Takes the maximised log-likelihoods and parameter counts of a set of candidate survival models fitted to the same data, together with one penalty count, and returns three vectors: BIC, the difference from the smallest BIC and the approximate posterior probability of each model. It chains HE-FM-BIC-001 and HE-FM-BIC-003, and the second and third steps use the whole vector produced by the step before, so the inputs are lists of models rather than the single values the formulae take. The count may be the number of uncensored events, as in HE-FM-BIC-001, or the number of patients, which gives the patient-count form recorded on the Akaike information criterion page.

  Inputs and outputs: lnL: Vector of maximised log-likelihoods, one per candidate model, as reported by the fitting software; required, all from the same patients, endpoint and data cut. Unit: none.; k: Vector of the number of estimated parameters of each model, in the same order; required, positive integers. Unit: count.; c: Penalty count, either the number of uncensored events or the number of patients, the same for every model; required, greater than 1. Unit: count.; BIC: Vector of BIC values. Unit: none.; Delta: Vector of BIC differences from the smallest BIC; 0 for the best-scoring model. Unit: BIC units.; p: Vector of approximate posterior model probabilities, summing to 1. Unit: probability.

  Assumption: All candidates are fitted by maximum likelihood to identical data with one count, and every model has the same prior probability. Working from the differences rather than the raw BIC values keeps the exponentials within floating-point range when BIC values run into the thousands.

  Worked example (Four curves with the event count of 120 deaths): The article's illustrative arm of 400 patients, in the order exponential, Weibull, log-normal and generalised gamma. The generalised gamma ranks first with a probability of about 0.52 and the Weibull follows with about 0.38. lnL = (-530.0, -524.0, -525.5, -521.3); k = (1, 2, 2, 3); c = 120; BIC = (1064.79, 1057.57, 1060.57, 1056.96); Delta = (7.83, 0.61, 3.61, 0); p = (0.0104, 0.3833, 0.0855, 0.5207)

  Worked example (Same curves with the patient count of 400): The BIC values and differences are the article's; the probabilities are computed here as an illustration and are not taken from the article. With the larger penalty the Weibull ranks first with about 0.50 and the generalised gamma second with about 0.37, while the log-normal stays exactly 3.00 behind the Weibull because the two have the same number of parameters. lnL = (-530.0, -524.0, -525.5, -521.3); k = (1, 2, 2, 3); c = 400; BIC = (1065.99, 1059.98, 1062.98, 1060.57); Delta = (6.01, 0, 3.00, 0.59); p = (0.0246, 0.4959, 0.1106, 0.3689)

  Excel: =-2*B2+C2*LN($G$1) in D2, =D2-MIN($D$2:$D$5) in E2 and =EXP(-E2/2)/SUMPRODUCT(EXP(-$E$2:$E$5/2)) in F2, each filled down to row 5. With one model per row, log-likelihoods in B2:B5, parameter counts in C2:C5 and the penalty count in G1, columns D, E and F return BIC, the BIC difference and the posterior probability.

  R: bic_table <- function(loglik, k, count) { bic <- -2*loglik+k*log(count); delta <- bic-min(bic); p <- exp(-delta/2)/sum(exp(-delta/2)); data.frame(bic, delta, p) } Vectorised over models; bic_table(c(-530, -524, -525.5, -521.3), c(1, 2, 2, 3), 120) reproduces the event-count example.

  Python: def bic_table(loglik, k, count): bic = [-2*li+ki*math.log(count) for li, ki in zip(loglik, k)]; m = min(bic); delta = [b-m for b in bic]; e = [math.exp(-d/2) for d in delta]; s = sum(e); return bic, delta, [x/s for x in e] Uses the math module, where math.log is the natural logarithm, and returns the three lists in the order of the inputs.

  Test (Probabilities sum to one): The posterior probabilities of all candidates add to 1. Expected result: TRUE. Excel check: =ABS(SUM(F2:F5)-1)<1E-9

  Test (Ratio of two probabilities equals the approximate Bayes factor): The generalised gamma probability divided by the Weibull probability equals exp of half their BIC difference, about 1.36 under the event count, as HE-FM-BIC-002 gives. Expected result: TRUE. Excel check: =ABS(F5/F3-EXP((D3-D5)/2))<1E-9

  Common error (Comparing probabilities computed with different counts or candidate sets): The generalised gamma probability is about 0.52 with the event count and about 0.37 with the patient count, so probabilities from analyses that used different counts, or different candidate curves, cannot be compared. The count and the candidate set are reported with the probabilities.

  Source: Burnham KP, Anderson DR. Multimodel inference: understanding AIC and BIC in model selection. Sociological Methods & Research. 2004;33(2):261-304. Page 275, which gives BIC and the posterior model probabilities from BIC differences under equal prior probabilities. Volinsky CT, Raftery AE. Bayesian information criterion for censored survival models. Biometrics. 2000;56(1):256-262. Abstract, on the event count in the penalty.

  BIC = -2 * lnL + k * log(c); Delta = BIC - min(BIC); p = exp(-Delta / 2) / sum(exp(-Delta / 2))

Try this function

Implementations

• Excel

  Event-count BIC in one cell

  Excel uses LN, the natural logarithm, on named cells holding the log-likelihood, the parameter count and the number of events.

  =-2*LogLik+Params*LN(Events)

Assumptions

• One event count applied to every candidate survival curve

  Every candidate curve is fitted by maximum likelihood to the same patients, endpoint and data cut, and the same count d is used for all of them. When arms are fitted separately, each arm has its own count, and the analysis states which count it used.

• Event count reflects how fast the information grows

  Kass and Raftery explain that the count in the penalty should be the rate at which the Hessian of the log-likelihood grows, the number of data values contributing to the observed information. In censored survival data that is the number of uncensored observations rather than the number of patients.

Worked examples

• Generalised gamma curve in an arm with 120 deaths

  In the article's illustrative arm of 400 patients with 120 deaths, the three-parameter generalised gamma with a maximised log-likelihood of -521.3 has an event-count BIC of about 1056.96, the lowest of the four candidates. With the patient count it would score 1060.57 and rank second.

  lnL = -521.3; k = 3; d = 120; BIC_d = 1056.96

• Weibull curve in an arm with 120 deaths

  The two-parameter Weibull with a log-likelihood of -524.0 scores about 1057.57, 0.61 behind the generalised gamma. Under the patient count it led by 0.59, so the choice of count reverses the order of the two leading curves.

  lnL = -524.0; k = 2; d = 120; BIC_d = 1057.57

• Exponential curve in an arm with 120 deaths

  The one-parameter exponential with a log-likelihood of -530.0 scores about 1064.79, 7.83 behind the best-scoring model.

  lnL = -530.0; k = 1; d = 120; BIC_d = 1064.79

Common errors

• Mixing patient and event counts across candidate curves

  Computing some candidates with the patient count and others with the event count distorts the comparison. In the 400-patient example, a Weibull BIC of 1059.98 from the patient count set against a generalised gamma BIC of 1056.96 from the event count puts the generalised gamma 3.02 ahead, positive evidence on the Kass and Raftery scale, instead of 0.61 ahead under one consistent count.

• Using the events of both arms for a single-arm BIC

  When arms are fitted separately, the count for each fit is the number of events in that arm. Taking the events of both arms for a model fitted to one arm overstates the penalty for that fit and pushes BIC further towards the simpler curves.

Sources

• Volinsky and Raftery event-count revision of the BIC penalty

  Volinsky CT, Raftery AE. Bayesian information criterion for censored survival models. Biometrics. 2000;56(1):256-262. Abstract, which proposes defining the penalty by the number of uncensored events instead of the number of observations and reports a better approximation to the exact Bayes factor in a simple censored data model.

  View source →

• Kass and Raftery on the count in the Schwarz penalty

  Kass RE, Raftery AE. Bayes factors. Journal of the American Statistical Association. 1995;90(430):773-795. Section 4.1.3, which defines the count as the rate at which the Hessian of the log-likelihood grows and notes that in survival analysis it has been taken as the number of uncensored observations, that is, deaths.

  View source →

Canonical Identity