SUCRA

SUCRA for Ranking Treatments in Network Meta-Analysis

SUCRA (surface under the cumulative ranking curve) condenses the rank probabilities that a network meta-analysis produces for one treatment into a single value, 1 if the treatment would always come first and 0 if it would always come last. It is a rescaled mean rank, so it shows where a treatment tends to sit among the competing options for one outcome. It does not show how large the differences between treatments are, how certain the evidence is, or whether the top-ranked treatment is good value. In health technology assessment the relative effects and their joint uncertainty carry the decision, and SUCRA is a reading aid for the clinical evidence that does not enter a cost-effectiveness model.

Rank probabilities from the posterior distribution

In a Bayesian network meta-analysis every relative effect has a posterior distribution, usually sampled by Markov chain Monte Carlo. At each sampled iteration the treatments can be put in order of their sampled effects, and the share of iterations in which a treatment holds rank $k$ estimates the probability that it has that rank. The NICE Decision Support Unit's Technical Support Document 2 (TSD 2) gives WinBUGS code that records, at every iteration, whether each treatment is best and which rank it occupies, with a switch for whether the outcome events are good or bad.

Salanti, Ades and Ioannidis extended the probability of being best to the probability of every rank, introduced graphs of these probabilities called rankograms, and proposed SUCRA as a numerical summary of the rank distribution. The Cochrane Handbook (chapter 11) warns that inference based only on the probability of being ranked best, without accounting for its variability, is misleading. It names mean ranks, median ranks and cumulative ranking probabilities as measures that use all possible ranks and account for uncertainty in the ranking.

Calculating SUCRA from the cumulative ranking curve

For each treatment the rank probabilities form a discrete distribution that sums to one. Accumulating them rank by rank gives the cumulative ranking curve, and SUCRA is the area under that curve scaled to its maximum, in the notation of Rücker and Schwarzer.

$$F_i(r) = \sum_{k=1}^{r} P_i(k) \qquad \mathrm{SUCRA}_i = \frac{1}{n-1} \sum_{r=1}^{n-1} F_i(r)$$

where $n$ is the number of treatments in the network, $P_i(k)$ is the probability that treatment $i$ has rank $k$ with rank 1 the best, $F_i(r)$ is the probability that treatment $i$ has rank $r$ or better, and $\mathrm{SUCRA}_i$ is the surface under the cumulative ranking curve for treatment $i$. The sum stops at $n-1$ because $F_i(n)$ equals one for every treatment and carries no information.

The expected rank of a discrete variable on the values 1 to $n$ equals $n$ minus the area under its distribution function up to $n-1$, which gives a direct link to the mean rank.

$$\mathrm{SUCRA}_i = \frac{n - E(R_i)}{n - 1}$$

where $E(R_i)$ is the expected (mean) rank of treatment $i$ across the posterior draws and $n$ is again the number of treatments. Rücker and Schwarzer describe SUCRA as the inversely scaled average rank, equal to one if the treatment always ranks first and zero if it always ranks last, and also as the average proportion of treatments worse than the treatment of interest. The SUCRA values in a network average 0.5. Salanti, Nikolakopoulou and colleagues add that mean ranks always give the same hierarchy as SUCRA, and that SUCRA measures the extent of certainty that a treatment beats all of its competitors.

Worked example with four treatments and one outcome

All numbers in this example are illustrative. A Bayesian network meta-analysis compares four treatments, A to D, on remission at 12 months, where a higher remission rate is better. The rank probabilities estimated from the posterior draws are shown below, and each row and each column sums to one.

1. Build the cumulative ranking curve. For A the cumulative probabilities are 0.55, then 0.55 + 0.25 = 0.80, then 0.80 + 0.15 = 0.95, and finally 1. For B they are 0.30, 0.70 and 0.90, for C 0.10, 0.35 and 0.80, and for D 0.05, 0.15 and 0.35.

2. Take the scaled area. With $n = 4$ the first three cumulative values are summed and divided by 3.

$$\mathrm{SUCRA}_A = \frac{0.55 + 0.80 + 0.95}{3} = \frac{2.30}{3} = 0.767$$

where 0.55, 0.80 and 0.95 are treatment A's cumulative probabilities of being at rank 1, rank 2 or better and rank 3 or better, and 3 is $n - 1$. In the same way B has 1.90 / 3 = 0.633, C has 1.25 / 3 = 0.417 and D has 0.55 / 3 = 0.183.

3. Check against the mean rank. A's mean rank is 1 × 0.55 + 2 × 0.25 + 3 × 0.15 + 4 × 0.05 = 1.70, and (4 minus 1.70) / 3 = 0.767, matching step 2. The mean ranks of B, C and D are 2.10, 2.75 and 3.45, which return 0.633, 0.417 and 0.183. The four SUCRA values sum to 2.000, an average of 0.5 as expected.

4. Read the result. A SUCRA of 0.767 says that A is expected to outrank 0.767 × 3 = 2.3 of its three competitors. Nothing in the table shows whether the A versus B difference in remission is large, trivial or uncertain.

5. Equal scores from different profiles. In a separate four-treatment network, treatment X has a probability of 0.5 of ranking first and 0.5 of ranking last, while treatment Y has 0.5 at rank 2 and 0.5 at rank 3. X has cumulative values 0.5, 0.5 and 0.5, and Y has 0, 0.5 and 1. Both sum to 1.5, so both have a SUCRA of 1.5 / 3 = 0.5 and a mean rank of 2.5. These are very different clinical prospects, yet SUCRA cannot tell them apart.

Analysis choices that move SUCRA values

SUCRA inherits every modelling choice behind the posterior, and some choices act on it directly. Reversing the outcome direction, the switch in the TSD 2 code, turns a SUCRA of $s$ into one minus $s$. Because the score is defined relative to the treatments in the network, adding or removing one changes $n$ and the competitors, so values from networks with different treatment sets are not comparable.

Salanti and colleagues note that SUCRA reflects the overlap between the treatment effect distributions. The larger the overlap, the more similar the values, and if all treatments had the same mean effect, whatever their uncertainty, every SUCRA would be 50%. A random-effects model, a vague prior on the between-study heterogeneity, or sparse data therefore tends to pull values towards the middle. Rücker and Schwarzer show the reverse risk. When two treatments have independent normal posteriors, the one with the better point estimate has a probability above 0.5 of being better whatever the variances, so a very imprecise treatment can still be ranked above a precisely estimated one.

The Cochrane Handbook notes that ranking probabilities may vary for different outcomes, so each outcome has its own SUCRA and a treatment can rank high on benefit and low on harm.

SUCRA, P-scores and other ranking metrics

A frequentist meta-analysis has no posterior distribution to rank, so Rücker and Schwarzer proposed the P-score, which needs only the point estimates and standard errors. It replaces the rank probabilities with pairwise measures of certainty that one treatment is better than another, averaged over all competitors.

$$\bar{P}_i = \frac{1}{n-1} \sum_{j \ne i} \Phi\!\left( \frac{\hat{\mu}_i - \hat{\mu}_j}{s_{ij}} \right)$$

where $\bar{P}_i$ is the P-score of treatment $i$, $\hat{\mu}_i$ and $\hat{\mu}_j$ are the estimated effects of treatments $i$ and $j$ on a scale where higher is better, $s_{ij}$ is the standard error of their difference, $\Phi$ is the standard normal distribution function and $n$ is the number of treatments. Each term is one minus a one-sided p-value, so the P-score is the mean certainty that treatment $i$ beats another treatment. In their diabetes and depression examples SUCRA values and P-scores were nearly identical, and they prove the two are identical if the true probabilities are known. They concluded that neither offers a major advantage over looking at the credible interval or confidence interval for each comparison.

The probability of being best looks only at rank 1, while SUCRA also counts the chance that a treatment produces unfavourable outcomes. Chiocchia and colleagues re-analysed 232 networks of four or more interventions published between 1999 and 2015 and measured agreement between hierarchies with Kendall's tau and related statistics. Agreement was high overall and highest between SUCRA and the relative treatment effect. In the original publications, 57.3% of the networks reported no ranking metric, 35.8% reported the probability of being best and 6.9% reported SUCRA. Salanti and colleagues argue that a review should first state its treatment hierarchy question, the criterion for preferring one treatment over others, and then choose the metric that answers it.

Reading SUCRA in health technology assessment

The Cochrane Handbook treats ranking as secondary to relative effects. Reviews that aim to rank interventions should say so in their objectives, and ranking is best treated as a secondary objective that supplements the relative effects, with confidence in the ranking also considered when it is reported. Imprecision in a ranking shows as overlap between the distributions of rank probabilities.

In economic evaluation the link from evidence synthesis to the decision runs through the relative effects, not the ranks. TSD 2 notes that significance tests and interval estimates tend to be set aside when a decision maker follows the rules of cost-effectiveness analysis. When the network contains closed loops, TSD 2 asks for the posterior samples themselves, or another method that carries their correlations, to be used in the decision model, because distributions built from marginal summaries are not adequate. The joint draws then pass through the model in probabilistic sensitivity analysis and give expected costs, QALYs and net monetary benefit for each option. SUCRA plays no part in that calculation.

A treatment can top the SUCRA order on efficacy and still not be cost effective once its price, adverse events and downstream costs are counted. Mbuagbaw and colleagues give reasons why a clinician should not simply choose the treatment with the highest SUCRA, among them that a treatment best for a benefit outcome may be worst for a harm outcome and that cost may also bear consideration. Where several outcomes must be weighed together, the trade-off belongs to explicit methods such as multi-criteria decision analysis or the economic model.

Uncertainty in rankings and common misreadings

SUCRA is itself uncertain. Trinquart and colleagues re-analysed 58 network meta-analyses covering 1,308 randomised trials and 404 interventions. On a 0 to 100% scale, the median width of the 95% credible interval of SUCRA was 65%, and in 28% of networks there was a probability of 50% or more that the best-ranked treatment was not the best. They found no evidence of a difference between the best-ranked intervention and the second and third in 90% and 71% of comparisons respectively.

1. Reading SUCRA as the probability of being best. In the example, A has a SUCRA of 0.767 but a 0.55 probability of being best, and X has a 0.5 probability of being best with the same SUCRA as Y, which is never best.
2. Reversing the scale. Writing the score as the mean rank minus one, divided by $n - 1$, gives (1.70 minus 1) / 3 = 0.233 for A, the expected share of competitors that outrank A, not the share A outranks.
3. Treating a gap in SUCRA as a gap in effect. Mbuagbaw and colleagues point out that SUCRA does not consider the magnitude of differences between treatments and does not capture the possibility that chance explains them.
4. Ignoring certainty of evidence. Mbuagbaw and colleagues note that the same SUCRA values can arise from a large body of trials with few limitations or from a small one with serious risk of bias, imprecision, inconsistency or indirectness.
5. Using the order as a reimbursement rule. HTA decisions rest on relative effects, costs and their uncertainty, so SUCRA belongs beside a forest plot or league table of effects with their intervals.

Sources

• Chaimani A, Caldwell DM, Li T, Higgins JPT, Salanti G. Chapter 11: Undertaking network meta-analyses [last updated October 2019]. In: Higgins JPT, Thomas J, Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-11
• Chiocchia V, Nikolakopoulou A, Papakonstantinou T, Egger M, Salanti G. Agreement between ranking metrics in network meta-analysis: an empirical study. BMJ Open. 2020;10(8):e037744. https://doi.org/10.1136/bmjopen-2020-037744
• Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU Technical Support Document 2: A generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. Decision Support Unit; August 2011, last updated September 2016. https://www.sheffield.ac.uk/sites/default/files/2022-02/TSD2-General-meta-analysis-corrected-2Sep2016v2.pdf
• Mbuagbaw L, Rochwerg B, Jaeschke R, Heels-Andsell D, Alhazzani W, Thabane L, Guyatt GH. Approaches to interpreting and choosing the best treatments in network meta-analyses. Systematic Reviews. 2017;6(1):79. https://doi.org/10.1186/s13643-017-0473-z
• Rücker G, Schwarzer G. Ranking treatments in frequentist network meta-analysis works without resampling methods. BMC Medical Research Methodology. 2015;15:58. https://doi.org/10.1186/s12874-015-0060-8
• Salanti G, Ades AE, Ioannidis JPA. Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial. Journal of Clinical Epidemiology. 2011;64(2):163-171. https://doi.org/10.1016/j.jclinepi.2010.03.016
• Salanti G, Nikolakopoulou A, Efthimiou O, Mavridis D, Egger M, White IR. Introducing the treatment hierarchy question in network meta-analysis. American Journal of Epidemiology. 2022;191(5):930-938. https://doi.org/10.1093/aje/kwab278
• Trinquart L, Attiche N, Bafeta A, Porcher R, Ravaud P. Uncertainty in treatment rankings: reanalysis of network meta-analyses of randomized trials. Annals of Internal Medicine. 2016;164(10):666-673. https://doi.org/10.7326/M15-2521