Area Under Curve

Area Under Curve as a Measure of Discrimination in Economic Evaluation

In health economics the phrase "area under the curve" has several meanings. The meaning used here is the area under a receiver operating characteristic (ROC) curve, also reported as the C-statistic, which summarises how well a diagnostic test or risk prediction model separates people who have, or will have, an outcome from people who do not. It matters to economists because tests and risk scores decide who is treated, and the costs and health effects in a model follow from those decisions. The statistic rests on a simple probability, can be calculated from patient pairs or from the ROC curve, has a survival version, and a high value does not on its own show that a test or model is worth using, as an illustrative ten-patient example shows. The survival and pharmacokinetic meanings of the phrase are linked in the first section.

Which area under which curve

Several quantities in health economics are areas under curves, and they share only the mathematics of integration. In survival modelling, the mean survival of a population is the area under its whole survival function; when some patients are still alive at the end of follow-up, only a mean restricted to that time can be estimated directly, which is the restricted mean survival time, and fitted curves are extrapolated to estimate the total area, as described under the area under curve approach. Partitioned survival models build health-state occupancy from such curves, and quality-adjusted life years weight time by health-state utility, as set out under the QALY calculation method. In clinical pharmacology, the area under a plasma concentration and time curve measures the extent of drug exposure, covered under pharmacokinetics.

These areas carry units, such as years or concentration multiplied by time. The discrimination AUC is a probability between 0 and 1 with no units, and it is the quantity discussed in the remaining sections.

The area as a probability of correct ranking

The discrimination AUC has a direct reading that does not need the curve at all. Hanley and McNeil showed that the area under an ROC curve equals the probability that a randomly chosen person with the condition is rated or ranked with greater suspicion than a randomly chosen person without it, the same quantity estimated by the Wilcoxon rank statistic. For a score $S$ from a test or model, the definition is as follows.

$$\text{AUC} = P(S_1 > S_0) + \tfrac{1}{2} P(S_1 = S_0)$$

where $S_1$ is the score of a randomly chosen person with the outcome, $S_0$ is the score of a randomly chosen person without it, and $P(\cdot)$ denotes probability. The second term gives half credit to tied scores, which corresponds to the diagonal segment drawn through tied points on the ROC curve.

A score unrelated to the outcome achieves 0.5, the area under the diagonal line of random guessing, and a score that places every person with the outcome above every person without it achieves 1. A value below 0.5 means the score ranks in the wrong direction; reversing it gives one minus the original AUC. Because only the order of the scores matters, any transformation that keeps the order, such as adding a constant to every predicted risk, leaves the AUC unchanged. A related Gini index, twice the area between the ROC curve and the diagonal, satisfies $\text{Gini} + 1 = 2 \times \text{AUC}$, as Fawcett notes; this index is a different quantity from the Gini coefficient of income or health inequality.

Calculating AUC from patient pairs or from the ROC curve

With observed data, every person with the outcome is compared with every person without it. The estimate is the share of those pairs in which the person with the outcome has the higher score. Written as a formula, the estimate is as follows.

$$\widehat{\text{AUC}} = \frac{1}{n_1 n_0} \sum_{i=1}^{n_1} \sum_{j=1}^{n_0} \psi(s_i, s_j)$$

where $n_1$ and $n_0$ are the numbers of people with and without the outcome, $s_i$ and $s_j$ are their scores, and $\psi$ equals 1 when $s_i > s_j$, one half when $s_i = s_j$ and 0 otherwise.

The same number comes from plotting the empirical ROC curve, with sensitivity against the false positive rate (one minus specificity) at every threshold, and adding the trapezoids beneath it. Fawcett explains that trapezoids rather than rectangles are used so that equally scored cases are averaged.

An estimated AUC should carry a confidence interval. Hanley and McNeil gave closed-form approximations for its standard error and for the sample size needed, and DeLong and colleagues proposed a nonparametric method for comparing correlated AUCs when two tests are applied to the same patients. When only part of the curve is relevant, such as a screening test that must keep false positives low, the partial AUC restricts the area to a range of specificity or sensitivity; the pROC authors describe the McClish standardisation, which rescales it to 1 for a perfect curve and 0.5 for no discrimination.

The C-statistic for censored survival outcomes

Many models used in health economics predict time to an event rather than a yes or no outcome. Steyerberg and colleagues describe the concordance (C) statistic as the most commonly used measure of discriminative ability for regression models, identical to the ROC area for a binary outcome and extended to survival outcomes, and Harrell, Lee and Mark describe an index of this kind for survival data.

With censoring, not every pair can be ordered. If one patient is censored before the other has an event, the order of their event times is unknown. A popular version ignores such pairs, and Steyerberg and colleagues note that the result then depends on the censoring pattern. A variant by Gonen and Heller avoids this but holds only within a Cox proportional hazards model, and time-dependent versions measure discrimination up to a stated time. These versions can give different values for the same model, so a survival C-statistic is comparable only when the method and follow-up period match.

Worked example with ten patients and a risk score

The figures below are illustrative. A risk model gives a predicted probability of a cardiovascular event to ten patients, four of whom have the event during follow-up.

1. Count the pairs. There are 4 × 6 = 24 pairs of one event patient and one non-event patient. The patients at 0.80 and 0.60 score above all six non-event patients, giving 6 each. The patient at 0.40 scores below the non-event patient at 0.50 and above the other five, giving 5. The patient at 0.30 scores below 0.50, ties with the non-event patient at 0.30 (half credit) and scores above the other four, giving 4.5.

$$\widehat{\text{AUC}} = \frac{6 + 6 + 5 + 4.5}{4 \times 6} = \frac{21.5}{24} = 0.896$$

where $\widehat{\text{AUC}}$ is the estimated probability that an event patient has a higher predicted risk than a non-event patient.

2. Confirm with the ROC curve. As the threshold falls, the curve rises from (0, 0) to (0, 0.5), moves across to (1/6, 0.5), rises to (1/6, 0.75), runs diagonally through the tie to (2/6, 1) and then across to (1, 1). The area is a rectangle, a trapezoid and a second rectangle, as follows.

$$\text{AUC} = \tfrac{1}{6} \times 0.5 + \tfrac{1}{6} \times \frac{0.75 + 1}{2} + \tfrac{4}{6} \times 1 = 0.0833 + 0.1458 + 0.6667 = 0.896$$

where each term is the area of one segment under the empirical ROC curve. The two methods agree, and the diagonal segment supplies the half credit for the tie.

3. Shift every predicted risk upwards. In a second version of the model every predicted risk is 0.10 higher, so the order is unchanged and its AUC is still 0.896. Neither version is calibrated on average. Four of the ten patients (0.40) have the event, against a mean predicted risk of 0.34 in the original and 0.44 in the shifted version. At an illustrative treatment threshold of 0.35, the original model flags four patients (three with the event and the non-event patient at 0.50), while the shifted model flags six (all four with the event and the non-event patients now at 0.60 and 0.40).

4. Follow the decision into costs. With an illustrative treatment cost of £400 per patient flagged, the original model leads to £1,600 of treatment per ten patients (4 × 400 = 1600) and the shifted model to £2,400 (6 × 400 = 2400). The AUC is identical, yet a cost-effectiveness model that uses the threshold to select patients would give different costs, events prevented and incremental results. Whether the extra treatment is worthwhile depends on its costs and health effects, which the AUC does not contain.

Where discrimination enters economic evaluation and HTA

In health technology assessment the AUC is rarely the input a model needs. The NICE manual (PMG36) describes diagnostic test accuracy statistics as intermediate measures that, when incorporated into models, can predict future health outcomes, and asks for a linked-evidence model, specifying the links between diagnosis, treatment and final outcomes, when direct evidence on final outcomes is missing. Such a model follows people through the test result they receive, so it needs the sensitivity and specificity at the threshold used in practice. PMG36 also notes that sensitivity and specificity are usually correlated and asks for correlation between parameters to be reflected in the probabilistic analysis.

Risk prediction models raise the same issue. Collins and Altman validated QRISK2-2011 and the NICE version of the Framingham equation in a cohort of over two million patients from UK general practice, comparing the two in people aged 35 to 74. Among men in that age group the AUCs were 0.771 and 0.750, yet at the traditional 20% threshold for designating high risk, QRISK2-2011 identified 110 men per 1,000 and the Framingham equation 206, because the Framingham equation over-predicted 10-year risk in men by about 5%. A difference in AUC of about 0.02 says little about a near doubling of the treated population, which drives the cost side of any prevention model.

Vickers, Van Calster and Steyerberg note that sensitivity, specificity and the area under the curve do not show whether a model, marker or test should be used. Decision curve analysis instead calculates the clinical net benefit of acting on a model, true positives less weighted false positives, across a range of threshold probabilities. That clinical measure is distinct from the net monetary benefit of cost-effectiveness analysis, although both judge a decision at a stated trade-off.

Misreadings of the AUC and the C-statistic

The statistic is easy to compute and widely reported, which makes some errors common. The points below are the ones most likely to mislead an economic evaluation.

A good AUC says nothing about calibration. Steyerberg and colleagues note that, as a rank order statistic, the C-statistic is insensitive to systematic errors in calibration such as differences in average outcome, and Van Calster and colleagues add that a poorly calibrated model can be less clinically useful than a well calibrated competitor with a lower AUC. Predicted risks that enter a model as probabilities, or that set treatment thresholds, need calibration checked separately.

Small changes in AUC can hide large changes in risk. Cook showed that a biomarker with an odds ratio of 3 may have little effect on the C-statistic yet could shift an individual's estimated 10-year cardiovascular risk from 8% to 24%. A model comparison based on the AUC alone can therefore miss changes that alter who is treated.

Association, risk and discrimination are different properties. A strong odds ratio does not guarantee good discrimination. Pepe and colleagues showed that a marker with an odds ratio of 3 that flags 10% of people without the condition will correctly identify only 25% of those with it.

The AUC does not change with prevalence, but value does. Fawcett notes that ROC curves are insensitive to changes in the proportion of positive and negative cases. If a test produces the same score distributions among people with and without the condition in a screening population and a specialist clinic, its AUC is the same in both, while its positive predictive value and cost per case detected differ widely. In practice case mix also differs between settings, which can change the AUC as well.

Apparent performance is optimistic. An AUC measured on the data used to develop a model tends to overstate how it will perform elsewhere, partly because of overfitting and partly because new settings differ in case mix. In the testicular cancer case study of Steyerberg and colleagues, the C-statistic of the model without the tumour marker fell from 0.818 at development to 0.785 at external validation. Economic evaluations that borrow a published risk equation should look for external validation, or internal validation by bootstrapping or cross-validation.

The same name, a different quantity. An "AUC" reported in a pharmacokinetic study as drug exposure, or in a survival analysis as the area under a survival curve, is an exposure or a mean time, and cannot be read on the 0.5 to 1 scale.

Sources

• Collins GS, Altman DG. Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2. BMJ. 2012;344:e4181. https://pmc.ncbi.nlm.nih.gov/articles/PMC3380799/
• Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935. https://pubmed.ncbi.nlm.nih.gov/17309939/
• DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-845. https://pubmed.ncbi.nlm.nih.gov/3203132/
• Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed. Oxford: Oxford University Press; 2015.
• European Medicines Agency, Committee for Medicinal Products for Human Use. Guideline on the investigation of bioequivalence (CPMP/EWP/QWP/1401/98 Rev. 1). London: EMA; 2010. Section 1.1. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-investigation-bioequivalence-rev1_en.pdf
• Fawcett T. An introduction to ROC analysis. Pattern Recognition Letters. 2006;27(8):861-874. Sections 4.2, 5 and 7. https://doi.org/10.1016/j.patrec.2005.10.010
• Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29-36. https://pubmed.ncbi.nlm.nih.gov/7063747/
• Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine. 1996;15(4):361-387. https://pubmed.ncbi.nlm.nih.gov/8668867/
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Sheffield: Decision Support Unit; 2011, last updated March 2013. Section 2. https://www.sheffield.ac.uk/nice-dsu/tsds/survival-analysis
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 4.6.12, 4.6.14, 4.7.18 and 4.7.19. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2
• Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. American Journal of Epidemiology. 2004;159(9):882-890. https://pubmed.ncbi.nlm.nih.gov/15105181/
• Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Müller M. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. https://pmc.ncbi.nlm.nih.gov/articles/PMC3068975/
• Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138. https://pmc.ncbi.nlm.nih.gov/articles/PMC3575184/
• Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17:230. https://pmc.ncbi.nlm.nih.gov/articles/PMC6912996/
• Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6. https://pmc.ncbi.nlm.nih.gov/articles/PMC4724785/