Classification Accuracy

Classification Accuracy and the Cost of Errors

Classification accuracy, often reported simply as accuracy or the proportion correct, is the most quoted single summary of a test, a clinical coder or a prediction rule, and its complement is the error rate. In measurement more widely, accuracy means closeness to a true value, a sense covered here through trueness and precision. Health economists meet it in audits of hospital coding that drives payment, in summaries of diagnostic tests and risk models, and in validation of routine data used to count patients and costs. The figure is easy to misread, because it treats every error alike and moves with the case mix. In the illustrative example below, two coding processes with the same 90% accuracy produce different payment totals. Sensitivity, specificity and predictive values are covered on the diagnostic accuracy page.

Counting correct classifications in two classes and in many

The calculation needs only a confusion matrix, the cross-table of assigned categories against reference categories. Alberg and colleagues define overall accuracy as the probability that an individual will be correctly classified by a test, a quantity also reported as diagnostic accuracy or test efficiency. Saito and Rehmsmeier list accuracy and its complement, the error rate, as the most widely used basic measures of classifier performance, defined as follows.

$$Acc = \frac{TP + TN}{TP + TN + FP + FN}, \qquad ERR = 1 - Acc$$

where $Acc$ is classification accuracy, $ERR$ is the error rate, $TP$ and $TN$ are true positives and true negatives, and $FP$ and $FN$ are false positives and false negatives, all judged against the reference standard.

The same idea extends to any number of categories. With more categories the confusion matrix is still square, and accuracy is the share of cases on its diagonal.

$$Acc = \frac{1}{N} \sum_{k=1}^{K} n_{kk}$$

where $K$ is the number of categories, $n_{kk}$ is the number of cases whose assigned and reference categories are both $k$, and $N$ is the total number of cases. Clinical coding is a common multi-class case in health economics. In their systematic review of discharge coding in Great Britain, Burns and colleagues define accuracy as the percentage agreement between codes assigned by independent review of case notes and the codes held in the routine dataset. They found a median overall accuracy of 83.2% against case notes, with results ranging from 50.5% to 97.8% between studies, and noted that some studies reported accuracy at three or four levels of the code. Finer categories give more ways to be wrong, so an accuracy figure means little until the categories are stated.

Why the proportion correct tracks prevalence

A binary classifier makes two kinds of error, and the case mix decides how often each kind of person is seen. Accuracy inherits that mix. Alberg and colleagues show that overall accuracy is a weighted average of sensitivity and specificity.

$$Acc = p \times Se + (1 - p) \times Sp$$

where $p$ is the prevalence of the condition in the group assessed, $Se$ is sensitivity and $Sp$ is specificity. They conclude that the measure becomes more problematic as the gap between sensitivity and specificity widens and as prevalence moves away from 50%, and that its dependence on prevalence makes it inferior to considering sensitivity and specificity together. Fixing both weights at one half gives the simple average of sensitivity and specificity, which equals accuracy at 50% prevalence and does not change with prevalence while sensitivity and specificity stay fixed.

Agreement can also arise by chance. McHugh describes how Cohen criticised per cent agreement in 1960 for ignoring chance agreement and introduced kappa, defined below.

$$\kappa = \frac{\Pr(a) - \Pr(e)}{1 - \Pr(e)}$$

where $\kappa$ is Cohen's kappa, $\Pr(a)$ is observed agreement, which equals classification accuracy when one rater is the reference, and $\Pr(e)$ is the agreement expected by chance from the two sets of category totals.

The equal-cost assumption inside a single accuracy figure

Every error counts once in the proportion correct, whatever it costs. Steyerberg and colleagues note that error rates are usually calculated by classifying people as positive when their predicted probability exceeds 50%, which implies equal weighting of false-positive and false-negative classifications. Vickers and Elkin make the link to decision analysis explicit. The threshold probability at which a patient would opt for treatment shows how the patient weighs the harm of a false positive against the harm of a false negative. Equal harms give a threshold of 0.5, the cut-off that maximises accuracy for well-calibrated predictions, and the decision curve analysis page derives the threshold for unequal harms.

In economic evaluation the two harms are rarely equal, since a missed cancer and an unnecessary biopsy differ in cost and in QALYs. Vickers and Elkin observe that metrics concerned with accuracy do not incorporate information on consequences. When a classification feeds a decision, the comparison belongs in a model that values each kind of error, through net monetary benefit or decision curve analysis.

Accuracy of a measurement or estimate, trueness and precision

Outside classification, accuracy has a wider meaning set out in the International Vocabulary of Metrology (VIM). It defines measurement accuracy as the closeness of agreement between a measured value and a true value, and notes that accuracy is not a quantity and is not given a numerical value, although it is related to both trueness and precision. Trueness is closeness between the average of infinitely many replicate measurements and a reference value, and the VIM states that it is inversely related to systematic error but not related to random error, pointing to ISO 5725 for measures of closeness. Precision is closeness between replicate measurements, usually expressed through imprecision measures such as the standard deviation or variance.

These terms carry over to the estimates an economic evaluation uses. A mean cost per episode, a QALY gain or a model output is accurate when it lies close to the true value. Lack of trueness is bias and imprecision is variance, and the mean squared error combines them. Morris, White and Crowther state that the mean squared error is the sum of the squared bias and the variance of an estimator, and that small bias may be accepted for a substantial reduction in variance, the trade examined on the bias-variance tradeoff page. Errors in recorded values that feed such estimates are covered under measurement error.

The word precision needs care. In the classifier literature that Saito and Rehmsmeier review, precision means the positive predictive value, the share of positive calls that are correct, a different idea from metrological precision.

Worked example with two coding processes and two payment totals

All figures are illustrative. A hospital codes 1,000 episodes into three payment groups with tariffs of £1,000 (Low), £3,000 (Medium) and £8,000 (High). Case-note review, taken as the reference, places 600 episodes in Low, 300 in Medium and 100 in High, so the correct payment is 600 × 1,000 + 300 × 3,000 + 100 × 8,000 = £2,300,000. Two coding processes are audited against the same notes.

1. Accuracy. Process A has 560 + 250 + 90 = 900 cases on the diagonal and process B has 540 + 260 + 100 = 900, so both score 900 / 1,000 = 0.90.

2. Chance-corrected agreement. For A the expected chance agreement is 0.6 × 0.6 + 0.3 × 0.3 + 0.1 × 0.1 = 0.46, so kappa is (0.90 − 0.46) / 0.54 = 0.815. For B it is 0.6 × 0.54 + 0.3 × 0.32 + 0.1 × 0.14 = 0.434, so kappa is (0.90 − 0.434) / 0.566 = 0.823. Kappa ranks B slightly higher.

3. Payment totals. A's errors cancel, because its coded totals match the reference, so it pays £2,300,000. B's errors all move episodes upwards, so it pays 540 × 1,000 + 320 × 3,000 + 140 × 8,000 = £2,620,000, an overpayment of £320,000, or 13.9%.

4. Bias and spread of the episode payment error. Treating each coded tariff as a measurement of the correct one, the error per episode is the coded tariff minus the reference tariff. For A, 80 episodes are wrong by £2,000 in either direction and 20 by £5,000, so the mean error is zero and the mean squared error is (80 × 4,000,000 + 20 × 25,000,000) / 1,000 = 820,000. For B, 60 episodes are over by £2,000 and 40 by £5,000 and the bias and mean squared error are as follows.

$$\text{Bias}_B = \frac{60 \times 2000 + 40 \times 5000}{1000} = 320, \qquad MSE_B = \frac{60 \times 2000^{2} + 40 \times 5000^{2}}{1000} = 1240000$$

where $\text{Bias}_B$ is the mean error per episode in pounds and $MSE_B$ is the mean squared error in squared pounds. The squared bias is 320 × 320 = 102,400, so the variance of B's errors is 1,240,000 − 102,400 = 1,137,600. The root mean squared error is about £906 per episode for A and £1,114 for B.

5. Interpretation. Identical accuracy hides a process with no bias in the total and a process that overstates it by 13.9%; kappa does not separate them either. What matters for a cost estimate is the direction and size of errors, which the confusion matrix shows and the single figure discards. Burns and colleagues report a related pattern in practice. An Audit Commission programme in all acute NHS trusts found an average Healthcare Resource Group (HRG) coding error of 9.4% with a net financial impact close to zero, although the local impact was significant in some cases.

Reporting an accuracy figure so others can use it

An accuracy figure is an estimate from a sample and carries sampling error. STARD 2015 asks reports of diagnostic accuracy studies to give estimates of diagnostic accuracy and their precision, such as 95% confidence intervals. For 900 correct out of 1,000, the binomial standard error is the square root of 0.9 × 0.1 / 1,000, or 0.0095, giving a normal-approximation 95% confidence interval of 0.881 to 0.919.

A usable report also states the reference standard, the population and its case mix, and the category definitions, including the code level for clinical coding. Publishing the full confusion matrix, which STARD asks for in the form of index test results by reference standard results, lets an analyst recompute accuracy, error direction and any cost consequence.

Neighbouring measures and when each applies

Classification accuracy judges labels after a cut-off has been applied. Related measures answer different questions.

• Diagnostic accuracy covers the family of measures for a binary test, conditional on disease status or on the test result, and their pooling across studies.
• Brier score and calibration. The Brier score and calibration judge predicted probabilities, which decision models need as risks, not labels.
• Area under the curve. The area under the curve measures ranking across all thresholds and fixes none.
• Model validation. Agreement of a decision model's outputs with observed data is a question for model validation.
• Criterion validity. Agreement of a routine-data algorithm with medical records, and its effect on cost estimates, is examined under criterion validity.

Common misreadings

The proportion correct is often treated as a fuller summary than it is. Each error below changes conclusions in costing or HTA.

• High accuracy for a rare condition. At low prevalence the figure is driven by specificity and can look strong for a classifier that misses most cases.
• Comparing figures across settings. Two accuracy values from populations with different prevalence or category definitions are not directly comparable.
• Equal accuracy read as equal consequences. As the example shows, the direction of errors can bias a total cost while accuracy stays the same.

Sources

• Alberg AJ, Park JW, Hager BW, Brock MV, Diener-West M. The use of "overall accuracy" to evaluate the validity of screening or diagnostic tests. Journal of General Internal Medicine. 2004;19(5 Pt 1):460-465. https://doi.org/10.1111/j.1525-1497.2004.30091.x
• Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. https://pmc.ncbi.nlm.nih.gov/articles/PMC4623764/
• Burns EM, Rigby E, Mamidanna R, Bottle A, Aylin P, Ziprin P, Faiz OD. Systematic review of discharge coding accuracy. Journal of Public Health. 2012;34(1):138-148. https://doi.org/10.1093/pubmed/fdr054
• Joint Committee for Guides in Metrology. International vocabulary of metrology: basic and general concepts and associated terms (VIM). 3rd ed. JCGM 200:2012. Sections 2.13 to 2.15. https://www.bipm.org/documents/20126/2071204/JCGM_200_2012.pdf
• McHugh ML. Interrater reliability: the kappa statistic. Biochemia Medica. 2012;22(3):276-282. https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
• Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Statistics in Medicine. 2019;38(11):2074-2102. https://doi.org/10.1002/sim.8086
• Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One. 2015;10(3):e0118432. https://doi.org/10.1371/journal.pone.0118432
• Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138. https://doi.org/10.1097/EDE.0b013e3181c30fb2
• Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. https://doi.org/10.1177/0272989X06295361