Kappa Statistic

Kappa Statistic for Agreement Between Reviewers and Raters

Introduced by Jacob Cohen in 1960, the kappa statistic compares how often two raters give the same category with how often they would match if each assigned categories independently at their own rates. A value of 1 means perfect agreement, 0 means agreement no better than chance and a negative value means less agreement than chance would produce. In health technology assessment it is a common summary of how consistently two reviewers screen studies, extract data or judge risk of bias, and of how far proxy reports of health states match patients' own reports. Weighted kappa handles ordered categories and Fleiss' kappa larger groups of raters, but kappa depends on how common each category is, so a single value can mislead without its agreement table.

Agreement beyond chance in two-rater judgements

Two raters who classify the same items will agree on some of them by luck alone. Kappa removes that expected agreement from both the observed agreement and the maximum possible agreement, so the result is the share of the achievable improvement over chance that the raters actually reached. The agreement expected by chance comes from the marginal totals of the cross-tabulation.

$$\kappa = \frac{p_o - p_e}{1 - p_e}, \qquad p_e = \sum_{i=1}^{k} p_{i+}\, p_{+i}$$

where $p_o$ is the observed proportion of items on which the two raters chose the same category (the sum of the diagonal cells divided by the number of items), $p_e$ is the proportion of agreement expected if the raters were independent, $k$ is the number of categories, $p_{i+}$ is the proportion of items the first rater placed in category $i$ and $p_{+i}$ is the proportion the second rater placed in category $i$.

Cohen noted that in the typical rating study there is no criterion for the correctness of judgements. Kappa therefore measures agreement, the reproducibility of a judgement, and says nothing directly about whether either rater is right. Two reviewers who share the same misunderstanding of an eligibility criterion can reach a high kappa while both excluding studies that belong in the review.

Where kappa appears in systematic reviews and economic evaluation

The Cochrane Handbook asks for at least two people working independently to decide whether each study meets the eligibility criteria, and judges it desirable, though not mandatory, that title and abstract screening is also done in duplicate. For data extraction, the Handbook notes that agreement on coded items before consensus can be quantified with kappa statistics, although this is not routinely done in Cochrane reviews, and that any such assessment should be limited to the most important data, such as key risk of bias judgements.

Agreement on risk of bias has been measured directly. Hartling and colleagues applied the first Cochrane risk of bias tool to 163 trials in children and found that weighted kappa between reviewers ranged from 0.13 to 0.74 across the tool's domains, with poorer agreement on items that required more judgement. Because a systematic review feeds the relative effects used in a cost-effectiveness analysis, unreliable screening or appraisal can change which trials reach a meta-analysis and how they are weighted.

The second common use is the reliability of the instruments that generate QALYs. Devine and colleagues compared EQ-5D responses from 565 pairs of care home residents and staff proxies. Cluster-adjusted kappa for the individual domains ranged from minus 0.03 to 0.26, whereas the intraclass correlation for the index score was 0.44 to 0.50 and for QALYs 0.59. The authors concluded that proxies appear acceptable for index scores and QALYs but less reliable for individual domains. The same calculation applies whenever two analysts classify the same records into categories, such as assigning patients from administrative data to the health states of a model.

Weighted kappa for ordered categories

Many health judgements are ordinal. EQ-5D levels, severity grades and the three risk of bias judgements of low, unclear and high all have an order, and a disagreement between adjacent levels is less serious than one between the extremes. Cohen introduced weighted kappa in 1968 to give partial credit for near agreement.

$$\kappa_w = \frac{\sum_{i,j} w_{ij}\, p_{ij} - \sum_{i,j} w_{ij}\, p_{i+}\, p_{+j}}{1 - \sum_{i,j} w_{ij}\, p_{i+}\, p_{+j}}$$

where $p_{ij}$ is the proportion of items placed in category $i$ by the first rater and category $j$ by the second, $p_{i+}$ and $p_{+j}$ are the two raters' marginal proportions, and $w_{ij}$ is an agreement weight equal to 1 on the diagonal and below 1 elsewhere. Linear weights set $w_{ij} = 1 - |i-j|/(k-1)$ and quadratic weights set $w_{ij} = 1 - (i-j)^2/(k-1)^2$. Setting every off-diagonal weight to zero returns the unweighted kappa.

The choice of weights changes the answer, and different schemes can give quite different values for the same table. A quadratically weighted kappa has been shown to equal a form of intraclass correlation coefficient. Linear weights can be read as a weighted average of the kappas from the two-by-two tables formed by collapsing adjacent categories. Devine and colleagues used linear weights for the EQ-5D domains and recommended that studies report which weighting was used and why.

Fleiss' kappa and designs with more than two raters

Cohen's kappa is defined for exactly two raters who each rate every item. Fleiss proposed in 1971 a kappa-like coefficient, an extension of Scott's pi, for studies in which each item is rated by the same number of raters drawn at random from a larger pool. Because it assumes a new sample of raters for each item, Fleiss' kappa is not appropriate for fully crossed designs in which the same raters rate everything. For those designs, the mean of Cohen's kappa across all pairs of raters is one accepted summary.

Assumptions behind the chance correction

The chance term assumes that the ratings are made independently, so that under complete independence of the raters kappa would be zero. Raters who confer or see each other's decisions inflate agreement in a way the statistic cannot detect. Items must also be independent. In the care home study the observations were clustered within homes, and ignoring the clustering made most domain kappas appear to show better agreement than the cluster-adjusted values.

Kappa from a sample is an estimate and should carry a confidence interval. It can be calculated on very few items, but the interval is then so wide that it may include no agreement at all. A pilot of a handful of abstracts can therefore show where criteria are unclear but cannot establish that screening is reliable.

Worked example with two reviewers screening abstracts

The numbers in this example are illustrative. Two reviewers independently screen 200 titles and abstracts for a review that will inform an economic model, each deciding include or exclude. In screen A, both include 20 records, only the first reviewer includes 10, only the second includes 5 and both exclude 165.

Observed agreement is $p_o = (20 + 165)/200 = 0.925$. The first reviewer includes 30 of 200 records (0.15) and the second 25 of 200 (0.125), so the agreement expected by chance follows.

$$p_e = 0.15 \times 0.125 + 0.85 \times 0.875 = 0.7625, \qquad \kappa = \frac{0.925 - 0.7625}{1 - 0.7625} = \frac{0.1625}{0.2375} = 0.684$$

where 0.85 and 0.875 are the two proportions of exclusions.

In screen B, on a different topic where eligible studies are rarer, both include 4, only the first includes 8, only the second includes 7 and both exclude 181. Observed agreement is again $(4 + 181)/200 = 0.925$. The inclusion proportions are now 12 of 200 (0.06) and 11 of 200 (0.055).

$$p_e = 0.06 \times 0.055 + 0.94 \times 0.945 = 0.8916, \qquad \kappa = \frac{0.925 - 0.8916}{1 - 0.8916} = \frac{0.0334}{0.1084} = 0.308$$

where 0.94 and 0.945 are the two proportions of exclusions.

Positive agreement divides the joint inclusions by the average number of inclusions per reviewer, 20 divided by 27.5 in screen A and 4 divided by 11.5 in screen B. Negative agreement does the same for exclusions, 165 divided by 172.5 and 181 divided by 188.5. In screen B the reviewers flagged 19 distinct records but agreed on only 4, so a single screener would have discarded several records the other judged relevant. The low kappa correctly signals that problem, and the positive agreement shows where it lies.

Prevalence and bias paradoxes

Feinstein and Cicchetti described two paradoxes in the two-by-two table. In the first, a high observed agreement can be drastically lowered by a substantial imbalance in the marginal totals, as in screen B. In the second, kappa is higher when the imbalance in the marginal totals is asymmetrical rather than symmetrical. Later accounts call these the prevalence problem, which tends to make kappa unrepresentatively low when one category dominates, and the bias problem, which tends to make it unrepresentatively high when the raters' marginal distributions differ.

Byrt, Bishop and Carlin showed that kappa can be written in terms of observed agreement and separate indices of bias and prevalence, and recommended reporting those indicators whenever agreement studies are compared. Cicchetti and Feinstein argued that kappa should always be accompanied by the proportions of positive and negative agreement, which they likened to sensitivity and specificity for concordance. Alternative coefficients such as Gwet's AC1 use the same structure with a different chance term, and no single variant corrects for both prevalence and bias.

Reading kappa values and benchmark conventions

Landis and Koch proposed verbal labels that are now widely quoted, with 0 to 0.20 called slight agreement, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and 0.81 to 1 almost perfect. These bands are conventions, and their cut-offs are debated. Krippendorff suggested stricter cut-offs for content analysis, discounting conclusions below 0.67, and McHugh notes that the same scale lets quite modest agreement be described as substantial.

For decision use, what matters is the consequence of disagreement. Screening disagreements are usually resolved by discussion or a third person, so a modest kappa mainly signals that criteria need clarifying and that single screening would be risky. Agreement on a model input matters more, because a misclassified health state carries its error into costs and QALYs. Comparing kappa values across studies can mislead unless prevalence, bias and the weighting scheme are reported alongside them.

Boundaries with percentage agreement, intraclass correlation and accuracy

Raw observed agreement ignores chance, which is large when one category dominates. The intraclass correlation is a common reliability measure for continuous scores such as utility values or costs, and test-retest reliability applies either measure to the same rater or instrument over time. When a reference standard exists, diagnostic accuracy measures such as sensitivity and specificity answer the different question of whether ratings are correct. Krippendorff's alpha extends chance-corrected agreement to nominal, ordinal, interval and ratio data and copes better with missing ratings.

Misreadings of kappa in practice

A low kappa does not always mean careless raters, since rare categories depress it even when observed agreement is high. Studies sometimes report kappa without the agreement table, the weighting scheme or the variant used, which prevents readers from judging whether prevalence or bias explains the value.

Sources

• Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. Journal of Clinical Epidemiology. 1993;46(5):423-429. https://doi.org/10.1016/0895-4356(93)90018-V
• Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology. 1990;43(6):551-558. https://doi.org/10.1016/0895-4356(90)90159-M
• Devine A, Taylor SJC, Spencer A, et al. The agreement between proxy and self-completed EQ-5D for care home residents was better for index scores than individual domains. Journal of Clinical Epidemiology. 2014;67(9):1035-1043. https://doi.org/10.1016/j.jclinepi.2014.04.005
• Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology. 1990;43(6):543-549. https://doi.org/10.1016/0895-4356(90)90158-L
• Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in Quantitative Methods for Psychology. 2012;8(1):23-34. https://doi.org/10.20982/tqmp.08.1.p023
• Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012. https://doi.org/10.1136/bmj.b4012
• Lefebvre C, Glanville J, Briscoe S, et al. Chapter 4: Searching for and selecting studies (last updated March 2025). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5.1. Cochrane; 2025. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04
• Li T, Higgins JPT, Deeks JJ. Chapter 5: Collecting data (last updated October 2019). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-05
• McHugh ML. Interrater reliability: the kappa statistic. Biochemia Medica. 2012;22(3):276-282. https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
• Rapallo F. Analysis of the weighted kappa and its maximum with Markov moves. Psychometrika. 2022;87(4):1270-1289. https://doi.org/10.1007/s11336-022-09844-y
• Tan KS, Yeh YC, Adusumilli PS, Travis WD. Quantifying interrater agreement and reliability between thoracic pathologists: paradoxical behavior of Cohen's kappa in the presence of a high prevalence of the histopathologic feature in lung cancer. JTO Clinical and Research Reports. 2024;5(1):100618. https://doi.org/10.1016/j.jtocrr.2023.100618