Intraclass Correlation

Intraclass Correlation in Reliability Studies and Clustered Health Data

The intraclass correlation compares how far measurements differ between the units of interest, such as patients or general practices, with how far they differ within those units. In health economics it answers two practical questions, whether a utility instrument or clinical rating gives the same result when repeated, and how strongly patients in the same practice or hospital resemble one another in a cluster randomised trial. One variance ratio sits behind both uses, although the designs and the values regarded as acceptable differ. Shrout and Fleiss showed that different forms of the coefficient can give quite different results from the same data, so a reported value carries little meaning until its form is stated.

A ratio of variance components

The simplest model writes each measurement as an overall mean plus the unit's true departure plus residual noise. Two measurements on the same unit share that departure, and their correlation is the intraclass correlation.

$$x_{ij} = \mu + b_i + w_{ij}, \qquad \rho = \frac{\sigma_b^2}{\sigma_b^2 + \sigma_w^2}$$

where $x_{ij}$ is the $j$th measurement on unit $i$ (a patient in a reliability study, a cluster such as a practice in a trial), $\mu$ is the overall mean, $b_i$ is the departure of unit $i$'s true mean from $\mu$ with variance $\sigma_b^2$, $w_{ij}$ is the residual with variance $\sigma_w^2$, and $\rho$ is the population intraclass correlation.

Shrout and Fleiss define the intraclass correlation as the correlation between one measurement on a target and another measurement on the same target, which is often, but not always, the target variance divided by the sum of it and other variance components. Terwee and colleagues describe reliability coefficients for health status questionnaires in the same way, as variation between patients over total variation. The value is 0 when units do not differ beyond noise and approaches 1 as within-unit variation shrinks. The same parameter appears as the common correlation in an exchangeable correlation structure.

With $k$ measurements on each of $n$ units, a one-way analysis of variance gives the estimator below.

$$\text{ICC}(1,1) = \frac{BMS - WMS}{BMS + (k - 1)\,WMS}$$

where $BMS$ is the between-units mean square on $n - 1$ degrees of freedom, $WMS$ is the within-unit mean square on $n(k - 1)$ degrees of freedom and $k$ is the number of measurements per unit. Shrout and Fleiss note that it is biased, and that applying it to two-way data tends to underestimate the true correlation. Whenever $BMS$ falls below $WMS$ the estimate is negative, a chance result when the true value is near zero.

Six forms for rating and repeat-measurement studies

Reliability studies usually have the same raters or occasions for every patient, which allows a two-way analysis that separates systematic differences between raters or occasions from residual error. Shrout and Fleiss frame the choice as three decisions, whether a one-way or two-way analysis fits the design, whether differences between raters' mean levels count against reliability, and whether the unit of analysis is a single rating or the mean of several.

$$\text{ICC}(2,1) = \frac{BMS - EMS}{BMS + (k - 1)EMS + k(JMS - EMS)/n}, \qquad \text{ICC}(3,1) = \frac{BMS - EMS}{BMS + (k - 1)EMS}$$

where $JMS$ is the mean square between raters or occasions on $k - 1$ degrees of freedom, $EMS$ is the residual mean square on $(n - 1)(k - 1)$ degrees of freedom, $n$ is the number of patients, and $BMS$ and $k$ are as before. ICC(2,1) treats raters as a random sample and counts their systematic differences as error, so it measures absolute agreement. ICC(3,1) treats the raters as the only ones of interest and ignores mean differences, so Shrout and Fleiss interpret it as consistency rather than agreement.

The average-measure forms give the reliability of a mean of $k$ ratings, which is higher than that of a single rating whenever the single-rating value is positive. Shrout and Fleiss give ICC(1,k) as $(BMS - WMS)/BMS$ and, where the rater by target interaction can be assumed absent, ICC(3,k) as $(BMS - EMS)/BMS$, and note that ICC(3,k) is equivalent to Cronbach's alpha, the usual index of internal consistency reliability. In their example of four judges rating six targets, the six forms ranged from 0.17 for ICC(1,1) to 0.91 for ICC(3,4).

Koo and Li set out the ten forms defined by McGraw and Wong, classified by model, by type (single or mean measurement) and by definition (consistency or absolute agreement), with the six Shrout and Fleiss forms as a subset. They note that two-way random and mixed models with the same definition share one formula, so a mixed model with absolute agreement is computed as ICC(2,1). Terwee and colleagues prefer the agreement form, ICC(A,1) in McGraw and Wong's notation, for health status questionnaires because systematic differences are part of measurement error.

Choosing the form for utility and outcome measurement

In economic evaluation the coefficient is used to judge whether an outcome measure behind quality-adjusted life years is stable. For test-retest reliability of a self-completed patient-reported outcome, Koo and Li recommend a two-way mixed-effects model with absolute agreement, because repeated measurements cannot be regarded as randomised samples and measurements would be meaningless without agreement between them. The single-measurement type fits when each patient completes the instrument once.

Devine and colleagues compared EQ-5D responses from 565 care home residents with those of staff proxies, using a one-way random-effects intraclass correlation and adjusting for clustering by care home. Agreement was moderate for index scores (0.44 to 0.50) and for QALYs over one year (0.59, 95% confidence interval 0.50 to 0.67, in 248 pairs), while cluster-adjusted kappa for individual domains ranged from minus 0.03 to 0.26. The authors judged proxies an acceptable source for index scores and QALYs but less reliable for domains. A cost-utility analysis that mixes self-reported and proxy health state utility values therefore carries extra measurement error.

Terwee and colleagues give a positive rating for reliability when the coefficient, or weighted kappa, is at least 0.70 in a sample of at least 50 patients. Koo and Li suggest at least 30 heterogeneous subjects and, where possible, at least 3 raters.

Worked example with repeated utility scores

All numbers in this example are illustrative. Five patients with a stable chronic condition complete a preference-based questionnaire twice, two weeks apart. Their index scores are 0.60, 0.65, 0.70, 0.75 and 0.80 at the first visit and 0.68, 0.65, 0.76, 0.77 and 0.89 at the second. Five patients is too few for a real study but keeps the sums visible.

1. Differences and means. The retest minus test differences are 0.08, 0.00, 0.06, 0.02 and 0.09, with mean 0.05. Patient means are 0.64, 0.65, 0.73, 0.76 and 0.845, and the grand mean is 0.725.

2. Mean squares. The between-patients mean square is $BMS = 2 \times (0.085^2 + 0.075^2 + 0.005^2 + 0.035^2 + 0.12^2)/4 = 2 \times 0.0285/4 = 0.01425$. With two occasions, the occasions mean square is $JMS = 5 \times 0.05^2/2 = 0.00625$, the residual mean square is $EMS = (0.03^2 + 0.05^2 + 0.01^2 + 0.03^2 + 0.04^2)/8 = 0.006/8 = 0.00075$, and the within-patient mean square is $WMS = 0.0185/10 = 0.00185$, where 0.0185 is the sum of the squared differences.

3. Three coefficients. Consistency gives ICC(3,1) $= 0.0135/0.015 = 0.900$. Absolute agreement gives ICC(2,1) $= 0.0135/(0.015 + 2 \times 0.0055/5) = 0.0135/0.0172 = 0.785$. The one-way form gives ICC(1,1) $= 0.0124/0.0161 = 0.770$. The Pearson correlation between visits is 0.913, close to the consistency value, because it also ignores the average rise of 0.05.

4. Confidence interval. The Shrout and Fleiss interval for ICC(3,1) uses $F_0 = BMS/EMS = 19.0$ on 4 and 4 degrees of freedom, whose 97.5th percentile is 9.605. Then $F_L = 19.0/9.605 = 1.978$ and $F_U = 19.0 \times 9.605 = 182.5$, and the 95% interval runs from $(1.978 - 1)/(1.978 + 1) = 0.33$ to $(182.5 - 1)/(182.5 + 1) = 0.99$.

5. Error in utility units. The two-way variance components are 0.00675 for patients (0.0135 divided by 2), 0.0011 for occasions (0.0055 divided by 5) and 0.00075 for residual error, which reproduce ICC(2,1) $= 0.00675/0.0086 = 0.785$. Following de Vet and colleagues, the standard error of measurement is the standard deviation times the square root of one minus the reliability, $SEM = \sqrt{0.0086} \times \sqrt{1 - 0.785} = 0.043$, and the minimal detectable change is $MDC = 1.96 \times \sqrt{2} \times 0.043 = 0.119$.

6. A more varied sample. If first-visit scores had been 0.50, 0.60, 0.70, 0.80 and 0.90 with the same differences, the between-patients mean square rises to 0.05275 while the other two are unchanged. Agreement becomes ICC(2,1) $= 0.052/(0.0535 + 0.0022) = 0.052/0.0557 = 0.934$, yet the standard error of measurement stays 0.043.

7. Interpretation. The coefficient moves with the form and the spread of patients, while the error in utility units does not. A change smaller than about 0.12 in one patient cannot be separated from measurement error. Random error averages out in arm means, but a systematic rise of 0.05 between occasions would appear as a utility gain in any before-and-after comparison.

Clustering in trials and economic evaluation

When the units are clusters, the coefficient describes how much patients in the same practice resemble each other, the cluster effect. The CONSORT extension gives the inflation factor, or design effect, $DE = 1 + (m - 1)\rho$ for clusters of size $m$, notes that $\rho$ is typically small (often below 0.05) and only estimated with error after a trial, and asks for a coefficient for each primary outcome to be reported.

The design effect and effective sample size pages carry the conversion further, and a mixed effects model estimates the variance components directly.

Boundaries with nearby statistics

Nearby statistics answer different questions. Terwee and colleagues call the Pearson correlation inadequate for reliability because systematic differences are not taken into account, as the worked example shows.

De Vet and colleagues note that the standard error of measurement gives a number in the same units as the original measurement. The intraclass correlation is unitless and, as step 6 shows, depends on how much patients differ. Reliability assessment covers limits of agreement and related statistics.

Misreadings in reliability and trial reports

Most errors arise from reporting a single number without the design behind it. The same data can make an instrument look excellent or mediocre depending on the form chosen and the spread of patients.

1. Not naming the form. Shrout and Fleiss observed that researchers often fail to report which form they used, and Terwee and colleagues found the same in questionnaire studies.
2. Judging the point estimate alone. Koo and Li advise reading the 95% confidence interval against their bands (below 0.5 poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good, above 0.9 excellent). The interval of 0.33 to 0.99 in the worked example spans all four.
3. Treating reliability as a fixed property of an instrument. Koo and Li note that a low value can reflect a lack of variability among the sampled subjects, as step 6 shows.
4. Generalising mean-rating reliability to single raters. Shrout and Fleiss describe a study whose mean ratings over 30 observers had reliability of 0.90 while single ratings had reliability below 0.30.
5. Copying a misprinted formula. The 2016 Koo and Li summary table printed the denominator of the one-way single-measure form with $k + 1$ where Shrout and Fleiss have $k - 1$, an error the journal corrected in a 2017 erratum.

Sources

• Campbell MK, Piaggio G, Elbourne DR, Altman DG. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661. https://doi.org/10.1136/bmj.e5661
• de Vet HCW, Terwee CB, Ostelo RW, Beckerman H, Knol DL, Bouter LM. Minimal changes in health status questionnaires: distinction between minimally detectable change and minimally important change. Health and Quality of Life Outcomes. 2006;4:54. https://doi.org/10.1186/1477-7525-4-54
• Devine A, Taylor SJC, Spencer A, Diaz-Ordaz K, Eldridge S, Underwood M. The agreement between proxy and self-completed EQ-5D for care home residents was better for index scores than individual domains. Journal of Clinical Epidemiology. 2014;67(9):1035-1043. https://doi.org/10.1016/j.jclinepi.2014.04.005
• Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016;15(2):155-163. https://doi.org/10.1016/j.jcm.2016.02.012
• Koo TK, Li MY. Erratum to "A guideline of selecting and reporting intraclass correlation coefficients for reliability research". Journal of Chiropractic Medicine. 2017;16(4):346. https://doi.org/10.1016/j.jcm.2017.10.001
• Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin. 1979;86(2):420-428. https://doi.org/10.1037/0033-2909.86.2.420
• Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, Bouter LM, de Vet HCW. Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology. 2007;60(1):34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012