Test-Retest Reliability

Test-Retest Reliability of Health Outcome Instruments

Test-retest reliability asks whether a questionnaire or clinical measure gives the same result twice when the person completing it has not changed. In health economics the question matters most for the patient-reported outcome instruments that supply health state utility values and, through them, QALYs. If repeat scores in stable patients wander, part of any measured change is noise, and responder rules based on individual change can misclassify patients. A test-retest study yields both a reliability coefficient and an error estimate in the instrument's own units.

Stability as the condition that defines the study

The COSMIN consensus on measurement properties gives an extended definition of reliability as the extent to which scores for patients who have not changed are the same for repeated measurement, with "over time (test-retest)" as one of those conditions. Terwee and colleagues, in their quality criteria for health status questionnaires, call this reproducibility, the degree to which repeated measurements in stable persons provide similar answers. Unlike internal consistency reliability, the replication is a second occasion rather than a different set of items.

The COSMIN manual places reliability in classical test theory, where each observation is a true score plus error, and describes it as the proportion of total variance due to true differences between patients. It adds that the assumption behind both reliability and measurement error studies is that patients are stable on the construct between the repeated measurements. Terwee and colleagues separate agreement, which concerns how close repeated scores are in the unit of the scale, which matters for detecting change, from reliability, which concerns how well patients can be distinguished from each other despite measurement error.

Designing a test-retest study for a patient-reported outcome

The COSMIN risk of bias box for reliability opens with three design standards, which ask whether patients were stable in the interim period, whether the time interval was appropriate and whether test conditions were similar. Under its worst score counts principle, the lowest rating on any standard sets the rating of the study.

On stability, COSMIN suggests that evidence could come from a global rating of change completed by patients or physicians, and advises that where an intervention is given in the interim period many patients can be assumed to have changed. On the interval, the manual states that it should be long enough to prevent recall bias and short enough to ensure that patients have not changed, and that about 2 weeks is often considered appropriate for patient-reported outcome measures. Terwee and colleagues suggest 1 or 2 weeks. Marx and colleagues randomised 70 stable patients with knee disorders to repeat four knee rating scales and the SF-36 at either 2 days or 2 weeks and found no statistically significant difference in reliability between the intervals. COSMIN also treats it as an important flaw if the first administration influenced the second, for example if patients could see their earlier answers.

Statistics for scores, categories and error

For continuous scores such as a utility index COSMIN prefers the intraclass correlation and, for a straightforward test-retest design, the two-way random effects model, which takes systematic differences between time points into account. It rates a Pearson correlation as doubtful unless there is evidence that no systematic change occurred, because correlation ignores systematic error. For dichotomous or nominal items it prefers Cohen's kappa, and for ordinal items such as the five-level EQ-5D-5L dimensions a weighted kappa with the weights described (see kappa statistic). COSMIN treats percentage agreement as a parameter of measurement error rather than of reliability.

$$\text{ICC}_{\text{agreement}} = \frac{\sigma^2_p}{\sigma^2_p + \sigma^2_o + \sigma^2_e}$$

where $\sigma^2_p$ is the variance between patients, $\sigma^2_o$ is the variance between the two occasions (the systematic shift), and $\sigma^2_e$ is the residual error variance, all estimated from a two-way analysis of variance of the paired scores.

The error side is expressed in scale units. Terwee and colleagues define the standard error of measurement as the square root of the error variance, either including systematic differences (agreement) or excluding them (consistency), and argue that systematic differences belong in the error. Bland and Altman analysed repeatability through the differences between paired measurements, which should average zero when one method is used twice, with 95% expected to be less than two standard deviations of the differences, the repeatability coefficient of the British Standards Institution.

$$\text{LoA} = \bar{d} \pm 1.96\, s_d, \qquad SEM_{\text{agreement}} = \sqrt{\sigma^2_o + \sigma^2_e}, \qquad SDC = 1.96 \times \sqrt{2} \times SEM$$

where $\bar{d}$ is the mean retest minus test difference, $s_d$ is the standard deviation of those differences, LoA are the 95% limits of agreement, $SEM$ is the standard error of measurement, and $SDC$ is the smallest detectable change for one individual. The factor $\sqrt{2}$ appears because a change score involves two measurements, each carrying error, as de Vet and colleagues explain for the minimal detectable change.

Thresholds for reliability and detectable change

COSMIN rates reliability as sufficient when the intraclass correlation or weighted kappa is at least 0.70, and Terwee and colleagues add a sample of at least 50 patients. For measurement error, COSMIN gives a positive rating when the smallest detectable change or the limits of agreement are smaller than the minimal important change. The manual explains that changes below the smallest detectable change or inside the limits of agreement are likely to be measurement error, while larger changes are real but not necessarily meaningful to patients, a question answered by the minimal clinically important difference, preferably estimated with anchor-based methods.

Terwee and colleagues note that the smallest detectable change for a group is the individual value divided by the square root of the group size. A threshold too coarse for one patient can therefore be adequate for mean change in a trial arm, although a systematic shift between occasions does not shrink with group size.

Worked example with a utility index and a global rating of change

All numbers in this example are illustrative. Nine outpatients with stable disease answer a preference-based questionnaire at two visits 14 days apart, adding a global rating of change question at the second visit. Eight report no change. Their index scores are 0.52, 0.60, 0.65, 0.70, 0.74, 0.80, 0.85 and 0.90 at the first visit and 0.56, 0.58, 0.68, 0.70, 0.79, 0.77, 0.87 and 0.93 at the second. The ninth reports being much better and moves from 0.55 to 0.75.

1. Differences in the stable patients. The retest minus test differences are 0.04, minus 0.02, 0.03, 0.00, 0.05, minus 0.03, 0.02 and 0.03. They sum to 0.12, so $\bar{d}$ = 0.12/8 = 0.015.

2. Spread of the differences. The squared deviations from 0.015 sum to 0.0058, so $s_d = \sqrt{0.0058/7} = \sqrt{0.000829} = 0.0288$.

3. Limits of agreement. With 1.96 × 0.0288 = 0.0564, the limits are 0.015 minus 0.0564 = minus 0.041 and 0.015 plus 0.0564 = 0.071.

4. Reliability coefficient. A two-way analysis of variance gives mean squares of 0.033186 between patients, 0.000900 between occasions and 0.000414 for residual error. The variance components are $\sigma^2_p$ = (0.033186 minus 0.000414)/2 = 0.016386, $\sigma^2_o$ = (0.000900 minus 0.000414)/8 = 0.000061 and $\sigma^2_e$ = 0.000414. The three components sum to 0.016861, and the agreement coefficient is 0.016386 divided by 0.016861, or 0.97.

5. Error in utility units. $SEM_{\text{agreement}} = \sqrt{0.000061 + 0.000414} = \sqrt{0.000475} = 0.0218$, and $SDC$ = 1.96 × 1.4142 × 0.0218 = 0.060. For a group of 100 patients the detectable change is 0.060/10 = 0.006.

6. Including the patient who changed. Adding the ninth patient, whose difference is 0.20, raises $s_d$ to 0.067 and the individual $SDC$ to about 0.14, and lowers the agreement coefficient to 0.84. One real change, wrongly treated as noise, more than doubles the apparent measurement error.

7. Interpretation. The instrument separates these patients well, yet a single patient's change of 0.05 cannot be distinguished from error, because it is below 0.060. A responder rule set at a gain of 0.05 would classify some unchanged patients as responders. A mean change of 0.05 in a group of 100 patients, by contrast, lies far beyond the group value of 0.006.

Evidence from EQ-5D-5L studies

In a systematic review of EQ-5D-5L studies published in 2011 to 2019, Feng and colleagues found nine papers on test-retest reliability. Eight found agreement for the index value excellent and one found an intraclass correlation of 0.7, while five found only fair kappa for some dimensions, tending to be lowest for pain/discomfort and highest for mobility. The authors concluded that the results show potential problems with stability at item level but not at index level.

A general population study listed in the same review's table, though not in its count, points the other way. Purba and colleagues retested 206 Indonesian adults after a mean of 17.45 days. Agreement on the five dimensions was high (Gwet's agreement coefficient 0.85 to 0.99, percentage agreement 90% to 99%), yet the intraclass correlation was 0.37 for the index and 0.45 for the visual analogue scale. In the full sample 44.07% reported no problems on any dimension, and the authors explained the low index coefficient by the skewed, low-variance data, noting that the intraclass correlation relies on variance and is more favourable in patient samples with less ceiling effect.

Consequences for cost-utility analysis

In a randomised cost-utility analysis, random error that is unrelated to treatment arm widens the variance of the mean QALY difference without shifting the arm means, so its cost is precision and sample size rather than bias. A systematic shift between occasions, such as the 0.015 in the example, is indistinguishable from a real gain when there is no control group.

A responder definition that feeds a model's response probabilities needs the smallest detectable change to be below its threshold, or stable patients will be counted as responders. In a mapping study, random error in the clinical measure used as a predictor weakens its estimated relationship with utility, a standard regression dilution effect that understates differences between health states.

Boundaries with neighbouring measurement properties

Several neighbouring properties are easily confused with test-retest reliability. For internal consistency, the COSMIN manual regards a standard error of measurement calculated from Cronbach's alpha as inappropriate because it ignores variance between time points, and Feng and colleagues treat internal consistency as irrelevant for the EQ-5D, a preference-based measure. Inter-rater and intra-rater designs, compared under reliability assessment, replicate across or within assessors.

Responsiveness is the mirror image, the ability to detect change in patients who have changed, and the smallest detectable change from test-retest data shows which observed changes exceed error. A highly reproducible score can still measure the wrong thing, which is a question for construct validity.

Common misreadings of test-retest results

Most errors come from forgetting the stability assumption or from reading one number without its sample. Each carries a cost in a model or trial analysis.

1. Assuming stability from elapsed time. COSMIN asks for evidence such as a global rating of change, and step 6 shows the cost of one changed patient.
2. Reporting a correlation as reliability. Bland and Altman called the use of correlation for agreement misleading, and a constant upward shift leaves the correlation untouched.
3. Treating the coefficient as a fixed property of the instrument. The EQ-5D-5L index shows excellent agreement in most reviewed studies and poor agreement in a general population with a large ceiling.
4. Using the group threshold for individuals. The individual smallest detectable change is larger by the square root of the group size.
5. Equating detectable with important change. COSMIN states that a change beyond measurement error is not necessarily meaningful to patients.

Sources

• Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307-310. https://doi.org/10.1016/S0140-6736(86)90837-8
• de Vet HCW, Terwee CB, Ostelo RW, Beckerman H, Knol DL, Bouter LM. Minimal changes in health status questionnaires: distinction between minimally detectable change and minimally important change. Health and Quality of Life Outcomes. 2006;4:54. https://doi.org/10.1186/1477-7525-4-54
• Feng YS, Kohlmann T, Janssen MF, Buchholz I. Psychometric properties of the EQ-5D-5L: a systematic review of the literature. Quality of Life Research. 2021;30(3):647-673. https://doi.org/10.1007/s11136-020-02688-y
• Marx RG, Menezes A, Horovitz L, Jones EC, Warren RF. A comparison of two time intervals for test-retest reliability of health status instruments. Journal of Clinical Epidemiology. 2003;56(8):730-735. https://doi.org/10.1016/S0895-4356(03)00084-2
• Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of patient-reported outcome measures (PROMs): user manual. Version 1.0, February 2018. https://www.cosmin.nl/wp-content/uploads/COSMIN-syst-review-for-PROMs-manual_version-1_feb-2018.pdf
• Purba FD, Hunfeld JAM, Iskandarsyah A, Fitriana TS, Sadarjoen SS, Passchier J, Busschbach JJV. Quality of life of the Indonesian general population: test-retest reliability and population norms of the EQ-5D-5L and WHOQOL-BREF. PLoS ONE. 2018;13(5):e0197098. https://doi.org/10.1371/journal.pone.0197098
• Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, Bouter LM, de Vet HCW. Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology. 2007;60(1):34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012