Internal Consistency Reliability

Internal Consistency Reliability in Health Outcome Questionnaires

Internal consistency reliability asks whether the items that make up one scale or subscale of a questionnaire rise and fall together closely enough for their sum to be read as a single score. It is estimated from one administration of the instrument, usually with Cronbach's alpha, and it is one of the measurement properties checked before a patient-reported outcome score is used as a trial endpoint or a predictor in a mapping function. A low coefficient means much of the summed score is item-specific noise, while a very high one can mean items repeat each other. Neither shows that the scale measures the intended construct, and the coefficient is only meaningful for items that are meant to reflect one underlying quantity. Stability over time and agreement between raters belong to the wider reliability assessment family.

Which scales internal consistency applies to

Terwee and colleagues, in their quality criteria for health status questionnaires, describe internal consistency as the extent to which items in a scale or subscale are correlated and so measure the same concept. They call it an important property for questionnaires that measure one construct with several items, and not relevant where items are separate aspects of a complex clinical phenomenon that need not correlate, with the Apgar score as their example.

The COSMIN manual for systematic reviews of patient-reported outcome measures (PROMs) frames the same condition through measurement models. In a reflective model every item is a manifestation of the same construct, so the items are expected to be highly correlated and interchangeable. In a formative model the items together form the construct and need not correlate. The manual's first question for any internal consistency study is whether the scale consists of effect indicators, and it suggests a simple test of asking whether all item scores would be expected to change when the construct changes.

This matters for instruments used in economic evaluation. Gamst-Klaussen, Gudex and Olsen, analysing EQ-5D-5L responses from 7,933 people in six countries, found that the instrument appears to include both causal indicators (pain or discomfort and anxiety or depression) and effect indicators (self-care and usual activities), with mobility mostly causal. They also summarise earlier work finding that symptom items in the cancer-specific EORTC QLQ-C30, such as nausea and shortness of breath, behave as causal indicators. Whether items form one dimension is a question for structural validity, which is examined before internal consistency.

Split halves and coefficient alpha

The oldest single-administration approach splits a scale into two halves, correlates the half scores and projects the result to the full length with the Spearman-Brown formula. The answer depends on which split is chosen. Cronbach's 1951 paper in Psychometrika showed that a general coefficient, alpha, is the mean of all split-half coefficients from the different ways of splitting a test, computed by the Flanagan-Rulon formula, which gives the Spearman-Brown value only when the two halves have equal variances, and that the Kuder-Richardson coefficient for dichotomous items is a special case of alpha.

$$\alpha = \frac{k}{k-1}\left(1 - \frac{\sum_{i=1}^{k} \sigma_i^2}{\sigma_X^2}\right)$$

where $k$ is the number of items in the scale, $\sigma_i^2$ is the variance of item $i$ across respondents and $\sigma_X^2$ is the variance of the summed scale score. When items covary strongly, the total-score variance is much larger than the sum of the item variances and alpha approaches 1. When items are unrelated, the two are about equal and alpha approaches 0.

The standardised form shows directly how alpha depends on scale length as well as on how strongly items correlate.

$$\alpha_{std} = \frac{k\,\bar{r}}{1 + (k-1)\,\bar{r}}$$

where $\bar{r}$ is the mean of the correlations between all pairs of items and $k$ is again the number of items. This is the Spearman-Brown formula with the average inter-item correlation in place of the reliability of one item, so the same average correlation yields a higher coefficient in a longer scale.

Coefficient omega and the assumptions behind alpha

Alpha equals the reliability of the summed score only under a restrictive measurement model. Tavakol and Dennick explain that alpha assumes a tau-equivalent model, in which each item measures the same latent trait on the same scale, and that heterogeneous items break this assumption, making alpha in practice a lower-bound estimate. Sijtsma calls alpha a lower bound to the reliability, in many cases a gross underestimate, and shows that it is unrelated to the internal structure of the test.

Dunn, Baguley and Brunsden reviewed these deficiencies and proposed coefficient omega as a practical alternative. Omega is computed from a one-factor model in which items may load unequally on the common factor.

$$\omega = \frac{\left(\sum_{i=1}^{k}\lambda_i\right)^2}{\left(\sum_{i=1}^{k}\lambda_i\right)^2 + \sum_{i=1}^{k}\theta_i}$$

where $\lambda_i$ is the loading of item $i$ on the common factor and $\theta_i$ is the error variance of item $i$. With standardised loadings of 0.8, 0.7, 0.8 and 0.4 (illustrative), the loadings sum to 2.7, the error variances (one minus each squared loading) sum to 2.07, and omega is 7.29 / 9.36 = 0.779. Omega is only as credible as the fitted factor model behind it.

The COSMIN manual accepts either Cronbach's alpha or omega for continuous scores and alpha or the Kuder-Richardson coefficient KR-20 for dichotomous scores.

Item diagnostics during scale development

Two item-level statistics usually accompany the scale coefficient. The corrected item-total correlation correlates each item with the sum of the remaining items, and alpha if item deleted recomputes the coefficient with that item removed. Tavakol and Dennick describe deleting items whose correlation with the total score approaches zero, and they note that an alpha that is too high may mean some items test the same question in a different guise, citing a recommended maximum of 0.90.

These statistics are prompts for judgement and do not settle what the scale should contain. An item that lowers alpha may cover a part of the construct that patients consider important, which is a question of content validity. Removing it can narrow the scale and reduce its responsiveness to treatment.

Thresholds and how COSMIN rates the evidence

Conventional cut-offs vary. Terwee and colleagues report the 0.70 to 0.90 criterion proposed by Nunnally and Bernstein, observe that many good questionnaires exceed 0.90, and give a positive rating when factor analysis was applied and alpha lies between 0.70 and 0.95, calculated separately for each subscale. Their criteria also ask for a factor analysis sample of seven times the number of items and more than 100 respondents.

The 2018 COSMIN criteria rate internal consistency sufficient when there is at least low evidence for sufficient structural validity and alpha is at least 0.70 for each unidimensional scale or subscale. The earlier upper limit of 0.95 was deleted, since the manual treats that criterion as relevant in the development phase of a PROM and not when an existing one is evaluated. Without adequate structural evidence the rating is indeterminate, and the manual states that a coefficient reported for a multidimensional total score, such as the sum of four subscales, cannot be interpreted and can be ignored.

Worked example with a four-item fatigue subscale (illustrative)

All numbers in this example are illustrative. A disease-specific questionnaire contains a four-item fatigue subscale, each item scored 1 to 5, and six patients complete it during piloting. Six respondents are far too few in practice and only keep the arithmetic visible.

1. Item and total variances. Using sample variances with divisor 5, the item variances are 2.000, 1.467, 2.167 and 2.000, which sum to 7.633. The totals have mean 12.5 and variance 21.1.

2. Coefficient alpha.

$$\alpha = \frac{4}{3}\left(1 - \frac{7.633}{21.1}\right) = 1.333 \times 0.638 = 0.851$$

where 4 is the number of items, 7.633 is the sum of the item variances and 21.1 is the variance of the total score. The value passes both the Terwee and COSMIN cut-offs, subject to structural evidence of one dimension.

3. Item diagnostics. The corrected item-total correlations are 0.908, 0.653, 0.860 and 0.406. Item 4 stands out, correlating only 0.117 with item 2. Without item 4 the three remaining item variances sum to 5.633 and the three-item total has variance 14.7.

$$\alpha_{-4} = \frac{3}{2}\left(1 - \frac{5.633}{14.7}\right) = 1.5 \times 0.617 = 0.925$$

where the subscript shows that item 4 has been removed. Deleting the item raises alpha, but if item 4 asks about fatigue on waking, which patients may describe as distinct, the choice is about content and the coefficient cannot make it.

4. Effect of scale length. The six inter-item correlations (0.817, 0.865, 0.500, 0.748, 0.117 and 0.480) average 0.588. The standardised formula gives the four-item value and projects an eight-item scale with the same average correlation.

$$\alpha_{std,4} = \frac{4 \times 0.588}{1 + 3 \times 0.588} = \frac{2.352}{2.764} = 0.851, \qquad \alpha_{std,8} = \frac{8 \times 0.588}{1 + 7 \times 0.588} = \frac{4.704}{5.116} = 0.919$$

where the second subscript is the number of items. Doubling the length lifts the coefficient by almost 0.07 with no improvement in how well any single item measures fatigue, which is why long scales tend to show high alphas.

5. Score precision. Under classical assumptions the standard error of measurement is the score standard deviation multiplied by the square root of one minus the reliability.

$$SEM = 4.593 \times \sqrt{1 - 0.851} = 4.593 \times 0.386 = 1.77$$

where 4.593 is the standard deviation of the total score (the square root of 21.1) and 0.851 is the four-item alpha. On a scale running 4 to 20, an individual score therefore carries an error of roughly 1.8 points, which matters when judging change in one patient.

Uses in trials, mapping and economic evaluation

Internal consistency reaches health economic analysis through the scores it supports. A subscale with low reliability adds error variance to a trial outcome, which widens confidence intervals and lowers power for the same sample size. When disease-specific subscale scores are used as explanatory variables in a mapping function to predict a utility value, error in those scores attenuates the estimated coefficients, so the predicted utilities vary less across patients than the true ones. The mechanism is described under measurement error.

Preference-based measures need a different reading. Gamst-Klaussen and colleagues note that health state utility instruments have an important role in cost-effectiveness analyses, and their results show the EQ-5D mixes causal and effect indicators. The index value for an EQ-5D health state comes from preference weights in a value set and does not assume that the five dimensions reflect one latent trait, so a low alpha across the five dimensions is not evidence of a poor instrument.

When an outcome instrument is chosen for a trial or model, a high alpha in a development paper is weak grounds on its own for preferring one instrument, since test-retest reliability, measurement error, construct validity and responsiveness also count.

Misreadings of alpha in outcome measurement

Most errors come from treating alpha as a general seal of quality. These misreadings recur in instrument reports and outcome choices.

1. Reading alpha as evidence of one dimension. Terwee and colleagues point out that alpha can be high when two or more subscales with high alphas are combined, so the coefficient gives no information on the number of subscales.
2. Treating a higher value as always better. Very high values usually reflect redundant items, which add respondent burden without adding information, or simply a long scale.
3. Computing alpha for formative or mixed item sets. Symptom checklists and multi-attribute utility classification systems need not have correlated items.
4. Quoting alpha for a total score that spans subscales. The COSMIN manual regards such a coefficient as uninterpretable.
5. Transferring a coefficient between populations. Alpha depends on the variance of scores in the sample, so a value from a heterogeneous development sample may not hold in a narrower trial population.
6. Using alpha in place of reliability over time. Stability in patients whose health has not changed needs a repeated design and an intraclass correlation coefficient.

Sources

• Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. https://doi.org/10.1007/BF02310555
• Dunn TJ, Baguley T, Brunsden V. From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology. 2014;105(3):399-412. https://doi.org/10.1111/bjop.12046
• Gamst-Klaussen T, Gudex C, Olsen JA. Exploring the causal and effect nature of EQ-5D dimensions: an application of confirmatory tetrad analysis and confirmatory factor analysis. Health and Quality of Life Outcomes. 2018;16:153. https://doi.org/10.1186/s12955-018-0975-y
• Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs): user manual. Version 1.0. Amsterdam: COSMIN; February 2018. https://www.cosmin.nl/wp-content/uploads/COSMIN-syst-review-for-PROMs-manual_version-1_feb-2018.pdf
• Sijtsma K. On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika. 2009;74(1):107-120. https://doi.org/10.1007/s11336-008-9101-0
• Tavakol M, Dennick R. Making sense of Cronbach's alpha. International Journal of Medical Education. 2011;2:53-55. https://doi.org/10.5116/ijme.4dfb.8dfd
• Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, Bouter LM, de Vet HCW. Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology. 2007;60(1):34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012