Cross-Cultural Validity

Cross-Cultural Validity of Translated Outcome Measures in Multinational Evaluation

Multinational trials and cost-utility analyses routinely pool questionnaire answers collected in several languages, and the pooled numbers mean something only if a given level of health produces the same answers whatever version a patient happened to complete. Cross-cultural validity is the property that tests this, and the COSMIN initiative, which set the standard vocabulary for the quality of patient-reported outcome measures, names it cross-cultural validity or measurement invariance and places it inside construct validity. For a health economist the stakes are concrete. An EQ-5D language version in which one dimension is answered differently at the same level of health shifts the utilities that feed QALYs, and the shift can reach an ICER whenever the groups being compared did not complete the same versions. Cross-cultural validity concerns the descriptive questionnaire, so it is a separate question from which country's value set should score the answers.

What COSMIN means by cross-cultural validity and measurement invariance

The COSMIN taxonomy, agreed through an international Delphi study and published by Mokkink and colleagues in 2010, groups measurement properties into three domains, namely reliability, validity and responsiveness. Interpretability appears in the taxonomy as well, but COSMIN treats it as an important characteristic of an instrument and not as a measurement property. Validity contains content validity, criterion validity and construct validity, and construct validity in turn contains structural validity, hypotheses testing and cross-cultural validity.

COSMIN defines cross-cultural validity, under the joint label cross-cultural validity\measurement invariance, as the degree to which the performance of the items on a translated or culturally adapted patient-reported outcome measure (PROM) is an adequate reflection of the performance of the items of the original version. The COSMIN manual describes measurement invariance as the requirement that respondents from different groups with the same level of the latent trait respond similarly to a particular item. The 2018 COSMIN guideline for systematic reviews, reported by Prinsen and colleagues, says the property should be evaluated when a PROM is or will be used in different cultural populations, and applies the same analyses to other groups, such as different gender or age groups or different patient populations. The label therefore covers more than translation, although translated versions are where health economists meet it most often.

Translation as development and invariance testing as evidence

Producing a new language version and showing that it works are different steps. COSMIN states that the translation process itself is not a measurement property but part of the development of a new version of a PROM. A careful forward and back translation, reviewed by a committee and tested in cognitive interviews, makes equivalence more likely, but it does not show that the items behave alike once real respondents answer them.

Good practice for the translation step has its own literature. In 2005 an ISPOR task force reported by Wild and colleagues set out principles of good practice for translating and culturally adapting patient-reported outcome measures, drawing together the methods and guidelines published up to then. Cross-cultural validity, in COSMIN's sense, is the empirical check that follows such a process, using response data from the original and the new versions.

Testing invariance with differential item functioning and multiple-group factor analysis

COSMIN recognises two families of analysis. Differential item functioning (DIF) analysis asks, item by item, whether group membership predicts the response once the level of the underlying trait is held constant. Multiple-group confirmatory factor analysis (MGCFA) asks whether the factor structure and the factor loadings are the same in each group. Equal loadings are not enough for comparing mean scores across versions, which also needs scalar invariance, meaning equal item intercepts, or equal thresholds for ordinal items, so that people with the same level of the trait are expected to give the same answers in every version.

Under the COSMIN criteria for good measurement properties a result is rated sufficient when no important differences are found between group factors such as age, gender or language in multiple-group factor analysis, or no important DIF is found, with McFadden's $R^2$ below 0.02. It is rated insufficient when important differences or DIF are found, and indeterminate when no multiple-group factor analysis or DIF analysis was performed.

COSMIN standards for a study of cross-cultural validity

COSMIN judges the design of the study as well as its result. The COSMIN Risk of Bias checklist, published by Mokkink and colleagues in 2018 for systematic reviews of PROMs, adapted the original COSMIN checklist and its four-point rating system so that each study on a measurement property is rated for risk of bias. The cross-cultural validity box opens with a design question, whether the samples were similar for relevant characteristics apart from the group variable, and it grades sample size for multiple-group factor analysis in multiples of the number of items, with seven times the number of items needed for the top rating.

Evidence for the EQ-5D-5L across language versions in Singapore

In Singapore patients in one health system can complete the EQ-5D-5L in English, Chinese or Malay. Wang and colleagues enrolled a convenience sample of patients with type 2 diabetes from a public primary care institution and compared the three versions, each patient completing the version in the language they chose. They used multiple linear regression for the index, calculated with an interim scoring algorithm, and for the EQ VAS, judging equivalence by whether the 90 per cent confidence interval of the difference fell within a margin set in advance. Multiple logistic regression compared the responses to each of the five items by language.

The index scores were equivalent between the Chinese and English versions and between the Malay and English versions, but equivalence could not be shown between the Malay and Chinese versions. On the items, responses did not differ between languages, except that patients completing the Chinese version were more likely than those completing the Malay version to report no problems with mobility. The authors reported in 2015 that the study gave evidence for measurement equivalence of the language versions in Singapore. These are comparisons of scores between groups who chose different versions. They do not hold the level of the trait constant, as DIF analysis does, and do not test loadings or intercepts, as multiple-group factor analysis does, so under the COSMIN criteria the study is indeterminate evidence of cross-cultural validity. Equivalent mean scores are consistent with invariance without demonstrating it, and the mobility difference may reflect item bias, real differences in health between the language groups, or both.

Item performance across languages and the choice of value set

Cross-cultural validity and the choice of value set answer different questions, and an analysis needs both. The first asks whether the descriptive system records the same health state in the same way across versions. The second asks whose preferences turn that state into a health state utility value, and the practice of using a country-specific value set rests on preferences differing between countries, not on any defect in the questionnaire.

The size of the value set effect can be large. Gerlinger and colleagues compared six national EQ-5D-5L value sets, for Canada, England, Japan, Korea, the Netherlands and Uruguay, together with ten crosswalk value sets, across all 3,125 health states. The median difference between the highest and lowest country index for a state was 0.417, and they recommended in 2019 that multinational clinical studies use country-specific value sets to evaluate treatment effects. Invariance evidence does not justify borrowing another country's value set, and a domestic value set cannot repair an item that performs differently in translation. A complete multinational analysis therefore checks that the language versions are invariant and then scores each patient's answers with the value set that fits the decision.

Worked example of a DIF check and its effect on an ICER (illustrative)

All numbers are illustrative. An analyst tests the mobility item of a five-item questionnaire for DIF between two language versions, using ordinal logistic regression. McFadden's pseudo $R^2$ compares a fitted model with a model that contains only intercepts.

$$R^2_{McF} = 1-\frac{\ln L_M}{\ln L_0}$$

where $R^2_{McF}$ is McFadden's pseudo $R^2$, $\ln L_M$ is the log-likelihood of the fitted model and $\ln L_0$ is the log-likelihood of the intercept-only model. COSMIN's criteria state only the threshold of 0.02. In logistic regression DIF practice the figure is applied to the change in pseudo $R^2$ when group terms are added to a model that already holds the trait score, and the lordif software described by Choi, Gibbons and Crane uses a change of 0.02 as its default criterion. The example follows that reading.

1. Trait-only model. With $\ln L_0 = -1000$ and $\ln L_M = -700$, the result is 1 − 700/1000 = 0.300.

2. Trait plus language for the self-care item. The log-likelihood rises to −688, so $R^2_{McF}$ is 1 − 688/1000 = 0.312. The gain is 0.312 − 0.300 = 0.012, below 0.02, so on this reading the self-care item shows no important DIF.

3. Trait plus language for the mobility item. For the mobility item the trait-only model also gives 0.300, and adding language raises the log-likelihood to −672, so $R^2_{McF}$ is 1 − 672/1000 = 0.328. The gain is 0.328 − 0.300 = 0.028, above 0.02, so on the same reading the mobility item is flagged for DIF.

4. Effect on a cost-utility result. Suppose a single-arm study of a new treatment used version B, where the mobility item flags fewer problems at the same health, and the comparator data came from a registry using version A. The true utility gain is 0.04 for 2 years, a QALY gain of 0.04 × 2 = 0.08, and the incremental cost is GBP 2,400, giving GBP 2,400 ÷ 0.08 = GBP 30,000 per QALY. If DIF adds 0.02 to the apparent gain, the ICER falls as follows.

$$\text{ICER} = \frac{2400}{0.06 \times 2} = 20000$$

where 2,400 is the incremental cost in GBP, 0.06 is the apparent utility gain including the DIF shift and 2 is the duration in years. A treatment that sits at GBP 30,000 per QALY appears to cost GBP 20,000, which could change a decision at an illustrative threshold of GBP 25,000. In a randomised trial with the same language mix in both arms most of this shift would cancel in the incremental QALYs, which is why DIF matters most in unanchored and cross-country comparisons.

Where cross-cultural validity sits among neighbouring properties

Cross-cultural validity borrows its methods from other properties but answers its own question. Structural validity of a measurement instrument asks whether scores reflect the dimensions of the construct within one population, and MGCFA extends that test across groups. Content validity asks whether items are relevant, comprehensive and comprehensible, which in a new culture is examined during translation and cognitive interviewing.

Language bias is unrelated despite the name, since it concerns which studies a systematic review finds, not how a questionnaire performs. Cross-cultural validity is also distinct from the comparability of utilities across countries, which depends on the value set.

Misreadings of cross-cultural validity evidence

A completed translation protocol is often cited as if it settled cross-cultural validity, but COSMIN classes translation as development, and only response data can show invariance. Similar mean scores across countries do not show invariance either, because true differences in health and item bias can offset each other.

Invariance is specific to the groups and versions tested. The Singapore study supported equivalence for some pairs of versions and not for others, so evidence for one pair does not transfer to a third language. A non-significant test in a small sample is weak evidence, which is why COSMIN rates sample size, and a statistically significant DIF term in a very large sample may still fall below the 0.02 threshold for importance. Invariance across languages says nothing about whether a value set elicited in one country reflects preferences in another, and a pooled multinational utility remains a choice about whose preferences count.

Sources

• Choi SW, Gibbons LE, Crane PK. lordif: an R package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations. Journal of Statistical Software. 2011;39(8). https://doi.org/10.18637/jss.v039.i08
• Gerlinger C, Bamber L, Leverkus F, Schwenke C, Haberland C, Schmidt G, Endrikat J. Comparing the EQ-5D-5L utility index based on value sets of different countries: impact on the interpretation of clinical study results. BMC Research Notes. 2019;12(1):18. https://doi.org/10.1186/s13104-019-4067-9
• Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology. 2010;63(7):737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006
• Mokkink LB, de Vet HCW, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, Terwee CB. COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. Quality of Life Research. 2018;27(5):1171-1179. https://doi.org/10.1007/s11136-017-1765-4
• Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs): user manual. Version 1.0. Amsterdam: COSMIN; February 2018. https://www.cosmin.nl/wp-content/uploads/COSMIN-syst-review-for-PROMs-manual_version-1_feb-2018.pdf
• Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, Terwee CB. COSMIN guideline for systematic reviews of patient-reported outcome measures. Quality of Life Research. 2018;27(5):1147-1157. https://doi.org/10.1007/s11136-018-1798-3
• Wang Y, Tan NC, Tay EG, Thumboo J, Luo N. Cross-cultural measurement equivalence of the 5-level EQ-5D (EQ-5D-5L) in patients with type 2 diabetes mellitus in Singapore. Health and Quality of Life Outcomes. 2015;13:103. https://doi.org/10.1186/s12955-015-0297-2
• Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, Erikson P. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures. Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value in Health. 2005;8(2):94-104. https://www.ispor.org/heor-resources/good-practices/article/principles-of-good-practice-for-the-translation-and-cultural-adaptation-process-for-patient-reported-outcomes-(pro)-measures