Interviewer Bias

Interviewer Bias in Health State Valuation and Willingness-to-Pay Studies

Interviewer bias arises when the person administering a survey or valuation task changes the answers through the way questions are put, explained, probed or recorded. In health economics it matters most where preferences are elicited in interviews, above all in the composite time trade-off tasks behind EQ-5D-5L value sets and in contingent valuation surveys of willingness to pay. Because a value set scores every EQ-5D-based QALY in an appraisal, an interviewer effect in one valuation study can travel into many cost-utility results. The EuroQol Group now treats interviewer performance as something to monitor and correct during fieldwork, and NICE's handling of the first English EQ-5D-5L value set shows what is at stake when data quality is in doubt.

How an interviewer shifts answers

Interviewer bias is a form of information bias, because the error enters during measurement and not through who is sampled. Survey methodologists observe that estimates tend to vary between interviewers even when training is standardised. Elliott and colleagues attribute this to verbal or nonverbal signals, usually unintended, and to interviewer characteristics that reveal preferences and expectations.

Two components are worth separating. A shift shared by all interviewers, such as rushing every respondent through a difficult task, biases the mean. Differences between interviewers create correlation among the answers each interviewer collects, which survey statisticians call interviewer variance and which widens the uncertainty around every estimate. Both are forms of measurement error, but only the precision lost to the second shrinks when the work is spread across more interviewers.

Preference elicitation is especially exposed. In the EQ-VT protocol the interviewer explains composite time trade-off with a wheelchair example, and the EuroQol Group's overview notes that most instructions are not prompted on screen, the task is hard for some respondents and it must be delivered in a standardised and neutral way. Respondent engagement depends on the engagement the interviewer shows.

Interviewer effects in EQ-5D-5L valuation studies

Some of the first wave of EQ-5D-5L valuation studies, in Canada, China, England, the Netherlands and Spain, produced composite time trade-off data with spikes at values such as minus 1, 0 and 1, lower than expected values for mild states, few worse-than-dead values and many inconsistencies. These problems clustered in particular interviewers, which led the EuroQol Group to hypothesise that they were interviewer effects.

In the English study, Devlin and colleagues used 48 interviewers and reported different proportions of worse-than-dead values depending on which interviewer a participant saw. NICE's position statement (ECD16, published 2017, last updated 2019) did not recommend that value set, citing concerns from independent quality assurance and four expert reports about the quality and reliability of the data and the modelling methods. For several years NICE analyses therefore mapped EQ-5D-5L responses onto the 3L value set.

NICE's interim methods statement (PMG51, 27 August 2026) replaced ECD16 and makes a new UK EQ-5D-5L value set the reference case for adults. That study, reported by Rowen and colleagues, conducted 1,200 interviews, 91.8% by videoconference, with independent assessment of data quality during fieldwork.

Quality control in the EQ-VT protocol

From EQ-VT version 1.1, every interview is screened against four criteria. An interview is flagged if the interviewer did not explain the worse-than-dead task during the wheelchair example, spent less than three minutes on that example, or completed the ten time trade-off tasks in under five minutes in total, or if the respondent valued the worst state 55555 at least 0.5 above the lowest-valued state.

If 40% or more of an interviewer's first ten interviews are flagged, those interviews are removed and the interviewer is retrained. A second failure after ten further interviews removes the interviewer from the team. The threshold is set at 40% because some flagged interviews hold genuine responses from respondents who decide quickly.

Ramos-Goñi and colleagues compared studies before and after this process. Protocol violations fell from 87% to 5% in Spain and from 20% to 8% in the Netherlands, interviewers performed more homogeneously, inconsistent respondents fell by about 23% in both countries, and values were less clustered. In Egypt, three interviewers were excluded for interviewer effects (113 interviews), and in the 1,180 interviews analysed, by nine interviewers, the variance attributed to interviewers stayed small. A conversation analysis of Italian EQ-VT recordings by Varriale and colleagues found negative patterns that included forcing an answer, speaking too fast and moving on without checking understanding. The EuroQol overview suggests that discrete choice experiment tasks may be less exposed to interviewer effects, or that problems are simply easier to see in time trade-off data.

Removing the interviewer and the effect of mode

Self-completion removes the interviewer but also removes the help that keeps respondents engaged. Norman and colleagues randomised 135 adults to a face-to-face or a remote electronic time trade-off. The electronic tool produced larger standard deviations and more responses at minus 1, 0 and 1, and the resulting scores differed by more than 0.1 for 100 of the 243 EQ-5D health states.

Hill and colleagues found composite time trade-off data from unassisted online surveys less reliable than data from face-to-face or remote-assisted interviews, with higher utilities for moderate and severe states, while discrete choice data were equally reliable across modes. Peasgood and colleagues found mean values equivalent for mild states and not significantly different for serious states between online and face-to-face interviews, and Estévez-Carrillo and colleagues found no evidence that videoconferencing reduced data quality.

Interviewers in willingness-to-pay surveys

Contingent valuation raises a different risk, social desirability. The NOAA panel on contingent valuation, chaired by Arrow and Solow, warned that a respondent unwilling to pay might feel pressure to give the right or good answer to an in-person or telephone interviewer. It recommended that major studies include experiments that assess interviewer effects, such as a sealed ballot box, while still preferring face-to-face interviews to mail surveys.

Maguire, comparing the same survey by telephone, mail and in person, found some evidence of social desirability bias in the telephone version. Both sources come from environmental economics, and the reasoning carries over to health care, where paying for treatment or prevention may look like the approved answer.

Measuring interviewer variance

When respondents are randomly assigned to interviewers, a design called interpenetration, interviewer variance can be estimated with random-effects or hierarchical models that nest responses within interviewers. Its effect on the precision of a mean follows the cluster-sampling formula cited by Elliott and colleagues.

$$\text{deff} = 1 + \rho_{int}(m - 1)$$

where $\text{deff}$ is the factor by which interviewer clustering multiplies the variance of an estimated mean, $\rho_{int}$ is the correlation between answers collected by the same interviewer, and $m$ is the average number of interviews per interviewer.

Small correlations matter because interviewers conduct many interviews. Elliott and colleagues, citing telephone surveys analysed by Groves and Magilavy, note that correlations of 0.002 to 0.02 imply design effects of 1.04 to 1.38 at 20 interviews per interviewer and 1.10 to 1.98 at 50. Without interpenetration, clustering may reflect which respondents each interviewer met, so between-interviewer differences are not proof of interviewer effects.

Worked example with a valuation study and an ICER (illustrative)

All numbers are illustrative. A valuation study completes 1,000 interviews, and the correlation between answers collected by the same interviewer is 0.02. With 20 interviewers, each conducts 50 interviews.

$$\text{deff} = 1 + 0.02 \times (50 - 1) = 1.98$$

where 0.02 is the within-interviewer correlation and 50 is the number of interviews per interviewer. The effective sample size is 1,000 ÷ 1.98, about 505 interviews. With 50 interviewers conducting 20 each, the design effect is 1 + 0.02 × 19 = 1.38 and the effective sample is about 725. Spreading the work across more interviewers recovers precision, but neither design removes a shift shared by all interviewers.

Now suppose a treatment improves a health state for 3 years at an incremental cost of GBP 6,000, ignoring discounting. If the true utility gain is 0.10, the QALY gain is 0.30 and the ICER is GBP 20,000 per QALY. If interviewers who rush the task flatten values so that the gain appears as 0.08, the QALY gain becomes 0.24.

$$\text{ICER} = \frac{6000}{0.08 \times 3} = 25000$$

where 6,000 is the incremental cost in GBP, 0.08 is the biased utility gain and 3 is the duration in years. The ICER rises by a quarter. At an illustrative threshold of GBP 22,000 per QALY the biased values would turn an acceptable treatment into a rejected one, and a probabilistic sensitivity analysis would not reveal the error, because it samples around the values as estimated.

Boundaries with related biases

Detection bias concerns outcome assessment that differs between trial arms because assessors know the treatment. The Cochrane Handbook's RoB 2 guidance treats the participant as the outcome assessor for a patient-reported outcome, even when a blinded interviewer asks the questions and completes the questionnaire. Interviewing that differed between arms would instead fall under the same domain's question on whether measurement or ascertainment of the outcome could have differed between groups.

Anchoring bias in elicitation concerns expert judgements pulled towards starting values, and hypothetical bias concerns the gap between stated and real payments. Interviewer bias can occur with either. Recall bias comes from the respondent's memory, while interviewer bias comes from the person asking.

Common misreadings of interviewer bias

A study with low interviewer variance can still be biased, because a shared shift does not show up as variance. A high proportion of flagged interviews does not prove misconduct either, since the EQ-VT threshold allows for quick but genuine respondents and the Egyptian study notes that respondent characteristics can mimic interviewer effects.

Unassisted online surveys and videoconference interviews are different modes. The first removes the interviewer and, in the studies above, lowered the reliability of time trade-off data, while the second keeps the interviewer and the need for quality control.

Sources

• Al Shabasy S, Roudijk B, Abbassi M, Finch A, Stolk E, Farid S. The Egyptian EQ-5D-5L extensive pilot study: lessons learned. PharmacoEconomics. 2023;41(3):329-338. https://doi.org/10.1007/s40273-022-01208-9
• Arrow K, Solow R, Portney PR, Leamer EE, Radner R, Schuman H. Report of the NOAA Panel on Contingent Valuation. Federal Register. 1993;58(10):4601-4614.
• Devlin NJ, Shah KK, Feng Y, Mulhern B, van Hout B. Valuing health-related quality of life: an EQ-5D-5L value set for England. Health Economics. 2018;27(1):7-22. https://doi.org/10.1002/hec.3564
• Elliott MR, West BT, Zhang X, Coffey S. The anchoring method: estimation of interviewer effects in the absence of interpenetrated sample assignment. Survey Methodology. 2022;48(1):25-48.
• Estévez-Carrillo A, Dewilde S, Oppe M, Ramos-Goñi JM. Exploring the comparability of face-to-face versus video conference-based composite time trade-off interviews. The Patient. 2022;15(5):521-535. https://doi.org/10.1007/s40271-022-00573-z
• Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial (last updated October 2019). In: Higgins JPT, Thomas J, Chandler J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024.
• Hill SR, Gibson A, Oluboyede Y, Longworth L, Bennett B, Shaw JW. A methodological study to compare alternative modes of administration with value EQ-5D using preference-elicitation techniques. Value in Health. 2024;27(6):784-793. https://doi.org/10.1016/j.jval.2024.02.020
• Maguire KB. Does mode matter? A comparison of telephone, mail, and in-person treatments in contingent valuation surveys. Journal of Environmental Management. 2009;90(11):3528-3533. https://doi.org/10.1016/j.jenvman.2009.06.005
• National Institute for Health and Care Excellence. Use of the EQ-5D-5L value set for England: NICE position statement (ECD16). Published 2 November 2017, last updated 31 October 2019; superseded.
• National Institute for Health and Care Excellence. Interim methods statement: implementing the EQ-5D-5L value set (PMG51). Published 27 August 2026. https://www.nice.org.uk/process/pmg51
• Norman R, King MT, Clarke D, Viney R, Cronin P, Street D. Does mode of administration matter? Comparison of online and face-to-face administration of a time trade-off task. Quality of Life Research. 2010;19(4):499-508. https://doi.org/10.1007/s11136-010-9609-5
• Peasgood T, Bourke M, Devlin N, Rowen D, Yang Y, Dalziel K. Randomised comparison of online interviews versus face-to-face interviews to value health states. Social Science and Medicine. 2023;323:115818. https://doi.org/10.1016/j.socscimed.2023.115818
• Ramos-Goñi JM, Oppe M, Slaap B, Busschbach JJ, Stolk E. Quality control process for EQ-5D-5L valuation studies. Value in Health. 2017;20(3):466-473. https://doi.org/10.1016/j.jval.2016.10.012
• Rowen D, Mukuria C, Bray N, et al. A United Kingdom value set for the EQ-5D-5L. Value in Health. 2026;29(5):858-869. https://doi.org/10.1016/j.jval.2026.03.008
• Stolk E, Ramos-Goñi JM, Ludwig K, Oppe M, Norman R. The development and strengthening of methods for valuing EQ-5D-5L, an overview. In: Devlin N, Roudijk B, Ludwig K, editors. Value Sets for EQ-5D-5L: A Compendium, Comparative Review and User Guide. Cham: Springer; 2022. Chapter 2. https://doi.org/10.1007/978-3-030-89289-0_2
• Varriale C, Andrulli G, Meregaglia M, Rencz F, Finch AP. Behind the scenes: a qualitative investigation of interviewers' performance in EQ-5D valuation studies. PharmacoEconomics Open. 2024;8(3):389-401. https://doi.org/10.1007/s41669-024-00486-7