Core Outcome Set Development

Core Outcome Set Development and Its Consensus Methods

A core outcome set is only as good as the study that agreed it, and that study is a piece of applied research with its own design choices and biases. The development study gathers candidate outcomes from published trials and from patients, asks several stakeholder groups to rate them over repeated survey rounds, and settles the final list at a meeting. Choices such as who is invited and how consensus is defined can change which outcomes every later trial in the condition is expected to report. For health economists the process matters because the outcomes it selects shape what evidence will exist for future syntheses and cost-utility models, and because the same consensus machinery has been used to agree core sets of resource use items. The COMET Handbook, version 1.0, published by Williamson and colleagues in 2017, recommends a four-step development process, and COMET has since added standards for the design, protocol and reporting of these studies.

Stages of a development study

The COMET Handbook recommends a four-step process for developing a core outcome set. A typical study combines a review of the outcomes already used, a structured survey of stakeholders and a final consensus meeting, with a separate study later on how to measure each outcome.

The scope comes first. It states the condition, population, interventions and setting the set is meant for, since a set for trials of surgery in adults may not suit trials of drug treatment in children. A systematic review of trials in the area then lists the outcomes that have been measured, sometimes supplemented by interviews with patients so that outcomes missing from the trial literature can be added.

The long list goes into a Delphi survey, in which participants rate each outcome anonymously, see feedback on how others rated it and rate again. Outcomes that reach a prespecified level of agreement are carried forward, and a consensus meeting with representatives of each group discusses the outcomes still in doubt and ratifies the final set. The whole study is a specialised form of consensus building, in which the judgement agreed concerns which outcomes matter most to the people affected by the condition.

The developers of COMET's own standards followed a similar route. The COS-STAD standards were produced through an open survey that generated candidate items, a two-round Delphi survey with nearly 250 participants from stakeholder groups, and a consensus discussion with a management group that represented several of those groups.

Who takes part and why the mix matters

The COMET Handbook lists health service users, health care practitioners, trialists, regulators, industry representatives, policy-makers, researchers and the public as possible stakeholders, and says the groups chosen and their numbers depend on the scope, existing knowledge, the methods and feasibility. As an example of scale, the Handbook cites a Delphi survey for a core outcome set on preventing preterm birth that involved 32 parents and 163 health professionals.

Sinha, Smyth and Williamson reviewed Delphi studies used to choose outcomes for trials and, in 2011, recommended involving both patients and clinicians, avoiding the imposition of researchers' views on participants and minimising attrition between rounds.

How patients are recruited affects who stays. Barrington, Young and Williamson reviewed core outcome set studies published between 2015 and 2019 that included patients in Delphi scoring and found wide variation in scoring and feedback systems, recruitment, timing between rounds, reminders and incentives. They reported minimal description of participant characteristics, little translation of surveys into local languages, and an indication that recruitment through treatment centres achieved higher round-two response rates than recruitment through patient organisations.

Health economists fall within the researchers in the Handbook's list, but a panel built for clinical effectiveness may contain none of them. Whether outcomes needed for economic evaluation, such as resource use or preference-based quality of life, survive the rating depends on how patients and clinicians judge their importance.

Scoring outcomes and defining consensus

Development studies commonly ask participants to rate each outcome on a nine-point scale, and the COMET Handbook describes the usual banding, in which 1 to 3 means an outcome is of limited importance, 4 to 6 important but not critical, and 7 to 9 critical. The Handbook says that feedback to participants in later rounds is the mechanism through which a reduced number of outcomes is agreed against prespecified criteria.

The definition of consensus is a design choice and has to be written before the results are seen. COS-STAD used, for its own Delphi survey, a rule that at least 70 per cent of voting participants in each stakeholder group gave a score of 7 to 9. The COMET and COSMIN Delphi study on selecting instruments, reported by Prinsen and colleagues, defined consensus as 70 per cent or more agreeing and fewer than 15 per cent disagreeing.

Two further choices matter alongside the threshold. Requiring the threshold within every group gives each group a veto, while pooling all respondents gives the largest group the most weight. Adding a ceiling on low scores excludes outcomes that a minority rates as unimportant, even when most rate them critical.

Worked example with two stakeholder groups (illustrative)

All numbers in this example are illustrative. A Delphi survey for a core outcome set in a chronic condition has 100 patients and 50 health professionals in its final round. The protocol must say whether consensus means at least 70 per cent scoring 7 to 9 in each group or in the pooled panel.

Step 1. Return to usual activities, group rule. Seventy-seven patients score the outcome 7 to 9, which is 77 per cent, and 28 professionals do so, which is 56 per cent. Under a rule requiring 70 per cent in each group, the outcome fails.

Step 2. Return to usual activities, pooled rule. Across the whole panel the share scoring 7 to 9 is computed as follows.

$$p_{pooled} = \frac{77 + 28}{100 + 50} = \frac{105}{150} = 0.70$$

where $p_{pooled}$ is the proportion of all respondents scoring the outcome 7 to 9, 77 and 28 are the patients and professionals giving such scores, and 100 and 50 are the group sizes. The outcome just meets a pooled 70 per cent rule.

Step 3. Adding a ceiling on low scores. Suppose 5 patients and 10 professionals score the outcome 1 to 3. The pooled share is 15 ÷ 150 = 0.1, or 10 per cent, which passes a fewer-than-15-per-cent ceiling, but within professionals the share is 10 ÷ 50 = 0.2, or 20 per cent, which fails if the ceiling applies to each group.

Step 4. A resource use outcome. Forty patients and 35 professionals score health care resource use 7 to 9, which is 40 per cent and 70 per cent. The pooled share is 75 ÷ 150 = 0.5, so the outcome fails under both rules.

Interpretation. The same ratings put return to usual activities inside or outside the set depending on a rule chosen before the survey, and COS-STAD lists the consensus process as one of its three domains. The resource use outcome fails either way, so an economic evaluation alongside future trials would still need to collect resource use as an additional outcome.

Standards for design, protocol and reporting

COMET has published three statements that apply to development studies, each agreed through a Delphi survey of stakeholder groups. COS-STAD, reported by Kirkham and colleagues in 2017, sets 11 minimum standards for the design of a core outcome set study, grouped under scope, stakeholders and the consensus process. COS-STAP, published in 2019, is a 13-item checklist for the study protocol and applies to studies deciding which outcomes belong in a set, without covering how those outcomes should be defined or measured.

COS-STAR, published in 2016, is an 18-item checklist for reporting a completed study, developed through a two-round Delphi survey of nearly 200 participants and a consensus meeting. NICE's guideline manual (PMG20) asks reviewers to use COS-STAD and COS-STAR to judge whether an identified core outcome set is suitable, which gives the standards a practical role in NICE guideline reviews.

Biases in the Delphi process

The Delphi survey reduces the dominance of loud voices in a meeting, but it introduces its own sources of bias. Methodological trials nested inside core outcome set studies have tested some of them.

Brookes and colleagues ran three randomised trials inside Delphi surveys for two core outcome sets in surgery and a core information set, reported in 2016. In round two, participants saw feedback from their own stakeholder group only or from all groups shown separately. The type of feedback did not change how often or how much participants re-rated items, but it did change which items were retained, with 3 to 22 per cent of items discordant between arms. Agreement between patients and professionals on retained items was 74 to 94 per cent with feedback from all groups and 65 to 82 per cent with peer-only feedback.

The order of questions matters too. In a trial reported by Brookes and colleagues in 2018, within a Delphi survey for oesophageal cancer surgery, patients gave more patient-reported outcomes a score in the top band of 7 to 9 when that section came last than when it came first, a difference of 23.7 per cent. Fewer professionals responded when clinical items came first.

Attrition between rounds can shift a panel towards the groups most willing to continue, and the Handbook advises developers to consider how many participants to invite into the Delphi survey with drop-out in mind. Who designs the long list also matters, since an outcome that is never offered cannot be rated, and wording chosen by researchers can carry their own views into the ratings.

Consensus methods applied to resource use

The same development methods have been used for economic data. Thorn and colleagues ran an electronic Delphi survey of health economists with experience of UK economic evaluations to agree core items for a standardised resource use measure. Forty-five participants completed round one, in which 26 of 60 items were dropped, 34 were kept and no new items were added; 42 participants (93.3 per cent) completed round two, and a final meeting selected 10 core items, with others suitable for optional bolt-on modules.

The asthma programme begun by Hounsome and colleagues followed the review stage of the same route, and Roukas and colleagues reported in 2020 that its next stage would use Delphi methods to address the lack of stakeholder involvement in identifying economic parameters. These projects show that the development process can serve the costing side of economic evaluation directly, though their panels and purposes differ from those of a clinical core outcome set.

Limits of agreement as evidence

Consensus records what a panel judged important at one time, and its authority rests on the process followed. No external test shows that the chosen outcomes are the right ones. Thresholds such as 70 per cent are conventions, and a set agreed under one rule might differ under another, as the worked example shows.

A development study that agrees what to measure leaves the choice of instrument open. The COMET and COSMIN guideline reported by Prinsen and colleagues treats instrument selection as a further consensus task, starting with conceptual considerations, a search for existing instruments and an assessment of their quality. Until that step is done, trials complying with the set may still report the same outcome in ways that cannot be pooled. The value of the work for evidence synthesis, and for the economic models that draw on it, therefore depends on both stages.

Sources

• Barrington H, Young B, Williamson PR. Patient participation in Delphi surveys to develop core outcome sets: systematic review. BMJ Open. 2021;11(9):e051066. https://doi.org/10.1136/bmjopen-2021-051066
• Brookes ST, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17:409. https://doi.org/10.1186/s13063-016-1479-x
• Brookes ST, et al. Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial. Trials. 2018. https://doi.org/10.1186/s13063-017-2405-6
• Hounsome N, Fitzsimmons D, Phillips C, Patel A. Developing core economic outcome sets for asthma studies: a protocol for a systematic review. BMJ Open. 2017;7(8):e017054. https://doi.org/10.1136/bmjopen-2017-017054
• Kirkham JJ, et al. Core Outcome Set-STAndards for Reporting: the COS-STAR Statement. PLoS Medicine. 2016;13(10):e1002148. https://doi.org/10.1371/journal.pmed.1002148
• Kirkham JJ, Davis K, Altman DG, et al. Core Outcome Set-STAndards for Development: the COS-STAD recommendations. PLoS Medicine. 2017;14(11):e1002447. https://doi.org/10.1371/journal.pmed.1002447
• Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Tunis S, Williamson PR. Core Outcome Set-STAndardised Protocol Items: the COS-STAP Statement. Trials. 2019;20(1):116. https://pubmed.ncbi.nlm.nih.gov/30744706/
• National Institute for Health and Care Excellence. Developing NICE guidelines: the manual (PMG20). Chapter 4, Developing review questions and planning the evidence review. Last updated 23 October 2025. https://www.nice.org.uk/process/pmg20/chapter/developing-review-questions-and-planning-the-evidence-review
• Prinsen CAC, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, Williamson PR, Terwee CB. How to select outcome measurement instruments for outcomes included in a "Core Outcome Set", a practical guideline. Trials. 2016;17(1):449. https://doi.org/10.1186/s13063-016-1555-2
• Roukas C, Quayyum Z, Patel A, Fitzsimmons D, Phillips C, Hounsome N. Developing core economic parameter sets for asthma studies: a realist review and an analytical framework. BMJ Open. 2020;10(10):e037889. https://doi.org/10.1136/bmjopen-2020-037889
• Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Medicine. 2011;8(1):e1000393. https://doi.org/10.1371/journal.pmed.1000393
• Thorn JC, et al. Core items for a standardized resource use measure: expert Delphi consensus survey. Value in Health. 2018;21(6):640-649. https://pmc.ncbi.nlm.nih.gov/articles/PMC6021557/
• Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280. https://doi.org/10.1186/s13063-017-1978-4