Quality Score

Quality Score in Study Appraisal and Health Economic Reviews

A quality score collapses the answers to an appraisal checklist into one number, such as a total on the three-item Jadad scale or a mark out of 100 on the Quality of Health Economic Studies (QHES) instrument, and uses it to rank, exclude or weight studies. Early appraisal tools for systematic reviews of trials worked this way, but the evidence since Jüni and colleagues in 1999 shows that the total depends on which scale is used and does not track bias. Cochrane now judges risk of bias domain by domain, and the CHEERS 2022 authors strongly discourage scoring their checklist, while the Philips checklist for models is not meant to settle a model's quality on its own. In health technology assessment the question that matters is whether a flaw could change the cost-effectiveness conclusion, and a total cannot answer it.

How checklist items become a single number

Many such scales were built for randomised controlled trials. The Jadad scale, developed on reports of pain trials, kept three items, the third of which concerns the explanation of withdrawals.

Most scales give each item a point, or a weighted number of points, and add them.

$$S_j = \sum_{i=1}^{K} w_i x_{ij}$$

where $S_j$ is the quality score of study $j$, $K$ is the number of checklist items, $w_i$ is the weight given to item $i$, and $x_{ij}$ equals 1 if study $j$ meets item $i$ and 0 otherwise. An unweighted count sets every $w_i$ to 1.

One weighted score built for health economics is the QHES, named in a companion paper by Ofman and colleagues. Chiou and colleagues chose 16 criteria and estimated their weights from a conjoint analysis survey of 120 health economists, giving a 100-point scale.

Evidence that summary scores mislead

Jüni and colleagues applied 25 scales to the same 17 trials of low-molecular-weight heparin against standard heparin for preventing postoperative thrombosis. Pooled relative risks among the trials each scale called high quality ranged from 0.63 to 0.90. For six scales the high-quality trials showed no significant benefit while the low-quality trials did, for seven the pattern was reversed, and for 12 the strata were similar. Summary scores were not significantly associated with treatment effects, but open rather than blinded outcome assessment exaggerated the effect by 35% on average (95% CI 1% to 57%), so the authors advised assessing each methodological aspect individually.

Herbison and colleagues applied 43 quality scores to the trials in 65 meta-analyses from 18 Cochrane reviews and concluded that none appeared to measure quality validly, since none did better at bringing the pooled result closer to a reference standard set by a large trial. Clark and colleagues found poor agreement between raters on the Jadad scale, with a kappa statistic of 0.37 to 0.39, rising to 0.53 to 0.59 once the withdrawals item was removed.

Greenland and O'Rourke explain why. Quality, meaning whatever produces more valid results, has many dimensions that may be non-additive, nonlinear and specific to the topic. Even if it could be captured in one dimension, using a score to change study weights in a meta-analysis would bias the estimate, justified only if the bias were outweighed by a gain in precision. The Cochrane Handbook adds that the old scales lacked a theoretical basis and mixed risk of bias with imprecision, relevance, applicability, ethics and completeness of reporting.

From scales to domain-based risk of bias judgements

The first Cochrane risk of bias tool covered seven domains, including random sequence generation, allocation concealment and blinding of outcome assessment, and had no scoring system. Its successor, RoB 2, has a fixed set of domains covering the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome and selection of the reported result. Signalling questions in each domain feed an algorithm that proposes low risk, some concerns or high risk.

The overall judgement follows the worst domain rather than the average. A result is at low risk only if every domain is at low risk, and at high risk if any domain is at high risk. A trial with blinding and sound allocation concealment cannot buy back a high-risk judgement for missing outcome data, which an additive score allows.

The Handbook states that methods for weighting studies by risk of bias are not developed well enough to recommend. It suggests restricting the primary analysis to studies at low risk with a sensitivity analysis adding the rest, stratifying results by risk of bias, or carrying the judgements into the certainty of evidence through GRADE. The NICE technology appraisal manual (PMG36) likewise asks for risk of bias in each non-randomised study to be assessed with a validated tool such as ROBINS-I.

Quality checklists for economic evaluations and decision models

Reviews of economic evaluations commonly appraise studies with checklists. Watts and Li found 346 systematic reviews from 2010 to early 2018 that used 18 different checklists on economic evaluations. The most common was the 1996 Drummond and Jefferson guidance for authors and peer reviewers of economic submissions to the BMJ, and many checklists were used in ways their developers had not intended.

The Consensus on Health Economic Criteria (CHEC) list came from a three-round Delphi panel of 23 international experts as a minimum standard for assessing economic evaluations in systematic reviews, with each item answered yes or no. Philips and colleagues consolidated good practice guidance on decision-analytic models into a checklist. Their expert advisory group judged that it should not be used on its own to determine a model's quality or replace critical appraisal, and the authors noted that no generic checklist can settle whether a model's structure is appropriate, a matter for model validation.

CHEERS 2022 is a 28-item reporting checklist. Its authors state that it is not intended as a scoring tool or a tool to judge the appropriateness of methods, that it has no validated scoring system, and that scoring with it could mislead and is strongly discouraged.

The NICE guidelines manual (PMG20) shows a judgement-based alternative. Its economic evaluation checklist asks seven applicability questions ending in an overall judgement of directly, partially or not applicable, then 11 questions on study limitations ending in minor, potentially serious or very serious limitations. Each rating turns on whether the failed criteria could, or are highly likely to, change the conclusions about cost effectiveness, and the manual points to the Philips checklist for a closer look at modelling studies.

Worked example with two cost-utility analyses and three scoring schemes

The studies, items and weights in this example are illustrative. A reviewer appraises two cost-utility analyses of the same drug with a 10-item checklist. Seven items concern methods, one of which asks whether the comparator reflects current practice, and three concern reporting (price year, funding statement and structured abstract).

Study A meets every item except the comparator, because it compares the drug with no treatment when an active drug is standard care. Study B meets every methods item but fails all three reporting items. The reviewer computes $S_j$ under three schemes.

Under set 1, study A loses its 15 comparator points, leaving 100 minus 15, or 85, while study B loses 20 reporting points, leaving 80. Under set 2, study A loses 10 points to score 90 and study B loses 6 points to score 94. If the review treats 85 or more as high quality, the unweighted count (90% against 70%) and set 1 keep only study A, while set 2 keeps both.

Two of the three schemes rank study A first, yet its ICER answers a different question from the one the decision maker faces. On the NICE checklist it would be at best partially applicable, because the failed criterion could change the conclusion about cost effectiveness, while study B would be directly applicable with minor limitations. As in the Jüni study, the ranking moves with the weights, and no total reveals the flaw that matters.

Assumptions behind a total score and safer uses

Adding items assumes that each contributes a fixed amount to validity whatever else is true of the study and that strength in one area offsets weakness in another, although Greenland and O'Rourke suggest that quality may be non-additive and nonlinear. A score taken from published reports also measures reporting as much as conduct, and the Cochrane Handbook notes that information needed to judge risk of bias is often missing from articles.

For reporting, the CHEERS 2022 authors advise judging completeness item by item rather than counting the items met. To explain differences between results, Greenland and O'Rourke prefer modelling individual quality items, for example in a meta-regression, to weighting by a total. In a decision model, the safer course is to take a parameter from studies judged at low risk of bias or most applicable, and to show in a scenario or sensitivity analysis how far the result moves when other studies are added.

Boundaries with risk of bias, certainty of evidence and reporting quality

Risk of bias concerns whether a study's design, conduct or analysis could have distorted its result, judged per domain and per outcome. Quality assessment is the broader process, and critical appraisal adds relevance and applicability to the decision.

Certainty of evidence rates a whole body of evidence for an outcome, with risk of bias as one input. Reporting quality, the subject of CHEERS 2022, concerns what a paper discloses, and a fully reported study can still use a poor comparator or an unsuitable model structure. The same phrase is also used for provider performance measures such as hospital star ratings, a separate concept not covered here.

Common misreadings of quality scores

A high score does not mean a low risk of bias. Jüni and colleagues found no significant association between summary scores and treatment effects, while a single item, blinded outcome assessment, did predict exaggeration.

The share of CHEERS items met is not a measure of methodological quality, since a well reported evaluation can still be unusable for a given decision.

Weighting a meta-analysis by quality scores does not remove bias. It shifts the pooled estimate towards the studies the chosen scale favours, which can itself introduce bias.

Sources

• Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies (last updated August 2022). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07
• Chiou CF, Hay JW, Wallace JF, et al. Development and validation of a grading system for the quality of cost-effectiveness studies. Medical Care. 2003;41(1):32-44. https://doi.org/10.1097/00005650-200301000-00007
• Clark HD, Wells GA, Huët C, et al. Assessing the quality of randomized trials: reliability of the Jadad scale. Controlled Clinical Trials. 1999;20(5):448-452. https://doi.org/10.1016/s0197-2456(99)00026-4
• Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ. 1996;313(7052):275-283. https://doi.org/10.1136/bmj.313.7052.275
• Evers S, Goossens M, de Vet H, van Tulder M, Ament A. Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria. International Journal of Technology Assessment in Health Care. 2005;21(2):240-245.
• Greenland S, O'Rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001;2(4):463-471. https://doi.org/10.1093/biostatistics/2.4.463
• Herbison P, Hay-Smith J, Gillespie WJ. Adjustment of meta-analyses on the basis of quality scores should be abandoned. Journal of Clinical Epidemiology. 2006;59(12):1249-1256. https://doi.org/10.1016/j.jclinepi.2006.03.008
• Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial (last updated October 2019). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
• Husereau D, Drummond M, Augustovski F, et al. Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement: updated reporting guidance for health economic evaluations. BMC Health Services Research. 2022;22:114. https://doi.org/10.1186/s12913-021-07460-7
• Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Controlled Clinical Trials. 1996;17(1):1-12. https://doi.org/10.1016/0197-2456(95)00134-4
• Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11):1054-1060. https://doi.org/10.1001/jama.282.11.1054
• National Institute for Health and Care Excellence. Developing NICE guidelines: the manual (PMG20), chapter 7 and appendix H. London: NICE; 2014, last updated 23 October 2025. https://www.nice.org.uk/process/pmg20
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Ofman JJ, Sullivan SD, Neumann PJ, et al. Examining the value and quality of health economic analyses: implications of utilizing the QHES. Journal of Managed Care Pharmacy. 2003;9(1):53-61. https://doi.org/10.18553/jmcp.2003.9.1.53
• Philips Z, Ginnelly L, Sculpher M, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technology Assessment. 2004;8(36). https://doi.org/10.3310/hta8360
• Watts RD, Li IW. Use of checklists in reviews of health economic evaluations, 2010 to 2018. Value in Health. 2019;22(3):377-382. https://doi.org/10.1016/j.jval.2018.10.006