Quality of Evidence

Quality of Evidence in GRADE Ratings for Health Decisions

Quality of evidence is the rating a systematic review, guideline panel or HTA team gives to the trust it can place in an estimated treatment effect for one outcome. In the GRADE approach the rating starts from the study design, moves down for five kinds of problem and up for three kinds of strength, and ends at high, moderate, low or very low. It matters to health economists because every relative effect, baseline risk and resource-use estimate in a decision model rests on evidence that can be rated this way, and a precise-looking input can still be weak. GRADE now calls the same construct certainty of evidence, and how study judgements combine is covered under body of evidence.

Why grading moved beyond study design

The GRADE Working Group's 2004 appraisal of six grading systems describes the original Canadian Task Force approach as based on study design alone and judges it too simple, because judgements about trial quality, conflicting trials and convincing non-experimental results stayed implicit. It found important shortcomings in every system and argued that a systematic review should not sit in an evidence hierarchy, since a well-done review may contain only poor or inconsistent studies.

The first GRADE proposal, in The BMJ in 2004, defined the quality of evidence as the extent to which one can be confident that an estimate of effect is correct. It graded each outcome on study design, study quality, consistency and directness, with sparse data and reporting bias able to lower the grade and a strong association, a dose-response gradient or confounding that would shrink the effect able to raise it. In 2008 Guyatt and colleagues illustrated the stakes. Hormone replacement therapy was recommended for cardiovascular protection on inconsistent observational studies, evidence a formal rating would have called very low quality. Encainide and flecainide were licensed for suppressing asymptomatic arrhythmias, an indirect basis for a claim about sudden death, and a later randomised controlled trial showed that both drugs increased that risk.

Their paper separated quality of evidence from the strength of a recommendation, which also depends on the balance of benefits and harms, values and preferences and whether the intervention is a wise use of resources. In 2017 the GRADE Working Group stated that quality of evidence, confidence in the estimates and certainty of evidence all refer to the same concept.

Starting level and the eight factors

The GRADE Handbook starts a body of randomised trials without important limitations at high and a body of observational studies without special strengths or important limitations at low. Case series usually drop to very low because the source of control results is unclear. The rating is made separately for each outcome, so unblinded trials may be rated down on stroke, an outcome open to biased judgement, but not on all-cause mortality.

The Handbook treats the factors as additive yet states that GRADE is not a quantitative system, because each factor is a continuum. Where a body sits near the line on several factors, reviewers make one explicit judgement and explain borderline decisions in footnotes.

Five reasons to rate down

Each downgrading factor asks why the pooled estimate might mislead. All five are judged across the studies that contribute to one outcome.

Risk of bias across the studies that carry the estimate

For trials the Handbook lists failures of allocation concealment, blinding and accounting for patients and events, selective outcome reporting and stopping early for benefit, warning that trials stopped early with fewer than 500 events are likely to overestimate the effect substantially. For an observational study the main problems are inappropriate eligibility criteria or control groups, flawed measurement and failure to control confounding. The body is not judged by averaging studies. Reviewers weigh each study by its contribution to the estimate and rate down only when confident that a substantial risk of bias runs across most of the evidence. Study-level checklists come earlier, under critical appraisal.

Inconsistency that remains unexplained

Inconsistency means unexplained heterogeneity in relative effects. The Handbook's signals are widely varying point estimates, little overlap of confidence intervals, a low p-value in a heterogeneity test and a large I-squared.

$$I^2 = \frac{Q - df}{Q} \times 100\%$$

where $Q$ is Cochran's heterogeneity statistic, $df$ is its degrees of freedom (the number of studies minus one), and $I^2$ is set to zero when $Q$ is below $df$. The Handbook reads values below 40% as possibly low and 75% to 100% as possibly considerable, and warns that small studies can give a low value despite scattered estimates. When differences in population, intervention or outcome explain the variation, the review reports separate estimates instead of rating down, although most putative effects found by subgroup analysis prove spurious.

Indirectness of population, intervention, outcome or comparison

The Handbook names four sources of indirectness, namely differences in population, differences in intervention, surrogate outcomes and indirect comparisons. A surrogate endpoint generally costs one level, or two when it sits far from the patient-important outcome in the causal pathway. An indirect treatment comparison always loses at least one level because its similarity assumption is always in some doubt. Population differences call for restraint, since the 2004 GRADE paper notes that many interventions have much the same relative effects across most patient groups.

Imprecision judged against intervals and information size

GRADE treats the 95% confidence interval as the primary criterion. For guidelines, the rating goes down if clinical action would differ between the two limits. The optimal information size (OIS), the number of patients one adequately powered trial would need, is a second necessary criterion, and systematic reviews, which make no value judgements, rate down when the review holds fewer patients unless it has at least 2,000 and perhaps 4,000. When the OIS is met but the interval includes no effect, a risk ratio below 0.75 or above 1.25 is a rough guide to appreciable benefit or harm. A conventional calculation for two proportions gives the OIS per group.

$$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \left[ p_0(1-p_0) + p_1(1-p_1) \right]}{(p_0 - p_1)^2}$$

where $n$ is the number of patients per group, $p_0$ is the control-group risk, $p_1$ is the risk with the intervention under the chosen relative risk reduction, $\alpha$ is the two-sided significance level, the power is one minus $\beta$, and each $z$ is a standard normal quantile. This is statistical power reasoning applied to the whole review.

Publication bias

Publication bias comes from selective publication of whole studies, while selective reporting within a study counts under risk of bias. The Handbook advises suspicion when the evidence is a few small positive trials, and cites 74 antidepressant trials submitted to the FDA, in which 37 of the 38 the FDA viewed as positive were published against 14 of the 36 it viewed as negative. Funnel plot tests are suggested only with at least 10 studies, and because the absence of this bias is so hard to establish, GRADE rates down for it by at most one level.

Three reasons to rate up observational evidence

Rating up comes only after the downgrading factors and rarely when serious limitations remain. A risk ratio above 2 or below 0.5 from direct evidence with no plausible confounders may raise the rating one level, and one above 5 or below 0.2 with no serious problems of bias or precision two levels, but rarely when the interval overlaps substantially with smaller effects. A dose-response gradient may add one level, as with rising bleeding risk at higher INR values in patients taking warfarin. So may evidence that all plausible residual confounding would shrink the effect, as in the Handbook's example of higher death rates in for-profit hospitals whose not-for-profit comparators probably treated sicker patients.

Rating economic and resource-use evidence

GRADE applies the same criteria to resource use as to health outcomes. Brunetti and colleagues, in the GRADE guidelines series, ask panels to find evidence on resource items that differ between strategies, judge confidence in it using the same criteria used for health outcomes, and then value it in costs for the setting. The Handbook notes that quality can differ between resources, with drug use easier to estimate than professionals' time, and that a model built on low-quality estimates of harms, benefits and resources may be misleading.

The NICE guidelines manual (PMG20) rates certainty for each outcome with GRADE and also integrates a review of the quality of cost-effectiveness studies, whose inclusion depends on applicability, the amount of higher-quality evidence and certainty about cost effectiveness overall. In health technology assessment modelling, an input rated down for bias or indirectness has a confidence interval that reflects random error only, so its parameter uncertainty may need a wider distribution or a scenario analysis.

Worked example rating one outcome for a remote monitoring service

All numbers are illustrative. Five randomised trials of remote monitoring in heart failure report emergency admission at a year. Across the trials 94 of 600 monitored patients and 120 of 600 controls were admitted, a risk ratio of (94/600)/(120/600) = 0.78 with a 95% confidence interval of 0.61 to 1.00, and Cochran's Q is 5.0 on 4 degrees of freedom.

1. Start. Randomised trials begin at high.

2. Risk of bias. All trials concealed allocation and adjudicated admissions blind to group, and none stopped early. Not rated down.

3. Inconsistency. Here $I^2 = (5.0 - 4)/5.0 = 0.20$, or 20%, possibly low. Not rated down.

4. Indirectness. The trials enrolled recently admitted patients, while the service will enrol a regional register with a lower risk of 100 per 1,000. With no reason to expect a different relative effect, the evidence is not rated down and the lower risk enters the absolute effect.

5. Imprecision. A control risk of 0.20 and a 25% relative risk reduction give 0.15 with monitoring. With $z$ values of 1.960 and 0.842, the OIS is $(1.960 + 0.842)^2 \times (0.16 + 0.1275) / 0.05^2 = 902.9$ per group, about 1,806 patients and 903 × 0.20 + 903 × 0.15 = 316 events. The review has 1,200 patients and 214 events, and its interval reaches 1.00. Rated down one level.

6. Publication bias. Five trials are too few for funnel plot tests, but the registry search found every registered trial. Not rated down.

7. Result. Moderate quality for emergency admission.

At the register's risk, monitoring gives 0.78 × 100 = 78 admissions per 1,000, or 22 fewer, with limits of 0.61 × 100 = 61 (39 fewer) and none fewer. If the service costs GBP 45 per patient and an admission GBP 3,000, it recovers its cost when it prevents 45 / 3,000 = 0.015 admissions per patient, or 15 per 1,000. The point estimate gives a net saving of GBP 21 per patient (0.022 × 3,000 − 45 = 21). The interval spans a net cost of GBP 45 and a net saving of GBP 72 (0.039 × 3,000 − 45 = 72). A panel judging imprecision against this threshold would also rate down, but for the same single level. Because imprecision was the only concern, the interval already carries the main uncertainty, so a model can draw the risk ratio from it while the saving stays uncertain.

Where quality ratings are misread

Most errors treat the rating as a label on a study or a verdict on a technology. Four recur in HTA work.

• A paper rated instead of an outcome. Quality can differ between mortality, quality of life and resource use within the same trials.
• Downgrades counted mechanically. Borderline concerns need one explicit judgement, and one flaw should not be counted twice.
• Precision read as trustworthiness. A narrow interval from few events, especially from a trial stopped early, can overstate an effect.
• Clinical quality carried over to economic inputs. Resource use and costs need ratings of their own.

Sources

• Atkins D, Eccles M, Flottorp S, et al. Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches. BMC Health Services Research 2004;4:38. https://doi.org/10.1186/1472-6963-4-38
• Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328(7454):1490. https://doi.org/10.1136/bmj.328.7454.1490
• Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336(7650):924-926. https://doi.org/10.1136/bmj.39489.470347.AD
• Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence, imprecision. Journal of Clinical Epidemiology 2011;64(12):1283-1293. https://doi.org/10.1016/j.jclinepi.2011.01.012
• Brunetti M, Shemilt I, Pregno S, et al. GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence. Journal of Clinical Epidemiology 2013;66(2):140-150. https://doi.org/10.1016/j.jclinepi.2012.04.012
• Hultcrantz M, Rind D, Akl EA, et al. The GRADE Working Group clarifies the construct of certainty of evidence. Journal of Clinical Epidemiology 2017;87:4-13. https://doi.org/10.1016/j.jclinepi.2017.05.006
• Schünemann H, Brożek J, Guyatt G, Oxman A (editors). GRADE Handbook for grading quality of evidence and strength of recommendations using the GRADE approach. GRADE Working Group; updated October 2013. https://gdt.gradepro.org/app/handbook/handbook.html
• National Institute for Health and Care Excellence. Developing NICE guidelines: the manual (PMG20). Published 31 October 2014, last updated 23 October 2025. Chapters 6 and 7. https://www.nice.org.uk/process/pmg20