Systematic Review

Systematic Review in Health Technology Assessment and Economic Evaluation

A systematic review answers one stated question from every study that meets eligibility criteria fixed in advance, found by a documented search and appraised by explicit methods before any result is combined. Its defining feature is the method, so another team following the same protocol should arrive at the same studies and the same judgements. In health technology assessment it is the expected route to the clinical evidence behind an appraisal, and the NICE manual (PMG36) asks for evidence on outcomes to come from one. Reviews also supply the utilities, costs and earlier economic evaluations that a health economic model needs, although NICE accepts non-exhaustive searches for published economic evaluations and its Decision Support Unit favours sufficiency over exhaustiveness for model inputs. The method protects a decision against biased selection of studies, but it cannot repair weak or missing primary research.

What makes a review systematic

The Cochrane Handbook (version 6.5) describes a systematic review as an attempt to collate all the empirical evidence that fits pre-specified eligibility criteria, using explicit methods selected to minimise bias. Its starting argument is that ad hoc searching and selection invite bias, and that summaries of research need methods to avoid bias just as primary studies do. The review collects no new data, which places it within secondary research.

PRISMA 2020 adds a vocabulary that matters for counting. A record is a title or abstract indexed in a database, a report is any document supplying information about a study, and a study is the investigation itself, which may have several reports. The Cochrane Handbook treats studies as the primary units of analysis, so two journal articles from one trial count once.

Protocol and question before the evidence is seen

For questions about the effects of an intervention, the Cochrane Handbook notes that the PICO approach (population, intervention, comparison and outcome) is usually used. In an appraisal the frame is the decision problem set out in the scope, including every comparator the committee needs to see.

According to the Handbook, publishing a protocol written without knowledge of the available studies reduces the impact of the authors' biases and reduces duplication, and registration on the PROSPERO register promotes accountability. It recommends predefining a limited number of potential effect modifiers at the protocol stage, each with a rationale, and asks that later changes to the planned methods be declared transparently.

Searching for sensitivity and accepting low precision

The Cochrane Handbook asks that searches be as extensive as possible, to reduce the risk of publication bias, and states that a search of MEDLINE alone is not considered adequate. For Cochrane intervention reviews the minimum databases are the review group's specialised register where one exists, CENTRAL, MEDLINE and Embase where available. Restricting a search by language can bias the evidence base, as the language bias page shows in detail.

Search performance is described by two ratios defined in the Handbook.

$$\text{Sensitivity} = \frac{a}{a + b} \qquad \text{Precision} = \frac{a}{a + c}$$

where $a$ is the number of relevant reports the search retrieved, $b$ is the number of relevant reports it failed to retrieve, and $c$ is the number of irrelevant reports it retrieved. Sensitivity is also called recall.

Raising sensitivity lowers precision, and the Handbook advises maximising sensitivity while striving for reasonable precision. It estimates screening speed at 60 to 120 abstracts an hour, so a low-precision search costs less time than it first appears to. True sensitivity is unknown because $b$ cannot be counted, so the Handbook describes proxies such as relative recall, the share of eligible studies, found by a range of sensitive searches and indexed in a database, that the search of that database retrieved. For appraisals, PMG36 (section 3.3.25) expects attempts to identify evidence not in the public domain, such as unpublished trial data and trial data available only as abstracts.

Independent study selection and risk-of-bias appraisal

Selection runs from deduplicated records through title and abstract screening to full-text assessment. Cochrane's conduct standard MECIR C39 makes it mandatory for at least two people working independently to decide whether each study meets the eligibility criteria, with a process for resolving disagreements set in advance, while dual title and abstract screening is desirable but not mandatory. PMG36 (section 3.4.5) asks for more than one reviewer to assess all records retrieved and for a log of ineligible studies with reasons. The Handbook expects the selection to be recorded in enough detail to complete a flow diagram.

Included studies then pass through data extraction and risk-of-bias assessment. For randomised trials, Cochrane's RoB 2 tool uses five fixed domains, and the overall judgement for a result is the least favourable across them. These judgements feed one domain of the GRADE approach, which rates certainty of evidence across the body of evidence for each outcome.

Synthesis with or without meta-analysis

A systematic review need not pool anything. PMG36 (section 3.3.4) states that data from included studies can be synthesised but that synthesis is not required, while all evidence should be critically appraised. Pooling through meta-analysis is appropriate, according to section 3.4.8, if there are enough relevant and valid data using comparable outcome measures. When several comparators are not linked by any single trial, the review feeds a network meta-analysis.

PRISMA 2020 defines statistical synthesis broadly, covering meta-analysis of effect estimates and other methods such as vote counting based on the direction of effect, and it applies to reviews with no synthesis at all. The difference from a narrative review lies in the search, selection and appraisal, so a review summarised in prose can still be systematic.

What NICE expects from reviews in an appraisal

PMG36 (section 3.4.2) asks for evidence on outcomes to come from a systematic review, which it defines as locating, including, appraising and synthesising evidence systematically for a clearly formulated question. Section 3.4.4 asks for a review of the technology and relevant comparators according to a previously prepared protocol.

For economic evidence the manual is less strict about exhaustiveness. Section 3.3.26 asks for a review of published, relevant economic evaluations, searched by transparent and reproducible approaches until sufficient relevant evidence has been identified, and accepts that such reviews may not be exhaustive when further studies would only confirm what has been found. EQ-5D data taken from the literature must be identified by systematic and transparent methods (section 4.3.8), and cost sources from the literature are preferably identified through systematic review (section 4.4.10).

Reviews for model inputs other than treatment effects

NICE DSU Technical Support Document 13 reports the view that searches for model inputs should aim for sufficient information, so that further effort would add nothing to the analysis, an idea it likens to saturation more than to sensitivity. It asks that decisions about when to stop searching be transparent and justified, suggesting sensitivity analysis or value of information analysis to show where search effort matters most.

For health state utility values, NICE DSU TSD 9 recommends keeping the scope broad at first and observes that a sequence of searches may be needed in place of a single one. Where several sets of values meet the review criteria, the choice for the model has to be justified and the less relevant alternatives used in sensitivity analysis.

Worked example of a search for a three-drug comparison

All numbers in this example are illustrative. A review for an appraisal compares a new oral anticoagulant with two existing ones, and the database and trial register searches return 3,400 records.

1. Remove duplicates. Deleting 1,100 duplicates leaves 3,400 − 1,100 = 2,300 records to screen.
2. Screen titles and abstracts. Excluding 2,210 records sends 2,300 − 2,210 = 90 reports forward. Four cannot be retrieved, so 90 − 4 = 86 full texts are assessed.
3. Assess full texts. Excluding 63 reports with recorded reasons leaves 86 − 63 = 23 included reports.
4. Search other sources. The reference lists of the included reports yield 1 further report, of a trial the databases missed, giving 23 + 1 = 24 reports.
5. Link reports to studies. Matching registration numbers shows that the 24 reports describe 14 studies, so the synthesis uses 14 units.
6. Precision. Treating the 23 included database reports as the relevant reports retrieved, precision is 23 / 2,300 = 0.010, or 1.0%, and 2,300 / 23 = 100 records were screened for each included report.
7. Screening time. At 60 to 120 abstracts an hour, one person needs 2,300 / 120 = 19.17 to 2,300 / 60 = 38.33 hours. Two independent screeners double this to 2 × 19.17 = 38.33 to 2 × 38.33 = 76.67 person-hours.
8. Relative recall. The database strategy retrieved 13 of the 14 included studies, a relative recall of 13 / 14 = 0.929, or 92.9%.

Low precision of this order is what the Handbook associates with sensitive systematic review searching. The relative recall carries more information, because the missed trial exposes a gap. Here the trial was indexed under a drug-class term absent from the strategy, so the strategy is revised and rerun before synthesis. Counting the 24 reports as 24 studies would count some participants twice and make a pooled estimate look more precise than the evidence allows, an error the evidence table page follows through to an ICER.

Reporting a review and judging its conduct

PRISMA 2020 replaces the 2009 statement and consists of a 27-item checklist in seven sections, an abstract checklist and revised flow diagrams, and it can be used for updated and living reviews. Its authors state that it should not be used to assess the conduct or methodological quality of a review, and point to tools built for that purpose, AMSTAR 2 and ROBIS. For an HTA team using an existing review, the main checks are whether its question matches the local decision problem and whether its final search date is recent enough for the comparators now in use.

Systematic reviews beside neighbouring terms

Evidence synthesis is the wider activity of combining study results for decision use. A meta-analysis is a statistical step that may sit inside a systematic review or be run on studies chosen some other way. A clinical effectiveness review is the part of an HTA that applies systematic review methods to a technology's benefits and harms. A review of published economic evaluations uses the same machinery on cost-effectiveness studies, and NICE expects one in an appraisal even when a new model is built.

Limitations and common misreadings

The label describes a method, so a review is only as useful as its question, search and appraisal. Several misreadings follow from forgetting this.

1. Equating systematic with meta-analysis. A review with no pooling is still systematic if its methods are.
2. Ignoring the final search date. A model built on an old review can miss newer trials and comparators.
3. Reading PRISMA compliance as quality. PRISMA 2020 addresses reporting, while AMSTAR 2 and ROBIS address conduct.
4. Expecting the review to fix weak studies. A precise pooled estimate from trials at high risk of bias is still rated down for certainty under GRADE.
5. Demanding full systematic methods for every model input. TSD 13 treats sufficiency, judged against a parameter's influence on the decision, as the more useful test, provided stopping decisions are reported.

Sources

• Higgins JP, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JP, Thomas J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
• Kaltenthaler E, Tappenden P, Paisley S, Squires H. NICE DSU Technical Support Document 13: Identifying and reviewing evidence to inform the conceptualisation and population of cost-effectiveness models. Decision Support Unit; May 2011. https://doi.org/10.15131/shef.data.33745504
• Lasserson TJ, Thomas J, Higgins JP. Chapter 1: Starting a review. In: Higgins JP, Thomas J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-01
• Lefebvre C, Glanville J, Briscoe S, et al. Chapter 4: Searching for and selecting studies. In: Higgins JP, Thomas J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.5.1. Cochrane; 2025. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Sections 3.3.4, 3.3.25, 3.3.26, 3.4.2, 3.4.4, 3.4.5, 3.4.8, 4.3.8 and 4.4.10. https://www.nice.org.uk/process/pmg36
• Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71
• Papaioannou D, Brazier J, Paisley S. NICE DSU Technical Support Document 9: The identification, review and synthesis of health state utility values from the literature. Decision Support Unit; October 2010. https://doi.org/10.15131/shef.data.33738742