Systematic Review

How a systematic review builds a reliable evidence base

A systematic review uses predefined, transparent methods to identify and synthesise evidence relevant to a specific question. This page explains how the question is defined, how studies are found and selected, how their credibility is assessed, and how the findings can inform health-economic analysis and decision making.

The value of a systematic review comes from the complete process rather than from the number of studies it includes. A review may identify substantial evidence, limited evidence or no eligible evidence, but its conclusions should remain traceable to the question, protocol, search, selection decisions and synthesis methods.

Starting with a clearly defined question

A systematic review begins with a question that is sufficiently precise to guide eligibility criteria, searching, data extraction and synthesis. Questions about intervention effects are often structured using population, intervention, comparator and outcome elements, but other frameworks may be appropriate for diagnostic, prognostic, qualitative, economic or policy questions.

A review question commonly specifies:

• Population: The people, conditions or settings to which the evidence should apply.
• Intervention or exposure: The technology, programme, policy, risk factor or experience being examined.
• Comparator: The relevant alternative.
• Outcomes: The clinical, economic, patient-reported or system outcomes required.
• Study designs: The types of evidence eligible for inclusion.
• Setting and jurisdiction: The context relevant to the decision.
• Time period: Any restrictions on dates, duration or follow-up.

The scope should reflect the intended use of the review. Overly narrow questions can omit relevant evidence, while very broad questions may produce findings that are difficult to combine or interpret.

Planning the review before seeing results

A protocol records the methods before study results influence review decisions. It reduces the risk that eligibility criteria, outcomes or analytical methods are changed because particular findings are preferred.

A protocol should normally describe:

• The review question and rationale.
• Eligibility criteria.
• Information sources and search strategy.
• Study-selection and data-extraction processes.
• Risk-of-bias assessment.
• Planned synthesis methods.
• Methods for investigating heterogeneity.
• Certainty assessment.
• Planned subgroup and sensitivity analyses.
• The process for recording amendments.

Prospective registration can make the planned review visible and help identify overlapping work. Registration does not guarantee methodological quality, and departures from the protocol should be documented and justified.

Finding the relevant evidence

The search should retrieve eligible evidence as completely as is practical. Searching only one database or using only obvious terms can miss relevant studies and distort the evidence base.

A comprehensive search may include:

• Bibliographic databases.
• Trial or study registries.
• Regulatory and health technology assessment sources.
• Conference records.
• Citation searching.
• Reference lists.
• Organisation and government websites.
• Grey literature.
• Contact with study authors or experts when justified.

The search should combine concepts from the review question using controlled vocabulary and free-text terms. Search dates, databases, platforms, limits and complete strategies should be recorded.

Restrictions by language, publication date or publication status require justification because they may introduce bias.

Selecting studies consistently

Study selection determines which evidence contributes to the review. Reviewers remove duplicates, assess titles and abstracts, retrieve potentially eligible full texts and apply predefined criteria.

The process commonly follows these steps:

1. Combine search results from all sources.
2. Remove duplicate records while preserving an audit trail.
3. Screen titles and abstracts against eligibility criteria.
4. Retrieve potentially eligible reports.
5. Assess full-text reports using the same criteria.
6. Record reasons for exclusion at the full-text stage.
7. Link multiple reports describing the same study.
8. Document the final study set in a study-flow record.

Using more than one reviewer can reduce errors and unsupported judgement. One study may produce several reports, and treating each report as a separate study can double-count participants and findings.

Extracting data

Data extraction converts study information into a structured evidence set. The extraction form should reflect the review question and planned synthesis rather than collect information without purpose.

Relevant data may include:

• Study design and location.
• Participant characteristics.
• Intervention and comparator details.
• Sample size and follow-up.
• Outcome definitions and measurement times.
• Effect estimates and uncertainty.
• Missing data and withdrawals.
• Funding and conflicts of interest.
• Information needed for risk-of-bias assessment.
• Resource use, costs, utility values and economic outcomes.

Extraction should preserve units, time points, denominators and analytical populations. Conversions or derived values should be documented.

Assessing risk of bias

Risk-of-bias assessment examines whether features of a study could cause its result to differ systematically from the truth. It is not the same as assigning a general quality score.

Relevant domains may include:

• Bias arising from randomisation or allocation.
• Bias caused by deviations from intended intervention.
• Bias from missing outcome data.
• Bias in outcome measurement.
• Bias in selection of the reported result.
• Bias caused by confounding.
• Bias in participant selection or exposure classification.
• Bias associated with selective publication.

Judgements should be supported by evidence and relate to the specific result used in synthesis. A study may have different risks for different outcomes or analyses.

Bringing the evidence together

All systematic reviews require a structured synthesis, but not every review includes a statistical meta-analysis. A narrative synthesis may compare study characteristics, direction and size of effects, limitations and reasons for different findings.

A meta-analysis may be appropriate when studies address sufficiently similar questions and their results can be combined meaningfully. Before pooling, reviewers should consider populations, interventions, comparators, outcomes, follow-up, effect measures, design and bias.

When meta-analysis is inappropriate, the review should explain why and use a transparent alternative. Counting statistically significant studies is not a reliable synthesis method.

Understanding heterogeneity

Heterogeneity refers to differences between studies that may affect results. It can be clinical, methodological or statistical.

Potential sources include:

• Patient characteristics.
• Intervention delivery or intensity.
• Comparators.
• Outcome definitions.
• Follow-up time.
• Study design or risk of bias.
• Healthcare systems, prices or clinical practice.

Subgroup analysis, sensitivity analysis or meta-regression may help when supported by enough evidence and a credible rationale. Analyses added after inspecting results should be identified as exploratory.

Judging certainty in the evidence

Certainty expresses confidence that an estimated effect or conclusion is suitable for its intended interpretation. It is assessed across the body of evidence for an outcome.

Considerations may include:

• Risk of bias.
• Inconsistency.
• Indirectness.
• Imprecision.
• Risk of missing results or studies.
• Other factors that strengthen or weaken confidence.

A statistically precise pooled result can still provide uncertain evidence if studies are biased or indirect. A review can also provide useful evidence without meta-analysis when limitations and consistency are assessed transparently.

Using systematic reviews in health economics

Systematic reviews provide evidence for economic evaluations, health technology assessments, budget impact analyses and economic models. Different questions may be needed for treatment effects, resource use, costs, utility values, epidemiology and model structure.

A review supporting a model may identify:

• Baseline event risks.
• Relative treatment effects.
• Disease-progression estimates.
• Adverse-event frequencies.
• Health-state utility values.
• Resource-use estimates.
• Unit costs.
• Adherence and discontinuation.
• Existing economic evaluations.
• Evidence relevant to structural assumptions.

Evidence suitable for one model input may not be suitable for another. Clinical and economic transferability should be evaluated rather than assumed.

Reviewing economic evaluations

A systematic review of economic evaluations examines how previous studies compared costs and outcomes. Its purpose is not simply to collect published cost-effectiveness ratios because results depend on the decision problem, model, evidence, prices and assumptions.

Important characteristics include:

• Population, intervention and comparator.
• Perspective and type of economic evaluation.
• Time horizon and discount rates.
• Model structure.
• Sources of clinical and economic inputs.
• Health outcomes.
• Currency and price year.
• Uncertainty methods.
• Incremental costs and outcomes.
• Conclusions and limitations.

Differences should be explained rather than averaged without justification.

Keeping the review current

A systematic review describes evidence identified through its final search date. New studies, corrected reports or changed methods can make conclusions incomplete or outdated.

Reviewers should state the final search date, whether searches were rerun, whether updating is planned and which findings are most likely to change.

Reporting transparently

Transparent reporting allows readers to understand what was planned, done, found and concluded. Reporting guidance supports completeness but does not replace sound methods.

A complete report should identify:

• The question and eligibility criteria.
• The protocol and amendments.
• Full search methods.
• Study selection.
• Included-study characteristics.
• Risk-of-bias assessments.
• Synthesis methods and results.
• Certainty assessment.
• Limitations.
• Funding and competing interests.

A reporting checklist cannot correct an incomplete search, inappropriate synthesis or biased selection process.

A simplified example

Suppose a health service is considering a digital programme to reduce hospital readmissions. A systematic review defines eligible patients, programme characteristics, comparators, readmission outcomes and study designs before searching.

The review finds that programmes differ in intensity, staffing and follow-up. Rather than pooling every result, reviewers group clinically similar programmes, assess bias, examine consistency and identify estimates suitable for economic modelling.

The review may conclude that the programme probably reduces short-term readmissions but that long-term outcomes and implementation costs remain uncertain. A model can use supported short-term evidence while representing remaining gaps through assumptions and uncertainty analysis.

How systematic reviews differ from related approaches

• A meta-analysis is a statistical method that may form part of a systematic review.
• A scoping review maps a broad evidence base and its gaps.
• A rapid review streamlines selected methods and should disclose the changes.
• A narrative review may provide expert interpretation without reproducible selection methods.
• A health technology assessment evaluates broader consequences and may contain systematic reviews.
• An umbrella review synthesises existing systematic reviews.

These approaches should be selected according to the decision question, evidence, timeframe and intended use.

Common errors

Common errors include:

• Changing eligibility criteria after seeing results.
• Searching too few sources without justification.
• Omitting important search terms.
• Applying unexplained restrictions.
• Double-counting multiple reports from one study.
• Excluding studies without clear reasons.
• Using an inappropriate risk-of-bias tool.
• Treating reporting quality as risk of bias.
• Pooling materially different studies.
• Counting significant studies rather than synthesising effects.
• Ignoring missing results or publication bias.
• Failing to consider certainty.
• Failing to report deviations or conflicts of interest.

Interpreting a systematic review

Findings should be interpreted in relation to the question, eligibility criteria, search date, evidence and methods. A carefully conducted review can reduce bias and clarify uncertainty, but it cannot repair fundamental weaknesses in the available studies.

A useful review shows what the evidence suggests, how confidently it supports the decision and where uncertainty remains.