Narrative Review

Narrative Review in HTA and Economic Modelling

The narrative review is the traditional review article, in which an author or panel chooses which studies to discuss and how much weight each deserves, with no protocol that a second team could follow to reach the same set of studies. Its strength is interpretation and its weakness is that the selection of evidence is invisible to the reader. In health economics the distinction matters because the relative effect, utility values and costs in a cost-effectiveness analysis must come from evidence that a health technology assessment committee can check, so NICE asks for a systematic review of clinical evidence, and a narrative review is better suited to framing a problem than to supplying model inputs. Studies in 1987 and 1992 found that review articles rarely reported their methods and that expert advice could lag behind the trials.

What separates a narrative review from a systematic review

The Cochrane Handbook describes a systematic review as an attempt to collate all the empirical evidence that fits pre-specified eligibility criteria, using explicit methods chosen to minimise bias. It names bias as the main problem with ad hoc searching and selection of research. The editors who built the SANRA scale use narrative review for an attempt to summarise the literature in a way that is not explicitly systematic, where the minimum requirement for calling a review systematic concerns the literature search. Greenhalgh and colleagues describe a narrative review as a scholarly summary with interpretation and critique, which can still be conducted in a disciplined way.

A long review by a leading expert that cites hundreds of papers is still a narrative review if the reader cannot see how those papers were found and chosen.

Early evidence that unstructured reviews can mislead

In 1987 Cynthia Mulrow assessed 50 reviews published between June 1985 and June 1986 in four major medical journals against eight explicit criteria. Only one clearly specified its methods of identifying, selecting and validating the included information, 43 integrated the evidence by qualitative synthesis and only 3 used quantitative synthesis. She concluded that medical reviews did not routinely use scientific methods to identify, assess and synthesise information.

Antman and colleagues then compared recommendations in review articles and textbook chapters on myocardial infarction with a cumulative meta-analysis of the randomised trials, updated as each new trial was published. Review articles often failed to mention important advances or were slow to recommend effective preventive measures, and some experts went on recommending treatments that had no effect on mortality or were potentially harmful. The Cochrane Handbook cites this study in its account of why systematic methods give more reliable findings.

Why HTA bodies require systematic reviews for clinical evidence

A cost-effectiveness model converts a relative treatment effect into incremental QALYs, so a biased selection of trials passes directly into the ICER. The NICE manual (PMG36) asks that evidence on outcomes come from a systematic review, which it defines as systematically locating, including, appraising and synthesising the evidence for a clearly formulated question. Such a review should follow a pre-defined protocol and use explicit, valid and replicable methods that minimise the risk of biased selection of studies.

When there is not enough relevant and valid data for pairwise or network meta-analysis, the analysis may be restricted to a narrative overview that critically appraises individual studies and presents their results, and the committee will then be particularly cautious in drawing conclusions about relative clinical effectiveness. That narrative overview still rests on a systematic search and appraisal, which is what separates it from a narrative review.

Model inputs where a full systematic review is not demanded

A decision model needs many inputs besides the treatment effect, including baseline risks, health state utility values, resource use and unit costs. PMG36 asks that EQ-5D data taken from the literature be identified by methods that are systematic and transparent, with the choice of data set justified, and that resource use and cost data be shown to have been identified systematically, preferably through systematic review where costs come from the literature. Where several plausible sources exist, the choice for the base case should be justified and sensitivity analysis should explore the alternatives.

The NICE Decision Support Unit's Technical Support Document 13 addressed the same question in 2011, under the 2008 NICE methods guide, which required a full systematic review for treatment effects and a systematic, transparent and justifiable approach for other parameters. The DSU states that its technical support documents are not formal NICE guidance or policy. TSD 13 observes that systematic review search methods are designed for single, focussed, predefined questions, whereas the information needs of a model emerge during its development. It reports the argument that searching should then seek sufficient information to understand the uncertainty in the model, and notes that the consequences of not searching exhaustively had not been tested empirically.

Its guiding principles mark the line between a pragmatic parameter search and a narrative review. Sources should be identified systematically, transparently and with justification, so that it is clear they were not found serendipitously, opportunistically or preferentially, every source used in the model should be referenced, and the factors behind the choice of source should be stated. Paisley later proposed a minimum requirement of comprehensive searching for treatment effects and, for other parameter types, at least one bibliographic database plus specialist and non-standard sources where appropriate. A targeted search in an HTA submission is therefore a documented, limited search, and an undocumented narrative selection meets neither these principles nor the PMG36 requirement that literature-based inputs be identified systematically and transparently.

Where narrative reviews still earn a place

Narrative reviews remain useful where the task is understanding more than estimation. Greenhalgh and colleagues argue that systematic and narrative reviews serve different purposes and should be treated as complementary, with the narrative review contributing interpretation and critique. The SANRA authors make a similar point, that systematic reviews are better at answering specific questions and narrative reviews at addressing a topic in wider ways.

In health economics this fits early stages of an assessment. A narrative account of a disease area, its care pathway and the debates around it can help analysts frame the decision problem and draft a conceptual model, which TSD 13 suggests building from sources such as clinical guidelines and clinical experts. Such reviews cannot stand in for a systematic review when a number enters the base case.

Judging the quality of a narrative review

The authors of SANRA, the Scale for the Assessment of Narrative Review Articles, reported in 2019 that no instrument existed to assess the quality of narrative reviews. Revised by three journal editors, the scale has six items, each scored from 0 to 2 for a maximum of 12. The items cover the importance of the topic, the aims of the review, the description of the literature search, referencing, the presentation of the level of evidence and the presentation of relevant endpoint data.

In its validation the editors rated 30 consecutive non-systematic review manuscripts submitted to a general medical journal. The mean score was 6.0 out of 12, Cronbach's alpha was 0.68 and the intra-class correlation for inter-rater reliability was 0.77. The authors see it as a complement to editorial judgement and called for further testing of its validity.

Worked example with vote counting and a pooled estimate

The numbers in this example are illustrative. Five trials compare a new treatment with standard care for preventing hospital admission. All five report a risk ratio of 0.80. Four small trials have a standard error of 0.15 on the log scale and one larger trial has 0.10, so only the larger trial is statistically significant, with z of about minus 2.23 against minus 1.49 for each small trial. A narrative review that tallies trials by statistical significance finds one positive and four null trials and calls benefit unproven.

The Cochrane Handbook describes this kind of vote counting based on statistical significance as unacceptable, because underpowered studies that do not rule out important effects are counted as showing no benefit. A fixed-effect model pools the same trials.

$$\ln \hat{RR} = \frac{\sum_i w_i \ln RR_i}{\sum_i w_i}, \qquad w_i = \frac{1}{SE_i^{2}}, \qquad SE_{pooled} = \frac{1}{\sqrt{\sum_i w_i}}$$

where $RR_i$ is the risk ratio in trial $i$, $SE_i$ is the standard error of its log risk ratio, $w_i$ is its weight and $\hat{RR}$ is the pooled risk ratio. The weights are 44.4 for each small trial and 100 for the large one, a total of 277.8, so the pooled standard error is 0.060. The pooled risk ratio is 0.80 with a 95% confidence interval of 0.71 to 0.90, which excludes no effect.

A one-year model then applies the risk ratio to a baseline admission risk of 0.25, with a treatment cost of GBP 300, an admission cost of GBP 4,000 and a QALY loss of 0.15 per admission.

$$\Delta C = c_T - p_0 (1 - RR) c_A, \qquad \Delta E = p_0 (1 - RR) q_A, \qquad ICER = \frac{\Delta C}{\Delta E}$$

where $\Delta C$ is the incremental cost in GBP, $\Delta E$ the incremental QALYs, $c_T$ the treatment cost, $p_0$ the baseline admission risk, $RR$ the risk ratio, $c_A$ the cost of an admission and $q_A$ the QALY loss from an admission.

The vote count turns a consistent effect into an apparent null and presents the treatment as a pure cost. The pooled analysis gives an ICER of GBP 13,333 per QALY, and its confidence interval feeds parameter uncertainty analysis instead of being discarded.

Boundaries with neighbouring terms

Literature review is the broader everyday term and covers both kinds. Greenhalgh and colleagues refer to Cynthia Mulrow's call for literature reviews to be done more systematically, so literature review is not a synonym for narrative review. Narrative synthesis is different again, a way of combining results inside a systematic review when meta-analysis is not possible. The Cochrane Handbook asks authors to report the specific methods used in place of meta-analysis instead of stating only that they did a narrative synthesis. The rapid review methods for model parameters listed in TSD 13 may truncate the search but still report methods, limitations and potential biases transparently.

Misreadings of the narrative review

Narrative does not mean careless. Greenhalgh and colleagues argue that the term systematic is not synonymous with high quality, and that the label sometimes lets a narrow data aggregation claim more authority than it deserves. The opposite error is a selective set of studies presented as if it were systematic, and TSD 13 reports criticism of HTA models for inadequate reporting of how evidence was identified and for not justifying how it was selected. An eminent author does not guarantee a complete or up-to-date selection, as the Antman comparison showed.

Sources

• Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992;268(2):240-248. https://pubmed.ncbi.nlm.nih.gov/1535110/
• Baethge C, Goldbeck-Wood S, Mertens S. SANRA, a scale for the quality assessment of narrative review articles. Research Integrity and Peer Review. 2019;4:5. https://doi.org/10.1186/s41073-019-0064-8
• Greenhalgh T, Thorne S, Malterud K. Time to challenge the spurious hierarchy of systematic over narrative reviews? European Journal of Clinical Investigation. 2018;48(6):e12931. https://doi.org/10.1111/eci.12931
• Kaltenthaler E, Tappenden P, Paisley S, Squires H. NICE DSU Technical Support Document 13: Identifying and reviewing evidence to inform the conceptualisation and population of cost-effectiveness models. Sheffield: Decision Support Unit, ScHARR; 2011. https://doi.org/10.15131/shef.data.33745504
• Lasserson TJ, Thomas J, Higgins JPT. Chapter 1: Starting a review (last updated August 2021). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-01
• McKenzie JE, Brennan SE. Chapter 12: Synthesizing and presenting findings using other methods (last updated October 2019). In: Higgins JPT, Thomas J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-12
• Mulrow CD. The medical review article: state of the science. Annals of Internal Medicine. 1987;106(3):485-488. https://doi.org/10.7326/0003-4819-106-3-485
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Paisley S. Identification of evidence for key parameters in decision-analytic models of cost effectiveness: a description of sources and a recommended minimum search requirement. PharmacoEconomics. 2016;34(6):597-608. https://doi.org/10.1007/s40273-015-0372-x