Retrieval-Augmented Generation

Retrieval-Augmented Generation for Health Economic Evidence

Retrieval-augmented generation (RAG) is a natural language processing method that joins a search step to a language model, so the model writes each answer from passages fetched out of a chosen document collection at the moment the question is asked, as well as from what it learned in training. In health economics the collection might hold HTA reports, the included studies of a systematic review or the documents behind the inputs of a decision model, and the passages used for an answer can be shown beside it and checked. The method sits inside many tools for artificial intelligence in health that search guidance, summarise evidence or propose values for data extraction, and an ISPOR working group lists it among the techniques for improving the accuracy of generative AI in health economics and outcomes research. Retrieval changes what the model reads before it writes, but a fluent answer with a source attached can still be wrong, and that gap limits how far an output can be trusted in an economic evaluation.

Two kinds of memory in one model

Lewis and colleagues introduced the term in 2020 for models that combine two kinds of memory. The parametric memory is the knowledge held in the weights of a pre-trained language model, and the non-parametric memory is an index of text passages that a retriever searches. They argued that models relying on their weights alone cannot easily expand or revise what they know, cannot readily show the basis of a prediction and may produce hallucinations, meaning fluent statements that are false.

A retrieval index can be edited without retraining the model. In their test, a model given a 2016 index answered 70% of questions about world leaders correctly for the 2016 office holders, and with a 2018 index it answered 68% correctly for the 2018 office holders. With the indexes swapped, accuracy fell to 12% and 4%. For a health economist the lesson is that an assistant built over appraisal documents is only as current as its library. Replacing superseded guidance in the library changes what the assistant can retrieve without retraining the model, a maintenance task close to that of a living HTA.

Indexing, retrieval and generation

A systematic review of RAG in healthcare by Amugongo and colleagues describes three steps. Documents are first split into segments and indexed, which for dense retrieval means an encoder model turns each segment into a vector. The segments most similar to the query vector are then retrieved, and finally the language model generates a response from the query and the retrieved segments. In the original system each Wikipedia article was split into disjoint 100-word chunks, giving 21 million passages, and each chunk was stored as an embedding in an index searched by maximum inner product.

Retrievers differ in what counts as a match. Dense retrievers compare learned vectors and lexical retrievers such as BM25 score word overlap, a contrast covered in more depth on the semantic search page. In the 2020 experiments BM25 did best on a fact verification task whose claims were heavily centred on named entities, and dense retrieval did better on the other tasks. Queries built around drug names or clinical codes in HTA submissions may share that character. The MIRAGE benchmark of Xiong and colleagues found that combining several retrievers by rank fusion gave the best average score on its combined medical corpus.

The probabilistic form in the original paper

The 2020 models treat the retrieved passage as a hidden variable and average over the passages they retrieve. One probability is attached to each passage and another to each candidate answer. The retriever gives each passage a probability from the inner product of two vectors,

$$p(z \mid x) \propto \exp\left( \mathbf{d}(z)^{\top} \mathbf{q}(x) \right)$$

where $z$ is a candidate passage, $x$ is the question, $\mathbf{d}(z)$ is the vector a document encoder assigns to the passage, $\mathbf{q}(x)$ is the vector a query encoder assigns to the question, and the exponent is the inner product of the two vectors. The $k$ passages with the highest probability are kept. In the RAG-Sequence model the probability of a whole answer is then

$$p(y \mid x) \approx \sum_{z \in \text{top-}k} p(z \mid x) \, p(y \mid x, z)$$

where $y$ is the generated answer, $k$ is the number of passages kept, $p(z \mid x)$ is the retrieval probability renormalised over those $k$ passages, and $p(y \mid x, z)$ is the probability the generator gives the answer when it reads the question together with passage $z$. A second form, RAG-Token, averages over passages separately for each token, a word or part of a word, so one answer can draw on several passages. The retriever and generator were trained together without any labels saying which passage was correct.

Most systems in use today are simpler. The MIRAGE benchmark evaluated the vanilla architecture, in which retrieved passages are placed in front of the question in the prompt, because it is the most widely implemented. The formula shows the logic of the original design, in which each passage counts towards an answer in proportion to its retrieval probability.

Worked example with an illustrative evidence library

All passages, scores and probabilities here are hypothetical and illustrative. An analyst asks an assistant built over a library of model reports which utility value an appraisal used for the progressed disease state. Three passages in the library answer the question, all from the final report and all giving 0.60. The retriever keeps three passages, passage 1 from the final report text, passage 2 from a model of a different technology that used 0.72, and passage 3 from a table in the final report. Their inner product scores are 4.0, 3.0 and 2.0.

Step 1. Retrieval probabilities. The exponentials of the scores are 54.598, 20.086 and 7.389, which sum to 82.073. For passage 1,

$$p(z_1 \mid x) = \frac{54.598}{82.073} = 0.665$$

where $z_1$ is passage 1 and $x$ is the question. The same division gives 0.245 for passage 2 and 0.090 for passage 3.

Step 2. Generator probabilities. Reading passage 1, the generator gives answer A (0.60) a probability of 0.90 and answer B (0.72) a probability of 0.05. Reading passage 2 the figures are 0.20 and 0.75, and reading passage 3, a table the generator reads less reliably, they are 0.50 and 0.30.

Step 3. Averaging over passages. The probability of answer A is

$$p(A \mid x) \approx 0.665 \times 0.90 + 0.245 \times 0.20 + 0.090 \times 0.50 = 0.6925$$

where $A$ is the answer 0.60, and the probability of answer B is

$$p(B \mid x) \approx 0.665 \times 0.05 + 0.245 \times 0.75 + 0.090 \times 0.30 = 0.244$$

where $B$ is the answer 0.72. The remaining probability, $p_{\text{other}} = 1 - 0.6925 - 0.244 = 0.0635$, falls on other wordings.

Step 4. Retrieval quality. Precision is the share of retrieved passages that are relevant and recall is the share of relevant passages that are retrieved. Two of the three passages returned are relevant, so precision at three is 2/3, or 0.667. Two of the three relevant passages in the library were returned, so recall is also 2/3, or 0.667.

Step 5. Interpretation. Answer A is the most probable and matches the report. Almost a quarter of the probability, 0.244, goes to the wrong value 0.72, and 0.184 of it comes through passage 2, which is about a different model. In a prompt-based system that passage would sit in the prompt beside the correct ones. A reviewer should open passage 1 and the report table before 0.60 enters the model as a parameter input with its source recorded.

Choices that change the answer

A RAG system has several settings, and each changes what the model reads. Xiong and colleagues built the MIRAGE benchmark of 7,663 questions from five medical question answering datasets and tested 41 combinations of corpora, retrievers and language models. The PubMed corpus and their combined corpus were the only collections with which their toolkit beat chain-of-thought prompting without retrieval on all five tasks, so the choice of library changed the result.

The number of passages kept also matters. On one literature-based task, accuracy was high with a single passage and fell as more passages were added, because the extra passages were irrelevant, while on another task accuracy kept rising as more passages were added. Position matters too. Accuracy followed a U-shaped pattern, lowest when the passage holding the answer sat in the middle of the retrieved set, the effect called lost in the middle. Amugongo and colleagues describe a system that set a minimum similarity and replied that it could not find relevant information when no passage reached it, which reduced misleading answers, and frameworks that reorder passages so the most relevant sit closest to the prompt.

Evidence from medical question answering

Benchmark results show gains with clear limits. Across six language models the MIRAGE toolkit raised accuracy by up to 18% in relative terms over chain-of-thought prompting. For examination questions written to separate medical students, retrieval did not always beat fine-tuning on medical text, since the helpful passages might be difficult to retrieve. In a human evaluation of question generation in the 2020 paper, raters judged the retrieval model more factual than the baseline in 42.7% of comparisons and the baseline more factual in 7.1%.

The review by Amugongo and colleagues included 70 studies selected from 2,139 records. It found no standardised framework for evaluating RAG in healthcare, reported that most included studies did not address privacy, and cited a study showing that language models can still give incorrect answers and explanations with retrieval. These benchmarks are mainly multiple-choice clinical questions, so they say little directly about HTA tasks such as extracting costs or reading a model report.

Use in health technology assessment

The ISPOR working group taxonomy by Fleurence and colleagues reviews generative AI for systematic literature reviews, real-world evidence, economic modelling and dossier development, and describes RAG as retrieving more up-to-date or task-specific information than the underlying model was trained on. It also names a limitation, that the method may handle conflicting retrieved information poorly, which lowers output quality.

For health technology assessment at NICE, the institute sets out its position in the statement Use of AI in evidence generation (ECD11, published 15 August 2024). Any use of AI methods should augment human involvement, not replace it, and the statement gives careful technical and external validation, with the results presented, as an example. Submitting organisations should declare the use of AI, explain the choice of method and report how it was used, and they remain accountable for the content of a submission. The statement warns of cybersecurity risks including data poisoning and prompt injection. Since a RAG system places retrieved text into the prompt, the library is one route by which such content could reach the model. ECD11 also says that requests to use NICE content for AI purposes are subject to an approval process and a licensing arrangement, with a fee for international use, which applies to any plan to index NICE guidance. Among potential uses, it treats automated data extraction by language models as less well established than support for finding and screening studies.

Boundaries with search, fine-tuning and extraction

RAG adds generation to search. Semantic search returns ranked passages for a person to read, whereas a RAG system goes on to write an answer from them, so it can join facts from several passages but can also misstate them. Fine-tuning changes the model's weights by training on domain text. In the MIRAGE comparison fine-tuning added more accuracy than retrieval on the same base model, yet the authors judged retrieval the more flexible and cost-efficient way to improve medical question answering.

Extractive question answering returns a span copied from a retrieved passage. A generator is not bound to the passages, and the 2020 model gave correct answers in 11.8% of cases where the answer was in none of the retrieved passages, drawing on its weights. A correct answer of this kind is not supported by the passages shown, and an incorrect one is not caught by them either.

Limits and common misreadings

Four misreadings treat retrieval as if it were verification.

1. Reading a retrieved source as a true source. Lewis and colleagues note that any external knowledge source will probably never be entirely factual or free of bias, so a library of drafts, superseded guidance or unreviewed preprints passes those faults on.

2. Treating an attached passage as proof. The answer can come from the model's weights, so each figure should be traced to the exact passage and then to the original document.

3. Missing a conflict. When passages disagree, as passage 2 did in the worked example, a single fluent answer can hide the disagreement.

4. Overlooking privacy. Amugongo and colleagues warn that information in the retrieval dataset can be exposed through the answers. A library built from patient records therefore needs the same data governance as the records themselves.

Sources

• Amugongo LM, Mascheroni P, Brooks S, Doering S, Seidel J. Retrieval augmented generation for large language models in healthcare: a systematic review. PLOS Digital Health. 2025;4(6):e0000877. https://doi.org/10.1371/journal.pdig.0000877
• Fleurence RL, Wang X, Bian J, et al. A taxonomy of generative artificial intelligence in health economics and outcomes research: an ISPOR working group report. Value in Health. 2025;28(11):1601-1610. https://doi.org/10.1016/j.jval.2025.04.2167 (preprint text read at https://arxiv.org/abs/2410.20204)
• Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). 2020. https://arxiv.org/abs/2005.11401
• Manning CD, Raghavan P, Schütze H. Introduction to Information Retrieval. Cambridge University Press; 2008. https://nlp.stanford.edu/IR-book/
• National Institute for Health and Care Excellence. Use of AI in evidence generation: NICE position statement (ECD11). Published 15 August 2024. https://www.nice.org.uk/corporate/ecd11
• Xiong G, Jin Q, Lu Z, Zhang A. Benchmarking retrieval-augmented generation for medicine. In: Findings of the Association for Computational Linguistics: ACL 2024. 2024:6233-6251. https://aclanthology.org/2024.findings-acl.372/