Semantic Search

Semantic Search for Health Economic Evidence

Semantic search is a kind of information retrieval that finds records by what a query is about, so a search for cost-utility studies can reach papers indexed under the MeSH heading Cost-Benefit Analysis, and a pasted NICE recommendation can reach others on the same topic that use different words. In health economics a systematic review, a hunt for model inputs and a check of existing guidance all begin with finding the right records. Keyword search misses relevant items whenever authors, indexers and searchers describe the same thing in different terms. Controlled vocabularies close that gap with explicit synonyms and broader terms, while text embeddings learn closeness of meaning from large bodies of text, and the two fail in different ways.

Words, meanings and the vocabulary gap

In their 2003 paper on Semantic Search, Guha, McCool and Miller observed that traditional information retrieval rests almost purely on the occurrence of words in documents. Their systems used what a query denotes, such as a particular person or place, to add Semantic Web facts to ordinary results. They aimed at research searches, where a person gathers several documents about a real thing, rather than navigational searches for one known page.

Evidence work in health economics is almost always research searching. Thakur and colleagues call the weakness of lexical methods such as TF-IDF and BM25 a lexical gap, since they only retrieve documents containing keywords present in the query. The Office for National Statistics (ONS) team that worked with NICE found the same with the NICE website, whose keyword search would not capture all synonyms or changes in terminology over time.

Meaning drawn from a controlled vocabulary

The older route to meaning is a thesaurus or ontology that records which words name the same concept and how concepts nest. PubMed's Automatic Term Mapping matches untagged terms against a subject translation table holding British and American spellings, synonyms, brand names mapped to generic names and the entry terms of MeSH, the National Library of Medicine's controlled vocabulary. A matched term is searched as its MeSH heading, including the more specific headings beneath it, and in all fields.

The MeSH record for economic evaluation shows both the gain and the trap. The descriptor Cost-Benefit Analysis includes Cost-Utility Analysis, Economic Evaluation and Marginal Analysis among its concepts, so a mapped query for any of them reaches articles under one heading. A separate descriptor, Cost-Effectiveness Analysis, was introduced in 2023, and its history note points searchers to Cost-Benefit Analysis for 2015 to 2022, so a search for cost-effectiveness analysis studies from 2015 onwards needs both headings.

The W3C's Simple Knowledge Organization System (SKOS) gives such vocabularies a standard web form, with preferred, alternative and hidden labels for each concept. The SKOS Reference describes hidden labels as a way for a text search to match misspellings without showing them to users. HealthEconomics.wiki's own vocabulary, the Health Economics Ontology (HEO), is published on NCBO BioPortal as SKOS concepts with stable URIs arranged by broader terms, so a search tool could expand a query to narrower concepts in the same way MeSH does.

Meaning drawn from text embeddings

The newer route turns each query and document into a vector of numbers, a text embedding, placed so that texts with similar meaning sit close together. Reimers and Gurevych describe semantic search as mapping sentences into a vector space where similar sentences are close and comparing them with a measure such as cosine similarity. Because their Sentence-BERT model computes each vector once, finding the most similar pair among 10,000 sentences took about 5 seconds, against about 65 hours with BERT.

Karpukhin and colleagues note that synonyms or paraphrases with completely different tokens can still map to nearby vectors. Their dense retriever beat a strong BM25 system by 9% to 19% absolute in top-20 passage retrieval accuracy on open-domain question answering datasets, although BM25 kept an edge on rare, highly selective phrases. In biomedicine, Jin and colleagues at the National Library of Medicine trained MedCPT on 255 million query and article pairs from PubMed click logs, noting that most biomedical retrieval is keyword based and misses relevant articles that share no words with the query. They suggest such models could improve tools such as PubMed's Best Match, placing semantic ranking in front of a bibliographic database reviewers already use.

Semantic search of NICE recommendations

A UK algorithmic transparency record published on GOV.UK in December 2024 documents one working NICE system, NORMA (NICE-ONS Recommendation Matching Algorithm). The surveillance team checks whether new evidence affects recommendations across more than 300 guidelines, and a manual check for overlaps could take up to a week of an analyst's time per guideline, partly because the website's keyword search returned pages rather than recommendations.

NORMA ranks recommendations by the cosine similarity of averaged biomedical word vectors and shows each with a score between 0 and 1. TF-IDF, word2vec, fastText, ELMo and BERT were among the models tested first. One main measure was recall within the top 200 matches, a number NICE judged manageable to review.

The evaluation shows how meaning-based search can expose gaps in a manual one. The first reference set, built from NICE's existing searches, held 246 true matches across 74 sample queries, and after experts reviewed the model's results the gold standard held 875. The record still lists as a risk that NORMA may miss valid recommendations and states that manual oversight is needed.

Measuring whether a search finds what matters

Two measures from information retrieval carry over directly to evidence searching. Both compare what a search returned with what a full assessment counts as relevant.

$$P = \frac{TP}{TP + FP} \qquad R = \frac{TP}{TP + FN}$$

where $P$ is precision, $R$ is recall, $TP$ is the number of relevant records retrieved, $FP$ is the number of irrelevant records retrieved and $FN$ is the number of relevant records missed. Manning and colleagues note that professional searchers seek the highest possible recall and tolerate low precision to get it, which is the position of a reviewer who cannot afford to miss an eligible economic evaluation.

The ONS team judged that ROC curves could look overly optimistic, because a query had fewer than 10 true matches among roughly 20,000 recommendations, and added precision-recall curves and ranking measures. Benchmarks can mislead too. Thakur and colleagues showed that many test collections were judged only on candidates found by lexical systems, so relevant records found only by a dense model are scored as irrelevant, and on TREC-COVID 31.8% of one dense model's top-10 results had never been judged, against 6.4% for BM25.

Worked example with a cosine score and a review search

All vectors, counts and records here are hypothetical and illustrative. A query on the cost per QALY of a statin is embedded as $q = (0.8, 0.6, 0)$. Abstract A, a cost-utility analysis of a lipid-lowering drug, has $a = (0.6, 0.8, 0)$, and abstract B, a trial of the same drug reporting only cholesterol levels, has $b = (0, 0.6, 0.8)$. All three vectors have length 1.

$$\cos(q, d) = \frac{q \cdot d}{\lVert q \rVert \, \lVert d \rVert}$$

where $\cos(q, d)$ is the cosine similarity between query vector $q$ and document vector $d$, $q \cdot d$ is their dot product and $\lVert q \rVert$ and $\lVert d \rVert$ are their lengths. Abstract A scores $s_A = 0.8 \times 0.6 + 0.6 \times 0.8 + 0 \times 0 = 0.96$ and abstract B scores $s_B = 0.8 \times 0 + 0.6 \times 0.6 + 0 \times 0.8 = 0.36$, so A ranks first although it never uses the word statin.

A reviewer then compares two searches against a reference set of 40 relevant economic evaluations.

Step 1. Keyword recall is 28 / 40 = 0.70 and precision is 28 / 300 = 0.093.

Step 2. Semantic recall is 34 / 40 = 0.85 and precision is 34 / 300 = 0.113.

Step 3. The combined list finds 37 / 40 = 0.925 of the relevant records, with precision 37 / 450 = 0.082.

Step 4. Combining costs 150 more records to screen and still misses 3 studies, and any relevant study outside the reference set does not show in these figures at all, the blind spot Thakur and colleagues found in benchmarks. The semantic list adds candidates and does not replace the documented strategy.

Choices that change what a semantic search returns

Several design choices sit between a query and its results. Each should be reported when a search supplies evidence for a submission or a model.

• Vocabulary version. PubMed notes that MeSH is updated annually, so a mapped search run in different years can return different sets.
• Embedding model. Jin and colleagues note that dense models trained on general data do not generalise well to domain-specific tasks, and Thakur and colleagues found BM25 a strong baseline across 18 datasets.
• Document units. The ONS team improved results by grouping very short recommendations or adding guideline titles and headings, since some recommendations named their subject only in a heading.
• Hybrid ranking. Karpukhin and colleagues ranked passages by BM25 plus 1.1 times the dense score, which helped on some datasets.
• Cut-off. A top-N list, such as the top 200 used to evaluate NORMA, trades recall against reviewing time.

Semantic search beside related ideas

Semantic search is the retrieval task, a text embedding is one representation of meaning for it, and a vector database is the index that makes nearest-neighbour lookup fast at scale. Retrieval-augmented generation puts a semantic search in front of a large language model, so a retrieval error can pass into the generated answer. Ontologies and knowledge graphs state meaning explicitly, as MeSH does, rather than learning it from text. A Boolean strategy for a systematic review is written to be reported and repeated exactly, and semantic ranking supplements it alongside screening and data extraction tools within artificial intelligence in health.

NICE's position statement on AI in evidence generation (ECD11, 2024) lists generating search strategies and screening records among possible uses in evidence synthesis, and says this is not an endorsement. It asks submitting organisations to keep a capable and informed human in the loop and to declare any use of AI and how it was used, and it notes that using NICE content for AI purposes needs NICE approval and a licence.

Limits and common misreadings

Most errors come from treating a similarity score as a judgement of relevance or truth. Four recur in evidence work.

1. Reading similarity as relevance. Jin and colleagues report that a search for the gene MAP3K3 also returned articles mentioning only MAP3K7, and the same drift can pull in a trial of the wrong dose or a model in the wrong population.
2. Reading a score as evidence quality. A cosine score of 0.96 says two texts are about similar things, and nothing about whether a study is valid.
3. Treating semantic ranking as exhaustive. NORMA's own record states that it may miss valid recommendations, so a semantic list cannot replace a documented search in a health technology assessment submission.
4. Expecting an explanation. Jin and colleagues note that dense similarity scores are not explainable, unlike a keyword match, so the model and its version need reporting for the search to be checked.

Sources

• Baines D, Ashley M. Health Economics Ontology (HEO), version 1.3.1. NCBO BioPortal, accessed 2 November 2026. https://bioportal.bioontology.org/ontologies/HEO
• Guha R, McCool R, Miller E. Semantic search. In: Proceedings of the 12th International Conference on World Wide Web (WWW 2003). ACM; 2003:700-709. https://doi.org/10.1145/775152.775250
• Jin Q, Kim W, Chen Q, et al. MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics. 2023;39(11):btad651. https://doi.org/10.1093/bioinformatics/btad651
• Karpukhin V, Oguz B, Min S, et al. Dense passage retrieval for open-domain question answering. In: Proceedings of EMNLP 2020. Association for Computational Linguistics; 2020:6769-6781. https://arxiv.org/abs/2004.04906
• Manning CD, Raghavan P, Schütze H. Introduction to Information Retrieval, section 8.3. Cambridge University Press; 2008. https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-unranked-retrieval-sets-1.html
• Miles A, Bechhofer S, editors. SKOS Simple Knowledge Organization System Reference. W3C Recommendation, 18 August 2009. https://www.w3.org/TR/skos-reference/
• National Institute for Health and Care Excellence. Use of AI in evidence generation: NICE position statement (ECD11). 15 August 2024. https://www.nice.org.uk/corporate/ecd11
• Cabinet Office, Department for Science, Innovation and Technology and Government Digital Service. NICE: NORMA (NICE-ONS Recommendation Matching Algorithm). Algorithmic transparency record, GOV.UK, 17 December 2024. https://www.gov.uk/algorithmic-transparency-records/nice-norma-nice-ons-recommendation-matching-algorithm
• National Library of Medicine. MeSH descriptors D003362 (Cost-Benefit Analysis) and D000094703 (Cost-Effectiveness Analysis). Accessed 2 November 2026. https://id.nlm.nih.gov/mesh/D000094703
• National Library of Medicine. PubMed User Guide. Accessed 2 November 2026. https://pubmed.ncbi.nlm.nih.gov/help/
• ONS Data Science Campus. Supporting healthcare guidance recommendations using natural language processing. Accessed 2 November 2026. https://datasciencecampus.ons.gov.uk/supporting-healthcare-guidance-recommendations-using-natural-language-processing/
• Reimers N, Gurevych I. Sentence-BERT: sentence embeddings using Siamese BERT-networks. In: Proceedings of EMNLP 2019. Association for Computational Linguistics; 2019. https://arxiv.org/abs/1908.10084
• Thakur N, Reimers N, Rücklé A, Srivastava A, Gurevych I. BEIR: a heterogeneous benchmark for zero-shot evaluation of information retrieval models. arXiv:2104.08663; 2021. https://arxiv.org/abs/2104.08663