Text Embedding

Text Embedding for Evidence Search and Screening

A text embedding is a point in a space of a few hundred dimensions, placed there by a trained model for a word, sentence or paragraph, and the distance between two points stands in for how alike their meanings are. Health economists meet the idea when software ranks records for a systematic review, finds NICE recommendations that overlap with a guideline under review, or retrieves passages for a language model. The method finds related text even when no words are shared, yet the score measures resemblance only, so a close match says nothing about whether a statement is correct or still in force. The term is unrelated to the embedding effect in contingent valuation, where stated willingness to pay fails to rise with the scope of the good being valued.

Words as learned vectors rather than codes

Mikolov and colleagues observed that many language systems treated words as atomic units, indices in a vocabulary with no notion of similarity between them. Distributed representations replace each index with a dense vector learned from unlabelled text. Their two architectures, continuous bag-of-words and skip-gram, learned high-quality vectors from 1.6 billion words in less than a day.

The skip-gram model learns by predicting the words that surround each word. Zhang and colleagues write its objective as follows.

$$J = \frac{1}{T} \sum_{t=1}^{T} \sum_{c \in C_t} \log p(w_c \mid w_t), \qquad p(w_c \mid w_t) = \frac{\exp(u_{w_t}^{\top} v_{w_c})}{\sum_{j=1}^{W} \exp(u_{w_t}^{\top} v_{w_j})}$$

where $w_t$ is the word at position $t$ of a training text of $T$ words, $C_t$ is the set of words around $w_t$, $w_c$ is one of them, $u_{w_t}$ and $v_{w_c}$ are the learned vectors of the two words, $W$ is the vocabulary size, $p(w_c \mid w_t)$ is the probability of seeing $w_c$ near $w_t$ and $J$ is the objective that training maximises. Words that appear in similar company end up with similar vectors, so "hazard" and "risk" can sit close together without any rule saying they are related.

The authors measured closeness by cosine distance and suggested that such vectors could improve information retrieval and question answering, the uses that matter most for evidence work.

Biomedical vectors trained on PubMed and MeSH

Vectors learned from news text carry the usage of news, so biomedical work needs biomedical training text. Zhang and colleagues built BioWordVec from the titles and abstracts of 27,599,238 PubMed articles together with the Medical Subject Headings (MeSH). The fastText subword model they used builds each word vector from vectors for its character n-grams, so a rare compound such as "deltaproteobacteria" inherits meaning from "proteo" and "bacteria". Each released vector has 200 dimensions.

The MeSH part brings a knowledge graph into the training data. The authors ran SPARQL queries on the MeSH RDF data to build a graph of 28,436 main headings and 52,013 relations, then turned random walks along the graph into word sequences for training. They treated every edge as undirected and did not distinguish the types and direction of the relations, so a broader-narrower link counts the same as any other once the graph is folded into vectors. That typing is what an ontology keeps.

With the same word2vec model, vectors trained on PubMed and PubMed Central clearly outperformed vectors trained on Google News, and the cosine similarity between "mycosis" and "histoplasmosis" was 0.353 with the Google News vectors and 0.706 with BioWordVec. The authors add that it is difficult to say how high an absolute cosine score should be, so they judged the vectors by their correlation with expert similarity ratings.

Sentence and passage embeddings

A recommendation or an abstract needs one vector for the whole text. The simplest method averages the word vectors, which Zhang and colleagues describe as a common baseline for sentence similarity. BERT reads sentences in context but compares two sentences by passing both through the network together, and Reimers and Gurevych calculate that finding the most similar pair among 10,000 sentences this way takes about 50 million inference computations, roughly 65 hours.

Their Sentence-BERT model trains two copies of BERT with shared weights so that each sentence gets its own fixed-size vector, stored once and compared by cosine similarity. The same task then took about 5 seconds of embedding and about 0.01 seconds of comparison. Averaging the outputs of an untuned BERT gave poor sentence vectors, worse on semantic similarity tasks than averaged GloVe word vectors, so the encoder has to be trained for the purpose.

The same design sits inside retrieval-augmented generation, covered on its own page in this set. Lewis and colleagues encoded documents and questions with two BERT encoders, embedded each passage once and ranked passages by the inner product of the two vectors.

Measuring closeness with cosine similarity

Cosine similarity compares the direction of two vectors and ignores their length. Reimers and Gurevych found that negative Manhattan and Euclidean distances gave roughly the same results.

$$\cos(\mathbf{a}, \mathbf{b}) = \frac{\sum_{k=1}^{d} a_k b_k}{\sqrt{\sum_{k=1}^{d} a_k^2} \, \sqrt{\sum_{k=1}^{d} b_k^2}}$$

where $\mathbf{a}$ and $\mathbf{b}$ are two embedding vectors, $a_k$ and $b_k$ are their values in dimension $k$, $d$ is the number of dimensions and $\cos(\mathbf{a}, \mathbf{b})$ is the cosine similarity. The score is 1 when the vectors point the same way and 0 when they are at right angles. When every vector is scaled to length 1 on storage, the cosine equals the plain sum of products.

How NICE uses embeddings to find related recommendations

The UK algorithmic transparency record for NORMA, published on 17 December 2024, describes an internal NICE tool built with the Office for National Statistics Data Science Campus. Rule-based pattern matching extracts the recommendations from each clinical guideline, and a pre-trained biomedical model called Bio-embedding-intrinsic, which the record links to the BioWordVec paper, turns each one into the average vector of its nouns. A query is embedded the same way and compared with every stored vector by cosine similarity.

The Data Science Campus report explains the choices. Averaging the nouns gave consistently better results than averaging every word, and BERT sentence models were set aside because they needed a graphics processor to run at a reasonable pace and did not suit searches of one or two keywords.

The report adds that the scores of true and false matches overlapped so widely that recovering most known matches would have meant including thousands of non-matches.

Screening records for a systematic review

Machine learning tools for title and abstract screening in evidence synthesis first turn each record into numbers, often an embedding. Ali and colleagues tested TF-IDF, Doc2Vec and Sentence-BERT features with several classifiers on real systematic review datasets from education. Work saved at 95% recall ranged from 1.2% to 75.6% across datasets, so the same tool could save most of the effort on one review and almost none on another.

The separability of relevant and irrelevant records in embedding space, measured by the Davies-Bouldin score, strongly predicted the work saved (overall R = 0.81). The score is a ratio of within-cluster to between-cluster distance, so smaller values mean the relevant records stand further apart, an idea close to cluster analysis. The authors note that education may lack the standardised vocabularies of medicine, so health reviews may behave differently, and that their datasets were few.

Worked example ranking recommendations by similarity

The vectors in this example are illustrative, with three dimensions in place of the 200 used by BioWordVec. A surveillance query about lipid-lowering treatment is embedded as $\mathbf{q} = (0.6, 0.8, 0.0)$, and three stored recommendations are $\mathbf{r}_1 = (0.8, 0.6, 0.0)$ on statin dose, $\mathbf{r}_2 = (0.48, 0.6, 0.64)$ on lipid monitoring and $\mathbf{r}_3 = (0.0, 0.28, 0.96)$ on smoking cessation.

Step 1. Each vector has length 1. For the lipid monitoring vector the squares are 0.2304 + 0.36 + 0.4096 = 1.0, and the other three also sum to 1, so each cosine is the sum of products.

Step 2. The scores are 0.6 × 0.8 + 0.8 × 0.6 = 0.96 for statin dose, 0.6 × 0.48 + 0.8 × 0.6 = 0.768 for lipid monitoring and 0.8 × 0.28 = 0.224 for smoking cessation, which gives the ranking.

Step 3. A fourth recommendation is embedded NORMA-style as the mean of two noun vectors, "statin" (0.8, 0.6, 0.0) and "intolerance" (0.6, 0.0, 0.8), giving $\mathbf{m} = (0.7, 0.3, 0.4)$ with length $\sqrt{0.74} = 0.8602$.

$$\cos(\mathbf{q}, \mathbf{m}) = \frac{0.6 \times 0.7 + 0.8 \times 0.3 + 0.0 \times 0.4}{1 \times 0.8602} = \frac{0.66}{0.8602} = 0.767$$

where $\mathbf{q}$ is the query vector, $\mathbf{m}$ is the averaged vector and 0.8602 is its length. Without dividing by the length the score would be 0.66, so averaged vectors must be normalised before their scores are compared.

Step 4. For screening, suppose a search returns 2,000 records of which 40 are relevant, and ranking by similarity to known includes finds 38 of them in the first 500 records read. Recall is 38 out of 40, or 95%, the share screened is 500 out of 2,000, or 25%, and work saved at 95% recall, as Ali and colleagues define it, is 95% minus 25%, or 70%.

None of these scores shows that the statin recommendation is current or correct. Similarity ranks text for a human to read, and the reading decides.

Choices that change the result

Several settings sit between a document and its score. Each belongs in the methods of an analysis that relies on embeddings.

• Training corpus. Biomedical vectors outperformed news vectors on medical term pairs in the BioWordVec validation.
• What text is embedded. Very short recommendations relied on neighbouring text, and the Data Science Campus found that adding the guideline title and headings as context improved results.
• Model version. NORMA's model has stayed fixed, and the Data Science Campus suggested a refresh every two to three years, so a reported search needs the model name and version.

Text embeddings beside related ideas

An embedding is a building block of artificial intelligence in health. The transformer behind most large language models, the subject of another page in this set, uses learned embeddings to turn input tokens into vectors, a link to deep learning in health. Semantic search ranks documents by embedding similarity, and a vector database stores the vectors and finds near neighbours quickly.

An ontology such as HealthEconomics.wiki's HEO records that one concept is a type of another, while an embedding of the two definitions yields only a distance between them.

Misreadings that lead to wrong conclusions

Most errors come from treating a similarity score as a judgement about the evidence.

1. Reading closeness as truth. A score shows that two texts use language in similar ways. Nothing in the vector records whether a recommendation is still in force, and the Data Science Campus proposed a separate flag for whether each recommendation is live.
2. Reading the score as a probability. Cosine values have no fixed scale across models, and Zhang and colleagues note that the right absolute level is hard to judge, so a threshold tuned on one model does not transfer to another.
3. Assuming nothing is missed. NORMA misses a small share of recommendations, and work saved in screening varies widely between reviews, so recall needs checking on each project.
4. Using general vectors for specialist text. News-trained vectors gave weaker similarity on medical terms, and whole-word models such as word2vec learn rare or unseen terms poorly.

Sources

• Ali F, Tan AS, Wang SJ. Can machine learning help accelerate article screening for systematic reviews? Yes, when article separability in embedding space is high. Research Synthesis Methods. 2025;16:194-210. https://doi.org/10.1017/rsm.2024.16
• Cabinet Office, Department for Science, Innovation and Technology and Government Digital Service. NICE: NORMA (NICE-ONS Recommendation Matching Algorithm). Algorithmic transparency record, published 17 December 2024. https://www.gov.uk/algorithmic-transparency-records/nice-norma-nice-ons-recommendation-matching-algorithm
• Data Science Campus, Office for National Statistics. Supporting healthcare guidance recommendations using natural language processing. Published 31 January 2022. https://datasciencecampus.ons.gov.uk/supporting-healthcare-guidance-recommendations-using-natural-language-processing/
• Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.11401v4. https://arxiv.org/abs/2005.11401
• Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. arXiv:1301.3781v3, 2013. https://arxiv.org/abs/1301.3781
• Reimers N, Gurevych I. Sentence-BERT: sentence embeddings using Siamese BERT-networks. Proceedings of EMNLP-IJCNLP 2019. https://aclanthology.org/D19-1410/ (arXiv:1908.10084)
• Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in Neural Information Processing Systems 30 (NIPS 2017). arXiv:1706.03762v7. https://arxiv.org/abs/1706.03762
• Zhang Y, Chen Q, Yang Z, Lin H, Lu Z. BioWordVec, improving biomedical word embeddings with subword information and MeSH. Scientific Data. 2019;6:52. https://doi.org/10.1038/s41597-019-0055-0