Named Entity Recognition

Named Entity Recognition in Health Records and Evidence Reviews

Within natural language processing, named entity recognition (NER) is the information extraction step that marks each run of words naming something of a chosen type, such as a medicine, a diagnosis, a test or a hospital stay. The marked spans can then be counted and analysed much like a coded field. Health economists meet the task because health records keep large portions of clinical information as free text, and trial reports describe populations and outcomes in prose. A recognised mention is still only a piece of text with a label, so its accuracy has to be measured against human annotation before any count built on it enters a cost model or an HTA submission. Entity resolution, and record linkage as its cross-dataset form, is a different task that matches records about the same person or organisation.

Newswire origins and clinical entity types

The named entity task was devised for the sixth Message Understanding Conference (MUC-6), one of a series of evaluations of information extraction systems. Grishman and Sundheim describe it as identifying the names of all the people, organisations and geographic locations in a text, with time, currency and percentage expressions added in the final specification. Most sites scored over 90% for both recall and precision, and the best system reached 96% recall and 97% precision, although every test text came from the Wall Street Journal.

The CoNLL-2003 shared task fixed a four-type scheme and a one-word-per-line tagged data format. Tjong Kim Sang and De Meulder's types were persons, locations, organisations and a miscellaneous group, applied to English Reuters news stories and German newspaper text. The best English system reached an F1 of 88.76 and the best German system 72.41.

Health applications change the entity types. The 2010 i2b2 concept extraction task, as Hu and colleagues describe it, labels medical problems, treatments and tests, so that "lung cancer" is a problem, "chemotherapy" a treatment and "CT scan" a test. The EBM-NLP corpus of Nye and colleagues marks the participants, interventions and outcomes in about 5,000 abstracts of randomised controlled trials, and its outcome labels include resource use. Du and colleagues treated data extraction for health economics and outcomes research reviews as NER, with types such as study location and pneumococcal disease type.

Tagging text one word at a time

Most systems treat NER as sequence labelling, in which every word receives a tag. In the CoNLL-2003 data, O marks a word outside any entity and I-XXX a word inside an entity of type XXX, with B-XXX for the first word of an entity that directly follows another of the same type. Du and colleagues used the related BIO format, in which B marks the first word of every entity, and the illustrative sentence below is tagged that way with types a costing study might choose.

Hu and colleagues note that early clinical systems relied on lexical resources and rules, and that tools such as cTAKES and CLAMP add machine learning. Du and colleagues call the conditional random field a classic statistical sequence model widely used for NER before deep learning, and their neural tagger fed a bidirectional LSTM into a conditional random field that decoded the labels jointly. Transformer models such as BioClinicalBERT are fine-tuned on annotated notes, and large language models can be prompted to mark entities with few or no annotated examples.

Because annotated clinical text is scarce, many biomedical tools match text against a dictionary of known terms, which needs no manual annotation. Kraljevic and colleagues list the problems this meets in health records, including spelling mistakes, variation such as "kidney failure" against "failure of kidneys", and ambiguous strings such as "hr", which may mean hour, heart rate or neither.

Linking recognised names to SNOMED CT and other terminologies

A labelled span is easier to analyse once it is tied to one concept in a terminology. Kraljevic and colleagues call the combined task named entity recognition plus linking (NER+L) and built MedCAT, an open-source toolkit that links mentions to vocabularies such as SNOMED CT and UMLS, in which one concept can carry several names, such as heart failure, myocardial failure, weak heart and cardiac failure.

MedCAT also adds meta-annotations, because a concept may need to be ignored if it occurred in the past or is negated. It was trained self-supervised on the whole electronic patient record of King's College Hospital, about 17 million documents with 8.8 billion tokens, refined with clinician annotations and shared with University College London Hospitals and South London and Maudsley NHS Foundation Trusts. When the King's College Hospital model was shared with University College London Hospitals, average F1 was 0.044 lower than at King's College Hospital.

A linked code can be matched to a code list, a unit cost or a model health state. The same step could tie a recognised term to a published concept scheme such as HealthEconomics.wiki's HEO, a SKOS concept scheme published in OWL.

Uses in real-world evidence and evidence synthesis

Free text matters for real-world evidence because some details are recorded only in notes, and the NICE real-world evidence framework (ECD9) gives free-text data on cancer stage or biomarkers as an example. It notes that extracting data elements from unstructured data, such as doctor's notes, may help with data availability and granularity, and NICE added sections on validating and reporting such methods in September 2025. The framework asks for the extraction rules, including how negation and temporal context were handled. For algorithmic methods it also asks for the model and version, how training and test sets were built, annotator agreement, performance metrics, performance in important subpopulations and the potential impact of model errors on the intended analysis.

NICE's position statement on the use of AI in evidence generation (ECD11), published on 15 August 2024, notes that natural language processing approaches are being used to generate structured data from unstructured real-world data. It requires AI methods used for real-world data extraction and curation to be reported in detail as part of the ECD9 data suitability assessment, and it calls for AI to augment rather than replace human involvement. It describes data extraction from published studies by large language models as less well established than their use in screening.

For a systematic review, NER is one way to automate data extraction. In the economic burden review studied by Du and colleagues, 24 articles held 252 annotated entity mentions and the best micro-averaged lenient F1 was 0.648, against 0.890 for a human papillomavirus prevalence review with 4,498 mentions. The authors attribute weak results for some elements mainly to the small number of annotated examples, and data in tables fell outside the study.

Scoring a tagger against human annotation

Accuracy is judged against a reference standard built by human annotators. The NICE framework lists positive predictive value, or precision, and sensitivity, or recall, among the metrics, together with the F1 score that combines them.

$$P = \frac{TP}{TP + FP}, \qquad R = \frac{TP}{TP + FN}, \qquad F_1 = \frac{2PR}{P + R} = \frac{2\,TP}{2\,TP + FP + FN}$$

where $TP$ is the number of entities found by the system that match an annotated entity, $FP$ is the number found that match none, $FN$ is the number of annotated entities the system missed, $P$ is precision, $R$ is recall and $F_1$ is their harmonic mean.

What counts as a match changes the score. CoNLL-2003 counted an entity as correct only on an exact match, while relaxed scoring accepts any overlap with the same type. In Hu and colleagues' tests, with five annotated examples in its prompt, GPT-4 reached an F1 of 0.861 under relaxed matching but 0.593 under exact matching on synthetic discharge summaries, against 0.901 and 0.785 for a fine-tuned BioClinicalBERT.

The reference standard has its own error. Between two medical students annotating 200 EBM-NLP abstracts, the kappa statistic was 0.71 for participants, 0.69 for interventions and 0.62 for outcomes.

Worked example scoring extracted hospital admissions

The numbers in this example are illustrative. An analyst wants to cost emergency admissions recorded in a cohort's free-text notes, and two clinicians annotate a validation sample mentioning 50 distinct admissions.

Step 1. The NER system flags 45 admissions in the same notes, and under relaxed matching 40 of them overlap an annotated admission. So $TP = 40$, the 5 unmatched flags give $FP = 5$ and the 10 missed admissions give $FN = 10$.

Step 2. Precision is 40 / 45 = 0.889 and recall is 40 / 50 = 0.800.

$$F_1 = \frac{2 \times 40}{2 \times 40 + 5 + 10} = \frac{80}{95} = 0.842$$

where 40 is the number of true positives, 5 the false positives and 10 the false negatives.

Step 3. Only 34 of the 40 matches have exactly the annotated boundaries, so exact precision is 34 / 45 = 0.756, exact recall is 34 / 50 = 0.680 and exact F1 is 68 / 95 = 0.716. A count of admissions needs only the relaxed score, while a drug dose copied into a model needs exact boundaries, and the NICE framework says appropriate metrics may vary by concept.

Step 4. With an illustrative unit cost of GBP 3,000 per admission, the annotated sample costs 50 × 3,000 = GBP 150,000 and the system's count costs 45 × 3,000 = GBP 135,000. The ratio 45 / 50 = 0.90 equals recall divided by precision, 0.800 / 0.889 = 0.90, so the cost estimate is 10% too low.

Step 5. Had the system produced 10 false positives in place of 5, it would have flagged 50 admissions and matched the total while still missing 10 real ones. A correct total can hide offsetting errors, so scores at the level of single mentions say more than totals, and the NICE framework asks for the sampling unit of evaluation, patient-level or instance-level, to be reported.

Choices that change the extracted data

Several decisions sit between a clinic letter and a number in a model. Each belongs in the methods of a study that uses NER.

• Entity scheme and guidelines. Hu and colleagues found that adding annotation guideline rules to the prompt raised F1 for both GPT models.
• Training text. Scores fell when MedCAT moved between hospitals, and the MUC-6 and CoNLL-2003 scores came from news text.
• Context handling. Negated, historical or family history mentions have to be separated from current ones before counting.
• Repeated mentions. Nye and colleagues note that abstracts often name the same intervention several times, so counting mentions can overstate the number of distinct items.

Boundaries with nearby tasks

NER finds and types mentions within a document. Entity linking, as in MedCAT, maps each mention to a terminology code, while entity resolution and record linkage decide which records across datasets refer to the same patient or provider.

Text classification for evidence synthesis labels whole records, for example as eligible for a review, whereas NER labels spans inside them. Large language models are one way to perform NER. Nye and colleagues suggest such extraction might eventually support automated knowledge-base construction for trials. The field sits under artificial intelligence in health, and the input is usually the electronic health record or published papers.

Misreadings and limits

Most errors come from treating extracted spans as coded data of known quality.

1. Reading a high F1 as valid data. A score covers one text type and one annotation scheme, and the MUC-6 results came from Wall Street Journal text alone.
2. Ignoring context. A drug that was stopped or a diagnosis that was ruled out is still a mention, which is why NICE asks how negation and temporal aspects were handled.
3. Assuming a language model reads entity types the way annotators do. Hu and colleagues found GPT models labelling abnormal laboratory results as medical problems, outside the task definition. Du and colleagues add that a sentence-level score can miss entities that are wrong at document level, giving hallucination as an example.
4. Overlooking data protection. Kraljevic and colleagues note that sending hospital text to an external annotation service may be prohibited under data protection legislation, and Hu and colleagues used only synthetic or anonymised reports for privacy reasons.

Sources

• Du J, Wang D, Lin B, et al. Use of deep learning-based NLP models for full-text data elements extraction for systematic literature review tasks. Scientific Reports. 2025;15:19379. https://doi.org/10.1038/s41598-025-03979-5
• Grishman R, Sundheim B. Message Understanding Conference-6: a brief history. Proceedings of COLING 1996, the 16th International Conference on Computational Linguistics. 1996:466-471. https://aclanthology.org/C96-1079/
• Hu Y, Chen Q, Du J, et al. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association. 2024;31(9):1812-1820. https://doi.org/10.1093/jamia/ocad259
• Kraljevic Z, Searle T, Shek A, et al. Multi-domain clinical natural language processing with MedCAT: the Medical Concept Annotation Toolkit. Artificial Intelligence in Medicine. 2021;117:102083. https://doi.org/10.1016/j.artmed.2021.102083
• National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). Published 23 June 2022, sections on unstructured data added September 2025. https://www.nice.org.uk/corporate/ecd9
• National Institute for Health and Care Excellence. Use of AI in evidence generation: NICE position statement (ECD11). Published 15 August 2024. https://www.nice.org.uk/corporate/ecd11
• Nye B, Li JJ, Patel R, et al. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018:197-207. https://aclanthology.org/P18-1019/
• Tjong Kim Sang EF, De Meulder F. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003. 2003:142-147. https://aclanthology.org/W03-0419/