Record Linkage

Record Linkage of Health Records for Costs and Outcomes

Record linkage is the data preparation step that decides which records in two health datasets belong to the same person, so that a trial cohort or a disease register can be joined to hospital episodes and death registrations. It has its own statistics, because the identifiers that carry the join, such as names and dates of birth, are often mistyped, missing or changed over time. For health economists it matters because many resource use, cost and survival inputs now come from linked administrative data, and every error in the join passes into those inputs. A missed link can make a patient look as if no admission ever happened, and a false link can attach another person's costs or death to the wrong record.

Why health economists link patient records

The NICE real-world evidence framework (ECD9, 2022) says linkage is often done to extend the information available on individual patients, for example by combining a prospective cohort with hospital discharge or mortality records. Among its examples from past NICE guidance is a technology appraisal of benralizumab for severe eosinophilic asthma that estimated economic burden from CPRD GOLD data linked to Hospital Episode Statistics (HES).

The 2015 data resource profile of the Clinical Practice Research Datalink (CPRD) describes patient-level data from consenting English practices linked through a trusted third party to HES, Office for National Statistics mortality data, deprivation scores and disease registries. A cost-of-illness study that needs primary care contacts, admissions and deaths for the same people depends on such linkage. So does extending follow-up of a clinical trial through routine records, or adding stage and survival from a cancer registry to hospital costs.

Deterministic linkage on shared identifiers

Deterministic linkage applies fixed rules, so a pair is a match if the chosen identifiers agree and a non-match if they do not. Harron and colleagues give the example of hospital admission records for the same person in the UK, linked over time by a three-step deterministic algorithm on combinations of NHS number, date of birth, sex and postcode. A single personal identity number used across sectors, as in the Scandinavian countries, allows linkage with marginal error, but elsewhere identifiers tend to belong to one sector. Exact rules then miss matches whenever a value is mistyped or missing, while false matches stay rare because unrelated people seldom agree on a full set of identifiers by chance.

Probabilistic linkage and the Fellegi and Sunter model

Probabilistic linkage weighs the evidence from several imperfect identifiers. Fellegi and Sunter set out the theory in 1969. Each candidate pair has a comparison vector recording whether each field agrees, and the method compares how likely that pattern is among true matches and among true non-matches. The m-probability is the chance that a field agrees when the records belong to the same person, and the u-probability is the chance that it agrees when they belong to different people.

$$w_k = \log_2\left(\frac{m_k}{u_k}\right) \text{ if field } k \text{ agrees}, \qquad w_k = \log_2\left(\frac{1-m_k}{1-u_k}\right) \text{ if it disagrees}, \qquad W = \sum_{k} w_k$$

Here $w_k$ is the weight for field $k$, $m_k$ and $u_k$ are its m- and u-probabilities, and $W$ is the total match weight of the pair. Adding the weights assumes that agreement on one field is independent of agreement on another within matched and within unmatched pairs, a conditional independence assumption that Fellegi and Sunter state explicitly. Sayers and colleagues point out that it is often violated, for example by postcode and street name. With base 2 logarithms, one unit of weight doubles the likelihood ratio.

Pairs above an upper threshold are links, pairs below a lower threshold are non-links, and pairs between them are possible links sent for clerical review. Fellegi and Sunter proved that ordering pairs by $m/u$ and cutting at two thresholds minimises the share left for review at chosen error rates, and noted that the rule is equivalent to a likelihood ratio test. Sayers and colleagues add a diagnostic test analogy, in which the m-probability plays the part of sensitivity and the u-probability that of one minus specificity.

A weight is not yet a probability of a match. Bayes theorem converts it once the prior chance of a match is set.

$$\pi = \frac{E}{N_A N_B}, \qquad \text{posterior odds} = 2^{W} \times \frac{\pi}{1-\pi}, \qquad P(\text{match}) = \frac{\text{posterior odds}}{1 + \text{posterior odds}}$$

Here $\pi$ is the prior probability that a random pair is a match, $E$ is the expected number of matches, and $N_A$ and $N_B$ are the numbers of records in the two files. The m- and u-probabilities are rarely known and are estimated from previous experience, a gold-standard subset or the expectation maximisation algorithm. The README of Splink, an open-source Python package, describes a Fellegi and Sunter model estimated in this last way.

Blocking, cleaning and partial agreement

Comparing every possible pair soon becomes impractical. Sayers and colleagues note that two files of 10,000 people give 100,000,000 candidate pairs. Blocking compares only records that share a cheap feature, such as region or year of birth, and Fellegi and Sunter already discussed its effect on error levels. A person whose blocking field is wrong can then never be linked.

Cleaning before the merge removes avoidable disagreement. The steps Sayers and colleagues list include unifying case and date formats, removing titles and punctuation, handling nicknames and using the checksum in NHS numbers to find invalid values. String comparators score partial agreement, so that "Steele" and "Steel" count as close. Edit distance counts the insertions, deletions and substitutions needed to turn one string into another, and Jaro-Winkler gives extra weight to agreement at the start.

Worked example with a trial cohort and hospital records

All numbers in this example are illustrative. A trial of a new asthma treatment has 2,000 participants, and the analyst wants their hospital admissions from national hospital data to cost the two arms. The trial file holds no NHS numbers, so records are linked probabilistically after blocking on year of birth, using surname, day and month of birth, and postcode district.

1. Set the field weights. The m- and u-probabilities are 0.95 and 0.01 for surname, 0.98 and 0.003 for day and month of birth, and 0.85 and 0.02 for postcode district. The agreement weights are $\log_2(0.95/0.01) = \log_2(95) = 6.57$, $\log_2(0.98/0.003) = \log_2(326.67) = 8.35$ and $\log_2(0.85/0.02) = \log_2(42.5) = 5.41$. The disagreement weights are $\log_2(0.05/0.99) = -4.31$, $\log_2(0.02/0.997) = -5.64$ and $\log_2(0.15/0.98) = -2.71$.

2. Score three candidate pairs. Pair A agrees on all three fields, so $W = 6.57 + 8.35 + 5.41 = 20.33$. Pair B agrees on date and postcode but not surname, as when a participant has changed surname, so $W = -4.31 + 8.35 + 5.41 = 9.45$. Pair C agrees on surname only, so $W = 6.57 - 5.64 - 2.71 = -1.78$.

3. Convert to probabilities. In one block the trial file has 40 people and the hospital file 5,000, with 36 matches expected. The prior is $\pi = 36/(40 \times 5000) = 0.00018$, and the prior odds are also about 0.00018. For pair A the posterior odds are $O_A = 2^{20.33} \times 0.00018 \approx 237$, a match probability of 237/238 = 0.996. For pair B they are $O_B = 2^{9.45} \times 0.00018$, or 699 × 0.00018 = 0.126, a probability of 0.126/1.126 = 0.112. Pair C is near nil.

4. Interpret. Pair A is linked and pair C rejected. Pair B is a true match in this story, yet sits at about 11%, so without clerical review it becomes a missed link. Surname changes are commoner in some groups, so such misses are rarely spread evenly.

5. Follow the error into the economics. Suppose the true mean hospital cost is GBP 1,200 per participant in the control arm and GBP 800 in the treatment arm, and missed links lose 10% of admission costs in both arms. The observed means become 1,200 × 0.9 = 1,080 and 800 × 0.9 = 720, so the observed hospital saving is 1,080 − 720 = 360 against a true GBP 400. If the treatment costs GBP 2,000 per participant, the true incremental cost is 2,000 − 400 = 1,600 and the observed one 2,000 − 360 = 1,640. With a gain of 0.08 QALYs the ICER is 1,600/0.08 = GBP 20,000 per QALY in truth and 1,640/0.08 = GBP 20,500 as observed. Missed links falling equally on both arms still bias the result, here against a treatment whose value lies in avoided admissions.

How linkage error biases costs and outcomes

Linkage error takes two forms, missed links between records of the same person and false links between records of different people. Doidge and Harron show that the damage depends on how the link is used. When the existence of a link carries meaning, as when mortality is inferred from linkage to a register of deaths, missed links cause false-negative misclassification and false links false-positive misclassification, and the two can partly offset each other. When the link only brings in extra variables, missed links become missing data, and dropping unlinked records can cause selection bias unless the missed links are missing completely at random.

A further problem is specific to linked data. Two hospital admission records may count as one person admitted twice if linked and as two people admitted once if not. Doidge and Harron call this splitting and merging, and it distorts readmission rates and any cost per person built from several records. They argue that bias depends on how errors are distributed across the variables of interest, so an overall error rate is not enough to judge a dataset.

Harron and colleagues report that data quality varies with age, sex, ethnicity and health status, which can lower match rates in more vulnerable or deprived populations and distort subgroup and equity analyses. The match rate helps only when the number that should link is known, and the sensitivity and positive predictive value of a linkage describe its accuracy without showing how a result is biased.

Privacy, trusted third parties and reporting

Linkage needs identifiers, which raise confidentiality risks. Harron and colleagues describe separating linkage from analysis as generally regarded as best practice, so that a trusted third party sees identifiers while researchers receive only de-identified attribute data. The price is that linkage quality becomes a black box to the analyst, so they call for linkers to share details such as which identifiers produced each match, as NHS Digital did with a match rank for each linked hospital record.

Privacy-preserving record linkage compares encrypted identifiers. A plain hash makes "John" and "Jon" look completely different, so match-keys, built from parts of each identifier, and Bloom filters, which split a string into two-character pieces before encoding, are used to allow approximate comparison. Harron and colleagues note that errors in the original identifiers are carried into the encrypted ones, which makes such linkage harder to evaluate. Access rules belong to data governance and differ between jurisdictions.

The NICE framework (ECD9) warns that imperfect linkage could exclude patient records or lead to misclassification. When sources are linked it asks evidence developers to report who did the linkage, whether the method was deterministic or probabilistic with the variables used, and the performance characteristics of the linkage.

Record linkage beside entity resolution and linked datasets

Record linkage is the statistical tradition of matching people across files, and deduplication applies the same methods within one file. The broader task of finding any records or graph nodes that stand for the same real-world thing is entity resolution.

The output is a linked dataset, a patient-level file built from several sources. The NICE framework separates linkage from pooling, which extends sample size or coverage, and the semantic web sense of Linked Data is a separate idea. Linked records are real-world data, and their data quality still has to be judged for the decision at hand.

Common misreadings of linked data

One error is to treat the linkage as perfect. Harron and colleagues stress that linkage error in administrative data is inevitable because identifiers are imperfect and change over time, and that even small amounts can bias results substantially. A related error reads a missing link as an absent event, when the participant may simply not have been linked.

Threshold choice is a further source of error. Sayers and colleagues describe the placement of thresholds as a matter of trial and error, trading more true matches against more clerical review, and a sensitivity analysis that varies the m- and u-probabilities shows how far a result depends on them. A deterministic rule, though exact, mostly turns its errors into missed links.

Sources

• Fellegi IP, Sunter AB. A theory for record linkage. Journal of the American Statistical Association. 1969;64(328):1183-1210. https://doi.org/10.1080/01621459.1969.10501049
• Sayers A, Ben-Shlomo Y, Blom AW, Steele F. Probabilistic record linkage. International Journal of Epidemiology. 2016;45(3):954-964. https://doi.org/10.1093/ije/dyv322
• Harron K, Dibben C, Boyd J, Hjern A, Azimaee M, Barreto ML, Goldstein H. Challenges in administrative data linkage for research. Big Data & Society. 2017;4(2):2053951717745678. https://doi.org/10.1177/2053951717745678
• Doidge JC, Harron KL. Reflections on modern methods: linkage error bias. International Journal of Epidemiology. 2019;48(6):2050-2060. https://doi.org/10.1093/ije/dyz203
• Herrett E, Gallagher AM, Bhaskaran K, Forbes H, Mathur R, van Staa T, Smeeth L. Data resource profile: Clinical Practice Research Datalink (CPRD). International Journal of Epidemiology. 2015;44(3):827-836. https://doi.org/10.1093/ije/dyv098
• National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). Published 23 June 2022. https://www.nice.org.uk/corporate/ecd9/chapter/assessing-data-suitability
• moj-analytical-services. Splink README. GitHub repository, accessed 2 November 2026. https://github.com/moj-analytical-services/splink