Cluster Analysis

Cluster Analysis for Segmenting Patients, Costs and Health Systems Without a Predefined Outcome

Cluster analysis looks for groups in data when nobody has said in advance what the groups should be. In health economics it is used to split patient populations into segments with different care use and cost, to describe the high-cost minority, to find preference classes in stated-preference studies and to sort health systems into types. The main approaches are k-means, hierarchical clustering and the model-based latent class approach, and the answer depends on choices made by the analyst. Cluster analysis as a method for finding groups is distinct from cluster randomised trials and cluster sampling, where a "cluster" is a pre-existing unit such as a general practice.

Grouping without an outcome variable

Most statistical methods in economic evaluation predict an outcome, such as cost or survival, from covariates. Cluster analysis is unsupervised. It has no outcome variable and works only with the measurements that describe each unit, whether a patient, a hospital, a survey respondent or a country. Everything rests on a measure of how alike two units are, and Hastie, Tibshirani and Friedman stress that this can only come from subject-matter considerations and matters more than the choice of algorithm. For quantitative variables the most common choice is squared Euclidean distance.

$$d(x_i, x_{i'}) = \sum_{j=1}^{p} (x_{ij} - x_{i'j})^2$$

where $d(x_i, x_{i'})$ is the dissimilarity between units $i$ and $i'$, $x_{ij}$ is the value of variable $j$ for unit $i$, and $p$ is the number of clustering variables.

Combinatorial algorithms, including k-means and hierarchical clustering, assign each unit directly to a group without a probability model. Mixture models instead assume the population is made up of several component distributions, one per cluster, and fit that model by maximum likelihood or Bayesian methods; latent class analysis belongs to this family.

Where clustering is used in health economics

A population that looks homogeneous on average may contain groups with very different needs, costs or preferences. Four kinds of published application show how the method is used.

Patient segmentation. A systematic review by Yan and colleagues included 216 data-driven segmentation studies, in which latent class methods were most common (96 studies), followed by k-means (60) and hierarchical analysis (50). In England, Vuik, Mayer and Darzi applied k-means to six utilisation variables for 300,000 patients in primary and secondary care administrative data, after using hierarchical clustering on 10 random samples of 3,000 patients to judge the number of clusters. They found eight segments; one held 7% of the population but accounted for over a quarter of overall cost.

High-cost users. Davis and colleagues took the top 1% of continuously enrolled adult members of Kaiser Permanente Southern California by total healthcare expenditure in 2010 and applied latent class analysis to 53 hierarchical condition categories for 21,183 high-cost adults. Seven clinically distinct classes emerged, and relatively few patients stayed high cost over 4 years. Vuik and colleagues similarly split the high-risk top 5% of patients ranked by a risk score into four groups with different use of emergency, elective, outpatient and GP care.

Preference heterogeneity. In a discrete choice experiment, latent class models group respondents into classes with different preference weights. A latent class logit is a model-based mixture of choice models, in which classes differ in their coefficients on the observed choices, rather than a clustering of respondents on raw descriptors. Zhou, Thayer and Bridges reviewed 99 stated-preference studies in health using latent class analysis and found that over 90% used latent class logit models, model selection commonly used the Bayesian information criterion, the Akaike information criterion and the log-likelihood, and about 80% reported two or three classes.

Health system typologies. Reibling, Ariaans and Wendt combined indicators on supply, public-private mix, access regulation, primary care orientation and performance management for OECD countries, and a series of cluster analyses indicated at least five distinct types of healthcare system.

Choices that change the clusters

Cluster solutions depend on decisions taken before any algorithm runs. The same data can give different groupings depending on the variables, their scaling, the algorithm and the number of clusters kept, so a report should state each choice.

Variables and scaling

The variables define what "similar" means, since segmenting on care use and segmenting on diagnoses group patients differently. Variables with large variance dominate the distance, since each variable's influence on squared Euclidean distance is proportional to its variance. Rescaling amounts to a weighted distance.

$$D(x_i, x_{i'}) = \sum_{j=1}^{p} w_j (x_{ij} - x_{i'j})^2$$

where $D(x_i, x_{i'})$ is the weighted dissimilarity and $w_j$ is the weight on variable $j$; standardising sets $w_j$ in inverse proportion to the variance. Hastie, Tibshirani and Friedman warn that equal influence is not always right, because it can obscure groups that only some variables separate. Cost recorded in pounds will swamp counts of visits unless it is rescaled or kept out of the clustering and described afterwards.

Algorithms

The three main algorithms differ in how they form groups and what they need in advance. K-means alternates between assigning each unit to the nearest cluster mean and recomputing the means. It minimises the within-cluster sum of squares.

$$W(K) = \sum_{k=1}^{K} \sum_{i \in C_k} \| x_i - \bar{x}_k \|^2$$

where $W(K)$ is the within-cluster sum of squares with $K$ clusters, $C_k$ is the set of units in cluster $k$, $x_i$ is the vector of clustering variables for unit $i$, and $\bar{x}_k$ is the mean of cluster $k$. It needs $K$ and starting centroids in advance and can stop at a local minimum, so it is usually run from several starting values.

Agglomerative hierarchical clustering starts with each unit alone and repeatedly merges the two closest clusters, giving a tree (dendrogram) that can be cut at any height. Single linkage measures closeness by the nearest pair and tends to chain units together, complete linkage uses the furthest pair and group average the mean pairwise dissimilarity. Ward's method aims to minimise the within-cluster sum of squares, which is why Vuik and colleagues used it as a hierarchical counterpart to k-means.

Latent class analysis and other finite mixture models give each unit a posterior probability of belonging to each class.

$$\Pr(c_i = k \mid y_i) = \frac{\pi_k f_k(y_i)}{\sum_{h=1}^{K} \pi_h f_h(y_i)}$$

where $c_i$ is the class of unit $i$, $y_i$ its observed data, $\pi_k$ the population share of class $k$ and $f_k$ the distribution of the data within class $k$. Units are commonly allocated to their most probable class, but the probabilities show how uncertain each allocation is. In a Gaussian mixture with shared spherical variance, this soft assignment approaches k-means as the variance shrinks to zero.

Number of clusters and validation

The within-cluster sum of squares generally falls as clusters are added, even when measured on new data, so it cannot simply be minimised and cross-validation does not resolve the choice. Analysts look for a kink, or elbow, where the fall flattens; the gap statistic formalises this against data spread uniformly and, unlike many rules, can conclude there is only one cluster. The silhouette compares each unit's closeness to its own and the nearest other cluster.

$$s(i) = \frac{b(i) - a(i)}{\max\{a(i), b(i)\}}$$

where $s(i)$ is the silhouette of unit $i$, $a(i)$ its mean distance to the other members of its cluster and $b(i)$ its mean distance to the members of the nearest other cluster. Values run from minus one to one; values near one indicate dense, well-separated clusters and values around zero overlapping ones. Statistical fit is only part of validation, and Yan and colleagues list internal and external validity, interpretability, substantiality, stability, actionability and parsimony, and found that only 10 of 216 studies assessed stability.

Worked example with six patients and two measures of care use

The figures below are illustrative. Six patients are described by annual emergency department (ED) attendances and outpatient appointments, both counts on similar ranges, so raw values are used. Annual cost is held back from the clustering and attached afterwards.

1. Assign. K-means with two clusters starts from P3 (2, 0) and P4 (5, 4). For P2 the squared distances are as follows.

$$d(\text{P2}, \text{P3}) = (1 - 2)^2 + (2 - 0)^2 = 5, \quad d(\text{P2}, \text{P4}) = (1 - 5)^2 + (2 - 4)^2 = 20$$

where each $d$ is the squared Euclidean distance between two profiles, so P2 joins P3. The same comparison puts P1 and P3 with P2 in cluster A and P4, P5 and P6 in cluster B.

2. Update and stop. The cluster means become (1, 1) for A and (5, 5) for B. Every patient is nearer its own mean than the other (for example, P2 is at squared distance 1 from A and 25 from B), so nothing moves and the algorithm has converged.

3. Judge the number of clusters. With one cluster the mean is (3, 3); with two, each patient is measured from its own cluster mean.

$$W(1) = 13 + 5 + 10 + 5 + 18 + 5 = 56, \quad W(2) = (1 + 1 + 2) + (1 + 2 + 1) = 8$$

where $W(1)$ and $W(2)$ are the within-cluster sums of squares for one and two clusters. Two clusters remove 48 of the 56 units of scatter, about 86%. Checking every three-cluster split, the best splits give 5, and two tie (P5 on its own with P4 and P6 together, or P3 on its own with P1 and P2 together), a further fall of only 3, so the kink is at two clusters.

4. Check separation. For P2, the Euclidean (not squared) distances are 1.414 and 2.236 to P1 and P3, and 4.472, 6.403 and 4.243 to P4, P5 and P6 respectively, so its silhouette is as follows.

$$s(\text{P2}) = \frac{5.039 - 1.825}{5.039} = 0.638$$

where 1.825 is the mean distance $a$ to its own cluster and 5.039 the mean distance $b$ to cluster B. Silhouettes for all six patients range from 0.619 to 0.716, averaging 0.657, well above zero, so the groups do not overlap.

5. Attach costs and interpret. Mean annual cost is £700 in cluster A and £6,400 in cluster B, which a budget model for a care coordination service aimed at cluster B could use as baseline costs. The clustering does not show that cluster B would respond to such a service or that its costs are avoidable; its higher cost largely follows from the higher use that defined it. Had cost in pounds been a third clustering variable without rescaling, the squared cost difference between P1 and P5 alone, 51,840,000, would dwarf their squared difference of 61 on the two counts, and the clusters would simply have been cost bands.

From segments to cost-effectiveness subgroups

Cluster results enter economic evaluation either as descriptions of how costs and needs vary, supplying segment-specific baseline costs or event rates, or as candidate subgroups for separate analysis. The second use meets the rules that assessment bodies apply to any subgroup analysis.

The NICE manual for technology appraisal (PMG36, section 4.9) asks for separate clinical and cost-effectiveness estimates for each relevant subgroup in cost-utility analyses. It says subgroups should preferably be identified from an expectation of differential clinical or cost effectiveness based on known, biologically plausible mechanisms, social characteristics or other clearly justified factors, and that post hoc data dredging for subgroup effects will be viewed sceptically. It lists subgroups based solely on individual utilities for health states and patient preference among those not considered relevant, which bears directly on latent preference classes, while stating in section 4.9.11 that "Consideration of subgroups based on differential cost may be appropriate in some circumstances", which is the opening for use- and cost-defined segments. It also asks for the precision of subgroup estimates to be reflected in the analysis of parameter uncertainty.

In cost-effectiveness analysis a cluster solution is therefore a hypothesis generator. A difference in treatment effect between clusters still has to be estimated, ideally from data that did not generate the clusters, as for any heterogeneous treatment effect, and uncertain latent class membership adds parameter uncertainty that a single most-likely assignment hides.

Cluster analysis, cluster trials and cluster sampling

The word "cluster" has three common meanings in health economics, and they answer different questions. In cluster analysis the groups are an output found in the data. Under cluster randomisation, pre-existing units such as general practices or hospitals are randomised as wholes, and the analysis must allow for correlation within them. In cluster sampling, a survey selects whole groups such as households or areas and measures their members, which again requires standard errors that reflect the grouping.

Supervised classification differs as well, because it assigns units to groups already defined by a known outcome. Principal component analysis is sometimes run before clustering to reduce many variables to a few, but it summarises variables rather than grouping units.

Where cluster findings mislead

Algorithms run with a fixed number of clusters return that many groups whether or not the data contain them. K-means asked for two clusters always produces two, and hierarchical methods impose a hierarchy whether or not one exists. A tidy dendrogram or plausible segment labels are not evidence of real types.

Solutions can also be unstable. Small changes in the data or the linkage rule can produce quite different dendrograms, k-means can settle in different local minima, and the people a segment describes change over time, as the high-cost study's finding on persistence showed. A segment defined in one year is not a fixed population for a lifetime model.

Clusters are not causal subgroups. A segment defined by high use will show high cost by construction, and differences between clusters in outcomes or treatment response are associations that need separate evidence before a programme targeted at one segment can be expected to change its costs. Clusters formed on variables measured after treatment cannot define subgroups for estimating that treatment's effect, because membership may itself be an effect of the treatment.

Sources

• Davis AC, Shen E, Shah NR, Glenn BA, Ponce N, Telesca D, Gould MK, Needleman J. Segmentation of high-cost adults in an integrated healthcare system based on empirical clustering of acute and chronic conditions. Journal of General Internal Medicine. 2018;33(12):2171-2179. https://doi.org/10.1007/s11606-018-4626-0
• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York: Springer; 2009. Sections 6.8 and 14.3. https://hastie.su.domains/ElemStatLearn/
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Section 4.9. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2
• Reibling N, Ariaans M, Wendt C. Worlds of healthcare: a healthcare system typology of OECD countries. Health Policy. 2019;123(7):611-620. https://doi.org/10.1016/j.healthpol.2019.05.001
• scikit-learn developers. Clustering. scikit-learn User Guide, section 2.3, version 1.9.1. Accessed 1 October 2026. https://scikit-learn.org/stable/modules/clustering.html
• Vuik SI, Mayer E, Darzi A. A quantitative evidence base for population health: applying utilization-based cluster analysis to segment a patient population. Population Health Metrics. 2016;14:44. https://doi.org/10.1186/s12963-016-0115-z
• Vuik SI, Mayer E, Darzi A. Enhancing risk stratification for use in integrated care: a cluster analysis of high-risk patients in a retrospective cohort study. BMJ Open. 2016;6(12):e012903. https://doi.org/10.1136/bmjopen-2016-012903
• Yan S, Kwan YH, Tan CS, Thumboo J, Low LL. A systematic review of the clinical application of data-driven population segmentation analysis. BMC Medical Research Methodology. 2018;18(1):121. https://doi.org/10.1186/s12874-018-0584-9
• Zhou M, Thayer WM, Bridges JFP. Using latent class analysis to model preference heterogeneity in health: a systematic review. PharmacoEconomics. 2018;36(2):175-187. https://doi.org/10.1007/s40273-017-0575-4