Pearson Correlation

Pearson Correlation in Health Economic Data and Models

Pearson's product-moment coefficient, usually written $r$, measures how closely two numerical quantities follow a straight line, for example a patient's costs in two consecutive years. In health economics it matters because the correlation between baseline and follow-up costs sets how much regression to the mean will flatter an uncontrolled evaluation, and the correlation between uncertain inputs shapes the spread of a probabilistic model. Rank-based alternatives and intervals are on the Correlation Coefficient page and correlated costs and effects on the Correlation page.

The product-moment formula and what it standardises

The coefficient is built from three sums of deviations from the means. The NIST Dataplot reference manual computes it this way, and Mukaka's guide for medical researchers gives the same sample formula.

$$r = \frac{S_{xy}}{\sqrt{S_{xx}\,S_{yy}}}, \qquad S_{xy} = \sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y}), \qquad S_{xx} = \sum_{i=1}^{n}(x_i - \bar{x})^2, \qquad S_{yy} = \sum_{i=1}^{n}(y_i - \bar{y})^2$$

where $x_i$ and $y_i$ are the paired values for patient or unit $i$, $\bar{x}$ and $\bar{y}$ are the sample means, $n$ is the number of pairs, $S_{xy}$ is the sum of cross-products of deviations and $S_{xx}$ and $S_{yy}$ are the sums of squared deviations. Dividing each sum by $n - 1$ turns $r$ into the sample covariance divided by the product of the two sample standard deviations.

Karl Pearson's 1896 memoir on regression and heredity likened the sum of products of deviations to the product-moment of dynamics and showed that, for normally correlated characteristics, $S(xy)/(n\sigma_1\sigma_2)$, with $x$ and $y$ as deviations and $\sigma$ using divisor $n$, is the most probable value of the coefficient. Mukaka describes the coefficient as dimensionless and lying between minus 1 and plus 1, so a change of currency unit leaves it unchanged. A nonlinear change of scale, such as taking logarithms of skewed costs, does change it, as the worked example shows.

Pearson correlation and the regression line

Correlation and simple linear regression use the same three sums. Bewick and colleagues distinguish them, with correlation quantifying the strength of the linear relationship and regression expressing it as an equation. The least squares slope of $y$ on $x$ is $S_{xy}/S_{xx}$, which can be rewritten in terms of $r$.

$$b_{y \mid x} = r\,\frac{s_y}{s_x}, \qquad r^2 = b_{y \mid x}\, b_{x \mid y}$$

where $b_{y \mid x}$ is the slope for predicting $y$ from $x$, $b_{x \mid y} = S_{xy}/S_{yy}$ is the slope for predicting $x$ from $y$, and $s_x$ and $s_y$ are the sample standard deviations. The coefficient is symmetric in the two variables, while the two regression lines differ unless the points lie exactly on a line. Bewick and colleagues add that with one predictor the coefficient of determination equals $r^2$, the proportion of the total variation in $y$ accounted for by the regression.

The slope formula has a direct consequence. In standard deviation units the predicted value of $y$ is $r$ times the standardised value of $x$, so unless $|r| = 1$ no prediction lies further from the mean than the value it was predicted from. This is the regression of Pearson's title, first described by Galton for the heights of parents and children, and it matters whenever patients are selected on an extreme baseline.

Regression to the mean in evaluations of high-cost patients

Case management schemes often enrol people with recent emergency admissions, and Roland and colleagues note that such programmes are sometimes judged only by the fall in admissions in the enrolled cohort, with no control group. For normally distributed data Linden gives the expected effect from the population mean $\mu$, standard deviation $\sigma$, pre-post correlation $\rho$ and the cut-off $\kappa$ that defines the high-risk group.

$$z = \frac{\kappa - \mu}{\sigma}, \qquad C = \frac{\phi(z)}{1 - \Phi(z)}, \qquad \text{RTM effect} = C\,\sigma\,(1 - \rho)$$

where $z$ is the standardised cut-off, $\phi$ and $\Phi$ are the standard normal density and distribution functions, $C$ is the mean standardised baseline value of those above the cut-off, and $\rho$ is the correlation between the baseline and follow-up measurements. Linden gives the expected baseline mean of the selected group as $\mu + C\sigma$ and its expected follow-up mean as $\mu + C\sigma\rho$, so the gap shrinks to zero only when $\rho = 1$. In the same framework the between-patient variance is $\rho\sigma^2$, which makes the pre-post correlation the share of total variance that is stable between patients, the quantity a test-retest reliability study estimates.

The effect is large in real data. In a cohort of chronically ill health plan members shown by Linden, with no intervention in place, the highest cost quintile averaged roughly USD 20,000 to USD 27,000 across three conditions in the first year and USD 7,000 to USD 10,000 in the second. In English hospital episode statistics, Roland and colleagues followed people aged 65 and over with at least two emergency admissions in 1997-8. Their emergency admissions per head of the original cohort, counting deaths, fell from 2.568 in the index year to 0.641 the next year and 0.194 in 2002-3, which the authors put down to mortality as well as regression toward the mean, and the group, which accounted for 38% of admissions in the index year, accounted for under 10% the next year. The authors concluded that comparison with a matched control group is necessary, which points to designs such as difference-in-differences in place of a simple before-and-after study.

Worked example with two years of patient costs

All numbers are illustrative. Six patients in a cost study have annual costs, in GBP 1,000s, recorded in two consecutive years.

1. Means and sums. The year 1 mean is 24/6 = 4 and the year 2 mean is 21/6 = 3.5. The squared year 1 deviations give $S_{xx}$ = 9 + 4 + 1 + 0 + 1 + 25 = 40, the squared year 2 deviations give $S_{yy}$ = 2.25 + 6.25 + 0.25 + 0.25 + 2.25 + 6.25 = 17.5, and the cross-products give $S_{xy}$ = 4.5 + 5.0 − 0.5 + 0 + 1.5 + 12.5 = 23.

2. The coefficient. The product 40 × 17.5 = 700 has square root 26.458, so $r$ = 23 / 26.458 = 0.869. Equivalently, the sample covariance is 23/5 = 4.6, the standard deviations are 2.828 and 1.871 (the square roots of 40/5 and 17.5/5), and 4.6 / (2.828 × 1.871) = 0.869.

3. Slopes and explained variation. The slope of year 2 on year 1 is 23/40 = 0.575, the same as 0.869 × 1.871/2.828. The reverse slope is 23/17.5 = 1.314, and 0.575 × 1.314 = 0.756, which is $r^2$. Year 1 cost accounts for about 76% of the variation in year 2 cost.

4. Regression to the mean for the costliest patient. Patient 6 cost GBP 9,000 in year 1, which is 5/2.828 = 1.77 standard deviations above the mean. The fitted line predicts 3.5 + 0.575 × 5 = 6.375, or GBP 6,375, which is 0.869 × 1.77 = 1.54 standard deviations above the year 2 mean. Patient 6 actually cost GBP 6,000, a fall of GBP 3,000 with no intervention at all.

5. A change of scale. Taking natural logarithms of both years' costs and repeating steps 1 and 2 gives $r$ = 0.759. Converting pounds to pence would have left 0.869 unchanged.

6. A cohort selected on high cost. Suppose a large historical cohort has mean annual cost GBP 3,000, standard deviation GBP 2,000 and a year-to-year correlation of 0.5, and a programme enrols everyone above GBP 7,000. Then $z$ = (7,000 − 3,000)/2,000 = 2, the normal density at 2 is 0.05399, the probability above 2 is 0.02275, and their ratio gives $C$ = 2.373. The expected baseline mean of those enrolled is 3,000 + 2.373 × 2,000 = GBP 7,746 and the expected follow-up mean is 3,000 + 2.373 × 2,000 × 0.5 = GBP 5,373, an apparent saving of GBP 2,373 per patient, or 31%, with no effect at all. With a correlation of 0.8 the follow-up mean is GBP 6,797 and the artefact shrinks to GBP 949, or 12%.

7. Interpretation. A high Pearson correlation between years still leaves room for a large fall among the patients selected for being extreme, and the size of that fall is proportional to one minus the correlation. A normal model with these figures places about 7% of patients below zero cost, and Linden found that with skewed data the normal formula underestimated the true effect, so for real costs it is a rough lower guide.

Assumptions behind the test and the interval

The NIST manual tests whether the correlation differs from zero with an F statistic on 1 and $n - 2$ degrees of freedom. Bewick and colleagues state that the test needs the same assumptions as regression, while interpreting a confidence interval for the correlation requires both variables to be Normally distributed. Mukaka calls Pearson's coefficient inappropriate when either variable is not normally distributed, because extreme values can exaggerate or dampen it, and recommends Spearman's coefficient for skewed or ordinal data, a description that often fits patient-level costs.

Pearson correlation in decision models

In a cost-effectiveness model the coefficient describes how uncertain inputs move together across simulations, and the NICE technology appraisal manual (PMG36, section 4.7.19) asks for evidence on the extent of correlation between individual parameters to be considered and reflected in the probabilistic analysis, with the assumptions clearly presented. Section 4.7.23 adds that threshold analysis is not suitable for parameters that are highly correlated with other influential parameters.

Correlation coefficients between parameters form a correlation matrix that is used to generate correlated normal draws for probabilistic sensitivity analysis. A Pearson correlation set on that normal scale generally changes once the draws are transformed into skewed distributions, as taking logarithms changed $r$ in step 5, so a model should state the scale it uses.

Boundaries with neighbouring measures

Several neighbouring measures answer different questions. Spearman's coefficient and Kendall's tau work on ranks and capture monotonic association. The partial correlation removes the linear influence of a third variable. For agreement between two ways of measuring the same quantity, such as self-reported and record-based resource use, Bewick and colleagues warn that a high correlation can be wrongly taken as agreement and recommend analysing paired differences as Bland and Altman proposed, while the intraclass correlation handles reliability.

Misreadings in health economic analyses

Most errors with Pearson's coefficient come from reading the number without the scatter plot or without knowing how the sample was chosen. The following recur in cost and outcome analyses.

1. Pooling groups or restricting the range. Bewick and colleagues note that subgroups such as men and women, or a single outlier, can inflate the coefficient, and that restricting one variable to a certain range may lead to an invalid estimate, so a correlation among high-cost patients only describes that group.
2. Treating blanks as zeros, or zeros as blanks. Microsoft's documentation for PEARSON states that empty cells are ignored while cells containing zero are included, so a missing cost recorded as 0 changes the result.

Sources

• Bewick V, Cheek L, Ball J. Statistics review 7: Correlation and regression. Critical Care. 2003;7(6):451-459. https://doi.org/10.1186/cc2401
• Linden A. Assessing regression to the mean effects in health care initiatives. BMC Medical Research Methodology. 2013;13:119. https://doi.org/10.1186/1471-2288-13-119
• Microsoft. PEARSON function. Microsoft Support; accessed 26 October 2026. https://support.microsoft.com/en-gb/office/pearson-function-0c3e30fc-e5af-49c4-808a-3ef66e034c18
• Mukaka MM. Statistics corner: a guide to appropriate use of correlation coefficient in medical research. Malawi Medical Journal. 2012;24(3):69-71. https://pmc.ncbi.nlm.nih.gov/articles/PMC3576830/
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 4.7.19 and 4.7.23. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2
• National Institute of Standards and Technology. Dataplot Reference Manual, entry CORRELATION (2018). Gaithersburg, MD: NIST Statistical Engineering Division. https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/correlat.htm
• Pearson K. Mathematical contributions to the theory of evolution. III. Regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London, Series A. 1896;187:253-318. https://doi.org/10.1098/rsta.1896.0007
• Roland M, Dusheiko M, Gravelle H, Parker S. Follow up of people aged 65 and over with a history of emergency admissions: analysis of routine admission data. BMJ. 2005;330(7486):289-292. https://doi.org/10.1136/bmj.330.7486.289