Regression Analysis

What regression analysis does

It can describe associations, adjust comparisons, predict outcomes or estimate causal effects when the design and assumptions justify that interpretation. This page explains the regression framework, estimation, interpretation, model families, diagnostics and health-economic applications.

Regression is not a single method and does not make an observational association causal by itself. The outcome type, data structure, estimand and study design determine which model and interpretation are defensible.

A regression model links an outcome to predictors

For observation (i), a multiple linear regression can be written as:

$$y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\varepsilon_i$$

where (y_i) is the outcome, (x_{ij}) are predictors, (\beta_0) is the intercept, (\beta_j) are coefficients and (\varepsilon_i) is the unexplained error. In matrix form:

$$y=X\beta+\varepsilon$$

The linearity assumption concerns linearity in the coefficients. Predictors can enter through transformations, polynomials, splines and interactions while the model remains linear in its estimated coefficients.

The coefficient meaning depends on the model

In an additive linear model, (\beta_j) is the expected difference in the outcome for a one-unit increase in (x_j), holding the other included predictors fixed. “Holding fixed” is a model-based conditional comparison; it does not necessarily describe a feasible intervention or a causal effect.

Coefficient interpretation changes with coding and transformation:

• A binary predictor coefficient compares its coded group with the reference group.
• A categorical predictor with several levels requires indicator variables or another explicit contrast system.
• A log-transformed outcome makes coefficients interpretable on a multiplicative scale after appropriate retransformation.
• A log-transformed predictor expresses change per proportional difference in that predictor.
• An interaction means the association of one predictor depends on the value of another.
• A spline represents a smoothly changing relationship rather than one constant slope.

Every reported coefficient should identify the outcome scale, predictor unit, reference category and adjustment set.

Ordinary least squares minimises squared residuals

Ordinary least squares, or OLS, estimates coefficients by minimising the residual sum of squares:

$$\operatorname{RSS}(\beta) =\sum_{i=1}^{n} \left(y_i-x_i^{\mathsf T}\beta\right)^2$$

When (X) has full column rank, the familiar algebraic expression is:

$$\hat{\beta} =(X^{\mathsf T}X)^{-1}X^{\mathsf T}y$$

Software should generally solve the least-squares problem using QR factorisation, Cholesky factorisation under suitable conditions or singular-value decomposition rather than explicitly calculating ((X^{\mathsf T}X)^{-1}). These methods can be more stable when predictors differ greatly in scale or are highly correlated.

The fitted values and residuals are:

$$\hat{y}=X\hat{\beta}$$ $$e=y-\hat{y}$$

Residuals are estimated discrepancies from the fitted model, not direct observations of the unobservable errors.

A worked linear-regression interpretation

Suppose annual healthcare cost is modelled as a function of age and treatment assignment:

$$\widehat{\operatorname{Cost}} =1200+35\times\operatorname{Age} -450\times\operatorname{Treatment}$$

where Treatment equals one for the intervention and zero for the comparator. For a 60-year-old receiving the comparator, predicted cost is:

$$1200+(35\times60)-(450\times0)=£3{,}300$$

For a 60-year-old receiving the intervention, predicted cost is:

$$1200+(35\times60)-(450\times1)=£2{,}850$$

The treatment coefficient represents an adjusted mean difference of (-£450) at the same age under this additive model. It should not be called a causal saving unless treatment assignment, confounding control and the other causal assumptions support that interpretation.

Statistical assumptions depend on the inferential goal

OLS coefficients can be useful under weaker conditions than those required for every textbook test or causal claim. Assumptions should therefore be linked to the specific target: unbiased conditional association, valid standard errors, prediction or causal estimation.

Important considerations include:

• The conditional mean is correctly specified: (E[y\mid X]=X\beta).
• Observations or clusters provide the independence structure assumed by the analysis.
• Predictors are not perfectly collinear.
• Error variance is constant if conventional homoscedastic standard errors are used.
• Distributional assumptions used for small-sample tests or prediction intervals are adequate.
• Influential observations do not dominate the result without explanation.
• Measurement error, selection and missingness do not invalidate the intended interpretation.

Normality of the outcome itself is not a general requirement for OLS. Distributional assumptions usually concern errors, conditional outcomes or the large-sample approximation used for inference.

Standard errors quantify sampling uncertainty under stated assumptions

Under homoscedastic independent errors with variance (\sigma^2), the OLS coefficient covariance matrix is:

$$\operatorname{Var}(\hat{\beta}\mid X) =\sigma^2(X^{\mathsf T}X)^{-1}$$

An estimated coefficient interval is commonly written as:

$$\hat{\beta}_j \pm t_{\nu,1-\alpha/2}\operatorname{SE}(\hat{\beta}_j)$$

Robust or sandwich standard errors can provide valid large-sample inference under some forms of heteroscedasticity, while cluster-robust methods account for dependence within defined clusters. These adjustments change estimated uncertainty, not the coefficient model, and they do not repair confounding, selection bias or an incorrect conditional mean.

Goodness of fit is not evidence of causality or usefulness

The coefficient of determination measures the fraction of sample outcome variation represented by the fitted linear model:

$$R^2 =1- \frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2} {\sum_{i=1}^{n}(y_i-\bar{y})^2}$$

Adjusted (R^2) penalises the mechanical increase that follows from adding predictors. A high (R^2) does not establish unbiased coefficients, causal identification or accurate out-of-sample prediction. A low (R^2) does not make a precisely estimated treatment effect unimportant when individual outcomes are inherently variable.

Fit measures should be interpreted alongside calibration, discrimination where relevant, residual behaviour, uncertainty and the intended decision use.

Categorical variables require explicit reference groups

A categorical predictor with (K) unordered levels is often represented by (K-1) indicator variables when the model contains an intercept. One omitted level becomes the reference category. Each displayed coefficient compares its category with that reference while holding other included predictors fixed.

Using all (K) indicators with an intercept creates perfect multicollinearity, sometimes called the dummy-variable trap. Alternative contrast coding can estimate different comparisons, but the fitted model is equivalent when the coding spans the same design space.

Ordinal categories should not automatically be coded as equally spaced numbers. Doing so imposes a constant linear difference between adjacent levels, which may not reflect their meaning.

Interactions make effects conditional

An interaction allows the association of one predictor with the outcome to depend on another predictor. For a treatment indicator (T) and continuous variable (Z):

$$E[y\mid T,Z] =\beta_0+\beta_1T+\beta_2Z+\beta_3(TZ)$$

The treatment contrast at value (Z=z) is:

$$E[y\mid T=1,Z=z] -E[y\mid T=0,Z=z] =\beta_1+\beta_3z$$

With an interaction present, (\beta_1) is the treatment contrast at (Z=0), which may be clinically meaningless if zero is outside the observed range. Centring (Z) at a meaningful value changes the interpretation of lower-order coefficients without changing fitted values.

Non-linearity should be modelled rather than ignored

A straight-line relationship may be implausible across the full range of age, baseline risk, dose or disease severity. Polynomial terms, restricted cubic splines and other smooth functions can represent curvature. Arbitrarily categorising a continuous predictor discards information and creates artificial thresholds.

For a spline basis (B_k(x)), a flexible model can be written as:

$$E[y\mid x] =\beta_0+ \sum_{k=1}^{K}\beta_k B_k(x)$$

The number and placement of knots or smoothing penalty should be chosen and validated transparently. Flexible modelling reduces functional-form bias but increases the need for adequate sample size and protection against overfitting.

Generalised linear models extend regression beyond continuous outcomes

Generalised linear models, or GLMs, relate the conditional mean of an outcome to a linear predictor through a link function:

$$g\!\left(E[y_i\mid x_i]\right)=x_i^{\mathsf T}\beta$$

The outcome distribution and link should reflect the data and estimand. Common examples include:

The distribution and link are separate choices. A log link does not require that the raw outcome itself follows a lognormal distribution.

Logistic-regression coefficients need careful translation

For a binary outcome with probability (p_i), logistic regression specifies:

$$\log\!\left(\frac{p_i}{1-p_i}\right) =x_i^{\mathsf T}\beta$$

Exponentiating a coefficient gives an odds ratio:

$$\operatorname{OR}_j=e^{\beta_j}$$

An odds ratio is not a risk ratio, and the same odds ratio can imply different absolute risk differences at different baseline risks. For decisions, adjusted predicted probabilities, marginal risks and risk differences are often easier to interpret than odds alone.

Cost data require more than a default transformation

Healthcare costs are often right-skewed, non-negative and may include many zeros. OLS can still estimate an adjusted mean cost difference under an adequate conditional-mean specification, but inference and prediction may require robust methods. Gamma GLMs, log-link models, two-part models and other approaches may be appropriate depending on zeros, tails and the estimand.

Simply logging cost and exponentiating a fitted mean generally does not recover the arithmetic mean cost because:

$$E[C\mid X] \ne \exp\!\left(E[\log C\mid X]\right)$$

Retransformation requires attention to residual variation and model assumptions. Model choice should be assessed through diagnostics, predictive performance and the policy quantity required, not skewness alone.

Repeated and clustered observations need dependence-aware models

Patient outcomes may be clustered within centres, clinicians or regions, and repeated measurements from the same person are correlated. Ignoring this dependence can understate uncertainty and may misrepresent the mean structure. Cluster-robust standard errors, generalised estimating equations and mixed-effects models address different inferential goals.

Mixed models introduce random effects, such as a random intercept (b_j) for cluster (j):

$$y_{ij}=x_{ij}^{\mathsf T}\beta+b_j+\varepsilon_{ij}$$

Population-average and cluster-specific coefficients need not have the same interpretation, especially for non-linear links. The method should match whether the target concerns average policy effects, centre-specific prediction or variance components.

Confounding control must follow the causal question

If regression is used to estimate a causal effect, covariate choice should be guided by the assumed causal structure and target estimand rather than automated significance testing. Adjustment for a common cause of treatment and outcome can reduce confounding, while adjustment for a mediator, collider or post-treatment variable can introduce bias.

A causal regression analysis should state:

• The intervention or exposure contrast.
• The target population and time point.
• The potential-outcome estimand, such as an average treatment effect.
• The covariates needed for conditional exchangeability.
• Positivity, consistency and interference assumptions.
• How missingness, measurement error and treatment changes are handled.

Even a correctly fitted regression cannot identify an effect when the relevant causal assumptions are false or unsupported.

Prediction and explanation require different validation

A prediction model is judged primarily by performance for new observations, not by whether every coefficient has a causal interpretation. Training performance is optimistic when model development and evaluation use the same data. Internal validation estimates this optimism; external validation tests transport to a different population, setting or time.

Prediction assessment can include:

• Calibration, comparing predicted and observed outcomes across relevant ranges.
• Discrimination for outcomes where ranking matters.
• Overall accuracy measures suited to the outcome and loss function.
• Decision-curve or utility analysis when prediction will guide action.
• Temporal or geographic external validation.
• Recalibration when baseline risk changes but predictor effects remain usable.

Random train–test splitting can be inefficient in modest datasets. Resampling methods such as bootstrap validation or cross-validation often use available data more effectively for internal assessment.

Missing data can change both estimates and target populations

Complete-case regression is valid only under assumptions that are often stronger than “few values are missing.” Missingness can alter covariate balance, precision and the population represented by the analysis. Missing indicators, single imputation and automatic row deletion are not interchangeable solutions.

Multiple imputation can propagate uncertainty from missing values when the imputation model and missing-at-random assumption are appropriate. The imputation model should include the outcome, analysis variables and useful predictors of missingness. Sensitivity analysis is needed when departures from the assumed missingness mechanism could change conclusions.

Diagnostics reveal where the fitted model is strained

Diagnostics should be selected for the outcome, estimator and purpose. A single residual plot or goodness-of-fit test is not a complete validation. Graphical examination often reveals patterns that a global statistic misses.

For linear regression, useful diagnostics include:

• Residuals versus fitted values for functional form and changing variance.
• Residuals versus individual predictors for omitted non-linearity.
• Quantile–quantile plots when normal-error inference or prediction intervals are important.
• Leverage and influence measures for observations with unusual predictor values or large impact.
• Variance inflation and condition diagnostics for collinearity.
• Cluster and time patterns for dependence.
• Out-of-sample residuals for predictive validation.

An influential observation should be checked for data error and substantive relevance, not removed automatically because it changes the result.

Regression outputs support health-economic analysis

Regression is used throughout health economics to estimate adjusted costs, outcomes, utilisation and treatment effects. It can also provide prediction equations and correlated parameter estimates for decision models. The model's target and uncertainty must remain aligned with the downstream use.

Applications include:

• Adjusting trial-based costs and QALYs for baseline imbalance.
• Estimating healthcare expenditure and resource-use relationships.
• Mapping condition-specific measures onto preference-based utility values.
• Estimating treatment effects from observational or quasi-experimental designs.
• Modelling survival, event rates and repeated outcomes.
• Predicting missing outcomes or extrapolating beyond observed follow-up under explicit assumptions.
• Building metamodels for complex simulations and value-of-information calculations.
• Risk adjustment for provider or population comparisons.

When regression estimates feed an economic model, the full variance–covariance matrix may be needed to preserve correlation among coefficients in probabilistic sensitivity analysis.

Common errors can make precise results misleading

Regression software can return estimates for a poorly posed question, so statistical output must be checked against design, data and intended interpretation.

• Interpreting association as causation ignores confounding and selection.
• Selecting covariates only by p-values can omit confounders or include harmful adjustments.
• Assuming linearity without checking can bias effects and predictions.
• Reporting odds ratios as risk ratios exaggerates effects when outcomes are common.
• Logging cost and naively exponentiating predictions can bias mean costs.
• Ignoring clustering or repeated measures can understate uncertainty.
• Treating non-significance as evidence of no effect confuses imprecision with equivalence.
• Using (R^2) as the sole measure of model quality ignores calibration and external validity.
• Extrapolating outside the observed predictor range relies on untested functional form.
• Categorising continuous predictors can discard information and create unstable cut-points.
• Reporting only coefficient estimates hides uncertainty, dependence and model diagnostics.

A defensible regression-analysis workflow

A high-quality analysis begins with the question and estimand, not with an automated model-selection routine. Every modelling decision should remain traceable from the data-generating process to the final health or economic interpretation.

1. Define the objective: Distinguish description, prediction, adjustment and causal estimation.
2. Specify the estimand: State the outcome scale, contrast, population, time point and required marginal or conditional quantity.
3. Understand the data: Check sampling, coding, units, missingness, clustering, repeated measures and outcome support.
4. Plan predictors: Use subject knowledge and causal reasoning rather than univariable significance screening.
5. Choose the model: Match the mean structure, distribution, link and dependence assumptions to the outcome and estimand.
6. Fit stably: Use tested numerical routines and examine convergence, rank, scaling and collinearity.
7. Diagnose assumptions: Evaluate residuals, functional form, variance, influence, dependence and missing-data implications.
8. Quantify uncertainty: Use standard errors, intervals or resampling methods suited to the design and model.
9. Validate the intended use: Assess independent fit, causal sensitivity or out-of-sample performance as appropriate.
10. Report transparently: Present coding, reference groups, transformations, coefficients, absolute predictions, diagnostics, limitations and reproducible implementation details.

Regression analysis is most useful when its model, assumptions and interpretation answer the same clearly defined question. Numerical precision and statistical significance cannot compensate for a mismatched estimand, biased design or unsupported causal claim.