Weighted Least Squares Regression

Weighted Least Squares Regression for Unequal Variances in Health Economics

Weighted least squares fits a regression by minimising a sum of squared residuals in which each residual is multiplied by a weight, so that precisely measured observations pull the fitted line harder than noisy ones. When the weights are the inverses of known error variances, it is the most precise linear unbiased estimator, which is why it sits behind inverse-variance meta-analysis and meta-regression. In health technology assessment these pooled effects feed decision models, so the weights reach the cost-effectiveness result through the estimate and its standard error. The same idea appears in heteroscedastic cost data and in survey-weighted regression, where "weight" means something different.

Why equal weighting fails when variances differ

Linear regression fitted by ordinary least squares treats every observation as equally informative. The NIST/SEMATECH e-Handbook of Statistical Methods describes the assumption as a constant error standard deviation across all values of the predictors, and notes that a procedure treating all data equally gives less precisely measured points more influence than they should have.

In health economics, trial estimates differ in precision with size and event counts, and Manning and Mullahy list heteroscedasticity among the common problems of skewed health expenditure data.

The weighted least squares estimator

The estimator chooses coefficients to minimise a weighted residual sum of squares. It is a direct extension of least squares approximation.

$$\hat{\beta}_{WLS} = \arg\min_{\beta} \sum_{i=1}^{n} w_i \left(y_i - x_i^{T}\beta\right)^2 = \left(X^{T} W X\right)^{-1} X^{T} W y$$

where $y_i$ is the outcome for observation $i$, $x_i$ is its vector of covariates, $\beta$ is the coefficient vector, $X$ is the design matrix, $y$ is the outcome vector and $W$ is a diagonal matrix of weights $w_i$. Setting $w_i = 1/\sigma_i^2$, the inverse of each observation's error variance, gives precision weights.

Solon, Haider and Wooldridge show the mechanics with group averages. If each observation is a mean over $J_i$ individuals, its error variance is proportional to the inverse of $J_i$, and ordinary least squares after multiplying each row by $\sqrt{J_i}$ is the minimum-variance linear unbiased estimator.

With exact inverse-variance weights, each standard error is the square root of a diagonal element of $(X^{T} W X)^{-1}$. General regression software instead treats weights as correct only up to a constant, which it estimates from the residuals.

Precision weights in meta-analysis

The Cochrane Handbook (chapter 10) explains that the inverse-variance method weights each study by one over the square of its standard error, a choice that minimises the imprecision of the pooled estimate. A fixed effect model pooled this way is weighted least squares on an intercept alone.

$$\hat{\theta}_{IV} = \frac{\sum_i w_i y_i}{\sum_i w_i}, \qquad SE(\hat{\theta}_{IV}) = \frac{1}{\sqrt{\sum_i w_i}}, \qquad w_i = \frac{1}{v_i}$$

where $\hat{\theta}_{IV}$ is the pooled effect, $y_i$ is the estimate from study $i$ (a log odds ratio for a ratio measure), $v_i$ is its within-study variance and $w_i$ is its weight. Viechtbauer notes that when true effects differ, the weighted fit estimates their weighted average, whereas unweighted least squares estimates their simple average.

A random effects model adds the between-study variance, tau-squared, to every study's variance. Viechtbauer describes estimating $\tau^2$ first and then the coefficients by weighted least squares with $w_i = 1/(v_i + \hat{\tau}^2)$.

Meta-regression as weighted least squares with known variances

The Cochrane Handbook says meta-regression differs from simple regression because studies are weighted by the precision of their estimates and the model should allow for residual heterogeneity the covariates leave unexplained. Thompson and Higgins describe weighting for both within-trial variances and residual between-trial heterogeneity as random-effects meta-regression. The NICE manual (PMG36, section 3.4.10) says known clinical heterogeneity may be explored using subgroup analyses and meta-regression.

The within-study variances are treated as known. Viechtbauer's comparison of the metafor function rma() with the general R function lm() explains that lm() assumes weights known only up to a proportionality constant, the residual variance, which it estimates and folds into the standard errors. Viechtbauer notes that Thompson and Sharp describe this as allowing for heterogeneity by a multiplicative factor, and they conclude that an additive component of residual heterogeneity should be used. General weighted regression software therefore reproduces the fixed-effect coefficients but not their standard errors.

Worked example with four trials and one covariate

All numbers in this example are illustrative. Four placebo-controlled trials report log odds ratios for hospital admission, with different planned treatment durations. A model for a health system funding only 6-month courses needs to know whether the effect depends on duration. The Cochrane Handbook advises that meta-regression should generally not be considered with fewer than ten studies, so four trials are used only to keep every step visible.

1. Pool with precision weights. The weights sum to 166 and the weighted effects to -53.1.

$$\hat{\theta}_{IV} = \frac{-53.1}{166} = -0.320$$

where -0.320 is the pooled log odds ratio, an odds ratio of 0.726, with a standard error of 0.0776. Trial B carries about 60% of the weight, against a quarter in the unweighted mean of -0.3375.

2. Fit the weighted slope. The weighted mean duration is 13.48 months, the weighted sum of squared deviations of duration is 4,711.4 and the weighted cross-product with the log odds ratios is -119.91.

$$\hat{\beta}_1 = \frac{-119.91}{4711.4} = -0.02545, \qquad \hat{\beta}_0 = -0.3199 - (-0.02545 \times 13.482) = 0.0232$$

where $\hat{\beta}_1$ is the change in log odds ratio per extra month of treatment and $\hat{\beta}_0$ is the intercept. With known variances the slope's standard error is the inverse square root of 4,711.4, or 0.01457, giving $z = -1.75$ and a two-sided p-value of about 0.08.

3. Compare general software. The weighted residual sum of squares is 0.173 on 2 degrees of freedom, so lm() estimates a scale factor of 0.0863 and multiplies the standard error by its square root.

$$SE_{lm} = 0.01457 \times \sqrt{0.0863} = 0.00428$$

where $SE_{lm}$ is the rescaled standard error. The trials fit the line more closely than their sampling variances imply, so lm() reports a t statistic near -5.9 (p about 0.03 on 2 degrees of freedom), a spurious finding. Larger residual scatter would inflate the standard error instead.

4. Add residual heterogeneity. With $\tau^2 = 0.01$ assumed for illustration, since these data would give a moment estimate of zero, the weights become 20, 50, 13.79 and 20. Trial B's share falls to about 48%, the slope becomes -0.0256 and its standard error 0.0164, a $z$ of about -1.56.

5. Carry the result into the model. At 6 months the fixed-effect line predicts a log odds ratio of -0.1295, an odds ratio of 0.879. With a comparator one-year admission risk of 0.20, the baseline odds are 0.25.

$$\text{odds}_1 = 0.25 \times 0.8786 = 0.2196, \qquad p_1 = \frac{0.2196}{1.2196} = 0.180$$

where $\text{odds}_1$ and $p_1$ are the odds and probability of admission with treatment. About 20 admissions are avoided per 1,000 patients, against about 46 with the pooled odds ratio of 0.726. The slope is imprecise and leans on trial A, so this effect suits a scenario analysis, not the base case.

Heteroscedastic cost and resource-use data

Patient-level costs are typically skewed, with excess zeros and heavy right tails. Mihaylova and colleagues report that generalised least squares estimators, or Huber and White variance estimates for ordinary least squares, are often used to obtain consistent estimates in such data.

The generalized linear model builds the weighting in. The same review explains that a GLM specifies a variance function and so accommodates skewness through variance weighting, and the R documentation for glm() states that its default fitting method is iteratively reweighted least squares. A gamma model for costs is solved as a sequence of weighted fits whose weights are updated from the current fitted means.

Group-level costs need care. Solon, Haider and Wooldridge show that when group averages share a common group-level error, weighting by group size can harm precision; in their state divorce-rate example the robust standard error estimates for ordinary least squares over the first eight years of effects were about half those for population-weighted least squares. They recommend testing for heteroscedasticity first, for example by regressing squared residuals on inverse group size, and reporting heteroscedasticity-robust standard errors either way.

Precision weights compared with survey and frequency weights

Software options called "weights" mix different ideas, and the right standard errors depend on which is meant. Frequency weights record that an observation stands for several identical individuals. A positive integer weight in R glm() is closer to a precision weight, since the documentation reads it as the response being the mean of that many unit-weight observations.

Survey weights are inverse probabilities of selection. Solon, Haider and Wooldridge separate two purposes. For descriptive population statistics, weighting is needed whenever the sample is not representative. For causal effects, inverse-probability weighting is needed for consistency when the sampling rate varies endogenously, and should then be paired with White heteroscedasticity-robust standard errors because the weighting itself induces heteroscedasticity. When sampling is exogenous, both estimators are consistent and the unweighted one may be more precise.

Choices that change the answer

Weights are almost always estimated. The NIST handbook warns that the theory assumes exactly known weights and that results can be badly and unpredictably affected when weights rest on few replicated observations. The Cochrane Handbook advises against inverse-variance methods for rare events, when within-study variances are least reliable.

Weighting can also change what is estimated. Solon, Haider and Wooldridge note that under misspecification weighted and unweighted estimates converge to different values, so a large gap between them is a warning sign.

Boundaries with nearby methods

Weighted least squares assumes uncorrelated errors. Generalised least squares extends it to correlated errors, and generalized estimating equations handle clustering when the correlation structure is uncertain. Robust standard errors from the sandwich estimator keep the ordinary least squares coefficients and correct only the inference. For sparse binary outcomes the Mantel-Haenszel method uses count-based weights instead of estimated variances.

Limitations and common misreadings

Most errors come from misreading what the weights mean. Three recur in health economics.

1. Running a meta-regression in general regression software. The coefficients match a fixed-effect fit, but the standard errors are rescaled by the residual variance and can be badly wrong.
2. Treating survey weights as precision weights. Inverse selection probabilities correct representativeness, not unequal variance, and need robust standard errors.
3. Assuming weighting always improves precision. With group-level error components, size weights can make estimates less precise, and the NIST handbook notes that a weighted fit that raises an outlier's influence can be far worse than an unweighted one.

Sources

• NIST/SEMATECH. Section 4.1.4.3 Weighted least squares regression. In: NIST/SEMATECH e-Handbook of Statistical Methods. National Institute of Standards and Technology; accessed 14 October 2026. https://www.itl.nist.gov/div898/handbook/pmd/section1/pmd143.htm
• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 10.3, 10.4.4 and 10.11.4. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10
• Viechtbauer W. Conducting meta-analyses in R with the metafor package. Journal of Statistical Software. 2010;36(3):1-48. https://doi.org/10.18637/jss.v036.i03
• Viechtbauer W. A comparison of the rma() and the lm(), lme(), and lmer() functions. The metafor package website; last modified 14 November 2023. https://www.metafor-project.org/doku.php/tips:rma_vs_lm_lme_lmer
• Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Statistics in Medicine. 1999;18(20):2693-2708. https://pubmed.ncbi.nlm.nih.gov/10521860/
• Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine. 2002;21(11):1559-1573. https://doi.org/10.1002/sim.1187
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 2022, last updated 13 October 2026. Section 3.4.10. https://www.nice.org.uk/process/pmg36/chapter/evidence-2
• Mihaylova B, Briggs A, O'Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Economics. 2011;20(8):897-916. https://doi.org/10.1002/hec.1653
• Manning WG, Mullahy J. Estimating log models: to transform or not to transform? Journal of Health Economics. 2001;20(4):461-494. https://doi.org/10.1016/s0167-6296(01)00086-8
• Solon G, Haider SJ, Wooldridge JM. What are we weighting for? NBER Working Paper 18859; 2013. Published in Journal of Human Resources. 2015;50(2):301-316. https://www.nber.org/papers/w18859
• R Core Team. glm: fitting generalized linear models. R stats package documentation; accessed 14 October 2026. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.html