Goodness of Fit

What goodness of fit means

It is assessed through discrepancies between observations and model expectations, using graphical diagnostics, summary measures, and formal tests appropriate to the outcome and model.

This page explains residuals, chi-square statistics, likelihood deviance, regression fit, calibration, discrimination, censored outcomes, and predictive validation. It also shows why no single fit statistic can establish that a health-economic or clinical model is correct.

Fit is always relative to an outcome and purpose

A model can fit the mean well but misrepresent variance, tails, time patterns, subgroup behaviour, or correlation. Goodness of fit must therefore be defined against the aspects of the data that matter for estimation, prediction, explanation, or decision making.

For a model that predicts $\widehat y_i$ for observed value $y_i$, the raw residual is:

$$e_i=y_i-\widehat y_i$$

Small residuals are desirable, but their pattern matters as much as their average size. Residuals that vary systematically with fitted values, time, treatment, or covariates signal model inadequacy even when one overall measure looks favourable.

Good fit does not prove the model is true

Many different models can reproduce the same observed data, especially in small samples or when important regions are sparsely observed. A model can also fit its development data closely because it has learned noise rather than a stable relationship.

Goodness-of-fit evidence supports adequacy under stated diagnostics and data; it does not establish causal validity, transportability, or correct extrapolation. Scientific plausibility, external evidence, validation, and decision relevance remain separate requirements.

Graphical diagnostics come first

Plots reveal structure that a single statistic can conceal. Appropriate graphics depend on the model, but they should compare observations and predictions on the scale relevant to the model's assumptions and use.

• Residuals versus fitted values reveal curvature, unequal variance, and systematic prediction error.
• Residuals versus time or observation order reveal drift, autocorrelation, seasonality, and step changes.
• Normal quantile plots assess whether residual tails and shape are compatible with a normal-error assumption when that assumption is required.
• Observed-versus-predicted plots show calibration across the prediction range and can identify compression toward the mean.
• Subgroup plots reveal poor fit hidden by an acceptable overall average.

Plots should include enough information to show density and uncertainty rather than only a smoothed line. A visually attractive diagonal can be misleading when most observations occupy a narrow region or when the axes obscure clinically important errors.

Residuals need model-specific scaling

Raw residuals have the outcome's units and may have different variances across observations. Standardised, studentised, Pearson, deviance, martingale, Schoenfeld, and quantile residuals serve different model families and diagnostic questions.

For ordinary linear regression with residual standard deviation estimate $s$ and leverage $h_{ii}$, an internally studentised residual is:

$$r_i=\frac{e_i}{s\sqrt{1-h_{ii}}}$$

Leverage adjusts for the fact that observations with unusual predictor values can have artificially small in-sample residuals. Large residual magnitude suggests poor local fit, while high leverage and influence require separate assessment.

Mean squared and root mean squared error

Squared-error measures summarise the magnitude of prediction discrepancies and penalise large errors more heavily than small ones. They retain the scale of the outcome after taking the square root.

$$MSE=\frac{1}{n}\sum_{i=1}^{n}(y_i-\widehat y_i)^2$$ $$RMSE=\sqrt{MSE}$$

The denominator $n$ is common for predictive RMSE. An estimate of residual variance in a fitted regression often instead divides the residual sum of squares by residual degrees of freedom, so the formula and purpose should be reported rather than calling both quantities “RMSE.”

A worked regression-fit example

Suppose observed outcomes are $(2,4,5,7)$ and fitted values are $(2.5,3.5,5.5,6.5)$. The residuals are $(-0.5,0.5,-0.5,0.5)$, giving a residual sum of squares of 1.

$$SSE =(-0.5)^2+(0.5)^2+(-0.5)^2+(0.5)^2 =1$$

The predictive root mean squared error is:

$$RMSE =\sqrt{\frac{1}{4}} =0.5$$

The observed mean is $\bar y=4.5$, and the total sum of squares is 13. Therefore:

$$R^2 =1-\frac{SSE}{SST} =1-\frac{1}{13} =0.9231$$

The model explains approximately 92.3% of the in-sample variation around the observed mean and has an in-sample RMSE of 0.5 outcome units. These values do not reveal whether the residuals are independent, homoscedastic, approximately normal, or equally accurate in new data.

What R-squared does and does not measure

The coefficient of determination compares residual variation with variation around a baseline mean model. It summarises in-sample reduction in squared error but does not diagnose why residual variation remains. In ordinary least-squares regression with an intercept, it is:

$$R^2 =1- \frac{\sum_i(y_i-\widehat y_i)^2} {\sum_i(y_i-\bar y)^2}$$

$R^2$ usually increases when predictors are added, even if they have little durable value. It does not establish calibration, causal validity, correct functional form, or out-of-sample performance and can be negative under some prediction settings or models without an intercept.

Adjusted R-squared penalises added parameters

Adjusted $R^2$ accounts for the number of fitted coefficients relative to sample size. It can decrease when an added predictor does not improve fit enough to offset the lost degrees of freedom.

$$\bar R^2 =1- (1-R^2) \frac{n-1}{n-p}$$

where $p$ is the number of estimated coefficients including the intercept under this convention. Adjusted $R^2$ remains an in-sample summary and is not a substitute for validation or a model-specific adequacy assessment.

Chi-square goodness-of-fit for counts

The Pearson chi-square goodness-of-fit statistic compares observed category counts with counts expected under a specified distribution. Each contribution scales the squared discrepancy by its expected count.

$$X^2 =\sum_{j=1}^{k} \frac{(O_j-E_j)^2}{E_j}$$

Under regularity conditions, $X^2$ is compared with a chi-square distribution. If $q$ parameters are estimated from the same data and the probabilities must sum to one, a common degrees-of-freedom calculation is $k-1-q$, although the exact rule depends on the fitted model and constraints.

A worked chi-square example

Suppose observed counts in three categories are $(50,30,20)$ and the prespecified expected counts are $(40,35,25)$. The expected total is 100, matching the observed total.

$$\begin{aligned} X^2 &=\frac{(50-40)^2}{40} +\frac{(30-35)^2}{35} +\frac{(20-25)^2}{25}\\ &=2.5+0.7143+1.0\\ &=4.2143 \end{aligned}$$

With no distributional parameters estimated from these counts, the reference degrees of freedom are $3-1=2$, giving a p-value of approximately 0.12. The result does not reject the specified distribution at a 5% level, but it does not prove that the distribution is correct or that discrepancies are unimportant for the intended use.

Sparse expected counts need special treatment

The chi-square approximation can be inaccurate when expected counts are small or categories are sparse. Pooling categories, using an exact procedure, or obtaining a reference distribution by simulation may be more appropriate, provided the choice preserves the scientific question.

Categories should not be combined after inspecting which combination produces a preferred p-value. Expected-count rules are diagnostics rather than universal laws, and the final method should reflect the data-generating process and parameter estimation.

Formal tests depend strongly on sample size

A goodness-of-fit test usually evaluates a null hypothesis that a specified discrepancy is compatible with the model. With a very large sample, a trivial deviation can produce a small p-value; with a small sample, a serious defect can remain undetected.

A non-significant result means that the test did not find sufficient evidence against the model under its assumptions. It does not mean the model fits well, and a significant result does not by itself identify which assumption failed or whether the failure is decision-relevant.

Likelihood deviance compares with a saturated model

For likelihood-based models, deviance measures the discrepancy between the fitted model and a saturated model that reproduces the observations as closely as the model family permits. Smaller deviance indicates closer in-sample fit on the likelihood scale.

D =2\left{ \ell_{sat}-\ell_{fit} \right}

The absolute deviance can sometimes be compared with a reference distribution, but validity depends on model family, sample structure, dispersion, and asymptotic conditions. Residual deviance should be interpreted alongside deviance residuals and targeted diagnostics rather than as a universal pass/fail number.

Deviance differences compare nested models

When one model is nested within another, the difference in deviances is a likelihood-ratio statistic. Under suitable regularity conditions, it can be compared with a chi-square distribution using the difference in parameter counts.

$$\Delta D =D_{reduced}-D_{full} =2(\ell_{full}-\ell_{reduced})$$

Boundary parameters, small samples, penalised fits, random effects, and data-driven model selection can invalidate the simple reference distribution. A better-fitting full model can still overfit and perform worse in external data.

Pearson goodness of fit in generalised linear models

For a generalised linear model, Pearson residuals scale observed-minus-fitted discrepancies using the model variance function. Their squared sum can provide an overall diagnostic of dispersion and fit.

$$X_P^2 =\sum_{i=1}^{n} \frac{(y_i-\widehat\mu_i)^2} {\widehat\phi V(\widehat\mu_i)}$$

Here $V(\widehat\mu_i)$ is the variance function and $\widehat\phi$ is the dispersion parameter under the adopted convention. Comparisons with degrees of freedom are approximate and must account for grouping, repeated observations, clustering, weights, and fitted parameters.

Logistic models need calibration and discrimination checks

Binary-outcome models should be assessed on several dimensions. Calibration measures agreement between predicted probabilities and observed event frequencies, while discrimination measures how well predictions separate people with and without the event.

An area under the receiver operating characteristic curve can be high even when probabilities are systematically too high or too low. Conversely, a well-calibrated model in a homogeneous population can have modest discrimination, so neither property substitutes for the other.

Calibration intercept and slope

Calibration-in-the-large can be assessed through an intercept, while the calibration slope examines whether predictions are too extreme or too compressed. In an external dataset, one common recalibration model uses the original linear predictor $LP_i$.

$$\operatorname{logit}{P(Y_i=1)} =\alpha+\beta LP_i$$

Ideal calibration corresponds to $\alpha=0$ and $\beta=1$. A slope below one often indicates predictions that are too extreme, while an intercept different from zero indicates systematic underprediction or overprediction, subject to the coding of the outcome.

Calibration plots need enough detail

A calibration plot compares observed and predicted risk across the prediction range, preferably using a flexible smooth curve with uncertainty. Grouped points can aid communication but can change substantially with the number and boundaries of groups.

The plot should show the distribution of predicted probabilities and avoid relying solely on deciles. Sparse tails may support little evidence about calibration exactly where high-risk decisions occur.

Grouped logistic goodness-of-fit tests have limitations

Tests that group predicted risks and compare observed with expected events, including Hosmer–Lemeshow-type tests, depend on arbitrary grouping and sample size. Different group definitions can yield different conclusions for the same predictions.

Such a test should not be the sole evidence of logistic-model adequacy. Calibration plots, intercept and slope, proper scoring rules, residual diagnostics, and external or resampled validation provide a more informative assessment.

The Brier score measures probabilistic accuracy

For a binary outcome, the Brier score is the mean squared difference between predicted probability and observed outcome. Lower values indicate better combined calibration and discrimination, but interpretation depends on event prevalence and the comparator.

$$Brier =\frac{1}{n} \sum_{i=1}^{n} (p_i-y_i)^2$$

A scaled Brier score or decomposition can add context, but every version should be defined. The score does not show where in the probability range the model fails, so calibration plots remain necessary.

Count models need dispersion diagnostics

Poisson models assume equality of the conditional mean and variance. Extra-Poisson variation can create overdispersion, while a narrower distribution can create underdispersion.

Pearson or deviance statistics relative to residual degrees of freedom can screen for dispersion problems, but clustering, excess zeros, omitted covariates, and misspecified functional form can produce similar patterns. Negative-binomial, zero-inflated, hurdle, robust-variance, or hierarchical approaches answer different data-generating questions and should not be chosen by one statistic alone.

Survival models require censoring-aware diagnostics

Ordinary residuals and $R^2$ do not transfer directly to censored time-to-event data. Cox–Snell, martingale, deviance, and Schoenfeld residuals address different aspects of survival-model adequacy.

Schoenfeld residuals can assess proportional hazards, while time-specific calibration compares predicted survival probabilities with censoring-adjusted observed outcomes. A proportional-hazards check is not an overall goodness-of-fit test, and acceptable proportionality does not validate the baseline hazard, functional forms, or extrapolated tail.

Flexible survival curves need tail validation

Health-economic models often extrapolate survival well beyond observed follow-up. Several parametric or spline models can fit the observed interval similarly but predict very different long-term survival and mean life-years.

In-sample likelihood, AIC, residuals, and visual fit should be combined with proportional-hazards assessment where relevant, external evidence, clinical plausibility, hazard shape, and sensitivity analysis. Good fit within follow-up does not justify an unsupported extrapolated tail.

AIC and BIC are comparison criteria

Akaike's information criterion and the Bayesian information criterion balance fitted likelihood against model complexity. They compare candidate models fitted to the same response data under compatible likelihood definitions.

$$AIC=-2\ell_{fit}+2p$$ $$BIC=-2\ell_{fit}+p\ln n$$

Lower values are preferred within the candidate set, but their absolute values are not universal goodness-of-fit scores. A model can have the lowest AIC and still fit poorly if every candidate omits an important pattern.

Pseudo R-squared measures are not interchangeable

Logistic, count, and survival models have several statistics labelled pseudo $R^2$. They are based on different likelihood, prediction, or explained-variation concepts and do not generally equal the proportion of outcome variance explained.

Reports must name and define the selected measure rather than presenting “$R^2$” without qualification. Values should not be compared across incompatible definitions, samples, outcome prevalences, or model families.

Influence and outliers are different from poor global fit

An outlying observation has an unusual outcome given the fitted model, while a high-leverage observation has unusual predictors. Influence measures how much an observation changes fitted parameters or predictions when its information is perturbed or removed.

Cook's distance, leverage, and change-in-estimate diagnostics can identify influential records in regression, but thresholds are contextual. An influential observation may be a valid and important part of the target population rather than an error to delete.

Internal fit and predictive validation are different

In-sample goodness of fit assesses how the model describes the data used to estimate it. Predictive validation asks how well the model performs for new observations from the intended target setting.

Cross-validation, bootstrap optimism correction, temporal validation, geographic validation, and fully external validation address increasingly distinct forms of transport. Data preprocessing, variable selection, and hyperparameter tuning must occur inside resampling loops to avoid leakage and overly optimistic results.

Overfitting can look like excellent fit

A highly flexible model can reduce training residuals by capturing random noise. Its apparent fit improves while its new-data calibration and prediction may deteriorate.

Penalisation, shrinkage, prespecified complexity, adequate sample size, and resampling can reduce or measure overfitting. Adding parameters until a fit test becomes non-significant is a data-driven strategy that invalidates ordinary inferential claims.

Bayesian models use predictive checks

Bayesian model assessment can compare observed data with replicated data generated from the posterior predictive distribution. Discrepancy measures and graphical checks reveal whether the fitted model can reproduce features that matter scientifically.

Posterior predictive p-values are not ordinary frequentist p-values and can be conservative because the observed data influence both parameter estimation and checking. The selected discrepancies should target known risks such as tails, zeros, subgroup patterns, or temporal dependence rather than only a generic average.

Fit of health-economic evidence models

Regression, survival, network meta-analysis, mapping, and cost models used in economic evaluation each require outcome-specific diagnostics. A fit measure from one component does not validate the entire decision model.

For example, a survival model may fit trial data well while the economic model mishandles treatment waning, subsequent therapy, utilities, or resource use. Component-level fit should be linked to structural validation, parameter verification, and decision-output checks.

Calibration of decision models

A decision model may estimate uncertain parameters so selected outputs reproduce epidemiological or clinical targets. Agreement with calibration targets is a model-fit criterion, but it is partly achieved by construction.

The report should show target definitions, uncertainty, weights, objective function, parameter bounds, convergence, alternative parameter sets, and goodness of fit for each target. Data reserved for external validation should not be reused as calibration targets and then presented as independent evidence.

A weighted calibration objective

When targets have different scales and precision, a weighted sum of squared discrepancies is one possible calibration objective. The weights determine which targets dominate the fitted parameter set.

$$J(\boldsymbol\theta) =\sum_{j=1}^{K} w_j \left[ M_j(\boldsymbol\theta)-T_j \right]^2$$

Here $M_j(\boldsymbol\theta)$ is the modelled target, $T_j$ is the observed target, and $w_j$ is its weight. Choosing $w_j$ as inverse variance can be defensible when uncertainties and independence assumptions are valid, but correlated targets require a covariance-aware objective.

Decision relevance can differ from statistical fit

A statistically detectable lack of fit may have negligible impact on incremental net benefit, while a small discrepancy near a clinical or reimbursement threshold may change the decision. Diagnostics should therefore examine both statistical adequacy and consequences for the decision output.

Decision-focused checks must not excuse a fundamentally invalid evidence model. They help prioritise defects by impact after the underlying statistical assumptions and data provenance have been assessed.

Implementing basic checks in Excel

Excel can calculate transparent residual, error, and chi-square checks for small teaching or audit datasets. Each row should contain the observed value, fitted or expected value, residual, squared residual, and any standardised contribution.

For observed values in B2:B101 and fitted values in C2:C101, residuals use =B2-C2, SSE uses =SUMXMY2(B2:B101,C2:C101), and predictive RMSE uses =SQRT(SUMXMY2(B2:B101,C2:C101)/COUNT(B2:B101)). A chi-square contribution uses =(Observed-Expected)^2/Expected, with explicit checks for zero or sparse expected counts.

Plots should use unrounded fitted values and should display residuals against fitted values and relevant covariates. Specialist statistical software is preferable for leverage-adjusted residuals, censored outcomes, clustered data, resampling, and model-specific reference distributions.

Validation and reproducibility checks

Goodness-of-fit calculations should themselves be validated. Incorrect grouping, degrees of freedom, prediction scale, or reuse of training data can reverse the interpretation.

• Data checks: Observed and fitted values refer to the same records, units, outcome definition, and time horizon.
• Scale checks: Predictions and residuals are calculated on the intended response, link, hazard, survival, or transformed scale.
• Parameter checks: Degrees of freedom reflect all parameters estimated from the evaluated data under the adopted model.
• Residual checks: Residual definitions match the model family and reproduce known special cases.
• Graph checks: Axes, smoothing, grouping, and uncertainty displays do not conceal sparse or influential regions.
• Validation checks: Development, tuning, internal validation, and external validation datasets remain distinct.
• Simulation checks: Reference distributions and posterior predictive replications use sufficient draws and reproducible settings.
• Replication checks: An independent implementation reproduces selected residuals, statistics, p-values, and calibration measures.

The analysis should prespecify which diagnostics are primary and how material lack of fit will be handled. Searching across many diagnostics and reporting only reassuring results creates selective validation.

Common errors and misunderstandings

Goodness of fit is often reduced to one p-value or headline statistic. That shortcut can conceal poor calibration, overfitting, subgroup failure, and unsupported extrapolation.

• Treating a non-significant test as proof of fit confuses absence of evidence with evidence of adequacy.
• Treating a small p-value as practical failure ignores sample size and the decision importance of the discrepancy.
• Using $R^2$ as universal validation applies a linear-regression summary outside its meaning.
• Selecting the lowest AIC as proof of good fit compares candidates without checking whether any candidate is adequate.
• Assessing only discrimination can approve systematically miscalibrated probabilities.
• Using grouped calibration alone makes conclusions depend on arbitrary bins.
• Deleting influential observations automatically can remove genuine members of the target population.
• Validating on training data overstates new-data performance and hides overfitting.
• Equating calibrated targets with external validation counts the same evidence twice.

Fit diagnostics identify discrepancies between data and a model under specific assumptions. They cannot determine whether the data are unbiased, the estimand is appropriate, or the model will remain valid in another population or future period.

A practical workflow

A defensible goodness-of-fit assessment begins with the model's intended use and the data-generating assumptions that support it. The workflow combines numerical measures, graphical diagnostics, validation, and decision consequences rather than seeking a single pass mark.

1. Define the purpose: State whether the model is intended for explanation, prediction, calibration, extrapolation, or decision analysis.
2. Specify the model assumptions: Identify the outcome distribution, link, variance, dependence, censoring, functional form, and parameter structure.
3. Choose targeted diagnostics: Select residuals, plots, scores, and tests that address the most consequential assumptions.
4. Calculate transparently: Preserve the prediction scale, degrees of freedom, grouping rules, and exact statistic definitions.
5. Inspect patterns: Examine residuals by fitted value, time, covariates, treatment, subgroup, and influential observations.
6. Assess calibration and discrimination: Evaluate both dimensions when the model produces risks or probabilities.
7. Validate predictions: Use resampling or independent data without leakage and quantify optimism and uncertainty.
8. Test alternatives: Compare defensible functional forms, distributions, tail assumptions, and model structures.
9. Evaluate decision impact: Determine whether identified lack of fit changes clinically or economically relevant conclusions.
10. Report completely: Present favourable and unfavourable diagnostics, uncertainty, limitations, and corrective actions.

What should be reported

A reproducible report identifies the dataset, sample exclusions, outcome, prediction horizon, model family, link, parameters, estimation method, handling of missingness and dependence, and every diagnostic used. It defines residuals and fit statistics, gives degrees of freedom and reference distributions, presents plots with uncertainty, and distinguishes development, internal validation, and external validation performance.

For health-economic use, the report should connect component-model fit to survival, costs, utilities, event risks, calibration targets, extrapolation, and incremental decision outcomes. It should explain which defects materially affect cost effectiveness or budget impact and which uncertainties remain outside the fitted model.