Goodness of Fit Test

Goodness of Fit Test: Concept Architecture

Orientation and learning roadmap

The page begins with a statistical model and the feature of observed data it is expected to reproduce. It then builds a count-based test, works through its statistic and probability, and compares it with distributional and predictive checks. The final sections explain why passing a test cannot by itself validate a health-economic decision model.

State what is being tested

A goodness of fit test compares observed data with a stated probability model under a null hypothesis. The test statistic measures a particular kind of discrepancy, and its reference distribution describes how surprising a discrepancy at least that large would be if the null model and testing assumptions held. The question must specify the observations, model, fitted parameters if any, and chosen discrepancy before a $p$ value has a useful interpretation.

Other tests emphasise different discrepancies, such as the maximum gap between an empirical and a specified continuous cumulative distribution. A goodness of fit result is always relative to its chosen test and its power to detect departures. It is not a universal certificate that the model is true.

Calculate a Pearson chi-square test

For $k$ mutually exclusive categories, let $O_j$ be an observed count and $E_j=np_j$ the count expected from the null category probability $p_j$. The Pearson statistic adds each squared count discrepancy divided by its positive expected count. Under a fully prespecified multinomial null with suitable sample size and non-sparse expected categories, the usual approximate reference has $k-1$ degrees of freedom.

$$X^2=\sum_{j=1}^{k}\frac{(O_j-E_j)^2}{E_j}, \qquad E_j=np_j>0, \qquad\sum_{j=1}^{k}O_j=\sum_{j=1}^{k}E_j=n.$$

Independence of the sampled units and a correct category definition are material assumptions. When model parameters are estimated from the same counts, the reference degrees of freedom may be reduced and the exact adjustment depends on the fitting and test construction. Sparse expected counts can make the chi-square approximation unreliable, prompting a defensible regrouping or an appropriate simulation or exact approach.

Check a four-category health example

Suppose 100 independent patients are classified into four exhaustive categories, and the prespecified null model assigns probability 0.25 to each. The observed counts below are illustrative rather than results from a study. Each expected count is 25, comfortably above the sparse-cell concern in this example.

The statistic is $X^2=1+1+0+0=2$, with $k-1=3$ degrees of freedom because no probabilities were estimated from these counts. The upper-tail probability is approximately 0.57; at a conventional 0.05 level, this sample does not reject the specified distribution. It does not show that every category probability truly equals 0.25, nor that a difference of practical importance has been excluded.

Choose the test for the data and model

A chi-square goodness of fit test uses binned counts and is different from a chi-square test of independence across two categorical variables. For a fully specified continuous distribution, a Kolmogorov–Smirnov statistic compares empirical and proposed cumulative distributions; its standard one-sample critical values do not remain valid unchanged when distribution parameters are estimated from the same data. The null distribution and any parameter-fitting correction must match the actual procedure.

The statistical test must be selected before using its output to defend an economic model. Comparing in-sample fit statistics among candidate models, assessing calibration of predicted risks, and checking external validity answer related but distinct questions. Clinical plausibility and the decision's time horizon can matter even when several models fit the observed data similarly.

Interpret rejection and non-rejection carefully

A small $p$ value signals data less compatible with the specified null model in the direction the statistic detects, subject to sampling and test assumptions. A large $p$ value may arise because the model is reasonably compatible with the data, the sample is too small, or the chosen statistic is insensitive to the departure that matters. The $p$ value is neither the probability that the model is true nor a measure of economic or clinical importance.

In a health-economic evaluation, an incorrect survival tail, baseline event risk, or subgroup pattern can change expected QALYs and costs despite acceptable fit to observed aggregates. Examine effect sizes, residual patterns, predicted versus observed quantities, competing specifications, and sensitivity of decisions to plausible alternatives. The decision model itself also requires structural, internal, external, and predictive validation where applicable.

Report enough to reproduce the check

Readers need to see the null distribution, data source, category boundaries or continuous statistic, parameter-estimation procedure, sample size, reference distribution, and calculated result. Report prespecified and exploratory checks separately, especially when several models or binning choices were tried. If a formal test is inappropriate for sparse or dependent data, explain the alternative rather than forcing a familiar $p$ value.

• Verify that categories are mutually exclusive and exhaustive and that observed and expected totals match.
• Inspect every expected count and document any justified pooling of categories.
• State whether parameters were prespecified or estimated from the tested sample.
• Provide the statistic, degrees of freedom or resampling design, and upper-tail probability where relevant.
• Examine clinically and economically consequential deviations beyond the single summary test.

Sources and technical basis

NIST documents the Pearson count-based statistic and the limitations of the standard Kolmogorov–Smirnov reference when parameters are estimated. ISPOR-SMDM good research practices distinguish broader decision-model transparency and validation from a single statistical fit check. The four-category numbers and calculation are original teaching values.

• NIST, Chi-Square Goodness-of-Fit Test.
• NIST, Kolmogorov–Smirnov Goodness-of-Fit Test.
• ISPOR-SMDM, Model Transparency and Validation.