Gaussian Process Regression

Gaussian Process Regression: Concept Architecture

Orientation and learning roadmap

The page starts with training observations and a covariance rule, then follows their role in prediction at an untried input. It works through a small conditional-Gaussian calculation and shows how a fitted process can emulate an expensive health-economic model. The later sections explain validation, computational limits, and how emulator uncertainty differs from uncertainty about the clinical evidence.

Specify the regression target and inputs

A Gaussian process (GP) places a joint probability model over function values at any finite collection of input settings. The analyst chooses a mean function $m(\mathbf x)$ and covariance function or kernel $k(\mathbf x,\mathbf x')$ to describe plausible patterns of the unknown relationship. Training data update that distribution, yielding a predictive mean and uncertainty at new inputs under the assumed model.

The observed response might be a measured health outcome, or it might be the output of a slow simulation evaluated at selected parameter combinations. These are different data-generating settings: empirical observations include measurement and sampling processes, while a deterministic simulator may have little output noise at exactly repeated inputs. The selected input region should cover the settings relevant to the decision, including meaningful interactions or boundaries.

Derive predictions from the covariance matrix

For $n$ training inputs, let $\mathbf K$ be their kernel matrix, $\mathbf k_*$ the vector of covariances with a new input $\mathbf x_*$, and $k_{**}=k(\mathbf x_*,\mathbf x_*)$. With independent zero-mean Gaussian observation noise of variance $\sigma_n^2$, conditional Gaussian calculations give the mean and variance of the latent function value at $\mathbf x_*$. The formulas condition on the selected kernel and its hyperparameters; uncertainty in those choices is not automatically represented.

$$\begin{aligned} \mu_*&=m(\mathbf x_*)+\mathbf k_*^{\mathsf T} (\mathbf K+\sigma_n^2\mathbf I)^{-1}(\mathbf y-\mathbf m),\\ v_*&=k_{**}-\mathbf k_*^{\mathsf T} (\mathbf K+\sigma_n^2\mathbf I)^{-1}\mathbf k_*. \end{aligned}$$

Here $\mathbf m$ contains the mean function evaluated at the training inputs. The variance $v_*$ describes uncertainty about the latent function conditional on the fitted model; the variance of a new noisy observed response also includes its specified observation-noise variance. For a deterministic simulator, a nugget may still be introduced for numerical stability or model discrepancy, but its meaning should be stated.

Check a small predictive calculation

Take a zero mean, two noiseless training responses $\mathbf y=(2,4)^{\mathsf T}$ in thousands of pounds, training covariance $\mathbf K=\mathbf I$, cross-covariance $\mathbf k_*=(0.5,0.5)^{\mathsf T}$, and prior variance $k_{**}=1$ in squared thousand-pound units. These values form a valid illustrative joint covariance and permit a transparent conditional calculation. They are not a calibrated kernel fitted to real health-economic results.

$$\mu_*=(0.5,0.5)\begin{pmatrix}2\\4\end{pmatrix}=3, \qquad v_*=1-(0.5^2+0.5^2)=0.5.$$

The conditional mean is £3,000 and the conditional standard deviation is $\sqrt{0.5}\approx0.707$ thousand pounds under this illustrative GP. That standard deviation measures predictive uncertainty conditional on this covariance model; it does not by itself capture treatment-effect uncertainty, model structure, or extrapolation beyond represented inputs. The calculation should not be interpreted as a cost-effectiveness estimate for a real intervention.

Emulate a slow economic model

An emulator is fitted to a designed set of input-output runs from a computationally costly model and used to predict outputs at additional parameter combinations more cheaply. Published health-economic applications have used GP metamodels for breast cancer screening and emulation of cancer simulation to make uncertainty analysis more tractable. The original economic model still specifies the patient pathways, costs, outcomes, and assumptions that the GP approximates.

For a fixed willingness-to-pay value, an analyst could train an emulator for each strategy's net monetary benefit or for an incremental net-benefit difference. Training separate cost and QALY emulators is another option, but their joint dependence must be handled if combined uncertainty is required. Neither a smooth prediction nor a small GP posterior variance establishes that the decision model's clinical parameters are correct.

Validate where the decision depends on the result

Hold-out runs of the original simulator at inputs not used in fitting can test whether predictions and uncertainty intervals are adequately calibrated in the relevant region. Check behaviour near decision boundaries, high-cost or low-probability parameter combinations, and regions that probabilistic sensitivity analysis samples often. A GP that fits training outputs closely can still be biased under an unsuitable kernel, mean function, or design.

• Define the input distributions, training design, output units, kernel, mean, and noise assumption.
• Inspect residuals and predictive coverage against held-out simulator runs or observed data, as appropriate.
• Compare key decision quantities, such as expected net benefit and probability of optimality, using original-model runs where feasible.
• Quantify the effect of emulator approximation error on conclusions and value-of-information calculations.
• Avoid interpreting low uncertainty near training points as evidence that inputs or causal assumptions are known precisely.
• Reassess hyperparameters and validation when the population, model structure, or input range changes.

Understand practical and conceptual limits

Exact GP fitting commonly requires matrix computations that grow rapidly with training-set size, so sparse or approximate approaches may be needed for large datasets. High-dimensional inputs, sharp discontinuities, categorical treatment choices, and stochastic simulator noise can make a standard smooth-kernel GP difficult to fit. Different kernels and transformed outputs can yield different predictions, especially when extrapolating away from training runs.

Gaussian process regression is a probabilistic regression or surrogate technique, not a substitute for randomisation, causal identification, or a complete cost-effectiveness model. Its posterior uncertainty conditions on chosen modelling assumptions; probabilistic sensitivity analysis represents uncertainty in economic-model inputs according to another specified distribution. If the GP is used inside that analysis, both layers should be described without double counting or silently dropping uncertainty.

Sources and technical basis

Rasmussen and Williams provide the standard GP regression framework and conditional predictive equations. Published health-economic applications demonstrate its use as a metamodel for expensive simulations and decision analysis. The numerical two-point calculation above is an original algebraic example, not empirical evidence.

• Rasmussen and Williams, Gaussian Processes for Machine Learning.
• Rojnik and colleagues, Gaussian process metamodeling in breast cancer screening value-of-information analysis.
• Evaluating parameter uncertainty in a simulation model of cancer using emulators.
• Strong, Oakley and Brennan, regression-based estimation of partial expected value of perfect information.