Monte Carlo Error

Monte Carlo Error: Concept Architecture

Orientation and learning roadmap

This page separates the random variation caused by finite simulation runs from uncertainty about the health-economic model itself. It develops a simple standard-error calculation, uses a probability-of-cost-effectiveness example, and explains how to check precision in probabilistic and patient-level simulations.

Where the error comes from

A Monte Carlo estimate averages or counts outcomes generated from random draws rather than evaluating every possible draw. Repeating the computation with a different random seed can produce a slightly different numerical answer even when the model, parameter distributions, and data are held fixed.

That between-run fluctuation is numerical sampling error, not evidence that a parameter is intrinsically uncertain or that the model's structural assumptions are correct. With independent draws and a finite variance, the Monte Carlo standard error (MCSE) of a sample mean decreases approximately as the inverse square root of the number of runs. More draws improve numerical precision but cannot remove bias from a wrong model, inappropriate parameter distribution, or flawed code.

Estimating precision for a sample mean

Let $g_1,\ldots,g_N$ be independent outputs of a single simulation experiment and $\overline g=N^{-1}\sum_{r=1}^{N}g_r$ its average. If $s_g$ is the sample standard deviation across those outputs, an estimated Monte Carlo standard error is $s_g/\sqrt N$ under independent sampling.

$$\widehat{\operatorname{MCSE}}(\overline g) =\frac{s_g}{\sqrt N}.$$

This standard error quantifies the numerical uncertainty in $\overline g$ as an approximation to the expectation under the chosen simulation distribution. It is distinct from the spread $s_g$ across possible outcomes, which can remain large as $N$ grows. Increasing $N$ fourfold approximately halves MCSE when independent sampling and stable variance conditions apply.

Worked example: probability of cost effectiveness

Suppose a probabilistic sensitivity analysis draws parameters independently 1,000 times and finds positive incremental net monetary benefit in 600 runs at a stated willingness-to-pay threshold. Treat each run's cost-effectiveness indicator as a Bernoulli outcome for this illustrative calculation.

The simulated probability is $\widehat p=600/1000=0.60$, with approximate MCSE

$$\widehat{\operatorname{MCSE}}(\widehat p) =\sqrt{\frac{\widehat p(1-\widehat p)}{N}} =\sqrt{\frac{0.60(0.40)}{1000}} \approx0.0155.$$

An approximate interval expressing only simulation precision is $0.60\pm1.96(0.0155)$, or about $0.570$ to $0.630$. If the same simulation design used 10,000 independent draws and the underlying probability remained near 0.60, its expected MCSE would be about $0.0049$; the actual estimate and standard error must be recalculated from those new draws. Neither interval represents the range of clinical effects or the uncertainty caused by uncertain evidence and assumptions.

Checks for economic-model outputs

The required number of draws depends on the output and precision needed for the decision. A cost-effectiveness probability near a decision threshold, an expected value of information result, and an incremental net-benefit mean can have different simulation-error tolerances.

The denominator in an MCSE calculation must be the number of effectively independent draws relevant to the reported statistic, not merely the number of lines of model output. For patient-level simulation nested inside parameter draws, report or assess the inner patient-history noise separately from the outer parameter-sampling error. Re-running with larger samples and inspecting stability is useful, but a predeclared tolerance on the decision-relevant output is clearer than a universal draw count.

Dependence and variance reduction

The formula $s_g/\sqrt N$ does not apply unchanged to serially correlated Markov chain Monte Carlo (MCMC) draws. In that setting, autocorrelation lowers effective information, and a suitable spectral, batch-means, or effective-sample-size-based MCSE should be used after checking chain behaviour and the estimator's assumptions.

Common random numbers, antithetic variables, or other variance-reduction methods can change the variance of a comparison without changing the underlying target expectation when applied correctly. Their precision should be measured for the actual paired or transformed statistic, rather than inferred from the variability of individual strategy outputs. A fixed seed improves reproducibility but does not make numerical error zero.

Interpreting a stable result

Results should state the estimand, algorithm, seed policy, number of draws, MCSE or equivalent precision diagnostic, and any dependence among draws. Reporting too many decimal places can suggest evidence precision that the simulation and input data do not support.

In health technology assessment, adequate numerical precision helps distinguish a real change in the model's expected output from variation between simulation runs. Even a negligible MCSE leaves parameter uncertainty, structural uncertainty, sampling error in source studies, and possible bias from implementation choices. Cost-effectiveness decisions should keep these uncertainty sources conceptually separate.

Source basis and further learning

The linked guidance motivates enough simulation runs for stable health-economic conclusions, while the statistical methods paper addresses MCSE when draws are dependent. The formulas above are for clearly specified estimands and sampling designs, so the precision calculation must match the implementation.

• NICE, NICE technology appraisal and highly specialised technologies guidance: the manual, economic evaluation. This recommends sufficient probabilistic simulations to minimise Monte Carlo error.
• ISPOR-SMDM Modeling Good Research Practices Task Force, “Model Parameter Estimation and Uncertainty Analysis”, Value in Health (2012). This separates model uncertainty and probabilistic analysis.
• Flegal, Haran, and Jones (2008), “Markov Chain Monte Carlo: Can We Trust the Third Significant Figure?”, Statistical Science, 23. This treats MCSE estimation for correlated MCMC output.