Monte Carlo Error

The statistical imprecision in a simulation estimate from using a finite number of iterations, distinct from the underlying parameter uncertainty being characterised.

Source: Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford University Press; 2006. doi:10.1093/oso/9780198526629.001.0001.

Monte Carlo error is the imprecision in a simulation's estimate that comes from using a finite number of random draws rather than infinitely many. Because it is sampling noise, averaging over more iterations steadies the estimate, and the error falls in proportion to the square root of the number of runs, so quadrupling the iterations roughly halves it. Unlike parameter uncertainty, which reflects real gaps in knowledge, Monte Carlo error is an artefact of the computation that enough runs can shrink as small as wanted. It is reducible by effort alone. Briggs and colleagues (2006) describe it.

Source: Briggs et al. 2006

Monte Carlo error is reduced by running more iterations, since it decreases roughly with the square root of the number of iterations, so quadrupling the iterations roughly halves the error. Variance reduction techniques, such as common random numbers or Latin hypercube sampling, can also reduce it for a given number of runs. The number of iterations is chosen so that the Monte Carlo error is small relative to the quantities of interest, judged by whether the estimates stabilise as iterations increase.

Source: Briggs, Claxton & Sculpher 2006

Monte Carlo error is the imprecision from using a finite number of simulation runs and can be made arbitrarily small by running more, whereas parameter uncertainty is the genuine uncertainty about the true values of the model's inputs, reflecting limited evidence, and does not shrink with more simulation runs. The simulation aims to characterise parameter uncertainty, and Monte Carlo error is the noise added by finite sampling in doing so. The two must be distinguished, since only Monte Carlo error is reduced by increasing iterations.

Source: Briggs, Claxton & Sculpher 2006

Monte Carlo error matters because if too few iterations are run, the estimates of quantities such as mean costs, mean effects, or the probability of cost-effectiveness carry noticeable random imprecision, which could affect conclusions or make results hard to reproduce. Ensuring Monte Carlo error is small relative to the quantities of interest gives confidence that reported results reflect the model and its parameter uncertainty rather than sampling noise. Checking and controlling Monte Carlo error is therefore part of conducting a reliable probabilistic analysis.

Source: Briggs, Claxton & Sculpher 2006

Monte Carlo error is assessed by examining whether the estimated quantities stabilise as the number of iterations increases, for instance by tracking running estimates and seeing whether they still move, and by estimating the standard error of key outputs, which reflects the imprecision from finite sampling. Repeating the simulation with different random seeds and comparing results also indicates the error. If the estimates still vary materially, more iterations are run until the Monte Carlo error is small relative to the quantities being reported.

Source: Briggs, Claxton & Sculpher 2006