Signature
R_req = (s_run / e_target)^2
| Inputs | Definition | Unit |
|---|---|---|
s_run | Standard deviation, with divisor one less than the number of pilot runs, of the run-level outcome such as mean cost or QALYs per person, or of its paired increment between strategies | units of the outcome |
e_target | Largest acceptable Monte Carlo standard error of the mean outcome over runs, above zero | units of the outcome |
R_req | Number of independent runs of the agent-based model whose mean outcome has a Monte Carlo standard error equal to the target, before rounding up | runs |
|---|
Function
Agent state update function of an agent-based model
Maps the state of agent i at step t, the set of agents in contact with it, the state of the environment, the intervention in force and a random input to the state of that agent at step t+1, the rule X_i(t+1) = F(X_i(t), N_i(t), E(t), A(t), U_i(t)) of the article. In an infection model the rule for a susceptible agent reduces to a per-step infection probability from its k infectious contacts, the complement of the binomial probability of no events (HE-FM-BINOM-002), followed by a comparison of the agent's own uniform random number with that probability. A per-contact probability built from a transmission rate uses the constant-rate conversion HE-FM-TP-001, and a probability observed over another interval is rescaled with HE-FM-TP-003; neither is restated here. Population outputs are averaged over independent runs, and the Monte Carlo standard error of such a mean is the standard deviation across runs divided by the square root of the number of runs, the form of HE-FM-ANTV-002 with one observation per run and the divisor of HE-FM-BSE-001. This package adds the number of runs needed for a target standard error of a mean and of a share of runs. Attack rates and the final size relation are in HE-FN-ATR-001 and HE-CF-ATR-001, the cohort simulation without interaction is HE-FN-CSIM-001, and expected onward infections and the reproduction number belong to the Basic Reproduction Number page. Notation follows the Agent-Based Model article.
Try this function
Implementations
Excel
Runs needed for a target Monte Carlo standard error in Excel
With the pilot standard deviation in a cell named RunSD, computed with STDEV.S over the pilot run results, and the target in TargetMCSE, CEILING rounds the result up to a whole number of runs. ROUND to six decimals first stops floating-point residue, such as 1600.0000000000002, being rounded up to 1,601. The tests below name the result cell RunsRequired.
=CEILING(ROUND((RunSD/TargetMCSE)^2,6),1)
Assumptions
Independent runs with separate random number seeds
Each run uses its own random numbers, so the run results are independent. Morris and colleagues advise setting the random number seed once per repetition and using separate streams of random numbers when chunks of the simulation run in parallel. Agents interact within a run, so the run, not the agent, is the unit of replication.
Pilot standard deviation taken as a conservative estimate
s_run comes from an initial small set of runs and is taken on the high side. Morris and colleagues base their count on a conservative estimate from an initial small simulation run and state that it is revised if the standard deviation turns out larger. The count fixes the standard error only. An interval built from that standard error also treats the mean over runs as close to normal, which needs a reasonably large number of runs.
Paired runs with common random numbers for an incremental outcome
For an incremental cost or QALY, s_run is the standard deviation of the paired differences between strategies within each run, computed with the same random numbers in both strategies (HE-AS-ABM-006). The spread of each strategy's own total ignores the positive correlation that common random numbers create and gives a larger count than needed.
Worked examples
Morris and colleagues' standard deviation of 0.2 and target of 0.005
In the worked simulation study of Morris and colleagues the standard deviation of the estimates is taken as at most 0.2 and the Monte Carlo standard error of bias is required to be below 0.005, which gives 1,600 repetitions. The same arithmetic applies to a mean output of an agent-based model.
s_run = 0.2; e_target = 0.005; R_req = 1600
Illustrative incremental cost per person with a spread of GBP 300 across paired runs
For an illustrative agent-based model whose pilot runs give a standard deviation of GBP 300 in the incremental cost per person between strategies, a Monte Carlo standard error of GBP 10 needs 900 runs.
s_run = 300; e_target = 10; R_req = 900
Halving the target standard error quadruples the agent-based model runs
Tightening the illustrative target to GBP 5 needs 3,600 runs, four times as many, because the standard error falls with the square root of the number of runs.
s_run = 300; e_target = 5; R_req = 3600
Common errors
Counting agents rather than runs as the replications
Dividing the spread of outcomes across the agents of one run by the square root of the number of agents, for example 5,000, treats the agents as independent when interaction makes their outcomes correlated, so the standard error is understated and too few runs are chosen. The standard error of a mean over runs comes from the spread across independent runs.
Doubling runs to halve the Monte Carlo standard error
Doubling the illustrative 900 runs to 1,800 gives a standard error of about GBP 7.07, not GBP 5. Halving the standard error needs 3,600 runs.
Sources
Morris and colleagues on the Monte Carlo standard error of a mean and the repetitions needed
Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Statistics in Medicine. 2019;38(11):2074-2102. Table 6, which gives the Monte Carlo standard error of estimated bias as the square root of the sum of squared deviations of the estimates from their mean divided by n_sim times (n_sim minus 1), and section 7.1, which takes the standard deviation of the estimates as at most 0.2 from an initial small simulation run, requires a Monte Carlo standard error of bias below 0.005 and concludes that 1,600 repetitions are needed. The checklist in Table 1 advises setting the random number seed once per repetition and using separate streams when running chunks in parallel.
Chhatwal and He on the number of Monte Carlo runs in agent-based models
Chhatwal J, He T. Economic evaluations with agent-based modelling, an introduction. PharmacoEconomics. 2015;33(5):423-433. Section 2.5, which states that agent-based models are typically stochastic, that first-order uncertainty can be higher than in state-transition or discrete event models and can be reduced by running the model several times with different random numbers, and that the sufficient number of Monte Carlo runs depends on the structure, the number of parameters and the time horizon and can be assessed qualitatively or by evaluation of the variance in model outcomes. Section 5.4 and Table 1 report 5,000 runs of each scenario in the case study.
Canonical Identity
Stable URI · Machine-readable · Resolvable · CC BY 4.0