Markov Microsimulation

Markov Microsimulation: Concept Architecture

Orientation and learning roadmap

Markov microsimulation represents health and economic outcomes as simulated individual trajectories through discrete states. This page explains why individual simulation is chosen, how transitions and rewards are generated, how memory and heterogeneity are represented, and how stochastic and parameter uncertainty differ. It then covers cohort comparison, cycle timing, variance reduction, validation, a worked example, implementation, and reproducible reporting.

Individuals move through a defined state space

Each simulated person occupies one modeled state at a time and faces a set of possible transitions at the next cycle. The state space must be mutually exclusive, collectively sufficient for the decision problem, and detailed enough to determine future risks and rewards.

If individual (i) is in state (S_{i,t}=s) at cycle (t), a transition may be drawn from:

[ P(S_{i,t+1}=s'\mid S_{i,t}=s,\mathbf x_i,t,\mathbf h_{i,t}), ]

where (\mathbf x_i) contains individual characteristics and (\mathbf h_{i,t}) contains any explicitly modeled history. The inclusion of history determines whether the implemented process remains strictly Markovian in the chosen state definition.

The Markov property depends on what the state records

A first-order Markov model assumes that the current modeled state contains all history needed to determine the next transition distribution. If future risk depends on prior events, time since entry, treatment exposure, or event counts not encoded in the state, the simple memoryless assumption is violated.

Microsimulation can retain person-level history variables or expand the state definition to restore the required conditional structure. Calling a model “Markov” does not make unrecorded history irrelevant.

Microsimulation is chosen for consequential heterogeneity

Individual-level simulation is useful when patient characteristics, correlated risks, prior events, or nonlinear outcomes materially affect transitions, costs, health, or treatment eligibility. It can preserve combinations of characteristics that a manageable cohort model would average or require many expanded states to represent.

Complexity should follow the decision problem rather than the availability of patient-level data. A cohort state-transition model is preferable when aggregation preserves the required relationships and yields the same decision-relevant results more transparently.

A cohort model moves proportions rather than people

A cohort state-transition model multiplies a state-occupancy vector by a transition matrix to obtain expected occupancy in the next cycle. A microsimulation instead draws a categorical transition for every simulated individual and aggregates the resulting trajectories.

[ \mathbf m_{t+1}=\mathbf m_tP_t ]

is a deterministic cohort update, while individual state draws introduce first-order simulation noise. With homogeneous Markov transitions, large microsimulated populations should converge toward the corresponding cohort expectation.

Individual trajectories are not MCMC draws

Markov microsimulation simulates the evolution of modeled people through health states. Markov chain Monte Carlo constructs a computational chain whose stationary distribution is a statistical target, commonly a Bayesian posterior.

Both use Markov chains and random numbers, but their states, purposes, diagnostics, and interpretations differ. Convergence of a patient microsimulation mean with population size is not the same as posterior convergence of an MCMC sampler.

Transition probabilities must form a valid distribution

For each person, current state, and cycle, all outgoing probabilities must be between zero and one and sum to one. Impossible transitions should have probability zero, and absorbing states should remain absorbing unless the conceptual model explicitly allows exit.

[ 0\leq p_{i,ss'}(t)\leq1, \qquad \sum_{s'}p_{i,ss'}(t)=1. ]

Validation should test these constraints across extreme covariate profiles and every parameter draw. Silently renormalizing invalid rows can conceal incompatible equations or coding errors.

Random numbers select categorical transitions

A uniform random number can select one destination from cumulative transition probabilities. If the outgoing probabilities are (p_1,\ldots,p_K), destination (k) is selected when the draw lies between cumulative boundaries (F_{k-1}) and (F_k).

[ F_k=\sum_{j=1}^{k}p_j, \qquad S_{i,t+1}=k\text{ when }F_{k-1}\leq U_{i,t}<F_k. ]

The destination ordering does not change exact probabilities but can interact with finite random streams and debugging. Boundaries must cover the full unit interval without gaps or overlaps.

A one-cycle example separates draws from expectations

Suppose five identical people begin Healthy and have one-cycle probabilities (0.75) of remaining Healthy, (0.20) of becoming Sick, and (0.05) of dying. Uniform draws (0.10,0.30,0.22,0.80,0.96) produce three Healthy, one Sick, and one Dead trajectory at cycle end.

The expected counts are (3.75), (1.00), and (0.25), but the finite sample must contain whole people, so its discrepancy is first-order Monte Carlo variation rather than evidence of incorrect transition probabilities.

Rewards convert trajectories into costs and health outcomes

State and transition rewards assign costs, utilities, life-years, resource use, or other outcomes to each simulated path. Rewards may depend on individual characteristics, calendar time, state duration, prior events, and treatment.

For cycle discount factor (d_t), total cost for person (i) can be written as:

[ C_i=\sum_{t=0}^{T}d_t \left[c(S_{i,t},\mathbf x_i,t)+c_{trans}(S_{i,t},S_{i,t+1})\right]. ]

Population results are usually estimated by averaging individual totals. Transition rewards must be timed consistently so events are not charged twice or omitted.

The worked sample shows outcome simulation error

Assign one-cycle costs of (100), (1{,}000), and (0) to Healthy, Sick, and Dead and utilities of (0.9), (0.6), and (0). The five simulated people generate total cost (1{,}300), mean cost (260), total QALYs (3.3), and mean QALYs (0.66) for the cycle.

The exact cohort expectations are mean cost (0.75(100)+0.20(1{,}000)=275) and mean QALYs (0.75(0.9)+0.20(0.6)=0.795). Increasing the simulated population reduces the Monte Carlo discrepancy but does not resolve uncertainty in the probabilities, costs, or utilities.

Baseline populations define the simulated decision cohort

Individuals may be generated from joint distributions, sampled from trial or registry records, or constructed as weighted representative profiles. The method must preserve important correlations among age, sex, disease severity, comorbidity, biomarkers, and other modifiers.

Independent sampling from marginal distributions can create unrealistic combinations and distort subgroup outcomes. Population sources, exclusions, weighting, missing-data handling, and transportability to the decision population should be explicit.

Transition risks can vary across people and time

Transition probabilities may be functions of baseline characteristics, attained age, cycle number, calendar time, treatment, prior outcomes, and state residence time. Regression models, survival functions, hazards, risk equations, or calibrated relationships can supply those risks.

When a probability is derived from a rate or hazard, the time interval must match the cycle length. For a constant continuous-time rate (r), the corresponding cycle probability over length (\Delta t) is:

[ p=1-e^{-r\Delta t}. ]

Using (p=r\Delta t) is only an approximation and can become materially inaccurate for large rates or intervals.

Competing transitions require coherent joint probabilities

Several events may compete to occur from the same state, such as progression, discontinuation, and death. Converting each cause-specific rate independently into a probability and then adding the results can yield invalid or biased transition probabilities.

With constant cause-specific rates (r_k) over a cycle, total rate (R=\sum_kr_k), the probability of any event is (1-e^{-R\Delta t}), and cause (k) receives its rate share:

[ p_k=\frac{r_k}{R}\left(1-e^{-R\Delta t}\right). ]

This formula depends on the constant-rate assumption and mutually exclusive first events. More complex within-cycle processes may require continuous-time simulation or finer temporal modeling.

Tunnel states and trackers represent memory differently

Tunnel states expand the state space to record time since entry for a limited number of cycles. Individual trackers retain attributes such as event count, prior treatment, or accumulated exposure without creating a separate health state for every history combination.

Both approaches change future transition or reward calculations and must be specified in the conceptual model. Trackers should not become hidden variables that determine outcomes without documentation or validation.

Recurrent events require explicit within-person dependence

Microsimulation can represent multiple admissions, relapses, adverse events, or treatment courses for the same individual. Event risk may depend on the number, timing, or severity of prior events, creating within-person correlation.

Resetting risk after every event without justification can underestimate clustering or long-term consequences. The model should define whether transitions use calendar time, gap time, time since disease onset, or time since the latest event.

Cycle length controls temporal approximation

Discrete cycles allow at most the modeled sequence of events within each interval. A long cycle can misorder events, miss rapid transitions, or distort time spent in states, while a shorter cycle increases computation and may require re-estimating probabilities.

Cycle length should reflect clinical timing, data resolution, and decision sensitivity. A convergence analysis across shorter cycles can identify material discretization error, but identical conclusions do not prove every within-cycle assumption is correct.

Half-cycle correction is not automatically sufficient

Half-cycle correction approximates events as occurring halfway through a cycle in some cohort state-transition calculations. In an individual simulation, event timing may instead be modeled at cycle boundaries, halfway points, or by sampled continuous times.

The correction must match how rewards and transitions are generated. Applying a generic half-cycle adjustment on top of explicitly simulated event times can double-correct timing.

Continuous-time microsimulation is a related alternative

An individual continuous-time state-transition model samples event times rather than advancing every person through equal cycles. It can represent irregular timing and competing events more naturally when hazards are available.

Continuous time does not eliminate structural assumptions, simulation noise, or the need to define history. It also differs from discrete-event simulation when the latter includes queues, shared resources, and interactions among entities.

First-order uncertainty is simulation noise

First-order uncertainty arises because individual transitions are randomly sampled even when all model parameters are fixed. Repeating the model with different random streams changes the estimated mean costs and outcomes around the same underlying expectation.

Increasing the number of simulated individuals reduces the standard error approximately with the square root of sample size under regular independent-person conditions. The simulation count should be chosen using precision of incremental and decision-relevant outputs, not a conventional round number.

Second-order uncertainty concerns unknown parameters

Second-order uncertainty represents incomplete knowledge about transition equations, costs, utilities, treatment effects, and other model inputs. Probabilistic sensitivity analysis draws a parameter set and evaluates outcomes conditional on that set.

When microsimulation is nested inside probabilistic sensitivity analysis, both parameter variation and first-order noise appear in outputs. The design must use enough individuals per parameter draw, or an efficient alternative, so inner simulation noise does not obscure parameter uncertainty.

Parameter heterogeneity is not parameter uncertainty

Heterogeneity describes real differences among individuals, such as age-specific mortality or severity-specific treatment response. Parameter uncertainty describes uncertainty about the equations or coefficients governing those differences.

Microsimulation can represent both, but they should be labeled and sampled separately. Reporting the spread of individual outcomes as if it were uncertainty about the population mean confuses variability with epistemic uncertainty.

Common random numbers improve strategy comparisons

Using matched random-number streams for the same simulated person under competing strategies can induce positive correlation between strategy outcomes. This often reduces variance in incremental costs and effects because shared chance events cancel in the comparison.

Common random numbers must be aligned by event and pathway so strategy differences do not shift the streams unpredictably. The method affects precision, not the expected value, when implemented correctly.

Antithetic and deterministic alternatives can reduce noise

Antithetic variates pair random draws such as (U) and (1-U) to encourage offsetting simulation errors. Quasi-random sequences or expected-value calculations may also improve efficiency in suitable models.

Variance-reduction methods must preserve the intended distribution and account for dependence when estimating standard errors. A lower-noise result is not necessarily less biased if structural or parameter inputs are wrong.

Incremental outcomes are estimated from paired trajectories

For strategies (A) and (B), the primary comparison uses differences in mean person-level outcomes. With common random numbers, person-matched differences can be averaged directly.

[ \Delta C=\frac{1}{N}\sum_{i=1}^{N}(C_{i,A}-C_{i,B}), \qquad \Delta E=\frac{1}{N}\sum_{i=1}^{N}(E_{i,A}-E_{i,B}). ]

Incremental net monetary benefit is (\lambda\Delta E-\Delta C). Simulation precision should be assessed for these incremental quantities rather than only for each strategy's totals.

Subgroups can be estimated without multiplying states

Because individual attributes remain attached to trajectories, results can be summarized for prespecified subgroups and used to explore heterogeneous cost-effectiveness. The subgroup model must contain credible effect modification and adequate evidence rather than merely produce many slices.

Post hoc searches across numerous characteristics create multiplicity and unstable small-sample results. Equity-relevant subgroup analysis should reflect the decision question, data support, and distributional consequences.

Calibration must account for simulation noise

Calibrating a stochastic microsimulation compares noisy model outputs with calibration targets. Optimization can chase random noise unless seeds, common random numbers, replication, emulation, or a noise-aware objective are used.

Calibration diagnostics should examine target fit, parameter identifiability, correlations, and out-of-sample behavior. A low objective value from one random seed does not establish a valid calibration.

Validation operates at several levels

Internal verification checks code, state logic, probability rows, rewards, and reproducibility. External validation compares appropriate model outputs with independent evidence, while cross-model validation investigates differences from other credible implementations.

Useful tests include deterministic extreme cases, absorbing-state behavior, closed-form or cohort equivalence cases, event counts, survival, state occupancy, cost identities, and conservation rules. Validation should target both mean outcomes and distributions when distributional outputs matter.

Simulation error needs an explicit stopping rule

The required population size depends on output variance and the accuracy needed for the decision. Running cumulative batches and tracking confidence intervals or Monte Carlo standard errors can show whether incremental outcomes have stabilized.

For independent simulated people with sample standard deviation (s_Y), the standard error of a mean is approximately:

[ SE(\overline Y)=\frac{s_Y}{\sqrt N}. ]

Paired strategy differences should use the standard deviation of person-level differences when common random numbers are used. Parameter-draw dependence and clustered patient structures require appropriate alternatives.

Individual output distributions need careful interpretation

Microsimulation can estimate distributions of lifetime cost, QALYs, event counts, or survival, not only means. These distributions may be useful for capacity planning, equity analysis, or risk communication.

An individual's simulated outcome is one realization conditional on the model and is not a prediction interval for a named patient. Tail estimates require more simulations and stronger validation than means.

Excel can teach small models but scales poorly

A spreadsheet can display one row per person and one column per cycle, with random draws, states, costs, and utilities visible for auditing. It becomes fragile when there are many cycles, histories, parameter draws, strategies, or requirements for controlled random streams.

Volatile functions such as RAND() change whenever the workbook recalculates and do not provide robust simulation governance. Production models are usually better implemented in version-controlled code with explicit seeds, tests, efficient arrays, and stored summaries.

A disciplined microsimulation workflow

Reliable microsimulation begins with a conceptual reason for preserving individual pathways and ends with stable incremental decision outputs. Every source of state dependence, heterogeneity, and randomness should be explicit. A practical sequence is:

1. Define the decision problem. Specify population, strategies, perspective, horizon, outcomes, and heterogeneity of interest.
2. Justify individual simulation. Show which person-level dependence or nonlinear relationship cannot be represented adequately by a simpler cohort model.
3. Design the states. Define mutually exclusive states, allowable transitions, absorbing states, trackers, and event timing.
4. Generate the population. Preserve relevant joint distributions, weights, exclusions, missingness, and transportability.
5. Parameterize transitions. Align equations, rates, probabilities, competing risks, covariates, time scales, and treatment effects.
6. Assign rewards. Specify state, transition, event, and terminal costs and outcomes with consistent timing and discounting.
7. Control randomness. Set independent streams, common random numbers, replications, and variance-reduction methods.
8. Verify the implementation. Test probability bounds, closed-form cases, cohort equivalence, traces, identities, and extreme inputs.
9. Quantify uncertainty. Separate first-order error, parameter uncertainty, heterogeneity, and structural uncertainty.
10. Report reproducibly. Preserve code, data, seeds, versions, algorithms, simulation counts, validation, and precision criteria.

Common errors and safeguards

Microsimulation can represent rich histories, but that flexibility also creates hidden assumptions and substantial computational noise. Strong governance makes the simulated pathway auditable from input generation to decision result.

• Choosing microsimulation without showing why a cohort model is inadequate.
• Calling a model memoryless while unrecorded history changes future risk.
• Sampling baseline attributes independently when their correlation affects outcomes.
• Allowing outgoing probabilities to be negative or sum to more than one.
• Converting competing rates into probabilities independently and adding them.
• Applying half-cycle correction despite explicitly simulated within-cycle event times.
• Reporting one stochastic run without a Monte Carlo precision assessment.
• Treating person-to-person variability as parameter uncertainty.
• Letting inner microsimulation noise dominate probabilistic sensitivity analysis.
• Using unmatched random streams and mistaking simulation noise for a strategy effect.
• Calibrating to a single seed and overfitting random variation.
• Reporting many subgroup results without credible effect modification or adequate simulation size.
• Using volatile spreadsheet randomness without saved seeds, draws, or reproducible outputs.
• Validating only mean costs while ignoring state occupancy, events, survival, and outcome distributions.

Reproducible reporting makes individual pathways auditable

A complete report identifies the model states, transition rules, memory variables, cycle length, event ordering, horizon, baseline population source, individual attributes, correlation structure, risks, competing-event method, rewards, discounting, and strategy-specific pathways. It should also state software and version, pseudorandom generator, seeds or streams, simulation size, replications, common-random-number design, variance reduction, runtime, and stopping criteria.

Results should separate first-order Monte Carlo error, parameter uncertainty, heterogeneity, and structural uncertainty. Verification, calibration, external validation, subgroup definitions, incremental outcomes, and the sensitivity of the decision to simulation size and cycle length should be reported rather than hidden behind a single cost-effectiveness estimate.