Cohort Simulation

Cohort Simulation of a State-Transition Model as Expected Proportions

Cohort simulation is the usual way to obtain results from a Markov model in health economic evaluation. Instead of following people one at a time, the model moves fractions of a notional population between health states, and each cycle's costs and quality-adjusted life years are read off those fractions. Because the fractions are probabilities, the result is an expected value per person, with no random noise to average away. It is one of several ways of evaluating a Markov process, built on a simple recursion with its own within-cycle counting rules, and the ISPOR-SMDM guidance prefers individual-level simulation in some circumstances.

Three ways to evaluate the same Markov process

The choice of evaluation method affects what the output contains. Sonnenberg and Beck's practical guide described three, namely a matrix algebra solution, a cohort simulation (one trial, many subjects) and a Monte Carlo simulation (many trials, a single subject each).

The matrix solution, the fundamental matrix, gives the expected number of cycles in each state in closed form, but it needs transition probabilities that are constant over time, the special case the guide calls a Markov chain. Cohort simulation removes that restriction by computing the model cycle by cycle, so probabilities can change with age or time since the start, which Sonnenberg and Beck give as an advantage of the simulation methods. They also called the matrix result exact and unaffected by cycle length, likening the two methods to summing blocks under a curve versus solving the integral. That needs qualifying, because the matrix result is exact for the discrete-time chain only. Their own guide notes that it counts membership at the start of each cycle and needs a half-cycle correction, so it equals a cohort run to absorption counted the same way, carries the same within-cycle error, and changes with the cycle length (worked example, step 5).

The cycle-by-cycle recursion

The cohort is held as a row vector of proportions, one entry per health state, and a transition matrix moves it forward one cycle at a time. The states must be mutually exclusive and collectively exhaustive, so every member of the cohort is in exactly one state in every cycle and each row of the matrix sums to one.

$$\mathbf{m}_{t+1} = \mathbf{m}_t \, \mathbf{P}_t$$

where $\mathbf{m}_t$ is the row vector of proportions in each state at cycle $t$, with $\mathbf{m}_0$ the starting distribution, and $\mathbf{P}_t$ is the matrix whose element $p_{ij}$ is the transition probability of moving from state $i$ to state $j$ during the cycle. Stacking the vectors row by row gives the cohort trace. Costs and health effects are then attached to states and summed as follows.

$$Y = \sum_{t=0}^{T} w_t \, \delta_t \, \mathbf{m}_t \, \mathbf{r}$$

where $Y$ is the expected total outcome per person (cost, life years or QALYs), $T$ is the number of cycles in the time horizon, $\mathbf{r}$ is the column vector of rewards per cycle in each state (for QALYs, the utility weight multiplied by the cycle length), $\delta_t$ is the discount factor for cycle $t$, and $w_t$ is the within-cycle weight that sets how occupancy at each recorded time point is counted. Each strategy is run separately, and its expected costs and QALYs feed the incremental analysis.

Why the output is an expected value

Each state in a cohort model is homogeneous, so everyone in it faces the same transition probabilities and rewards. The proportion in a state at cycle $t$ therefore equals the probability that one member of the cohort is in that state at that time, and the summed rewards are an expected value per person. The cohort size is bookkeeping only. Sonnenberg and Beck start 10,000 hypothetical patients and divide the accumulated utility by 10,000 at the end.

In a Monte Carlo evaluation of the same model, each person's next state is drawn with a random number, and Sonnenberg and Beck note that the mean of many runs will be similar to the cohort's expected utility. The cohort gives the value that mean approaches as more people are simulated, without first-order, between-person noise, but it reports means. Sonnenberg and Beck wrote that measures of variability are not possible with it; the ISPOR-SMDM report calls distributions from cohort models possible but technically more difficult, for example by analysing the Markov trace.

In probabilistic sensitivity analysis, each sampled parameter set is run through the cohort once (the R tutorial by Alarid-Escudero and colleagues uses 1,000 sets), so the spread across iterations reflects parameter uncertainty alone.

Counting rewards within a cycle

The recursion records the cohort at cycle boundaries, but people move at any time within a cycle. The ISPOR-SMDM report states that giving full reward at the start of a cycle overestimates expected values and giving none underestimates them, and recommends a half-cycle correction for costs and effectiveness in the first cycle, and in the final cycle if a lifetime horizon is not used. In the notation above, the common counting rules differ only in the weights $w_t$, as the table shows.

Elbasha and Chhatwal treated these as numerical integration methods. They showed that the standard half-cycle correction gives the same result as the trapezoidal rule and the life-table method, that errors need not cancel when incremental outcomes are calculated, and that a less accurate method can lead to a wrong decision; they recommended Simpson's 1/3 rule in the base case and found that every method improved greatly with a shorter cycle.

Worked example with a three-state trace over four annual cycles

The model is hypothetical and its numbers are illustrative. A cohort starts in Healthy. Each year 15% of the Healthy move to Sick and 5% die, while 30% of the Sick die and none recover; Dead is the absorbing state. Utility weights are 0.9 for Healthy and 0.6 for Sick, cycles last one year and nothing is discounted.

1. Run the recursion. Each row is the previous row multiplied by the transition matrix. For cycle 2, Sick receives 0.15 × 0.80 from Healthy and keeps 0.70 × 0.15 of its own members, giving 0.120 + 0.105 = 0.225.

Every row sums to one, the first check on any trace.

2. Count expected years in Healthy under each rule. End-of-cycle counting gives 0.8 + 0.64 + 0.512 + 0.4096 = 2.3616 years. Start-of-cycle counting gives 1 + 0.8 + 0.64 + 0.512 = 2.952 years. The half-cycle correction gives 0.5 × 1 + 0.8 + 0.64 + 0.512 + 0.5 × 0.4096 = 2.6568 years. Simpson's rule gives (1 + 4 × 0.8 + 2 × 0.64 + 4 × 0.512 + 0.4096)/3 = 2.6459 years.

3. Compare with a continuous-time benchmark. If leaving Healthy follows a constant hazard in continuous time, so that 80% remain after each full year, the expected time in Healthy over four years is the area under the survival curve $S_H(t) = 0.8^{t}$, calculated as follows.

$$\int_0^4 0.8^{t} \, dt = \frac{1 - 0.8^{4}}{-\ln 0.8} = \frac{0.5904}{0.22314} = 2.6459$$

where $t$ is time in years and $-\ln 0.8$ is the constant annual hazard of leaving Healthy implied by the 80% one-year probability. Against this benchmark the end-of-cycle count is about 10.7% too low, the start-of-cycle count about 11.6% too high, the half-cycle correction about 0.4% too high, and Simpson's rule agrees to four decimal places.

4. Turn the trace into QALYs. With the half-cycle correction, expected years in Sick are 0.5 × 0 + 0.15 + 0.225 + 0.2535 + 0.5 × 0.25425 = 0.7556. Expected QALYs over four years are 0.9 × 2.6568 + 0.6 × 0.7556 = 2.8445 per person, and expected life years are 2.6568 + 0.7556 = 3.4124.

5. Check the run to absorption. Because the probabilities are constant, a run to absorption can be checked by matrix algebra as follows.

$$\mathbf{N} = (\mathbf{I} - \mathbf{Q})^{-1}$$

where $\mathbf{Q}$ holds the transition probabilities between Healthy and Sick, $\mathbf{I}$ is the identity matrix of the same size, and each element of $\mathbf{N}$ is the expected number of cycles in a state for a given starting state, counted at the start of each cycle. Starting in Healthy, this gives 1/(1 − 0.8) = 5 cycles in Healthy and 0.15/(0.2 × 0.3) = 2.5 cycles in Sick, the same totals a cohort simulation reaches once the cohort is absorbed, counted at the start of each cycle. Subtracting half a cycle from the starting state gives a half-cycle-corrected life expectancy of 7.0 years and 0.9 × 4.5 + 0.6 × 2.5 = 5.55 QALYs. The matrix figure for Healthy depends on the cycle. With six-monthly cycles the probability of staying Healthy is about 0.8944 per cycle, giving 0.5/(1 − 0.8944) = 4.74 years, while the continuous-time value is 1/0.22314 = 4.48 years, close to the half-cycle-corrected 4.5.

Memory and tunnel states

A cohort simulation knows only the current state of each share of the cohort, so, as Sonnenberg and Beck put it, the process has no memory for earlier cycles, and any history that changes risk, utility or cost has to be written into the states, as the Markov assumption and tunnel state pages explain. Time since the start of the model, such as age, is easier because $\mathbf{P}_t$ can change from cycle to cycle, but each layer of history built into the states multiplies their number, which the ISPOR-SMDM report calls state explosion.

When a cohort is adequate and when individuals are needed

The ISPOR-SMDM guidance makes the choice turn on the state space. The modeller first lists every characteristic and piece of history, such as time in state, that affects transition probabilities, utilities or costs. A cohort simulation should be chosen if the problem can be represented with a manageable number of states that include all relevant characteristics and history, for its transparency, efficiency, ease of debugging and support for specific value of information analyses. If a valid representation would need an unmanageable number of states, the report recommends an individual-level model, adding that validity should not be sacrificed for simplicity.

The report also favours individual-level models when characteristics are continuous, since a cohort model has to group them into categories, and for dynamic strategies in which later decisions depend on a patient's current and past characteristics. Individual-level models take longer to run, which matters for probabilistic analysis, and must simulate enough people for stable expected values, costs a cohort simulation does not carry.

How cohort simulation relates to neighbouring concepts

Several pages describe this family of models from different angles. The cohort model page covers the model from construction to reporting and the Markov cohort model names the model type, while cohort simulation concerns how such a model is evaluated; the cohort trace is the table the simulation produces. Markov microsimulation, microsimulation and individual patient simulation describe the individual-level alternative. Monte Carlo simulation in a probabilistic analysis draws parameter values, which differs from the first-order draws that move individuals through a microsimulation.

Where cohort simulation is misread

Several errors recur. Treating the cohort size as a sample size is one, since a cohort of 1,000 is no more precise than a cohort of 1. Reading the trace as the path of typical patients is another. A value of 0.25 in Sick at cycle 4 is the probability that any one member is Sick at that point, and says nothing about which people are there or how they arrived.

Averaging parameters across a mixed population is a third. Illustratively, if half a cohort has an annual survival probability of 0.9 and half has 0.5, the expected cycles alive, counted at the start of each cycle, are 1/(1 − 0.9) = 10 and 1/(1 − 0.5) = 2, a mean of 6, whereas one cohort run at the average of 0.7 gives 1/(1 − 0.7) = 3.33. Outcomes are not linear in transition probabilities, so subgroups that differ in risk need separate cohorts or states, with results weighted afterwards. This is consistent with the ISPOR-SMDM advice that the starting cohort be defined by the characteristics that affect transition probabilities or state values.

Finally, the half-cycle correction is often applied mechanically or not at all, although Elbasha and Chhatwal found cases where it does more harm than good; the counting rule, cycle length and any correction belong in the reported methods and sensitivity analysis.

Sources

• Alarid-Escudero F, Krijkamp E, Enns EA, Yang A, Hunink MGM, Pechlivanoglou P, Jalal H. An introductory tutorial on cohort state-transition models in R using a cost-effectiveness analysis example. Medical Decision Making. 2023;43(1):3-20. Preprint read at https://arxiv.org/abs/2001.07824
• Elbasha EH, Chhatwal J. Theoretical foundations and practical applications of within-cycle correction methods. Medical Decision Making. 2016;36(1):115-131.
• Elbasha EH, Chhatwal J. Myths and misconceptions of within-cycle correction: a guide for modelers and decision makers. PharmacoEconomics. 2016;34(1):13-22.
• Siebert U, Alagoz O, Bayoumi AM, Jahn B, Owens DK, Cohen DJ, Kuntz KM. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Value in Health. 2012;15(6):812-820.
• Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338.