State Transition Model

State Transition Model Family Behind Markov Cohorts and Microsimulation

A state transition model is the broad class of decision model in health economics that describes a disease as a set of health states, the moves people can make between them and how likely each move is in a given period. The class covers the Markov cohort model, which follows a whole cohort as shares spread across the states, and individual-level microsimulation, which follows simulated patients one at a time with their own histories. The ISPOR-SMDM task force report by Siebert and colleagues (2012) calls state-transition modelling one of the most widespread techniques in health technology assessment and health economic evaluation, so choices made inside the family shape many of the cost per QALY estimates that reach reimbursement committees. The term is wider than "Markov model", and the difference matters because the memoryless property that limits a cohort model does not bind an individual-level one.

Elements shared by every state-transition model

Siebert and colleagues list the formal elements as states, transitions, an initial state vector, transition probabilities, a cycle length, state values (often called rewards), logical tests at the start of each cycle that decide the transitions, and termination criteria. The states must be mutually exclusive and collectively exhaustive, so a modelled person is in exactly one state in any cycle. Sonnenberg and Beck (1993), in the guide to Markov models that the task force cites for cohort models, add that a person can make only a single state transition during a cycle.

For a cohort run in discrete time the elements combine in one recursion, in which the shares at the end of a cycle are the shares at the end of the previous cycle multiplied by that cycle's transition matrix.

$$\mathbf{s}_t = \mathbf{s}_{t-1} P_t , \qquad E = \sum_{t=1}^{T} \frac{\mathbf{s}_t \mathbf{v}}{(1+r)^{t}}$$

where $\mathbf{s}_t$ is the row vector of state occupancy at the end of cycle $t$ (with $\mathbf{s}_0$ the initial state vector), $P_t$ is the transition matrix for cycle $t$ whose element $p_{ij}(t)$ is the probability that a person in state $i$ at the start of the cycle is in state $j$ at its end, $\mathbf{v}$ is the column vector of state values such as costs, utilities or the cycle length in years for each state alive, $r$ is the discount rate per cycle, $T$ is the number of cycles and $E$ is the expected discounted total. This simple form counts occupancy at the end of each cycle, and the half-cycle correction adjusts for transitions part way through a cycle. An individual-level model uses the same states and probabilities, but a random number decides each simulated person's move in each cycle, and expected values are averages over many simulated people.

Dimensions that separate one state-transition model from another

Siebert and colleagues describe state-transition modelling as a broad category with several dimensions. Models may or may not let transition probabilities depend on the states of other people, may allow transitions only at fixed intervals or in continuous time, and may follow a closed cohort or a dynamic population. The two common frameworks in health care, cohort Markov models and individual-based first-order Monte Carlo microsimulation, assume no interaction, follow a single closed cohort and allow transitions only at specified intervals.

The rows follow Table 1 and the surrounding text of the Siebert report. The cohort form rests on the Markov assumption that the next transition depends only on the current state, which the report calls its primary disadvantage, because past states and time spent in a state often drive what happens next. History can be added through extra states such as a tunnel state, at the risk of what the report calls state explosion.

Choosing between a cohort and an individual-level form

The Siebert report asks the modeller first to list every characteristic the model must carry, including relevant histories such as past states, time in state and time since the last event that determine transition probabilities, costs or utilities. Recommendation III-1 then sets the rule. If a manageable number of states can include the relevant history, a cohort simulation should be chosen for its transparency, efficiency, ease of debugging and support for value of information analysis. If a valid representation would need an unmanageable number of states, an individual-level model is recommended, and validity should not be sacrificed for simplicity.

Individual-level models handle continuous characteristics such as blood pressure without grouping them, and they can represent dynamic strategies in which later decisions depend on a patient's history. The report gives the example of a cervical screening algorithm that lengthens the interval after repeated negative tests. The cost is computation time, which matters most when probabilistic sensitivity analysis or value of information analysis needs many runs, and recommendation III-16 asks for enough simulated individuals to give stable expected values.

When a state-transition structure fits the decision problem

The Siebert report states that a state-transition model should be used in place of a decision tree when the problem needs time-dependent parameters, time to an event or repeated events, with examples such as recurrence after cancer treatment and a second myocardial infarction. The report notes that other techniques, such as discrete event simulation, also suit these problems.

The report also marks the limits of the two standard frameworks. A state-transition model is a reasonable choice when the problem can be framed in terms of states, interactions between individuals are not relevant and the population is a closed cohort. When an intervention against an infectious disease changes transmission in the target population, the companion task force report by Pitman and colleagues (2012) states that a dynamic transmission model is needed instead. Recommendation III-2 asks that sequential decisions, such as switching drugs after first-line failure, form part of each strategy's definition and are not built into the cycle structure.

Linked transitions compared with partitioned survival

NICE Decision Support Unit Technical Support Document 19 (TSD 19, Woods and colleagues, 2017) contrasts state-transition models with partitioned survival analysis, which its review of NICE appraisals found to be the most commonly used decision modelling approach in advanced or metastatic cancer. TSD 19 explains that a state-transition model derives state membership from transition probabilities, while a partitioned survival model reads it from separate survival curves. In a state-transition model, overall survival depends on every transition, so mortality is structurally linked to earlier progression events.

Treatment effects must therefore be specified for each transition, and TSD 19 notes that a benefit on overall survival can arise from an effect on progression alone, from effects on progression and on death after progression, or from a benefit on progression offset by harm after progression. The model needs time-to-event data for each transition, which trials reporting only progression-free and overall survival do not supply directly. Estimating them with multi-state survival analysis is feasible when individual patient data are available. TSD 19 also recommends that partitioned survival analysis should not be described as similar to a Markov approach or as "Markov like".

Worked example of a progression effect that changes survival

The example and all numbers are illustrative. A three-state model of an advanced cancer has the states progression-free (PF), progressed (P) and dead, with annual cycles over three years. Under the comparator a progression-free patient progresses with probability 0.30 and dies with probability 0.05 each year, so stays progression-free with probability 0.65, while a progressed patient dies with probability 0.40 and stays progressed with probability 0.60. The new treatment lowers the progression probability to 0.20, so staying progression-free rises to 0.75, and it has no direct effect on either death probability. Everyone starts progression-free.

For the comparator, cycle 1 gives 0.65 in PF, 0.30 in P and 0.05 dead. In cycle 2, PF is 0.65 × 0.65 = 0.4225, P is 0.65 × 0.30 + 0.30 × 0.60 = 0.375, and the dead share is 0.05 + 0.65 × 0.05 + 0.30 × 0.40 = 0.2025, so the shares sum to one. In cycle 3, PF is 0.4225 × 0.65 = 0.274625 and P is 0.4225 × 0.30 + 0.375 × 0.60 = 0.35175. For the treatment, cycle 2 gives PF of 0.75 × 0.75 = 0.5625 and P of 0.75 × 0.20 + 0.20 × 0.60 = 0.27, and cycle 3 gives PF of 0.5625 × 0.75 = 0.421875 and P of 0.5625 × 0.20 + 0.27 × 0.60 = 0.2745.

Counting one life year for each share alive at the end of each cycle, without discounting or half-cycle correction, the comparator gives 0.95 + 0.7975 + 0.626375 = 2.373875 life years and the treatment gives 0.95 + 0.8325 + 0.696375 = 2.478875, a gain of 0.105 per patient. Survival improves although no death probability changed, because the treatment keeps more of the cohort in the state with the lower death risk (0.05 against 0.40). This is the surrogacy relationship between progression and mortality that TSD 19 says such an effect implies, and its Recommendation 3 asks that any such relationship be reported and justified.

A second illustrative scenario shows the offsetting case. If the treatment also raised the death probability after progression to 0.50, P would be 0.75 × 0.20 + 0.20 × 0.50 = 0.25 after cycle 2 and 0.5625 × 0.20 + 0.25 × 0.50 = 0.2375 after cycle 3, so the shares alive would be 0.5625 + 0.25 = 0.8125 and 0.421875 + 0.2375 = 0.659375. Life years would be 0.95 + 0.8125 + 0.659375 = 2.421875, a gain over the comparator of 0.048, less than half the first estimate. A partitioned survival model would show neither pattern unless its overall survival curve did. Costs and utilities attached to PF and P turn the same traces into an ICER, as on the Markov model page.

Building and reporting a state-transition model credibly

The Siebert report gives recommendations that apply across the family. The starting cohort should be defined by the characteristics that affect transition probabilities or state values (III-3), and states should be homogeneous with respect to observed and unobserved characteristics that affect transition probabilities (III-6), since an unobserved factor such as an undiagnosed infection can cause heterogeneity bias. The time horizon should be lifetime when the intervention affects mortality, and cycle length should be short enough to represent the frequency of clinical events and interventions (III-8).

On parameters, the report asks that probabilities be converted between time units through rates, that the assumed relationship between disease-specific and background mortality be stated, and that double counting be checked when evidence covers both fewer disease events and lower mortality. TSD 19 asks that the structural assumptions of a state-transition model be reported, including the allowed transitions, the transitions on which treatments are assumed to differ, any dependence on patient history and any implied surrogacy relationships. Siebert and colleagues add that intermediate outputs such as 10-year risks, events per lifetime and modelled survival curves should be presented (III-18), so that clinical experts can judge face validity as part of model validation.

Where state-transition models are misread

Most errors come from treating one member as the whole family. Others overlook what the structure implies.

1. Treating the term as a synonym for a Markov cohort model. Individual-level state-transition models are not bound by the Markov assumption, so criticism of memorylessness applies to the cohort form only.
2. Calling a partitioned survival model a state-transition model. TSD 19 recommends keeping the terms apart, since partitioned survival has no structural link between endpoints.
3. Assuming a treatment effect on progression carries no survival consequence. The worked example shows the structure converting a progression effect into a survival gain.
4. Choosing an individual-level form for detail alone. Recommendation III-1 favours the cohort form whenever a manageable set of states can carry the relevant history.

Sources

• Pitman R, Fisman D, Zaric GS, Postma M, Kretzschmar M, Edmunds J, Brisson M. Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-5. Value in Health. 2012;15(6):828-834. https://doi.org/10.1016/j.jval.2012.06.011
• Siebert U, Alagoz O, Bayoumi AM, Jahn B, Owens DK, Cohen DJ, Kuntz KM. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Value in Health. 2012;15(6):812-820. https://doi.org/10.1016/j.jval.2012.06.014
• Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338. https://doi.org/10.1177/0272989X9301300409
• Woods B, Sideris E, Palmer S, Latimer N, Soares M. NICE DSU Technical Support Document 19: partitioned survival analysis for decision modelling in health care: a critical review. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2017. https://sheffield.ac.uk/nice-dsu/tsds/partitioned-survival-analysis