Markov Assumption

Markov Assumption and Memoryless Transitions

In a health economic model, the Markov assumption means that where a simulated person goes next depends on the health state they occupy now, and on model time when probabilities change with age, but never on how or when they arrived there. Everyone in a state shares one future, so history that changes risks, costs or utilities must be written into the states, or the assumption dropped. Briggs and Sculpher's introduction to Markov modelling names it as the limitation analysts should particularly be aware of, and the ISPOR-SMDM good practice report on state-transition modelling calls it the primary disadvantage of cohort models. One standard workaround in a cohort state-transition model is a tunnel state, which builds time since entry into the states themselves. The Markov chain and Markov model pages hold the theory and the full model.

Formal statement of what the next transition may depend on

The assumption is a statement about conditional probabilities. It fixes what the next transition may use. Writing the state occupied at cycle $t$ as $X_t$, the model is Markov when the earlier path adds nothing once the current state and cycle are known, so that

$$\Pr(X_{t+1} = j \mid X_t = i, X_{t-1}, \dots, X_0) = \Pr(X_{t+1} = j \mid X_t = i) = p_{ij}(t)$$

where $i$ and $j$ index the health states, $X_{t-1}, \dots, X_0$ is the path before cycle $t$, and $p_{ij}(t)$ is the transition probability from $i$ to $j$ in cycle $t$, an entry of that cycle's transition matrix. The discarded path holds two kinds of information, namely which states the person passed through and how long they have already spent in state $i$.

Sonnenberg and Beck's practical guide to Markov models in medical decision making lets the probabilities change over time in the general Markov process, so a time-inhomogeneous model with age-specific mortality is still Markov; the constant case, a time-homogeneous model, is what they called a Markov chain. The dependence on $t$ follows only the clock that started with the model, so two people in state $i$ at cycle $t$ face the same exit probability whether they arrived one cycle ago or ten. With a constant probability of staying, $p_{ii}$, the number of cycles $D$ spent in state $i$ on one visit (entry cycle counted) is geometric, with

$$\Pr(D = d) = p_{ii}^{\,d-1} (1 - p_{ii}), \qquad E[D] = \frac{1}{1 - p_{ii}}$$

where $d = 1, 2, \dots$ is the length of the stay in cycles and $E[D]$ is its mean. The chance of leaving is the same in every cycle of the stay, so a single state cannot represent a risk that falls or rises with time since entry. A model in which risk depends on time since entering the current state, with the clock reset at each transition, is called semi-Markov.

What the assumption means inside a cohort model

Sonnenberg and Beck put it clinically, saying that everyone in a state has the same prognosis regardless of previous history, so a separate state is needed for each subset of the cohort with a distinct utility or prognosis. Their examples split a disabled state by whether a bleed or an embolus caused it, and a well state into well with gallstones and well after cholecystectomy; if prognosis depends in any way on past history, each history needs its own state. They concede that the assumption is not followed strictly in medical problems, but is needed to model prognosis with a finite number of states.

The ISPOR-SMDM report turns this into a homogeneity requirement, under which every individual in a state has the same transition probabilities, so characteristics that determine them must not differ within the state. If prior states or time in the current state matter, the states should carry that history, as in its example of disease-free states with and without a prior myocardial infarction. Unobserved characteristics count too; ignoring them can create what the report calls heterogeneity bias. A cohort model tells individuals apart only by state, so whatever the states omit is lost to costs, utilities and transitions alike.

Where history matters through time since an event and prior events

The ISPOR-SMDM report explains why the assumption bites, since past states and time in the current state tend to be strong determinants of what happens next. Two patterns recur in economic evaluation.

Time since an event. Risks, costs and utilities often change with time since diagnosis, surgery or an acute event. Hawkins, Sculpher and Epstein note that a Markov model generally cannot make transition probabilities depend on time spent in a state; where risks depend on time in a treatment state, the simplification can cost the model its face validity and give misleading results.

Prior events and pathways. Prognosis can depend on a previous infarction, an earlier treatment line or how long it took to progress. NICE Decision Support Unit Technical Support Document 19 (TSD 19) notes that patients who take longer to progress may do better on post-progression treatment. Its review of 30 NICE cancer appraisals found that its seven state-transition models often assumed death after progression to be independent of treatment allocation, prior patient history and sometimes time in state; in a number of appraisals they did not predict within-trial overall survival well.

Standard workarounds and what each one carries

Each workaround stores the missing history where the model can see it. The ISPOR-SMDM report notes that history states risk state explosion, and TSD 19 lists tunnel states, semi-Markov approaches and patient-level simulation for transitions that depend on time in an intermediate state or in previous states.

Time-dependent probabilities are the cheapest fix. The tutorial on time-dependent cohort models by Alarid-Escudero and colleagues separates simulation-time dependence, on time since the model started, from state-residence time dependence. A cycle-specific matrix handles the first because the cohort ages together, but not time since onset when members fall ill at different times, for which the tutorial uses tunnel states, in whole cycles of the chosen cycle length.

Hawkins, Sculpher and Epstein added a time dimension to a Markov model, which they called semi-Markov; the tutorial indexes such an array by current state, next state and time in state, and warns that adding simulation time could need four dimensions. In continuous time, Williams and colleagues describe resetting the clock on entry to progression as semi-Markov.

Individual-level models are not limited by the Markovian property, the ISPOR-SMDM report says, because they simulate one person at a time and keep histories such as past states, time in state and time since the last event in tracker variables. It recommends a cohort model when a manageable number of states can carry all relevant history, and an individual-level model, such as Markov microsimulation, when it cannot.

Worked example of a preventive treatment judged with and without a tunnel state

All numbers are illustrative, cycles are annual and life-years are counted at the end of each cycle. People start event-free. Each year 10% have a non-fatal event under standard care (5% with a preventive treatment) and 2% die. In the year after an event the probability of death is 0.25, and 0.05 a year from then on, so a follow-up study of survivors would report five-year survival of 0.75 × 0.95 × 0.95 × 0.95 × 0.95 = 0.61088.

1. The tunnel model. The post-event state is split into a one-cycle tunnel state T1 (the year after the event) and a long-term state L. Under standard care the event-free row is (stay 0.88, to T1 0.10, die 0.02), T1 sends 0.75 to L and 0.25 to death, and L keeps 0.95.

2. The memoryless model. A single post-event state needs one annual death probability. Fitting it to the same five-year survival gives

$$q = 1 - S_5^{1/5} = 1 - 0.61088^{1/5} = 1 - 0.90613 = 0.09387$$

where $q$ is the constant annual probability of death after an event and $S_5 = 0.61088$ is five-year post-event survival. Year-two deaths among those with a year-one event fall from 0.025 to 0.0094 of the cohort, but the long-run risk is 0.09387 instead of 0.05.

3. Life-years per event. Each event brings the entry year plus survival after it, giving 1 + 0.75/0.05 = 16 years with the tunnel, and 1/q = 10.653 years without it. Despite matching five-year survival, the single state cuts post-event life by a third.

4. Life expectancy by strategy. Years event-free are 0.88/0.12 = 7.333 under standard care and 0.93/0.07 = 13.286 with treatment, and the lifetime chance of an event is 0.10/0.12 = 0.8333 against 0.05/0.07 = 0.7143. Life expectancy adds that chance times post-event years, as the table shows.

5. The bias. The gain is 5.952 extra event-free years less 0.1190 fewer events per person times the post-event years per event. The memoryless model undervalues those years and overstates the gain by about 0.64 life-years, or 15.7%. With an illustrative treatment cost of GBP 500 a year while event-free and no other costs, the incremental cost is GBP 6,643 in both models, so the cost per life-year gained is GBP 1,641 with the tunnel and GBP 1,418 without it, 13.6% too low.

6. Why recalibration does not help. Setting q = 1/16 = 0.0625 matches mean post-event survival and the undiscounted gain, but puts one-year post-event survival at 0.9375 instead of 0.75. Discounting each end-of-cycle count at an illustrative 3.5% a year, the gain is 1.520 discounted life-years with the tunnel, 1.386 with q = 0.0625 and 1.823 with q = 0.09387. No single probability fits both early and late risk; one tunnel state fits both.

Testing and reporting the assumption

Whether history matters is an empirical question, best settled with patient-level data. Williams and colleagues fitted a state-arrival extended model, with time in the previous state as a covariate for death after progression, to help decide whether to accept the Markov assumption; finding evidence against it, they used a semi-Markov model, which they also suggest when power to detect a violation is low and doubt remains.

The choice belongs in the model's documentation and its structural uncertainty analysis. TSD 19 recommends that a state-transition model's summary of structural assumptions cover any assumptions about the dependence of post-baseline transitions on prior patient history. Comparing predicted with observed overall survival is a direct check.

Boundaries and common misunderstandings

Most errors read the assumption too broadly or too narrowly. Each item marks a boundary.

• Markov does not mean constant. Probabilities that change with age or cycle keep a model Markov; dependence on the path or on time in state breaks it.
• A model-time curve is not time since entry. Applying a time-since-progression curve to the progressed state as a function of model time treats every progressor as having progressed at the start, which is right only if everyone enters together.
• Microsimulation is not history-aware by default. It relaxes the assumption only for the histories it stores and uses in its transition rules.
• Within-cycle corrections do not restore memory. A half-cycle correction fixes the timing of rewards within a cycle, not missing history.
• Neighbouring terms. Memorylessness also names the property of the exponential distribution. A hidden Markov model is Markov in unobserved states, and a partitioned survival model takes state membership from survival curves instead of transition probabilities.

Sources

• Alarid-Escudero F, Krijkamp E, Enns EA, Yang A, Hunink MGM, Pechlivanoglou P, Jalal H. A tutorial on time-dependent cohort state-transition models in R using a cost-effectiveness analysis example. Medical Decision Making. 2023;43(1):21-41. https://doi.org/10.1177/0272989X221121747
• Briggs A, Sculpher M. An introduction to Markov modelling for economic evaluation. PharmacoEconomics. 1998;13(4):397-409. https://doi.org/10.2165/00019053-199813040-00003
• Hawkins N, Sculpher M, Epstein D. Cost-effectiveness analysis of treatments for chronic disease: using R to incorporate time dependency of treatment response. Medical Decision Making. 2005;25(5):511-519. https://doi.org/10.1177/0272989X05280562
• Siebert U, Alagoz O, Bayoumi AM, Jahn B, Owens DK, Cohen DJ, Kuntz KM. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Value in Health. 2012;15(6):812-820. https://doi.org/10.1016/j.jval.2012.06.014
• Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338. https://doi.org/10.1177/0272989X9301300409
• Williams C, Lewsey JD, Mackay DF, Briggs AH. Estimation of survival probabilities for use in cost-effectiveness analyses: a comparison of a multi-state modeling survival analysis approach with partitioned survival and Markov decision-analytic modeling. Medical Decision Making. 2017;37(4):427-439. https://doi.org/10.1177/0272989X16670617
• Woods B, Sideris E, Palmer S, Latimer N, Soares M. NICE DSU Technical Support Document 19: partitioned survival analysis for decision modelling in health care: a critical review. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2017. https://sheffield.ac.uk/nice-dsu/tsds/partitioned-survival-analysis