Markov Model

A Markov model represents disease progression or another evolving process as movement among defined health states over repeated time cycles. This page explains how states, cycles and transition probabilities work together, how costs and health outcomes are accumulated, and how the model’s memory assumptions affect interpretation. It also shows how to construct and validate a cohort Markov model and when a different model structure may be more appropriate.

How a Markov model represents change over time

A Markov model divides the relevant clinical process into a finite set of health states. During each model cycle, every member of the modelled cohort occupies one state and may remain there or move to another state. Repeating this process produces a trace of how the cohort is distributed across the states over time.

Health states should be:

• Mutually exclusive, so each person occupies only one state at a particular time.
• Collectively exhaustive, so every person in the model can be assigned to a state.
• Clinically and economically meaningful, so differences in prognosis, costs or health-related quality of life can be represented.
• Detailed enough to answer the decision question, without adding distinctions that do not affect the analysis.

A simple chronic-disease model might contain Stable Disease, Progressed Disease and Death. More complex models may distinguish treatment response, complications, recurrence, severity or treatment history when those distinctions materially affect future outcomes.

Cycles, time horizon and cycle length

Time advances through repeated intervals known as cycles. During each cycle, patients may remain in their current state or transition to another permitted state. The model continues until it reaches the specified time horizon.

Cycle length should be short enough to represent clinically important changes without creating unnecessary computational detail. Weekly or monthly cycles may be appropriate for rapidly changing conditions, while annual cycles may be sufficient for slower disease progression.

The time horizon should be long enough to capture all material differences between the strategies. A lifetime horizon is often needed when an intervention changes survival, long-term complications or future quality of life.

The transition probability matrix

Transition probabilities describe the chance of moving from each current state to every possible state during one model cycle. In a cohort model, these probabilities are arranged in a transition matrix. Each row represents a current state, each column represents the state occupied in the next cycle, and every row must sum to one.

For a three-state model:

$$ \mathbf{P}= \begin{bmatrix} p_{SS} & p_{SP} & p_{SD}\ p_{PS} & p_{PP} & p_{PD}\ p_{DS} & p_{DP} & p_{DD} \end{bmatrix} $$

Where:

• (S) represents Stable Disease.
• (P) represents Progressed Disease.
• (D) represents Death.
• (p_{SP}) is the probability of moving from Stable Disease to Progressed Disease during one cycle.
• (p_{SS}) is the probability of remaining in Stable Disease.
• Death is normally an absorbing state, so (p_{DD}=1).

The probability of remaining in a state is often calculated as the remainder after all possible exit probabilities have been specified:

$$ p_{SS}=1-p_{SP}-p_{SD} $$

This calculation is valid only when the competing transition probabilities have been estimated on a coherent basis for the same cycle and population.

Updating the cohort through the model

The cohort trace records the proportion of the cohort occupying each state during every cycle. The initial state vector describes the cohort at the beginning of the model, and matrix multiplication produces the distribution in the next cycle.

If the state distribution is written as a row vector:

$$ \mathbf{s}{t+1}=\mathbf{s}{t}\mathbf{P} $$

Where:

• (\mathbf{s}_{t}) is the state distribution during cycle (t).
• (\mathbf{P}) is the transition probability matrix.
• (\mathbf{s}_{t+1}) is the state distribution during the following cycle.

After (n) cycles:

$$ \mathbf{s}{n}=\mathbf{s}{0}\mathbf{P}^{n} $$

The entries in each state vector should sum to one unless the model intentionally uses another accounting convention. Any loss or unexplained creation of cohort mass indicates a structural or calculation error.

Worked example: a three-state cohort model

Consider a cohort that begins entirely in Stable Disease. During each annual cycle, patients may remain stable, progress or die. Patients who have progressed may remain in Progressed Disease or die, while Death is absorbing.

The initial state vector is:

$$ \mathbf{s}_{0}=[1,\ 0,\ 0] $$

The illustrative transition matrix is:

After the first cycle:

$$ \mathbf{s}_{1}=[1,\ 0,\ 0]\mathbf{P} $$

$$ \mathbf{s}_{1}=[0.75,\ 0.20,\ 0.05] $$

After the second cycle:

$$ \mathbf{s}_{2}=[0.75,\ 0.20,\ 0.05]\mathbf{P} $$

$$ \mathbf{s}_{2}=[0.5625,\ 0.3100,\ 0.1275] $$

The second-cycle distribution means that 56.25% of the original cohort is expected to be in Stable Disease, 31.00% in Progressed Disease and 12.75% in Death. These are expected cohort proportions, not predictions of which specific individuals will occupy each state.

Attaching costs and health outcomes to states

State rewards assign costs, utilities or other outcomes to time spent in each health state. Multiplying state occupancy by the applicable reward produces the expected outcome for that cycle. Summing across cycles produces the model’s total expected cost and health outcome.

Suppose the annual state rewards are:

Using the end-of-cycle occupancy convention for this simplified example, expected first-cycle cost is:

$$ (0.75\times £2,000)+(0.20\times £10,000)+(0.05\times £0) $$

$$ =£3,500 $$

Expected first-cycle QALYs are:

$$ (0.75\times 0.85)+(0.20\times 0.55)+(0.05\times 0) $$

$$ =0.7475\text{ QALYs} $$

Expected second-cycle cost is:

$$ (0.5625\times £2,000)+(0.3100\times £10,000) $$

$$ =£4,225 $$

Expected second-cycle QALYs are:

$$ (0.5625\times 0.85)+(0.3100\times 0.55) $$

$$ =0.648625\text{ QALYs} $$

A real model must state whether rewards are attached at the beginning, middle or end of a cycle. The timing convention can affect results, particularly when cycle lengths are long or transitions occur rapidly.

State rewards and transition rewards

Not every cost or health consequence belongs to a health state. Some consequences occur only when a transition takes place, such as an acute hospital admission, procedure, treatment initiation or adverse event. These should be represented as transition rewards when attaching them permanently to a state would count them repeatedly.

• A state reward applies while a person occupies a health state.
• A transition reward applies when a particular movement between states occurs.
• An entry cost applies once when a person first enters a state or begins a strategy.
• A terminal cost may apply when death or another final event occurs.

The reward structure should prevent both omissions and double counting. For example, an acute progression cost should not be included as both a transition cost and part of every subsequent cycle in the progressed state unless the latter represents a separate continuing cost.

The Markov memory assumption

The conventional Markov property assumes that future transitions depend on the current state and the modelled time, but not on the full pathway used to reach that state. Two patients occupying the same state are therefore treated as having the same future transition probabilities unless the model explicitly distinguishes them.

This assumption may be unsuitable when future risk depends on:

• Time spent in the current state.
• The number of previous events.
• Age or calendar time.
• Prior treatment or treatment sequence.
• Time since treatment initiation.
• Previous complications or disease history.

The issue is not that Markov models can never represent history. The modeller can introduce additional states, tunnel states, time-dependent transition probabilities or individual-level simulation to retain relevant information. Each extension should be justified because added complexity can make the model harder to validate and explain.

Using tunnel states to retain temporary history

A tunnel state divides a health state into temporary stages so that transition probabilities, costs or utilities can depend on time since entry. Patients move through the tunnel stages in sequence and cannot remain indefinitely in an earlier stage.

For example, a Post-Event state could be divided into:

• Post-Event: First Cycle.
• Post-Event: Second Cycle.
• Post-Event: Subsequent Cycles.

This structure can represent a temporarily elevated mortality risk or a one-time recovery period. Tunnel states preserve limited history while retaining the general state-transition framework.

Converting rates into transition probabilities

Clinical evidence may report event rates rather than cycle probabilities. A rate and a probability are not interchangeable, particularly when events are frequent or cycle lengths are long. Under a constant-rate assumption, a rate (r) can be converted into a probability over time interval (t):

$$ p=1-e^{-rt} $$

Where:

• (r) is the event rate per unit of time.
• (t) is the model-cycle duration in the same time units.
• (p) is the probability of the event during the cycle.

This simple conversion is not sufficient when several competing events can occur. Competing risks should be handled jointly so that transition probabilities remain coherent and sum to no more than one.

Cycle length and within-cycle correction

A cohort model records patients at discrete cycle boundaries even though real events can occur at any time. Without adjustment, the model may behave as though all transitions occur at the beginning or end of a cycle. A within-cycle correction approximates the timing of transitions and rewards within each interval.

Half-cycle correction is a commonly used approximation, but it should not be applied automatically. The need for correction depends on cycle length, reward timing, transition timing and the numerical integration method used. Shortening the cycle or using a more suitable integration method may sometimes be preferable.

Discounting costs and health outcomes

Costs and health outcomes occurring in future cycles may need to be discounted to their present values. The discount rate, reference time and timing convention should follow the applicable jurisdictional guidance and remain consistent across the model.

For an outcome (V_t) occurring at time (t):

$$ PV(V_t)=\frac{V_t}{(1+r)^t} $$

Where:

• (PV(V_t)) is the present value.
• (r) is the annual discount rate.
• (t) is the time measured from the reference point.

If cycles are shorter than one year, the timing exponent must reflect the fraction of a year represented by each cycle.

Comparing alternative strategies

Each intervention or comparator normally requires its own transition probabilities, costs, utilities or other strategy-specific inputs. The model produces expected total costs and outcomes for every strategy over the same time horizon.

The strategy results can then be compared using:

• Incremental costs and incremental health outcomes.
• Incremental cost-effectiveness ratios where ratio interpretation is meaningful.
• Net monetary benefit or net health benefit.
• Dominance and extended-dominance analysis when several strategies are compared.
• Probabilistic results showing decision uncertainty.

The model structure should remain comparable across strategies unless a genuine clinical difference requires a different pathway. Structural differences should be documented rather than hidden inside separate calculations.

Cohort simulation and individual-level simulation

A cohort Markov model moves proportions of a cohort among states. An individual-level state-transition simulation follows one simulated person at a time and can retain patient characteristics and event history more flexibly. Both approaches may use Markov or extended state-transition logic, but they produce results differently.

An individual-level model is not automatically superior. The additional complexity should answer a material feature of the decision problem that cannot be represented adequately in a simpler cohort model.

When a Markov model is appropriate

Markov models are useful when disease progression or treatment consequences involve recurring risks and repeated movement among a manageable set of health states. They are frequently used for chronic disease, recurrence, long-term complications and lifetime economic evaluation.

A Markov model may be appropriate when:

• Patients can remain in a health state for several cycles.
• Events or risks recur over time.
• Long-term state occupancy drives costs or health outcomes.
• The modelled process can be represented using a manageable number of states.
• Relevant history can be represented through the states or justified extensions.

The model should be selected because its structure matches the decision problem, not simply because Markov models are familiar or widely used.

When another model structure may be preferable

A conventional Markov model may be unsuitable when event timing is highly individual, resources interact, transmission occurs between people, or clinically important history would require an unmanageable number of states. Alternative structures should be considered when they provide a clearer representation of the decision problem.

A hybrid structure may be appropriate, but each component should have a clear role and the transfer of patients, costs and outcomes between components must be validated.

Examining uncertainty

Markov-model results depend on uncertain transition probabilities, costs, utilities and structural assumptions. Sensitivity analysis examines whether those uncertainties affect expected outcomes or the preferred strategy.

Relevant analyses include:

• Deterministic sensitivity analysis of individual parameters.
• Scenario analysis of structural assumptions, time horizons and cycle lengths.
• Probabilistic sensitivity analysis of joint parameter uncertainty.
• Alternative assumptions about extrapolation, treatment effects and mortality.
• Structural sensitivity analysis using different state definitions or model forms.

Probability distributions and correlations should reflect the way model inputs were estimated. Transition probabilities drawn during probabilistic analysis must remain valid and preserve the required row totals.

Validating a Markov model

Validation examines whether the model is implemented correctly and provides a credible representation of the decision problem. It should be planned throughout development rather than performed only after the final results are produced.

Validation should include:

1. Check the conceptual model. Confirm that the states, transitions, strategies, population, perspective and time horizon reflect the decision problem.
2. Check the transition matrices. Verify that probabilities are valid, prohibited transitions remain zero and every row sums to one.
3. Check the cohort trace. Confirm that state occupancy remains non-negative and totals one in every cycle.
4. Check calculations. Reproduce selected cycles manually and confirm that costs, QALYs, discounting and within-cycle corrections are applied correctly.
5. Test extreme values. Set parameters to zero or one where meaningful and confirm that the model behaves as expected.
6. Compare external outcomes. Assess whether predicted survival, event rates or state occupancy are consistent with relevant observed evidence.
7. Review uncertainty. Confirm that sensitivity and probabilistic analyses preserve logical and mathematical constraints.

Calibration may be required when unobserved parameters are selected so that model outputs reproduce relevant empirical targets. Calibration does not replace validation, and calibrated parameters and targets should be reported transparently.

Implementing a cohort model in a spreadsheet

A spreadsheet implementation should separate transition probabilities, state rewards, cohort traces and accumulated outcomes. Visible checks make the model easier to audit and reduce the risk that a plausible-looking result conceals an invalid transition structure.

Transition matrices should be labelled clearly and kept separate for different strategies. Hard-coded adjustments buried inside formulas make validation and updating more difficult.

Common modelling errors

Markov models can produce stable numerical results even when the underlying structure is inappropriate. Structural reasoning, data consistency and validation are therefore as important as correct matrix multiplication.

Common errors include:

• Defining states that overlap or fail to cover the full cohort.
• Allowing transition-matrix rows to total something other than one.
• Applying annual probabilities directly to shorter model cycles.
• Treating rates as though they were probabilities.
• Ignoring competing risks when constructing transition probabilities.
• Using the Markov property when clinically important history affects future risk.
• Adding tunnel states without updating rewards and transition logic consistently.
• Counting transition costs again as recurring state costs.
• Applying an unjustified half-cycle correction.
• Using different time horizons or outcome definitions across strategies.
• Failing to verify that cohort occupancy remains non-negative and sums to one.
• Reporting deterministic precision without examining parameter and structural uncertainty.