Concept Architecture
Concept
Theoretically, Markov Chain is a stochastic process in which the probability of moving to a future state depends solely on the current state and not on the sequence of previous states. This property, known as the Markov property, provides the mathematical foundation for state-transition modelling. In health economics, Markov chains are used to represent disease progression, treatment pathways and long-term health outcomes through transitions between mutually exclusive health states.
Mathematically, a Markov chain is represented by a finite set of states and a transition probability matrix describing the probability of moving between states during each cycle. The future distribution of the process is obtained by repeated multiplication of the current state probability vector by the transition matrix. This framework estimates expected state occupancy, survival, costs and health outcomes over time.
In practice, transition probabilities are estimated from clinical trials, observational studies, registries or survival analyses. The transition matrix is incorporated into cohort or individual-level state-transition models, with repeated cycles generating estimates of life expectancy, quality-adjusted life years, healthcare costs and cost-effectiveness. Model validation includes assessment of transition probabilities, structural assumptions and calibration against observed outcomes.
Purpose
Used to model stochastic transitions between health states over time, providing the mathematical basis for estimating disease progression, health outcomes and economic consequences in state-transition models.
Mathematical Formulae
Primary Formula
State distribution update:
????? = ?????
where:
- ??? = probability distribution across health states at cycle t
- ?? = transition probability matrix
Supporting Formulae
Markov property:
P(X??? = j | X? = i, X???, ?, X?) = P(X??? = j | X? = i)
Transition matrix:
?? = [p??]
where
p?? = P(X??? = j | X? = i)
State distribution after n cycles:
??? = ?????�
Related Mathematical Methods
- Matrix algebra
- Stochastic processes
- State-transition modelling
- Markov modelling
- Eigenvalue analysis
- Probabilistic sensitivity analysis
Example
A chronic disease model contains three health states: Stable, Progressive Disease and Death.
The annual transition matrix is:
?? =
?0.85 0.10 0.05?
?0.00 0.80 0.20?
?0.00 0.00 1.00?
If the initial cohort is entirely in the Stable state,
??? = [1, 0, 0]
then after one cycle:
??? = ????? = [0.85, 0.10, 0.05]
These state occupancies are subsequently used to calculate expected costs and quality-adjusted life years for the cohort.
Excel Implementation
| Function | Example Formula | Health Economics Application |
|---|---|---|
| MMULT | =MMULT(StateVector,TransitionMatrix) | Calculate state distribution after each cycle |
| SUMPRODUCT | =SUMPRODUCT(StateVector,CostVector) | Calculate expected costs |
| SUMPRODUCT | =SUMPRODUCT(StateVector,UtilityVector) | Calculate expected QALYs |
| MUNIT | =MUNIT(NumberStates) | Create identity matrices during model development |
| POWER | =POWER(Matrix,n)* | Conceptually represent repeated state transitions (*implemented through repeated matrix multiplication rather than directly in Excel) |
VBA (Optional)
Automate repeated matrix multiplication over successive cycles to generate cohort traces and cumulative health economic outcomes.
Sources
- Norris JR. Markov Chains. Cambridge University Press.
- Kemeny JG, Snell JL. Finite Markov Chains. Springer.
- Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322?338.
- Siebert U, Alagoz O, Bayoumi AM, et al. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Medical Decision Making. 2012;32(5):690?700.
- Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford University Press; 2006.
- Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed. Oxford University Press.
Related Concepts (5)
Library
Publications
5
State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3 — Siebert, Alagoz, Bayoumi, Jahn, Owens, Cohen & Kuntz, Task Force Report 3 ed., 2012 (Value in Health / Medical Decision Making)
Best-practice guidance for cohort and individual-based state-transition (Markov) models, covering development, analysis, validation and reporting.
Journal ArticleView source →Markov Chain Monte Carlo in Practice — W. R. Gilks, S. Richardson & D. J. Spiegelhalter (editors), 1st Edition ed., 1996 (Chapman & Hall)
A foundational collection on Markov chain Monte Carlo methodology and practice, linking stochastic simulation with Bayesian inference and Gibbs sampling.
BookView source →Markov Chains — J. R. Norris, 1st Edition ed., 1997 (Cambridge University Press)
A rigorous introduction to discrete- and continuous-time Markov chains, transition structures, recurrence and long-run behaviour.
BookView source →Markov Processes for Stochastic Modeling — Masaaki Kijima, 1st Edition ed., 1997 (Chapman & Hall)
An applied mathematical treatment of discrete- and continuous-time Markov processes for modeling the transient behaviour of stochastic systems.
BookView source →The BUGS Book: A Practical Introduction to Bayesian Analysis — David Lunn, Chris Jackson, Nicky Best, Andrew Thomas & David Spiegelhalter, 1st Edition ed., 2012 (Chapman & Hall / CRC Press)
A practical guide to Bayesian modeling with BUGS, covering prior specification, posterior simulation, model checking and Markov chain Monte Carlo computation.
BookView source →
Frequently Asked Questions (7)
What is a markov chain?
A mathematical sequence of states in which the probability of moving to any next state depends only on the current state.
Source: Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338. doi:10.1177/0272989X9301300409.
What is a Markov chain?
A Markov chain is a mathematical sequence of states in which the probability of moving to any next state depends only on the current state, not on the sequence of states that preceded it. It describes a process that moves between states over time according to fixed transition probabilities, with the memoryless property that the present state alone determines the probabilities of the next. Markov chains are the mathematical basis of Markov models used in health economic evaluation to represent disease progression.
Source: Sonnenberg & Beck 1993
Who developed the theory of Markov chains?
Markov chains are named after the Russian mathematician Andrey Markov, who studied sequences of linked random events in the early twentieth century. He examined processes in which each outcome depends on the one immediately before it but not on the whole earlier sequence, and worked out their long-run behaviour. This idea of dependence limited to the present state is the property that defines a Markov chain and gives it its name. The theory now underlies models across many fields. Markov's work established it in the early 1900s.
Source: Markov 1906
What are the elements of a Markov chain?
A Markov chain consists of a set of possible states, transition probabilities specifying the chance of moving from each state to each other state in one step, and an initial distribution across the states. The transition probabilities from any state must sum to one, since the process must move to some state. Given these elements, the chain's evolution over time is determined probabilistically, with the distribution across states updated each step according to the transition probabilities.
Source: Sonnenberg & Beck 1993
What is the defining property of a Markov chain?
The defining property of a Markov chain is the Markov, or memoryless, property: the probability of moving to the next state depends only on the current state, not on the history of how it was reached. This means the future is conditionally independent of the past given the present state. Because of this property, the chain can be described entirely by its states and transition probabilities, without reference to history, which is what makes Markov chains tractable and underlies their use in modelling.
Source: Sonnenberg & Beck 1993
How does a Markov chain evolve over time?
A Markov chain evolves by moving between states step by step according to the transition probabilities, with the distribution across states updated at each step. Starting from an initial distribution, the probability of being in each state after one step is obtained by applying the transition probabilities, and repeating this traces the distribution over time. Over many steps, the distribution may settle toward a steady state or accumulate in absorbing states, depending on the chain's structure, describing the long-run behaviour of the process.
Source: Sonnenberg & Beck 1993
How are Markov chains used in health economics?
In health economics, Markov chains underlie Markov models, which represent disease progression as movement between health states over cycles according to transition probabilities. A cohort or individual is followed through the chain's states, accruing costs and health effects, to estimate expected outcomes over time. The memoryless property makes these models tractable for chronic and recurring conditions. Markov chains thus provide the mathematical framework for a widely used class of decision-analytic models in economic evaluation.
Source: Sonnenberg & Beck 1993
Trust Record
Verified by Dr Darrin Baines
British health economist
Professional identity: darrinbaines.org
Verification date: 7 Oct 2025
Content version: 1.0.0
Canonical Identity
- Persistent URI
- https://healtheconomics.wiki/concept/markov-chain
- Term code
- HE-EM-MM-011
Stable URI · Machine-readable · Resolvable · CC BY 4.0