Memorylessness

Memorylessness in Markov Models and Survival Distributions

A waiting time is memoryless when the chance of the event in the next interval is the same for someone who has already waited ten years as for someone who has only just started. In health economic modelling the idea turns up twice, as the waiting times of the exponential distribution and its discrete partner the geometric distribution, and as the Markov property of state-transition models. The two are closely tied, because a model state with a constant probability of leaving gives everyone in it a geometric distribution of time spent there, whatever the evidence says. A model can match the mean duration exactly and still misstate the tail, discounted costs and the expected future of people who have already spent years in a state. When a whole model can be treated as Markov is the subject of the Markov assumption page, and the arithmetic of a flat event rate is on the constant hazard page.

The memoryless property of a waiting time

The property is a statement about conditional probability. Given that the event has not happened by some time, the further wait has the same distribution as the whole wait seen from the start. Grinstead and Snell, in their Introduction to Probability, write the condition as follows.

$$\Pr(T > r + s \mid T > r) = \Pr(T > s)$$

where $T$ is the waiting time, such as time to event or time spent in a health state, $r$ is the time already waited and $s$ is any further length of time. They describe the probability of waiting $s$ more units as independent of the time $r$ already waited, state that the only continuous density with this property is the exponential, and show that the geometric distribution, used for waits counted in whole cycles, has it too.

The two survival curves are $S(t) = e^{-\lambda t}$ in continuous time and $S(d) = q^{d}$ in whole cycles, where $S$ is the survival function, $\lambda$ is a constant rate, $q$ is a constant probability of the wait continuing through one more cycle and $d$ counts cycles. In both cases the ratio $S(r + s)/S(r)$ equals $S(s)$, which is the property. A flat hazard rate is the same statement made instant by instant, since the hazard is the event rate among people still waiting.

The consequence that matters most in economic models is that the expected time still to wait does not shrink as time passes.

$$E[T - r \mid T > r] = E[T] = \frac{1}{\lambda}$$

where $E[T - r \mid T > r]$ is the mean remaining wait for someone who has already waited $r$, and $E[T]$ is the mean wait from the start. Someone who has been in a state for five years is expected to stay as long again as someone who arrived yesterday.

How the Markov property makes time in a state memoryless

Sonnenberg and Beck, in their practical guide to Markov models in medical decision making, describe the Markov property as the requirement that the behaviour of the process after any cycle depends only on its description in that cycle, so that the process has no memory for earlier cycles. The ISPOR-SMDM report on state-transition modelling describes transition probabilities that depend neither on past states nor on the time spent in the current state, and its comparison table labels cohort models "Markov assumption, memoryless".

In a time-homogeneous model the probability of staying in a state is the same in every cycle, so the number of cycles spent there on one visit follows a geometric distribution.

$$\Pr(D > d) = p_{ii}^{\,d}, \qquad E[D] = \frac{1}{1 - p_{ii}}, \qquad \operatorname{Var}(D) = \frac{p_{ii}}{(1 - p_{ii})^{2}}$$

where $D$ is the number of cycles spent in state $i$ on one visit, counting the entry cycle, $d = 0, 1, 2, \dots$ is a number of cycles, $p_{ii}$ is the transition probability of staying in state $i$ for one more cycle, $E[D]$ is the mean stay and $\operatorname{Var}(D)$ its variance. The chance of leaving, one minus $p_{ii}$, is the same in every cycle of the stay, and the most likely stay is a single cycle.

Continuous-time models behave the same way. Jackson's manual for the msm package, which fits multi-state models to panel data, states that in a time-homogeneous continuous-time Markov model a single sojourn in state $r$ is exponential with mean $-1/q_{rr}$, where $q_{rr}$ is the diagonal entry of the transition intensity matrix.

A time-inhomogeneous model, with probabilities that change with model time such as age, is still Markov.

Durations in economic models that carry memory

The ISPOR-SMDM report calls the assumption the primary disadvantage of cohort models and says it can be very limiting in clinical applications, where past states and time in the current state tend to be strong determinants of what happens next. Time on an expensive treatment and time before progression are often of that kind.

NICE Decision Support Unit Technical Support Document 14 (TSD 14) states that exponential models are only suitable if the observed hazard is approximately constant and non-zero. A single model state with a constant exit probability makes that exponential, or geometric, assumption about the duration it holds.

Checking a duration for memory

Memory in a duration shows up as an exit probability that changes with time already spent. With patient-level data, a simple check is conditional survival, the share of people still in a state at time $r$ who remain there a year later, which stays constant across $r$ only when the duration is memoryless. The log-cumulative hazard plot used to test a flat hazard is covered on the constant hazard page.

Panel data, in which people are seen only at clinic visits, need a model-based check. Jackson's manual notes that changes in deviance with the length of the interval between observations may indicate failure of the Markov assumption, and that a semi-Markov model, in which intensities depend on time spent in the current state, may then fit better.

Putting memory back into a model

Every remedy records time already spent somewhere the model can see. Sonnenberg and Beck describe tunnel states, temporary states visited in a fixed sequence, to apply an adjustment to utility or transition probabilities that lasts more than one cycle. Jackson's manual notes that semi-Markov models may in principle be fitted to panel data using phase-type distributions, which chain memoryless phases together.

Each sub-state stays memoryless and the expanded model stays Markov, while the time spent in the combined clinical state does not. Individual-level models such as Markov microsimulation avoid the property by keeping each history in tracker variables.

Worked example of time on treatment with and without memory

All numbers in this example are illustrative. A model holds patients in an On treatment state with annual cycles, and the evidence gives a mean time on treatment of 4 years. Treatment costs GBP 10,000 for each year begun on treatment, counted at the start of the year, and costs are discounted at an illustrative 3.5% a year. Two structures match the mean.

1. Set up the two structures. Model A uses a single state with a stay probability of 0.75, so the mean stay is 1/(1 − 0.75) = 4 years. Model B splits the state into two consecutive sub-states, an early phase and a late phase, each with a stay probability of 0.5 and so a mean of 2 years, giving the same total mean of 4 years. The time in Model B's combined state follows the distribution below.

$$\Pr(D = d) = (d - 1)\,0.5^{d}, \qquad \Pr(D > k) = \frac{k + 1}{2^{k}}$$

where $D$ is the number of years on treatment, $d = 2, 3, \dots$ is a possible duration, since each phase lasts at least one cycle, and $k = 0, 1, 2, \dots$ is a number of completed years.

2. Compare the spread. In Model A the variance is 0.75/0.25² = 12, a standard deviation of 3.46 years. In Model B each phase has variance 0.5/0.5² = 2, so the total variance is 4 and the standard deviation 2 years.

3. Compare the probability of stopping in each year. In Model A the probability of stopping in any year among people still on treatment is 0.25. In Model B it is $(d - 1)/(2d)$ in year $d$, rising towards 0.5 as more patients reach the late phase.

4. Compare the tail. The share still on treatment after 8 years is 0.75⁸ = 0.100 in Model A and 9/256 = 0.035 in Model B, about a third of Model A's share.

5. Compare the remaining time after 3 years. In Model A a patient still on treatment after 3 years expects 4 more years, the same as at the start. In Model B half the cohort is still on treatment at that point, and the share still in the early phase is 0.5³ = 0.125, a quarter of those remaining. Those in the early phase expect 2 + 2 = 4 more years and those in the late phase 2 more, so the expected remaining time is 0.25 × 4 + 0.75 × 2 = 2.5 years. The expected undiscounted future treatment cost for a patient still on treatment after 3 years is GBP 40,000 in Model A and GBP 25,000 in Model B.

6. Compare discounted cost. Undiscounted cost per patient is 4 × GBP 10,000 = GBP 40,000 in both models. Discounted, the totals differ.

$$C_A = 10{,}000 \sum_{k=0}^{\infty} \left( \frac{0.75}{1.035} \right)^{k} = 10{,}000 \times \frac{1.035}{0.285} = 36{,}316, \qquad C_B = 10{,}000 \sum_{k=0}^{\infty} (k + 1) \left( \frac{0.5}{1.035} \right)^{k} = 10{,}000 \times \left( \frac{1.035}{0.535} \right)^{2} = 37{,}426$$

where $C_A$ and $C_B$ are expected discounted treatment costs per patient in pounds, $k$ is the number of completed years, and each term is the probability of starting year $k + 1$ on treatment times the discount factor $(1.035)^{-k}$. The memoryless state puts more treatment time far in the future, where discounting shrinks it, so its discounted cost is GBP 1,110 lower, about 3.0%.

7. Interpretation. Both structures fit the same mean and give the same undiscounted cost, so a check against mean time on treatment cannot tell them apart. The GBP 15,000 gap in expected future cost after 3 years matters for stopping rules and continuation decisions.

What memorylessness means for decisions

The harm done by a memoryless state depends on what the decision uses. An expected undiscounted total with a constant reward per cycle depends only on the mean duration, so a single state calibrated to the right mean gets that state's undiscounted life-years and costs right. Discounting, rewards that change with time since entry, stopping rules and waning treatment effects all depend on the whole distribution of the duration. The choice between a single state and a split one is therefore part of the model's structural uncertainty, and running the alternative structure with the same mean as a scenario analysis shows how far the shape of the duration moves the incremental cost-effectiveness ratio.

Boundaries and common misreadings

Most confusion comes from treating the property of a process and the property of a single duration as the same thing.

1. A Markov model does not imply exponential overall survival. Each state can be memoryless while time to death across several states is not, as Model B shows with two memoryless phases producing a rising exit probability. Fitting an exponential to overall survival is a separate assumption from a Markov structure.
2. Pooled data can show memory that no individual has. The ISPOR-SMDM report warns that characteristics affecting transitions but unknown at decision time can create heterogeneity bias. A mixture of people with different constant exit rates shows a falling exit rate overall, because those with high rates leave first, so a declining hazard may call for subgroups and not tunnel states.
3. Memorylessness concerns transitions, not costs or utilities. A state can be memoryless in its transitions while its cost or utility should still change with time since entry, which needs the same splitting of states.

Sources

• Grinstead CM, Snell JL. Introduction to Probability. 2nd rev. ed. Providence, RI: American Mathematical Society; 1997. Sections 2.2, 4.2, 5.1 and 5.2 (online version dated 4 July 2006). https://math.dartmouth.edu/~prob/prob/prob.pdf
• Jackson C. Multi-state modelling with R: the msm package. Version 1.8.2. Cambridge: MRC Biostatistics Unit; 7 November 2024. https://cran.r-project.org/web/packages/msm/vignettes/msm-manual.pdf
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; June 2011, last updated March 2013. Section 3.2. https://doi.org/10.15131/shef.data.33745843
• Siebert U, Alagoz O, Bayoumi AM, Jahn B, Owens DK, Cohen DJ, Kuntz KM. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Value in Health. 2012;15(6):812-820. https://doi.org/10.1016/j.jval.2012.06.014
• Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338. https://doi.org/10.1177/0272989X9301300409