Transient State

Transient State in Markov Models of Disease

In a Markov model of disease, a state is transient when everyone in it will sooner or later move out and stay out, even if they pass through it several times on the way. Formally, the probability of ever being back in the state at a later cycle is less than one, and in a lifetime model with death as the absorbing state every living state meets that test. Transient states are where a cohort model does its accounting, because every cost and every QALY in the model is earned during time spent in them. The length of each stay and the number of returns can both be read from the transition probabilities, and both drive the results.

The return test and why every living state passes it

Durrett, in Probability Theory and Examples, classifies each state of a Markov chain by the probability that a chain starting there is in that state again at some later step. A state is recurrent when that probability equals one and transient when it is less than one.

$$\rho_{ii} = \Pr(T_i < \infty \mid X_0 = i)$$

where $X_n$ is the state occupied at cycle $n$, $T_i$ is the first cycle $n \ge 1$ at which $X_n = i$, and $\rho_{ii}$ is the probability that such a cycle exists. State $i$ is transient when $\rho_{ii} < 1$. Remaining in the state for one more cycle counts as being in it again, so a state with a high probability of remaining can still be transient.

For a chain with a finite number of states, Durrett gives a test that needs no calculation. A state is transient if it can reach some state that has no route back to it. Any other state lies in a closed set of states that all lead to one another, and the members of that set are recurrent. Grinstead and Snell, in Introduction to Probability, study absorbing chains, in which at least one state cannot be left and every state can reach such a state, and they call every state of an absorbing chain that is not absorbing transient. In a lifetime health economic model, death can be reached from every living state and nobody comes back from it, so every living state is transient.

That includes states that modellers call permanent. Sonnenberg and Beck, in their practical guide to Markov models in medical decision making, build an anticoagulation example with WELL, DISABLED and DEAD states in which the disability is permanent and a move back to WELL is not allowed, while people in DISABLED can still die. DISABLED is transient for the same reason as a state that people leave every few cycles. The general structure of such models is set out on the Markov model page, and the state that closes them on the absorbing state page.

How long a stay lasts and how many stays there are

Each stay in a transient state lasts a whole number of cycles. When the probability of remaining for one more cycle is the same in every cycle, the length of a stay follows a geometric distribution with a simple mean.

$$E[S_i] = \frac{1}{1 - p_{ii}}$$

where $S_i$ is the number of consecutive cycles spent in state $i$ during one stay, counted at the start of each cycle, and $p_{ii}$ is the one-cycle probability of remaining in state $i$. A state with $p_{ii}$ of 0.95 has an average stay of 20 cycles, yet it is still transient if its occupants can reach death.

The total time in a transient state also depends on how many separate stays there are. Durrett shows that a chain starting in a transient state spends only a finite expected number of later cycles there.

$$E_i[N(i)] = \frac{\rho_{ii}}{1 - \rho_{ii}}$$

where $N(i)$ is the number of cycles after the start at which the chain is in state $i$, and $E_i$ is the expectation for a chain that starts in state $i$. Durrett also proves that this expectation is infinite for a recurrent state, so a lifetime model has a finite life expectancy only because every living state in it is transient.

With constant transition probabilities, Grinstead and Snell collect these expectations for every pair of transient states in one matrix, the fundamental matrix, which the Markov chain and state occupancy pages work through in detail.

$$N = (I - Q)^{-1}$$

where $Q$ is the matrix of one-cycle transition probabilities among the transient states, $I$ is the identity matrix of the same size, and entry $n_{ij}$ of $N$ is the expected number of cycles spent in transient state $j$ by a chain that starts in transient state $i$, with the starting cycle counted when $i$ equals $j$. Grinstead and Snell also prove that $Q^n$ tends to zero as $n$ grows, which is the cohort draining out of the transient states.

Why costs and QALYs sit in transient states

Sonnenberg and Beck explain that evaluating a Markov process gives the average number of cycles spent in each state, and that the patient is given credit for the time spent in each state. Expected utility is the time in each state multiplied by that state's incremental utility and summed, and a separate value per cycle gives expected cost in the same way. Since DEAD must carry an incremental utility of zero, everything a lifetime cohort model reports is a weighted count of cycles in transient states, and life expectancy is the expected time in them before absorption.

Sonnenberg and Beck note that draining is never complete, so a cohort simulation is stopped once the cycle sum falls below a small threshold. They add that the fundamental matrix is equivalent to counting state membership at the start of each cycle, so its result is corrected by subtracting half a cycle from the membership of each starting state, the matrix form of the half-cycle correction.

Transient, temporary and tunnel states

Medical decision modelling also has states that can hold a person for at most one cycle. Sonnenberg and Beck call them temporary states, defined by transitions only to other states and not to themselves, and use them for events with short-term effects such as a stroke. An array of temporary states, each leading only to the next, makes up tunnel states, which apply a temporary adjustment to utilities or transition probabilities that lasts more than one cycle.

The word transient is sometimes used for exactly these states. In their tutorial on time-dependent cohort state-transition models, Alarid-Escudero and colleagues expand a state into as many transient states as the cycles needed to track residence time, and say that these are often called tunnel states, in which the cohort stays for exactly one cycle. When death can be reached from every living state, every temporary or tunnel state is transient in the return-probability sense, but most transient states are neither, since people can stay in them for many cycles. A model report that uses the narrow sense needs to say so.

Worked example with remission and relapse

All numbers in this example are illustrative. A cohort with a relapsing disease starts in Remission (R). Each year a person in Remission stays with probability 0.80, relapses (L) with probability 0.15 or dies with probability 0.05. A person in Relapse returns to Remission with probability 0.50, stays with probability 0.30, moves to Progressed (P) with probability 0.10 or dies with probability 0.10. A person in Progressed stays with probability 0.75 or dies with probability 0.25, and Dead is absorbing. Every row sums to one. Utilities are 0.80 in Remission, 0.50 in Relapse and 0.40 in Progressed.

1. Classify the states. Remission and Relapse lead to each other, but both lead to Dead, which has no route back, so both are transient. Progressed has no route back to either and also leads to Dead, so it is transient as well. Remission and Relapse can be entered more than once, while Progressed can be entered at most once.

2. Find the return probability for Remission. Each year in Relapse ends with a return to Remission with probability 0.50, a further year in Relapse with probability 0.30, or an exit from the loop with probability 0.20, so a person in Relapse gets back to Remission with probability 0.50 / 0.70 = 0.7143. Starting in Remission, the probability of being in Remission at some later cycle is 0.80 + 0.15 × 0.7143 = 0.9071, below one, so Remission is transient.

3. Count the years in Remission. The expected number of later years in Remission is 0.9071 / (1 − 0.9071) = 9.769. Adding the starting year gives 10.769 years.

4. Rebuild the same figure from stays. One stay in Remission lasts 1 / (1 − 0.80) = 5 years on average. A stay ends in relapse with probability 0.15 / 0.20 = 0.75, and a relapse leads back with probability 0.7143, so a new stay follows with probability 0.75 × 0.7143 = 0.5357. The expected number of stays is 1 / (1 − 0.5357) = 2.154, and 2.154 × 5 = 10.77 years, which matches step 3. The 2.154 stays produce 2.154 × 0.75 = 1.615 relapses, each lasting 1 / (1 − 0.30) = 1.429 years, or 2.308 years in Relapse. Each relapse ends in progression with probability 0.10 / 0.70 = 0.1429, so the probability of ever progressing is 1.615 × 0.1429 = 0.2308, and with an average stay of 1 / (1 − 0.75) = 4 years in Progressed the expected time there is 0.2308 × 4 = 0.923 years.

5. Add up the outcomes. Counted at the start of each cycle, life expectancy is 10.769 + 2.308 + 0.923 = 14.000 years, or 13.5 years after subtracting half a cycle from the starting state. Undiscounted QALYs are 10.769 × 0.80 + 2.308 × 0.50 + 0.923 × 0.40 = 10.138, made up of 8.615, 1.154 and 0.369, or 10.138 − 0.5 × 0.80 = 9.738 after the correction. The first row of the fundamental matrix for these three transient states is (10.769, 2.308, 0.923), and a cohort run of 5,000 annual cycles gives the same figures.

6. Change the return probability. Suppose a treatment raises the probability of a return to Remission out of Relapse to 0.60 and lowers the probability of staying in Relapse to 0.20, with progression and death unchanged. A person in Relapse now returns with probability 0.60 / 0.80 = 0.75, and the return probability for Remission becomes 0.80 + 0.15 × 0.75 = 0.9125. Years in Remission rise to 1 + 0.9125 / 0.0875 = 11.429, while Relapse falls to 2.143 years and Progressed to 0.857 years. Life expectancy becomes 14.429 years and undiscounted QALYs 11.429 × 0.80 + 2.143 × 0.50 + 0.857 × 0.40 = 10.557, a gain of 0.429 life years and 0.419 QALYs.

7. Interpretation. The treatment leaves the Remission row untouched, and the average stay in Remission is still 5 years. The whole gain works through the Relapse row, raising the return probability for Remission from 0.9071 to 0.9125, which lifts the expected number of remission stays from 2.154 to 2.286. Because the expected number of visits depends on one over one minus the return probability, small changes in a return probability close to one move the results a good deal, so the evidence behind re-entry transitions deserves as much scrutiny as the evidence behind exits.

Closed loops, constant probabilities and other modelling consequences

The classification depends on the structure of the matrix, so a structural change can turn transient states into recurrent ones. If the model above left out death from Remission and Relapse and had no progression, the two states would form a closed set. Durrett shows that a finite closed set always contains a recurrent state, and that every state in it is recurrent when its states all lead to one another. The cohort would then never drain, the matrix $I - Q$ would have no inverse, and expected time in Remission would grow without limit. Sonnenberg and Beck make the same point, that a Markov process needs at least one state that cannot be left in order to terminate, so such a model works only with a fixed time horizon.

The closed forms above need constant transition probabilities. Sonnenberg and Beck call a Markov process with constant probabilities a Markov chain, the case now described as a time-homogeneous model, and note that an absorbing chain of that kind can be solved exactly by matrix algebra. When mortality rises with age, a state stays transient while death remains reachable, but the closed forms no longer give exact answers, and time in each state comes from the cycle-by-cycle cohort trace of a time-inhomogeneous model. When the risk of leaving a state depends on how long a person has been in it, the constant-probability stay is the wrong model, and the state is split into tunnel states, as the Markov assumption page explains.

Misreadings of the term

Most errors come from reading the word in its everyday sense. Four recur often enough to check for in any model report.

1. Transient taken to mean short-lived. A state with a 95% chance of remaining each cycle has an average stay of 20 cycles and is still transient.
2. Transient taken to mean re-enterable. Progressed in the example can be entered at most once and is transient, so the definition turns on eventual departure, not on return.
3. Transient confused with temporary or tunnel states. Those states hold a person for one cycle at a time and are a small subset of transient states, although some tutorials use the word for them alone.
4. A closed loop of living states expected to empty. Without a route to death or another absorbing state the loop is recurrent, and lifetime totals do not converge.

Sources

• Alarid-Escudero F, Krijkamp E, Enns EA, Yang A, Hunink MGM, Pechlivanoglou P, Jalal H. A tutorial on time-dependent cohort state-transition models in R using a cost-effectiveness analysis example. Medical Decision Making. 2023;43(1):21-41. https://doi.org/10.1177/0272989X221121747
• Durrett R. Probability: Theory and Examples. 5th ed. Cambridge University Press; 2019. Section 5.3, Recurrence and transience. https://services.math.duke.edu/~rtd/PTE/PTE5_011119.pdf
• Grinstead CM, Snell JL. Introduction to Probability. 2nd ed. American Mathematical Society; 2003. Free version by PG Doyle, 2006. Section 11.2, Absorbing Markov chains. https://math.dartmouth.edu/~prob/prob/prob.pdf
• Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338. https://doi.org/10.1177/0272989X9301300409