State Occupancy

State Occupancy in Markov and Partitioned Survival Models

State occupancy is the amount of model time a cohort accumulates in each health state, built up cycle by cycle from the share of the cohort found in that state. In a cost-effectiveness model it is the quantity that turns transition probabilities or survival curves into life years, QALYs and costs, because each state's utility and cost are earned per unit of time spent there. Two models with identical transitions can still report different occupancy if they count the cohort at different points in the cycle, so the counting rule has to be reported with the result.

Occupancy at a moment and occupancy over time

Two related quantities share the name, and many errors come from mixing them. The first is the proportion of the cohort in a state at a given time. NICE Decision Support Unit Technical Support Document 19 (TSD 19) calls this state membership, and the multi-state survival literature, including Williams and colleagues, speaks of state occupancy probabilities. The second is the accumulated time in the state over the time horizon, which is what a model finally values.

The accumulated quantity is the sum, or in continuous time the integral, of the proportions over the horizon.

$$L_j = \Delta \sum_{t=0}^{T-1} m_{t,j} \qquad \text{or} \qquad L_j(\tau) = \int_{0}^{\tau} p_j(u)\, du$$

where $L_j$ is the expected occupancy of state $j$, $\Delta$ is the cycle length in years, $T$ is the number of cycles, $m_{t,j}$ is the proportion of the cohort in state $j$ at the start of cycle $t$, $p_j(u)$ is the probability of being in state $j$ at time $u$, and $\tau$ is the horizon. This discrete form counts each cycle by its starting proportion, one of several conventions. When $j$ is taken as the set of all alive states, $L_j(\tau)$ is the restricted mean survival time up to $\tau$.

Occupancy is an expectation over the cohort, so 2.5 years in a progressed state is an average over patients who never progress and patients who progress early and live long.

How a cohort model produces occupancy

A Markov cohort model moves the cohort's distribution forward one cycle at a time. The proportions in each state at the next cycle are the current proportions multiplied by the transition matrix, and the cycle-by-cycle record of those proportions is the cohort trace.

$$\mathbf{m}_{t+1} = \mathbf{m}_t P_t$$

where $\mathbf{m}_t$ is the row vector of state proportions at the start of cycle $t$ and $P_t$ is the transition matrix for that cycle, with rows for the state of origin and columns for the destination, each row summing to one.

Sonnenberg and Beck describe the bookkeeping. Each cycle, the fraction of the cohort in each state is multiplied by that state's incremental utility and summed to give the cycle sum, which is added to a running cumulative utility. Unadjusted life expectancy is the sum of the alive-state columns, counted at the end of each cycle, divided by the cohort size. In their prosthetic heart valve example, cohort members spend on average 1.5 cycles in the WELL state and 1.25 cycles in the DISABLED state, for an unadjusted life expectancy of 2.75 cycles.

Once occupancy is known for every state, totals follow by weighting it with each state reward.

$$E = \sum_{j} u_j L_j \qquad C = \sum_{j} c_j L_j$$

where $E$ is expected QALYs per person, $u_j$ is the utility of state $j$ for a full year, $C$ is expected cost per person, $c_j$ is the annual cost of state $j$, and $L_j$ is occupancy in years. This form holds when utilities and costs are constant within each state. When rewards change with age or time since entry, or when outcomes are discounted, each cycle's proportion is weighted by that cycle's reward and discount factor before summing, and the result is no longer a plain time in state.

Exact occupancy from the fundamental matrix

When transition probabilities are constant, occupancy over an unlimited horizon has a closed form. For absorbing Markov chains, Grinstead and Snell show that inverting the identity minus the submatrix of transitions among transient (alive) states gives the expected number of times the chain is in each transient state.

$$N = (I-Q)^{-1} = I + Q + Q^2 + \dots$$

where $Q$ is the matrix of one-cycle transition probabilities among the transient states, $I$ is the identity matrix of the same size, and entry $n_{ij}$ of $N$ is the expected number of cycles spent in transient state $j$ by a person starting in transient state $i$. The initial state is counted when $i$ equals $j$. Summing a row of $N$ gives the expected number of cycles before absorption, which in a model whose only absorbing state is death is life expectancy in cycles.

Sonnenberg and Beck note that this fundamental matrix shows, for each starting state, the expected length of time spent in each state, and that it is equivalent to counting state membership at the beginning of each cycle. The correction they give for a matrix result is to subtract half a cycle from the membership of each starting state. Because the method is restricted to constant transition probabilities, which rules out age-specific mortality, and has other practical drawbacks, Sonnenberg and Beck judged it largely superseded for medical applications. A modeller can still use it to check a simulated trace with the probabilities held constant.

Counting rules and the half-cycle correction

A cohort trace records proportions at cycle boundaries, while people in reality leave states continuously. Sonnenberg and Beck compare a Markov simulation with estimating the area under a survival curve by rectangles. Counting membership at the end of each cycle consistently underestimates that area and counting at the start consistently overestimates it.

The half-cycle correction assumes transitions occur on average halfway through each cycle. In the cohort trace this amounts to giving the first and last recorded rows half weight, which is the trapezoidal rule and equals the average of the start-of-cycle and end-of-cycle totals. The ISPOR-SMDM task force report on state-transition models by Siebert and colleagues recommends a half-cycle correction for costs and effectiveness in the first cycle, and in the final cycle when the horizon is not lifetime. Sonnenberg and Beck add that the size of the error depends on cycle length relative to average survival, so short cycles make the counting rule matter less.

Worked example with a three-state progression model

All numbers in this example are illustrative. A cohort starts progression free (PF) in a model with a progressed state (PD) and death, annual cycles and a five-year horizon. Each year, 70% of PF patients stay PF, 20% progress and 10% die, while 75% of PD patients stay PD and 25% die. Patients in PD cannot return to PF.

1. Build the trace. Starting from $\mathbf{m}_0 = (1, 0, 0)$, each row is the previous row multiplied by the transition matrix. For cycle 2, PF is 0.70 × 0.70 = 0.49 and PD is 0.70 × 0.20 + 0.20 × 0.75 = 0.29, leaving 0.22 dead.

The unrounded PD values are 0.3155 at cycle 3, 0.305225 at cycle 4 and 0.27693875 at cycle 5, and every unrounded row sums to one.

2. Count at the start of each cycle. Summing rows 0 to 4 gives PF occupancy of 1 + 0.7 + 0.49 + 0.343 + 0.2401 = 2.7731 years and PD occupancy of 0.2 + 0.29 + 0.3155 + 0.305225 = 1.110725 years.

3. Count at the end of each cycle. Summing rows 1 to 5 gives PF occupancy of 0.7 + 0.49 + 0.343 + 0.2401 + 0.16807 = 1.94117 years and PD occupancy of 1.110725 + 0.27693875 = 1.38766375 years.

4. Apply the half-cycle correction. Averaging the two counts gives PF occupancy of (2.7731 + 1.94117) / 2 = 2.357135 years and PD occupancy of (1.110725 + 1.38766375) / 2 = 1.249194 years. Total time alive within five years is 3.606 years. The two uncorrected PF counts sit 0.416 years either side of this value.

5. Value the occupancy. With illustrative utilities of 0.80 for PF and 0.50 for PD, PF contributes 0.80 × 2.357135 = 1.885708 QALYs and PD contributes 0.50 × 1.249194 = 0.624597 QALYs, so undiscounted QALYs total 1.885708 + 0.624597 = 2.510305. With illustrative annual costs of GBP 12,000 in PF and GBP 5,000 in PD, PF accounts for 12,000 × 2.357135 = GBP 28,286 and PD for 5,000 × 1.249194 = GBP 6,246, so undiscounted cost is 28,286 + 6,246 = GBP 34,532 per person.

6. Check the lifetime value with the fundamental matrix. Here $Q$ has PF to PF of 0.70, PF to PD of 0.20 and PD to PD of 0.75. Its inverse gives, for a person starting in PF, 1 / 0.30 = 3.333 cycles in PF and 0.20 / (0.30 × 0.25) = 2.667 cycles in PD, a life expectancy of 6.0 cycles on the start-of-cycle count. Subtracting half a cycle from the starting state gives 2.833 years in PF and 2.667 years in PD, or 5.5 years alive.

7. Interpretation. The five-year horizon captures 3.606 of the 5.5 expected life years, so a lifetime horizon would add about 1.9 years of occupancy, most of it in PD. Comparing state-by-state occupancy between arms would show whether a treatment's QALY gain comes from longer progression-free time or longer survival after progression.

Occupancy in partitioned survival and multi-state models

A partitioned survival model needs no transition probabilities. TSD 19 explains that the PFS curve gives the proportion progression free over time, the dead state is one minus the OS curve, and the progressed state is the difference between the OS and PFS curves at each time point. The area under the extrapolated OS curve estimates mean life expectancy, so occupancy of the progressed state is the area between the two curves.

TSD 19 notes that each endpoint is modelled independently, so the PFS and OS curves chosen for each arm alone determine progressed-state occupancy, which matters when most of a treatment's gain sits after progression.

Williams and colleagues compared the two approaches with a multi-state survival model in a cost-effectiveness setting. In their partitioned survival analysis, mean time in progression came from the difference in the areas under the PFS and OS curves, calculated by the trapezoidal rule in one-month steps. They note that state occupancy probabilities in a multi-state model combine the hazards for each transition into the state, so comparing the fit of models for single transitions does not assess the occupancy probabilities that are ultimately of interest.

What occupancy can and cannot show in an appraisal

Time spent in each state is a useful intermediate result. The ISPOR-SMDM task force recommends presenting intermediate outcomes that make results understandable, citing the average time spent without a stroke in a model of stroke prevention in atrial fibrillation. Such summaries also support validation against empirical survival curves, and inspection of the trace can expose programming errors.

Occupancy totals from a Markov model evaluated as a cohort are means. The same task force notes that a decision maker may need the distribution of an outcome, for example whether a one-year gain in life expectancy is shared evenly or concentrated in some patients. Distributions come most easily from individual-level simulation such as Markov microsimulation, where occupancy is the average of each simulated person's time in the state.

Occupancy does not count events. A state can hold a large share of the cohort for many cycles while receiving few new entrants, so incidence, event counts and one-off transition costs have to be tracked separately from time in state. Occupancy of a tunnel state is bounded by design, since Sonnenberg and Beck define temporary states as ones a patient can occupy for at most one cycle per visit.

Common errors with state occupancy

Many occupancy errors pass unnoticed because the trace still sums to one in every row. Most of them concern what is summed and how the total is labelled.

1. Reporting a proportion as time. A state membership of 0.30 at year five is not 0.30 years of occupancy, which needs the sum or integral over the horizon.
2. Summing every row of the trace. A five-cycle trace has six rows, so adding rows 0 to 5 in full counts six cycles of membership for a five-year horizon.
3. Using the fundamental matrix when probabilities vary. The matrix result assumes constant transitions and a start-of-cycle count, and needs the half-cycle adjustment to the starting state.
4. Calling discounted occupancy life years. Once a discount factor is applied, the total is a valuation input and no longer measures elapsed time.
5. Treating occupancy differences as patient-level effects. A difference between arms in mean PD time says nothing about how that time is spread across patients.

Sources

• Grinstead CM, Snell JL. Introduction to Probability. 2nd ed. American Mathematical Society; 2003. Free version by PG Doyle, 2006. Chapter 11, section 11.2 (Absorbing Markov chains), Theorems 11.4 and 11.5. https://math.dartmouth.edu/~prob/prob/prob.pdf
• Siebert U, Alagoz O, Bayoumi AM, Jahn B, Owens DK, Cohen DJ, Kuntz KM. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Value in Health. 2012;15(6):812-820. https://doi.org/10.1016/j.jval.2012.06.014
• Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338. https://doi.org/10.1177/0272989X9301300409
• Williams C, Lewsey JD, Mackay DF, Briggs AH. Estimation of survival probabilities for use in cost-effectiveness analyses: a comparison of a multi-state modeling survival analysis approach with partitioned survival and Markov decision-analytic modeling. Medical Decision Making. 2017;37(4):427-439. https://doi.org/10.1177/0272989X16670617
• Woods B, Sideris E, Palmer S, Latimer N, Soares M. NICE DSU Technical Support Document 19: Partitioned survival analysis for decision modelling in health care: a critical review. Decision Support Unit, ScHARR, University of Sheffield; 2 June 2017. https://orda.shef.ac.uk/articles/report/TSD_19_Partitioned_survival_analysis_as_a_decision_modelling_tool/33848653