State Reward

State Reward

A state reward is a cost, health outcome, utility or other consequence assigned to time spent in a state of a state-transition model. Multiplying the reward by the expected number of people or amount of time in that state lets an analyst accumulate outcomes over cycles. “Reward” is mathematical terminology: it can represent an undesirable cost or health loss as well as a benefit, provided the sign and units are clear.

Occupancy and reward vectors

In a cohort Markov model, let row vector $\boldsymbol{\pi}_t$ contain the proportions in each state at the start of cycle $t$. With row-stochastic transition matrix $P_t$, $\boldsymbol{\pi}_{t+1}=\boldsymbol{\pi}_tP_t$; each row of $P_t$ sums to one. Let column vector $\mathbf{r}_t$ contain a reward for one full cycle in each state, using the same state ordering as $\boldsymbol{\pi}_t$. Under a start-of-cycle occupancy convention, expected reward in cycle $t$ is

$$R_t=\boldsymbol{\pi}_t\mathbf{r}_t=\sum_s\pi_{t,s}r_{t,s}.$$

For health utility $u_{t,s}$ on a QALY scale and cycle length $\Delta t$ in years, a state reward is often $q_{t,s}=u_{t,s}\Delta t$ QALYs per full cycle. For a per-cycle cost $c_{t,s}$, expected cohort cost is $\sum_s\pi_{t,s}c_{t,s}$ per cohort member. If the cohort has $N$ members, multiply per-person expected totals by $N$ only when an aggregate budget is needed; do not accidentally apply $N$ twice.

The timing convention matters. Start-of-cycle occupancy implicitly applies the entire reward to the state occupied at the cycle's start, even if a person transitions during it. A half-cycle correction, tunnel-state construction or sufficiently short cycle can better approximate continuous accrual, depending on the model. State explicitly whether rewards accrue at cycle start, end, mid-cycle or from estimated time within states.

Two-cycle worked example

Consider three mutually exclusive states: Well, Ill and Dead. Use one-year cycles, start-of-cycle rewards, no discounting and no half-cycle correction solely to expose the arithmetic. Full-year utility weights are 0.90, 0.50 and 0; annual state costs are 100, 1,000 and 0 currency units. Assume initial distribution $\boldsymbol{\pi}_0=(0.80,0.20,0)$ and transition matrix with rows and columns in Well, Ill, Dead order:

The first two rows sum to one, and Dead is absorbing in this model. Multiplication gives $\boldsymbol{\pi}_1=(0.58,0.28,0.14)$: Well is $0.80(0.70)+0.20(0.10)=0.58$, Ill is $0.80(0.20)+0.20(0.60)=0.28$, and Dead is $0.80(0.10)+0.20(0.30)=0.14$. The probabilities again sum to one.

In cycle 0, expected QALYs are $0.80(0.90)+0.20(0.50)=0.82$ and costs are $0.80(100)+0.20(1{,}000)=280$. In cycle 1, expected QALYs are $0.58(0.90)+0.28(0.50)=0.662$ and costs are $0.58(100)+0.28(1{,}000)=338$. The two cycle-start rewards sum to 1.482 QALYs and 618 currency units per person. These totals do not imply that a person can live more than two years: the cohort average weights utility in each occupied year, and the example's occupancy convention counts cycle 1 survivors for its full cycle.

The example is deliberately stylised. Death during a cycle, nonuniform event timing and costs at death could change estimates under a more realistic convention. A death event might incur a one-time cost even though the absorbing Dead state carries zero ongoing cost; that event cost belongs on a transition, not as a perpetual Dead-state reward.

State versus transition rewards

A state reward accrues while occupying a state, such as annual care costs or utility-weighted survival. A transition reward is attached to a move from state $s$ to state $s'$, such as a one-time hospitalisation cost when entering Ill or a terminal cost upon entering Dead. For a transition reward matrix $H_t$, its expected contribution under a one-transition-per-cycle model is

$$T_t=\sum_s\sum_{s'}\pi_{t,s}P_{t,ss'}H_{t,ss'}.$$

If a one-time event cost is included in $H_t$, do not also apply it to every cycle in the destination state's ongoing cost. If an outcome depends on time since entering Ill or number of previous events, a single memoryless Ill state with one fixed reward may be inadequate; use tunnel states, individual simulation or history-dependent reward definitions as appropriate.

Costs and effects need distinct reward vectors and sources. A state might have a high care cost and low utility, while another has a lower cost and higher utility. Sign conventions vary across optimisation frameworks: a cost can be stored as a positive cost to minimise or as a negative reward to maximise. Document the convention before combining outputs or calculating incremental cost-effectiveness.

Accrual, discounting and time dependence

For horizon $T$, aggregate a per-cycle outcome as $\sum_{t=0}^{T-1}d_tR_t$ under a stated discount factor $d_t$ and timing convention. Costs and health outcomes may use separate applicable discount rates, and the year in which a cycle's reward occurs must be explicit. Discounting does not repair a timing error in state occupancy or a mismatched cycle length.

Rewards can depend on age, calendar year, time spent in a state, treatment arm or patient characteristics. For example, annual medication costs may change after discontinuation, and utility may decline with age. Specify whether reward changes reflect real changes in health or treatment versus price inflation, and avoid combining nominal costs from one year with real costs from another. The same state name can require different rewards under different strategies if treatment changes management while the health state remains otherwise comparable.

A shorter cycle can reduce approximation error but raises data and computation requirements; a half-cycle correction approximates within-cycle occupancy under assumptions that should be checked. If transitions are rare or the accrual timing has little impact, report a sensitivity comparison rather than assuming an adjustment is always material. Validate that state rewards are scaled to the cycle length: an annual utility multiplied by a monthly cycle must use $1/12$ year, while a one-time transition cost should not be divided by 12 merely because the cycles are monthly.

Health economic interpretation and checks

In cost-effectiveness analysis, attach QALY and cost rewards to each competing strategy, simulate occupancy under each strategy's transitions, then sum and compare discounted totals. Incremental QALYs and costs arise from both occupancy differences and differences in state or transition rewards. A strategy can improve survival while increasing time in a costly low-utility state, so neither survival alone nor a single cycle reward establishes cost-effectiveness.

Use evidence for state-specific utility and costs relevant to the population, perspective and period. Utilities should reflect the intended health state and avoid double counting adverse-event disutility already embodied in an observed health-state value. Costs should include relevant resource categories without counting the same admission as both a transition cost and an ongoing state cost. Assess parameter and structural uncertainty, especially when states conceal heterogeneous severity or when entry timing matters.

Check state ordering across $\boldsymbol{\pi}_t$, $P_t$ and reward vectors; row sums and nonnegative occupancy; survival and death accounting; units and cycle scaling; and whether ongoing rewards cease after death unless a justified post-death cost is represented separately. Reproduce simple hand calculations like the table before relying on a large model. Report the cycle convention, discount timing, half-cycle treatment, state and transition rewards, sources and uncertainty distributions.

Methodological sources

The ISPOR-SMDM State-Transition Modeling task force report covers cohort and individual state-transition model construction and validation. Sonnenberg and Beck's practical guide to Markov models in medical decision making explains how recurrent risks and timed outcomes are represented. The NICE health technology evaluations manual provides economic evaluation context for costs, health outcomes and time horizon. The matrix and reward values above are original teaching inputs, not an evidence-based clinical model.