Data Envelopment Analysis

Data Envelopment Analysis in Health Care Efficiency Measurement

Data envelopment analysis measures how far a hospital, clinic or health system sits from the best combinations of resources and activity achieved by comparable units in the same sample. Each unit receives a score, with 1 meaning that no observed peer, or weighted mix of peers, produces at least the same outputs with proportionally less input. Hollingsworth describes it as by far the most common method for analysing efficiency in health care, with over 400 applications in health care settings, and as the method behind more than 90% of frontier efficiency studies there. The results feed provider benchmarking and productivity reviews, so the set-up choices of inputs, outputs, returns to scale and comparator hospitals decide which organisations appear inefficient. The score speaks to technical efficiency relative to the sample, a narrower question than whether a health system spends its budget on the right services.

Efficiency as a weighted ratio of outputs to inputs

A hospital uses many inputs, such as medical staff, nurses, capital and drugs, to produce many outputs, such as inpatient episodes, outpatient attendances and emergency attendances. These cannot simply be added together, so the method forms a ratio of weighted outputs to weighted inputs. Jacobs, in a study of NHS trusts for the Centre for Health Economics at York, sets out the problem as choosing, for each hospital, the weights that give it the highest possible score, subject to no hospital in the sample scoring above 1 with those same weights.

The weights are not prices and are not known in advance. Hollingsworth explains that the linear programme picks, from all permissible sets, the weights that show each hospital in the best possible light against how the other hospitals use their inputs and outputs. The approach follows the linear programming formulation of Charnes, Cooper and Rhodes, which Simar and Wilson describe as resting on the definitions of technical and allocative efficiency given by Debreu and Farrell.

Technical efficiency, in the definition Hollingsworth uses, means minimising the inputs used to produce the chosen outputs, regardless of the value placed on those outputs. Allocative efficiency asks whether the outputs, or the input mix, create the most value, which needs prices or values that a standard model does not use. Hollingsworth notes that the method can also measure allocative efficiency, for example by comparing units with identical weights, although his account concentrates on technical efficiency.

The input-oriented envelopment model

The ratio problem is usually solved in its equivalent envelopment form, which builds a composite peer for each hospital. Jacobs gives the input-minimising version used for hospital 0 in a sample of $n$ hospitals.

$$\min_{\theta,\lambda} \theta \quad \text{subject to} \quad \sum_{j=1}^{n} \lambda_j x_{ij} \le \theta x_{i0}, \qquad \sum_{j=1}^{n} \lambda_j y_{rj} \ge y_{r0}, \qquad \lambda_j \ge 0$$

where $\theta$ is the efficiency score of hospital 0, $x_{ij}$ is the amount of input $i$ used by hospital $j$, $y_{rj}$ is the quantity of output $r$ produced by hospital $j$, and $\lambda_j$ is the weight placed on hospital $j$ in the composite peer. The composite must produce no less of every output than hospital 0 while using no more than the fraction $\theta$ of each of its inputs.

A hospital with $\theta = 1$ is efficient because no composite outperforms it. A score below 1 gives the proportional cut in all inputs that would place the hospital on the frontier, and the hospitals with positive $\lambda_j$ are its peers, whose combination supplies the targets. Jacobs chose this input orientation because cost was the variable over which trust managers had most control. An output-oriented model, which asks how far outputs could rise with the same inputs, can be estimated instead, and it suits settings where budgets are fixed and the question is how much more care could be delivered.

Constant or variable returns to scale

The model above assumes constant returns to scale, so a hospital twice the size of another should produce twice the output. Jacobs shows that variable returns to scale are obtained by adding one constraint.

$$\sum_{j=1}^{n} \lambda_j = 1$$

where the $\lambda_j$ are the peer weights defined above. The constraint forces each hospital to be compared with a weighted average of real hospitals rather than a scaled-up or scaled-down combination of them, so variable returns scores are always greater than or equal to constant returns scores.

Hollingsworth notes that the variable returns frontier draws more hospitals onto the frontier, while the constant returns model discriminates more between them, so analysts often run both. The gap between the two separates inefficiency due to operating at the wrong size from pure technical inefficiency. Scale efficiency is the ratio of the two scores.

$$SE_0 = \frac{\theta_0^{CRS}}{\theta_0^{VRS}}$$

where $SE_0$ is the scale efficiency of hospital 0, $\theta_0^{CRS}$ is its constant returns score and $\theta_0^{VRS}$ is its variable returns score. A value of 1 means the hospital operates at a scale where size causes no loss.

Worked example with five hospitals and one input

The figures below are illustrative. Five hospitals each use one input, running cost in GBP million, to produce one output, thousands of case-mix-adjusted inpatient episodes. With one input and one output the constant returns score is simply each hospital's output per GBP million divided by the highest value in the sample, which is hospital B at 0.75.

Hospital D is the only one inside the variable returns frontier. The cheapest mix of real hospitals that produces at least 25,000 episodes takes one quarter of A and three quarters of B, giving exactly 25,000 episodes.

$$\theta_D^{VRS} = \frac{0.25 \times 20 + 0.75 \times 40}{50} = \frac{35}{50} = 0.70$$

where the numerator is the cost of the composite peer in GBP million and 50 is the cost of hospital D. Under constant returns, D is compared with a scaled-down copy of B.

$$\theta_D^{CRS} = \frac{25 / 50}{30 / 40} = \frac{0.50}{0.75} = 0.6667$$

where 25 over 50 is D's output per GBP million and 30 over 40 is B's. Scale efficiency follows from the two scores.

$$SE_D = \frac{0.6667}{0.7000} = 0.952$$

where the numerator and denominator are D's constant and variable returns scores. The results say that D could in principle deliver its 25,000 episodes for GBP 35 million rather than GBP 50 million, a 30% reduction in cost, by matching the practice of A and B. Almost all of D's shortfall is pure technical inefficiency, since its scale efficiency is close to 1. Hospitals A, C and E are efficient under variable returns but not under constant returns, which points to A being too small and C and E too large to reach B's productivity. E is on the variable returns frontier by construction, because no other hospital produces as many episodes, so its score of 1 says little about how well it is run.

Choosing inputs, outputs and comparators

The specification decides the results more than the solver does. Hollingsworth stresses that the models and relationships need to be thought through with care because the techniques are very sensitive to the assumptions made.

Adding variables raises scores and puts more units on the frontier. In Jacobs's NHS study of 232 trusts, a model with one input and 17 outputs placed 110 trusts on the frontier with a mean score of 0.936, while a model with the same input and 3 outputs placed 8 there with a mean of 0.645. Hollingsworth gives a rule of thumb that the number of units should be at least three times the combined number of inputs and outputs, since scores are overestimated when the number of units is small relative to the number of variables.

Case mix has to be handled, since a hospital treating sicker patients needs more resources per episode. Hollingsworth argues that patients are not inputs that the hospital transforms, and that outputs are better adjusted for case severity, for example with DRG weights where payment systems collect them. Quality matters too. With throughput data alone, any inefficiency found may simply reflect quality that the data omit, and quality measures can be swamped by large throughput numbers unless weighted with care. The sample also has to compare like with like, because a single specialist or teaching hospital in a group of general hospitals can confound the frontier.

Reading the scores in policy and health technology assessment

An efficient score is a statement about the sample. Hollingsworth points out that a hospital deemed efficient is efficient only given the practices observed in that sample, and greater efficiency than any observed may be achievable. If every hospital in the sample performs poorly, some will still score 1.

Jacobs concluded that several specifications should be run to give ranges of inefficiency that act as signals, not point estimates. In her NHS data the best-agreeing DEA specifications correlated with the Department of Health cost indices at only around 0.5, and the methods did not appear to measure cost efficiency in the same way. Hollingsworth reports a UK project in which inefficient hospitals received input reduction targets, managers could sometimes explain a poor result by data deficiencies, and the useful content was often less the exact scores than the areas flagged for improvement and the comparators available for benchmarking.

DEA compares producers. It does not compare treatments for the same patients, which is the job of the cost-effectiveness frontier in health technology assessment, and a technically efficient hospital may still deliver services that are poor value for money. Changes over time can be tracked with a Malmquist index, which Hollingsworth regards as more informative than a single snapshot.

DEA compared with stochastic frontier analysis

The main alternative is stochastic frontier analysis, which estimates a frontier by regression and splits the error term into noise and inefficiency. The comparison below draws on Jacobs and Hollingsworth.

Neither set of assumptions can be tested. DEA assumes no measurement error or random variation in output, and stochastic frontier analysis assumes a particular error distribution. Hollingsworth cites researchers who recommend running both and examining hospitals that both methods flag as inefficient.

Where DEA results mislead

The method is deterministic, so every deviation from the frontier is read as inefficiency. Hollingsworth warns that small random variation alters the score of an inefficient hospital, while larger variation can move the frontier itself and change the scores of many hospitals. Jacobs notes that DEA relies heavily on outliers and does not smooth them away, so extreme data points, which other methods often purge, define the frontier.

The scores are also estimates with sampling error. Simar and Wilson note that the DEA efficiency estimate is biased by construction in finite samples, with the bias likely to be larger where data are sparse, and that it converges more slowly as the number of inputs and outputs grows, the curse of dimensionality. The bias can be estimated by bootstrapping, although they point out that a naive bootstrap, which resamples from the empirical distribution of the data, is inconsistent in this setting. Hollingsworth adds that confidence intervals are prudent unless the sample includes every relevant unit.

Second-stage analysis needs similar care. Many studies regress DEA scores on environmental factors such as case mix with tobit or ordinary least squares models. Simar and Wilson show that conventional inference in these regressions is invalid because the estimated efficiencies are serially correlated, and they propose truncated regression with bootstrap procedures instead.

Three misreadings recur. A frontier hospital is described as best practice, when it may simply be unusual or misreported. Throughput is treated as benefit, which can reward avoidable admissions. Scores from different samples or specifications are compared directly, although each frontier is specific to its data.

Sources

• Hollingsworth B. Health system efficiency: measurement and policy. In: Cylus J, Papanicolas I, Smith PC, editors. Health System Efficiency: How to Make Measurement Matter for Policy and Management. Health Policy Series 46. Copenhagen: World Health Organization on behalf of the European Observatory on Health Systems and Policies; 2016. p. 99-114. https://iris.who.int/server/api/core/bitstreams/2794c497-bcca-4a89-8161-b65f1b5cb998/content
• Jacobs R. Alternative Methods to Examine Hospital Efficiency: Data Envelopment Analysis and Stochastic Frontier Analysis. CHE Discussion Paper 177. York: Centre for Health Economics, University of York; 2000. https://www.york.ac.uk/media/che/documents/papers/discussionpapers/CHE%20Discussion%20Paper%20177.pdf
• Simar L, Wilson PW. Estimation and inference in two-stage, semi-parametric models of production processes. Journal of Econometrics. 2007;136(1):31-64. https://doi.org/10.1016/j.jeconom.2005.07.009 (text read: Institut de Statistique Discussion Paper 0307, Université catholique de Louvain, 2003, http://hdl.handle.net/2078.1/122906)