Efficiency Index

Efficiency Index in Health Care Performance Measurement

An efficiency index condenses the relationship between the resources a hospital, practice or health system uses and the care or health it produces into a single comparable number. In health economics the term covers several different constructions, including a cost ratio set against national average prices and a score showing how far a unit sits from an estimated best-practice frontier. Purchasers and researchers use such indices to flag possible waste and to benchmark providers, and in the English NHS a cost index of this kind has been published for NHS trusts. Each score depends on the inputs, outputs, comparators and model chosen, so the same hospital can look efficient under one specification and inefficient under another.

Three families of efficiency index

The Observatory volume on health system efficiency edited by Cylus, Papanicolas and Smith describes efficiency at its simplest as a ratio of resources consumed to valued outputs, and notes that this has produced many disconnected indicators, each showing one aspect of inefficiency. Three broad families can be distinguished, and they differ in what they compare a unit against.

The authors of the framework chapter point out that partial indicators use only a subset of inputs and outputs. Consultations per doctor, for example, ignore the other inputs, the doctor's other work and the health gain achieved, so none covers the whole conversion of inputs into valued outcomes. Unit cost measures give a view of a provider's overall technical efficiency relative to other providers but little guidance on why a gap arises. Every one of these indices measures some part of efficiency, and none measures all of it.

The weighted ratio behind a frontier score

Hollingsworth's chapter in the same volume starts from the single-input, single-output case, in which technical efficiency is output divided by input. A hospital, though, treats many kinds of case with several kinds of staff and equipment, so inputs and outputs have to be weighted before they can be added. When the weights are left fully flexible, the score for each unit is the ratio of a weighted sum of its outputs to a weighted sum of its inputs, with the weights chosen so that the score lies between 0 and 1.

$$E_j = \frac{\sum_{r=1}^{s} u_r \, y_{rj}}{\sum_{i=1}^{m} v_i \, x_{ij}}$$

where $E_j$ is the efficiency score of unit $j$, $y_{rj}$ is the quantity of output $r$ produced by unit $j$, $x_{ij}$ is the quantity of input $i$ it uses, $u_r$ and $v_i$ are the non-negative weights attached to output $r$ and input $i$, $s$ is the number of outputs and $m$ the number of inputs.

In data envelopment analysis the weights are not fixed in advance. The linear programming model picks, for each hospital, the set of weights that shows it in the best possible light relative to the other hospitals in the sample, and that most favourable score is reported, subject to no hospital in the sample scoring above 1 with the same weights. A unit scored 1 lies on the frontier formed by its peers, and a score below 1 measures its distance from that frontier. With an input orientation the score reads as the proportion of current inputs that the frontier suggests would suffice for the same outputs.

The assumption about returns to scale changes the scores directly. Constant returns to scale count the effects of operating size as inefficiency and discriminate more sharply than variable returns, which draw more hospitals onto the frontier. Stochastic frontier analysis splits the usual regression error into an inefficiency part and a noise part, so its scores depend on the assumed error distribution and functional form. Change over time is a separate question handled by the Malmquist index.

The NHS reference cost index

The national cost collection in England has published an expected-cost index for trusts, called both the reference cost index (RCI) and the National Cost Collection Index (NCCI) in the 2018/19 collection documents issued by NHS England and NHS Improvement. The user guide defines each trust's actual cost as its activity multiplied by its own unit cost and its expected cost as the same activity multiplied by the national mean unit cost. Its standard queries compute the index as total actual cost divided by total expected cost, multiplied by 100, using data adjusted by the market forces factor.

$$\mathrm{RCI}_j = 100 \times \frac{\sum_{h} q_{jh} \, c_{jh}}{\sum_{h} q_{jh} \, \bar{c}_h}$$

where $\mathrm{RCI}_j$ is the reference cost index of trust $j$, $q_{jh}$ is the trust's activity in currency $h$ (a healthcare resource group or other unit of activity), $c_{jh}$ is the trust's unit cost for that currency after market forces factor adjustment, and $\bar{c}_h$ is the national mean of the market forces factor adjusted unit costs for the same currency.

Because expected cost prices each trust's own activity at national average rates, the denominator carries the case mix, and a trust whose actual costs equal its expected costs scores exactly 100. The structure is the same as an observed-to-expected ratio, with cost in place of events. The market forces factor used for the index is rescaled so that the adjustment is cost neutral nationally.

A cost index of this kind measures cost per unit of classified activity and says nothing about outcomes. It also inherits the quality of the costing returns and of the patient classification, so a high score can reflect costing practice or coding as well as waste.

Worked example comparing a cost index with a frontier score

All numbers in this example are illustrative. Three hospitals treat two kinds of case, with national mean unit costs of GBP 2,000 for a simple case and GBP 6,000 for a complex one.

1. Unadjusted cost per case. Hospital A spends 3,500,000 / 1,200 = GBP 2,917 per case, B spends 4,400,000 / 1,100 = GBP 4,000 and C spends 2,900,000 / 1,200 = GBP 2,417. On this partial indicator B looks worst.

2. Expected cost. A's expected cost is 1,000 × 2,000 + 200 × 6,000 = GBP 3.2 million. B's is 600 × 2,000 + 500 × 6,000 = GBP 4.2 million. C's is 1,100 × 2,000 + 100 × 6,000 = GBP 2.8 million.

3. Cost index. A scores 100 × 3.5 / 3.2 = 109.4, B scores 100 × 4.4 / 4.2 = 104.8 and C scores 100 × 2.9 / 2.8 = 103.6. Once case mix is priced in, B rises to second place and A becomes the highest-cost hospital, about 9% above its activity priced nationally.

4. Frontier score. Treating expected cost as a single measure of cost-weighted output and actual cost as the single input, the ratios of output to input are 3.2 / 3.5 = 0.914 for A, 4.2 / 4.4 = 0.955 for B and 2.8 / 2.9 = 0.966 for C. Under constant returns to scale C defines the frontier. The input-oriented score is each ratio divided by C's, so A scores (3.2 × 2.9) / (3.5 × 2.8), which is 9.28 divided by 9.80, or 0.947. B scores (4.2 × 2.9) / (4.4 × 2.8), which is 12.18 divided by 12.32, or 0.989, and C scores 1.000.

5. Interpretation. If A matched C's cost per unit of cost-weighted activity, the model implies its activity could be delivered for 3.5 × 0.947 = GBP 3.31 million, about 3.5 × (1 − 0.94694) = GBP 0.186 million less. The two indices agree on the ranking, as they must with one input and one output, but not on the benchmark. C is at the frontier with a score of 1, yet its cost index of 103.6 places it 3.6% above national average cost, because a frontier score is relative to the best unit in the sample while the cost index is relative to the national mean. Outcomes enter neither calculation, so fewer readmissions at one hospital would go unseen.

Health system efficiency indices and the WHO ranking

Efficiency indices are also built for whole health systems. The chapter by Cylus and Pearson on cross-national comparisons describes the WHO analysis by Evans and colleagues of panel data from 191 countries, which related per capita health expenditure to disability-adjusted life expectancy after controlling for educational attainment. The index compared the health expected with no health care spending against the health expected if every system were as efficient as the best performer, and it ranked Oman most efficient and Zimbabwe least efficient.

The same chapter reports how fragile that ranking proved. Gravelle and colleagues found that 50 of the 191 countries had only one year of data, so those countries offered no variation over time for a fixed-effects model, and in a model using between-country variation instead Oman fell to 169th place. The chapter records that the WHO has not attempted a further performance ranking since. A system-level index can thus be driven by model specification as much as by performance.

Reading an index before acting on it

The construction limits what any index can support. A DEA frontier is made of the best units in the sample, so a unit scored 1 is efficient only relative to observed practice, and the method reads every shortfall as inefficiency, with no allowance for measurement error or random variation.

The empirical record in the English NHS points the same way. Jacobs compared the rankings of NHS trusts from the Department of Health's cost indices with those from DEA and SFA on the same data and concluded that the methods may measure different aspects of efficiency and that differences in scores may reflect noise and data deficiencies. Jacobs recommended ranges from several specifications as signalling devices in place of point estimates, and found the savings from raising poorer performers would be quite modest. Street estimated confidence intervals around hospital inefficiency from English cost functions and concluded that little confidence could be placed in individual point estimates, and that such techniques should not be used to set annual performance targets. Bootstrapping DEA scores, as a Nordic cross-country hospital study did, gives confidence intervals and a correction for small-sample bias.

Case mix is the other recurring problem. Hollingsworth describes adjusting outputs with diagnosis-related group weights, which is common for inpatient care but harder for outpatient and primary care, and a second-stage regression of DEA scores on patient characteristics, which raises statistical problems of its own. Case-mix adjustment improves comparability, but an index that omits quality can reward a provider that treats more cheaply and less well. Hollingsworth's 2008 review of 317 frontier efficiency studies, mostly using DEA, sets out criteria for judging whether a study is useful enough to act on.

Where efficiency indices sit among related measures

An efficiency index is mainly a tool for technical efficiency, the question of whether chosen outputs are produced with the fewest inputs. The framework chapter places allocative efficiency, the question of whether the right mix of services is produced given their value, at the centre of health technology assessment, where cost per QALY from cost-effectiveness analysis is the main criterion. A hospital can score 1 on a technical efficiency index while delivering services that are poor value, and a cost-effective treatment can be delivered wastefully. Provider benchmarking uses efficiency indices alongside process and quality comparisons.

Common misreadings of efficiency scores

Most errors with efficiency indices come from treating a model-relative score as an absolute verdict on a provider. The worked example above shows how far the reading of a score depends on its benchmark.

1. Reading a score of 1 as optimal care. It means the unit lies on the frontier estimated from this sample and these variables.
2. Reading the shortfall as a cash saving. A score of 0.947 implies a 5.3% input reduction under the model, before any account of fixed costs or quality.
3. Ranking by point estimate. Street found little confidence could be placed in point estimates for individual hospitals.
4. Ignoring outcomes. Activity and cost indices treat a readmission like any other episode unless outcomes are built in.

Sources

• Cylus J, Papanicolas I, Smith PC. A framework for thinking about health system efficiency. In: Cylus J, Papanicolas I, Smith PC, editors. Health system efficiency: how to make measurement matter for policy and management. Health Policy Series 46. Copenhagen: WHO Regional Office for Europe, on behalf of the European Observatory on Health Systems and Policies; 2016. Chapter 1. https://iris.who.int/server/api/core/bitstreams/2794c497-bcca-4a89-8161-b65f1b5cb998/content
• Cylus J, Pearson M. Cross-national efficiency comparisons of health systems, subsectors and disease areas. In: Cylus J, Papanicolas I, Smith PC, editors. Health system efficiency (as above). 2016. Chapter 7.
• Hollingsworth B. Health system efficiency: measurement and policy. In: Cylus J, Papanicolas I, Smith PC, editors. Health system efficiency (as above). 2016. Chapter 5.
• Hollingsworth B. The measurement of efficiency and productivity of health care delivery. Health Economics. 2008;17(10):1107-1128. https://doi.org/10.1002/hec.1391
• Jacobs R. Alternative methods to examine hospital efficiency: data envelopment analysis and stochastic frontier analysis. Health Care Management Science. 2001;4(2):103-115. https://doi.org/10.1023/a:1011453526849
• NHS England and NHS Improvement. User guide: National Cost Collection 2018/19. A guide to using the data. 2020. https://www.england.nhs.uk/wp-content/uploads/2020/08/National_Cost_Collection_2018-19_A_Guide_to_using_the_data.pdf
• NHS England and NHS Improvement. National Cost Collection 2019. For data relating to 2018/19: commentary on headlines and introduction to the available data. 2020. https://www.england.nhs.uk/wp-content/uploads/2020/08/1_-_NCC_Report_FINAL_002.pdf
• Street A. How much confidence should we place in efficiency estimates? Health Economics. 2003;12(11):895-907. https://doi.org/10.1002/hec.773