Hierarchy of Diagnostic Efficacy

Hierarchy of Diagnostic Efficacy from Image Quality to Societal Value

The hierarchy of diagnostic efficacy sorts evidence about a test into six levels, from technical performance through accuracy and clinical decisions to patient outcomes and the costs and benefits to society. Fryback and Thornbury set it out in 1991 for diagnostic imaging, holding that efficacy at each lower level is logically necessary for efficacy higher up but does not guarantee it. In health technology assessment the hierarchy explains why diagnostic accuracy is an intermediate result and why an economic model of a test has to follow people from the test result through management decisions to health outcomes. It also shows assessors where the evidence for a test stops, which decides whether a linked evidence model, a trial or further research comes next.

Six levels from technical quality to societal value

Fryback and Thornbury proposed the hierarchy as an organising structure for appraising the literature on imaging efficacy, crediting the earlier work of Lee Lusted. Each level answers a different question about the same test, usually with a different kind of study. The level names follow the common phases that Lijmer, Leeflang and Bossuyt found across published evaluation schemes.

Levels 3 and 4 describe what clinicians do with the result, so they depend on setting and local practice. Levels 5 and 6 are where health economics enters directly, because a QALY gain or an incremental cost-effectiveness ratio can only be estimated once the earlier links are known or assumed. Fryback and Thornbury allowed level 5 effects to be measured or computed, which anticipates the decision modelling now used for tests.

Why accuracy is necessary but not sufficient

The core claim of the hierarchy is logical. An unreadable image cannot be accurate, a result that changes nobody's thinking cannot change management, and a change in management helps the patient only if the treatment chosen works for the people the test now selects. Ferrante di Ruffano and colleagues describe the same chain as a test-treatment pathway running from the test result through diagnosis to the choice and delivery of treatment. They note that improvements in accuracy will not benefit patients unless they lead to changes in diagnoses and management, and they caution against using accuracy as a surrogate for patient health.

Empirical work supports that caution. Siontis and colleagues reviewed 140 randomised trials of diagnostic tests and found patient outcomes significantly improved in 28 of 153 comparisons (18%). Effects on patient outcomes showed no concordance with effects on the use of further tests or treatments, and did not correlate with the diagnostic odds ratio of the experimental test, so the authors concluded that diagnostic performance and effects on management decisions are not necessarily indicative of patient benefit.

Some routes to health effects bypass accuracy. Ferrante di Ruffano and colleagues list feasibility, interpretability, time to diagnosis and treatment, the clinician's confidence and the patient's experience of testing, so two equally accurate tests can produce different outcomes.

Linked evidence and test-treatment trials in HTA

Test-treatment trials give direct level 5 evidence by randomising patients to testing strategies and measuring outcomes after treatment. Ferrante di Ruffano and colleagues note that such trials often need sample sizes several orders of magnitude larger than trials of treatments, because only patients whose management differs between arms contribute to the contrast. A later review of 103 test-treatment trials published from 2004 to 2007 judged 101 (98%) to be at risk of bias.

The NICE HealthTech programme manual (PMG48, section 2.3.14) states that most outcomes of diagnostic technologies come after testing, because treatment or preventive measures are started, modified or stopped, and that evaluating tests in clinical trials is unusual. When direct data on final outcomes are lacking, it asks for a linked evidence modelling approach in which the links between test results, decisions about care and final outcomes are specified and justified. Section 2.3.15 names test accuracy as an intermediate outcome and, when only such outcomes support a value proposition, notes that it is beneficial for companies to provide information supporting their use.

Merlin and colleagues describe a linked evidence approach, mandated by the Australian federal government in 2005 for HTAs of medical tests and informed by Fryback and Thornbury's criteria, that links evidence on comparative accuracy, impact on clinical decision making and the effectiveness of the treatments that follow. An abridged version stops at the change in management when the new test finds patients with a similar spectrum of disease to those already treated. In Medical Services Advisory Committee assessments from August 2005 to March 2012, 96% of test evaluations used the full or abridged approach and 61% had the full linkage. Among evaluations judged problematic, 41% struggled with an imperfect reference standard and 18% with treatment effectiveness in test-positive patients when the new test was more accurate than its comparator.

In a decision tree the test splits the cohort into true and false positives and negatives, management attaches to each branch and treatment effects to each management choice. Errors at level 2, such as verification bias or an estimate read from the wrong part of a summary ROC curve, therefore carry through every later link.

Worked example with a more sensitive imaging test

All numbers in this example are illustrative. A new imaging test is proposed for 1,000 people with suspected disease, of whom 30% have it. The current test has sensitivity 0.80 and the new test 0.95, and both have specificity 0.90, so false positives do not change. The new test costs GBP 150 more per person, treating each extra case costs GBP 2,000, and treatment gains 0.40 QALYs per treated case. Discounting and test harms are ignored.

$$\Delta Q = N \, p \, (Se_1 - Se_0) \, m \, g, \qquad \Delta C = N \, c_T + N \, p \, (Se_1 - Se_0) \, m \, c_R$$

where $\Delta Q$ and $\Delta C$ are incremental QALYs and costs, $N$ is the number tested, $p$ the prevalence, $Se_1$ and $Se_0$ the new and current sensitivities, $m$ the share of extra true positives whose management changes, $g$ the QALY gain per extra case treated, $c_T$ the extra test cost per person and $c_R$ the treatment cost per extra case.

1. Reason from level 2 alone. Of 300 people with the disease, the new test detects 285 and the current test 240, so 45 extra cases are found. If every extra case is treated with full benefit ($m = 1$), the gain is 45 times 0.40, or 18.0 QALYs, at a cost of GBP 150,000 plus 45 times GBP 2,000, or GBP 240,000. The ICER is GBP 13,333 per QALY.

2. Add level 3 and 4 evidence. A management impact study finds that clinicians would already have treated 40% of the extra cases on clinical grounds, so management changes for 60% of them, or 27 people.

$$\Delta Q = 1000 \times 0.30 \times 0.15 \times 0.60 \times 0.40 = 10.8$$

where 10.8 is the QALY gain per 1,000 people tested. Costs fall to GBP 150,000 plus 27 times GBP 2,000, or GBP 204,000, and the ICER rises to GBP 18,889 per QALY.

3. Question the level 5 link. The treatment trials enrolled cases found by the current test. If the extra cases are milder and gain 0.20 QALYs each, the gain halves to 5.4 QALYs at the same cost, and the ICER doubles to GBP 37,778 per QALY.

4. Interpret. At an illustrative threshold of GBP 25,000 per QALY, the net monetary benefit per 1,000 people tested falls from GBP 210,000 to GBP 66,000 and then to minus GBP 69,000. The accuracy gain is identical in all three steps, so the decision turns on evidence at levels 4 and 5 that an accuracy study does not collect, which is where value of information analysis would direct further research.

Choices that decide where the chain of evidence breaks

Whether level 2 evidence is enough depends on how the new test differs from the old one. Lord, Irwig and Simes argue that accuracy studies suffice when a new test is safer or more specific than an old test with similar sensitivity. A more sensitive test detects extra cases, and treatment trials that enrolled only patients detected by the old test may not apply to them, unless the new test detects the same spectrum and subtype of disease or treatment response is similar across that spectrum.

Van den Bruel and colleagues make defining the test's place in the care pathway a step before assessing accuracy, since it decides whether existing evidence can be used or new evidence is needed. In a trial of PET in lung cancer described by Ferrante di Ruffano and colleagues, PET failed to change some management decisions because strong preferences for operating on all patients with stage IIIa disease exceeded the effect of the PET results. A management effect measured in one health system may not transfer to another.

Hierarchy of diagnostic efficacy compared with nearby frameworks

Lijmer, Leeflang and Bossuyt identified 19 phased schemes for medical tests published between 1978 and 2007, with substantial similarity and common phases matching the six levels. Sackett and Haynes framed diagnostic research as four phases of question, the last asking whether tested patients fare better in their ultimate health outcomes than similar patients who are not tested.

The efficacy hierarchy differs from a hierarchy of study designs, such as a level of evidence scale, which ranks designs by risk of bias. The efficacy hierarchy ranks the question being answered, so a well-conducted randomised controlled trial at level 4 still says nothing directly about level 5. Merlin and colleagues present the linked evidence approach as a way to evaluate the clinical utility of new tests without direct trial evidence, so clinical utility in that sense sits at levels 4 and 5.

Limitations and common misreadings

The hierarchy organises evidence rather than estimating anything. Most errors treat its levels as more rigid or informative than they are.

1. Reading the levels as a required sequence. Lijmer, Leeflang and Bossuyt conclude that the frameworks help distinguish study types but cannot be seen as a necessary sequence, and that test evaluation is more likely cyclic and repetitive than linear.
2. Treating a level 2 gain as a level 5 gain. Extra true positives produce health gains only through changed management and effective treatment.
3. Equating changed management with benefit. Siontis and colleagues found no concordance between effects on management and effects on patient outcomes.
4. Reading a null test-treatment trial as proof of no effect. Ferrante di Ruffano and colleagues warn that findings of no effect are often read as evidence of absence when the trial could not separate an ineffective test from a poorly implemented strategy.

Sources

• Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Medical Decision Making. 1991;11(2):88-94. https://doi.org/10.1177/0272989X9101100203
• Lijmer JG, Leeflang M, Bossuyt PM. Proposals for a phased evaluation of medical tests. Medical Decision Making. 2009;29(5):E13-E21. https://doi.org/10.1177/0272989X09336144
• Ferrante di Ruffano L, Hyde CJ, McCaffery KJ, Bossuyt PM, Deeks JJ. Assessing the value of diagnostic tests: a framework for designing and evaluating trials. BMJ. 2012;344:e686. https://doi.org/10.1136/bmj.e686
• Ferrante di Ruffano L, Dinnes J, Sitch AJ, Hyde C, Deeks JJ. Test-treatment RCTs are susceptible to bias: a review of the methodological quality of randomized trials that evaluate diagnostic tests. BMC Medical Research Methodology. 2017;17:35. https://doi.org/10.1186/s12874-016-0287-z
• Siontis KC, Siontis GC, Contopoulos-Ioannidis DG, Ioannidis JP. Diagnostic tests often fail to lead to changes in patient outcomes. Journal of Clinical Epidemiology. 2014;67(6):612-621. https://doi.org/10.1016/j.jclinepi.2013.12.008
• Lord SJ, Irwig L, Simes RJ. When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Annals of Internal Medicine. 2006;144(11):850-855. https://doi.org/10.7326/0003-4819-144-11-200606060-00011
• Merlin T, Lehman S, Hiller JE, Ryan P. The "linked evidence approach" to assess medical tests: a critical analysis. International Journal of Technology Assessment in Health Care. 2013;29(3):343-350. https://doi.org/10.1017/S0266462313000287
• Merlin TL, Hiller JE, Ryan P. Impact of the "linked evidence approach" method on policies to publicly fund diagnostic, staging, and screening medical tests. MDM Policy & Practice. 2016;1(1):2381468316672465. https://doi.org/10.1177/2381468316672465
• Van den Bruel A, Cleemput I, Aertgeerts B, Ramaekers D, Buntinx F. The evaluation of diagnostic tests: evidence on technical and diagnostic accuracy, impact on patient outcome and cost-effectiveness is needed. Journal of Clinical Epidemiology. 2007;60(11):1116-1122. https://doi.org/10.1016/j.jclinepi.2007.03.015
• Sackett DL, Haynes RB. The architecture of diagnostic research. BMJ. 2002;324(7336):539-541. https://doi.org/10.1136/bmj.324.7336.539
• National Institute for Health and Care Excellence. NICE HealthTech programme manual (PMG48). Published 14 July 2025, last updated 17 December 2025. Sections 2.3.14 and 2.3.15. https://www.nice.org.uk/process/pmg48