Moving Target Problem

Moving Target Problem in Health Technology Assessment

Health technology assessment (HTA) usually judges a technology as it stood when the trials were run, yet many devices, software products and surgical procedures go on changing while that evidence is collected and appraised. The results can then describe an earlier version, less experienced operators or a comparator that practice has already left behind. For health technology assessment bodies this creates a timing dilemma, since an early assessment measures an immature technology and a late one may arrive after the technology has spread and a randomised comparison is no longer feasible. Early assessment, coverage tied to evidence generation, version-aware guidance and living HTA are the main responses.

Why health technologies change while they are being assessed

The problem is discussed mainly for devices, procedures and software. Drummond and colleagues, summarising earlier work on devices, describe three sources of movement. Product modifications occur frequently and may affect the performance of the device. As physicians use the device there may be a learning effect, with consequent improvements in the effectiveness of the procedure. Device prices are often more dynamic than those of pharmaceuticals, which alters estimates of relative costs and cost-effectiveness.

Cook distinguishes a community or technology learning curve, reflecting refinement of the technology and technique, from the personal learning of individual surgeons. A late evaluation overcomes the first, while the second remains in smaller form. Tarricone and colleagues add that outcomes of procedures with medical devices depend heavily on end-users' experience, and that device innovation is often incremental, which makes repeated randomised trials of each version impractical.

Software moves faster still. Interviewing Australian professionals about clinical decision support software, Laka and colleagues report participants' view that the time needed to generate evidence usually exceeded the software development and upgrade cycle, so that evaluation could never keep up with its moving target, and that after an upgrade such software could move from providing information to specifying diagnoses and treatments.

The comparator can move as well. The UK By-Band trial was designed to compare gastric banding with gastric bypass for severe obesity. Registry data reviewed for its adaptation showed the band's share of band, bypass and sleeve operations falling from 32.6 per cent in 2008-2009 to 15.8 per cent in 2013, while sleeve gastrectomy rose from 9.0 to 28.1 per cent, so the trial added sleeve gastrectomy as a third arm. Thokala and colleagues note that one-off HTAs can go out of date as new data, new comparators and new methods appear, so the problem is not confined to devices.

Buxton's law and the timing of assessment

The dilemma of when to evaluate has a well-known summary. Rogers and colleagues quote Buxton's law as "it is always too early for rigorous evaluation until, unfortunately, it's suddenly too late", and they and Tarricone and colleagues attribute it to Buxton's 1987 chapter on the economic appraisal of heart transplants in the UK. Early in the life of a technology, results reflect prototypes and inexperienced operators. Later, clinicians and patients may already be convinced of its worth, equipoise is lost and recruitment to a randomised comparison becomes difficult.

Mowatt and colleagues studied the timing of HTA for fast-changing technologies such as laparoscopic cholecystectomy and teleradiology. They did not identify the precise point at which assessment should start, recommended that fast-changing technologies be reassessed from time to time, and noted that it remained unclear when a technology has changed enough to warrant reassessment. Rapid diffusion led by product champions, as with laparoscopic cholecystectomy, may limit the methods of evaluation that can then be used.

How the problem distorts a cost-effectiveness estimate

A cost-effectiveness analysis turns an effect estimate, a cost estimate and a comparator into an ICER. If the trial captured early learning-curve outcomes, the effect estimate may understate what experienced teams achieve. If a newer version has replaced the one trialled, the estimate refers to a product no longer sold. If the device price or the comparator has changed, both incremental cost and incremental effect shift. Drummond and colleagues report that these factors are often ignored in economic evaluations and suggest that coverage schemes linked to evidence may require re-analysis over time.

The moving target therefore adds to decision uncertainty in a particular way. Probabilistic analysis describes uncertainty about the technology as defined in the model, while the moving target problem asks whether the technology defined in the model is still the one being bought.

Worked example with a learning curve and a switching value

All numbers in this example are illustrative. A new device-based procedure is compared with standard care, and the payer uses an illustrative cost-effectiveness threshold of 20,000 GBP per QALY.

$$\mathrm{ICER} = \frac{\Delta C}{\Delta Q}, \qquad \mathrm{NMB} = \lambda \, \Delta Q - \Delta C$$

where $\Delta C$ is the incremental cost per patient in GBP, $\Delta Q$ is the incremental QALYs per patient, $\lambda$ is the threshold in GBP per QALY and NMB is net monetary benefit per patient.

Step 1 (early trial). Complications are more frequent while teams learn, so $\mathrm{ICER} = 1{,}500 / 0.05 = 30{,}000$ GBP per QALY and $\mathrm{NMB} = 0.05 \times 20{,}000 - 1{,}500 = -500$ GBP. On this evidence the procedure is not cost-effective.

Step 2 (experienced teams). Fewer complications lower the incremental cost and raise the gain, so $\mathrm{ICER} = 1{,}200 / 0.08 = 15{,}000$ GBP per QALY and $\mathrm{NMB} = 0.08 \times 20{,}000 - 1{,}200 = 400$ GBP. The decision reverses once the learning curve has been climbed.

Step 3 (switching values for later versions). Holding the incremental cost at 1,200 GBP, the QALY gain at which the decision changes is $\Delta Q^{*} = 1{,}200 / 20{,}000 = 0.06$. Holding the gain at 0.08 QALYs, the largest incremental cost consistent with adoption is $\Delta C^{*} = 0.08 \times 20{,}000 = 1{,}600$ GBP, so a later version could cost up to 400 GBP more per patient.

Step 4 (interpretation). The same technology gives opposite answers depending on when it is measured, which is Buxton's dilemma in numbers. A threshold analysis of this kind lets guidance state the performance a future version must keep, here at least 0.06 QALYs at an incremental cost of 1,200 GBP, without a full reassessment of every update.

Responses from early HTA to living HTA

The responses act at different points in the technology's life cycle. Each accepts that one assessment will not settle the question.

1. Early and iterative assessment. An HTAi working group defined early health technology assessment as an HTA that informs later development, research or investment decisions by evaluating the potential value of a conceptual or actual technology, and noted that it would highlight a fast-moving therapeutic or competitive environment and build that uncertainty into the analysis. Sculpher, Drummond and Buxton proposed economic evaluation in four stages giving progressively firmer estimates of cost-effectiveness.
2. Staged and adaptive study designs. The IDEAL framework describes five stages for surgical innovation, from idea through development, exploration and assessment to long-term study, with an IDEAL-D variant for devices. Cook describes handling a learning curve by design, for example requiring surgeons to have done a set number of procedures before taking part, or by analysis.
3. Coverage linked to evidence generation. Drummond and colleagues describe coverage with evidence development as temporary reimbursement conditional on further data collection, and recommend that device schemes log when product modifications and learning effects occur. Walker and colleagues argue that the choice between such options should weigh the value of information from further research and the costs of reversing the decision.
4. Version-aware national guidance. The NICE HealthTech programme manual (PMG48) allows evidence on predecessor versions when evidence on the current version is limited, recommends threshold analyses to find switching values for technologies likely to have rapid iteration and multiple new versions, and lets guidance state the performance needed for a technology to remain cost-effective. Its early-use assessments allow use while further evidence is generated, with listed uncertainties normally resolvable within 3 years. Under the NICE technology appraisal manual (PMG36), the managed access agreement route is for medicines only, with data collection of up to 5 years.
5. Living assessment. Living HTA plans updates from the outset at set intervals or trigger points, and Laka and colleagues recommend a living approach for digital technologies that evolve rapidly.

Boundaries with nearby ideas

The learning curve is one mechanism behind the moving target problem, and decision uncertainty is the wider idea of which it is one source. Technology obsolescence concerns a technology already superseded, while the moving target problem concerns change during and soon after the first assessment. Diffusion of innovation describes the spread that closes the window for evaluation in Buxton's law.

Leuchter, Turner and Ouyang use the phrase in a narrower sense for predictive artificial intelligence, where practice changes during model development shift the outcome being predicted and prediction-triggered interventions after deployment alter or censor it. That sense concerns a model's performance measures, while the HTA sense concerns whether evidence still matches the version in use.

Limitations and common misreadings

The term is descriptive and has no agreed measure. How much change should trigger reassessment remains a judgement for each HTA body.

1. Devices cannot be assessed in randomised trials. Cook reports that randomised trials have played a role in assessing surgical innovations, with scope for greater use, and that large surgical trials have become more common once the technology has stabilised.
2. A later version is always better. Laka and colleagues note that software changes can directly affect patient safety, so each version needs its own evidence or a judgement that earlier evidence still applies.
3. Waiting for stability is a safe default. Diffusion before evaluation may limit the methods that remain usable, and once equipoise is lost a randomised comparison may no longer recruit.
4. Only devices are moving targets. Thokala and colleagues note that one-off HTAs of any technology can go out of date as comparators and evidence change.

Sources

• Drummond M, Federici C, Reckers-Droog V, et al. Coverage with evidence development for medical devices in Europe: can practice meet theory? Health Economics. 2022;31(Suppl 1):179-194. https://doi.org/10.1002/hec.4478
• Cook JA. The challenges faced in the design, conduct and analysis of surgical randomised controlled trials. Trials. 2009;10:9. https://doi.org/10.1186/1745-6215-10-9
• Tarricone R, Boscolo PR, Armeni P. What type of clinical evidence is needed to assess medical devices? European Respiratory Review. 2016;25(141):259-265. https://doi.org/10.1183/16000617.0016-2016
• Laka M, Carter D, Merlin T. Evaluating clinical decision support software (CDSS): challenges for robust evidence generation. International Journal of Technology Assessment in Health Care. 2024;40(1):e16. https://doi.org/10.1017/S0266462324000059
• Rogers CA, Reeves BC, Byrne J, et al. Adaptation of the By-Band randomized clinical trial to By-Band-Sleeve to include a new intervention and maintain relevance of the study to practice. British Journal of Surgery. 2017;104(9):1207-1214. https://doi.org/10.1002/bjs.10562
• Thokala P, Srivastava T, Smith R, et al. Living health technology assessment: issues, challenges and opportunities. PharmacoEconomics. 2023;41(3):227-237. https://doi.org/10.1007/s40273-022-01229-4
• Mowatt G, Bower DJ, Brebner JA, Cairns JA, Grant AM, McKee L. When and how to assess fast-changing technologies: a comparative study of medical applications of four generic technologies. Health Technology Assessment. 1997;1(14). https://doi.org/10.3310/hta1140
• Grutters JPC, Bouttell J, Abrishami P, et al. Defining early health technology assessment: building consensus using Delphi technique. International Journal of Technology Assessment in Health Care. 2025;41(1):e34. https://doi.org/10.1017/S0266462325100123
• Sculpher M, Drummond M, Buxton M. The iterative use of economic evaluation as part of the process of health technology assessment. Journal of Health Services Research and Policy. 1997;2(1):26-30. https://doi.org/10.1177/135581969700200107
• Hirst A, Philippou Y, Blazeby J, et al. No surgical innovation without evaluation: evolution and further development of the IDEAL framework and recommendations. Annals of Surgery. 2019;269(2):211-220. https://doi.org/10.1097/SLA.0000000000002794
• Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, risk sharing, or patient access scheme? A framework for coverage decisions. Value in Health. 2012;15(3):570-579. https://doi.org/10.1016/j.jval.2011.12.013
• National Institute for Health and Care Excellence. NICE HealthTech programme manual (PMG48), sections 2 and 3. London: NICE; 2025, last updated 17 December 2025. https://www.nice.org.uk/process/pmg48
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36), section 6.4. London: NICE; 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Leuchter RK, Turner WB, Ouyang D. Evaluating translational AI: a two-way moving target problem. NEJM AI. 2025;2(12). https://doi.org/10.1056/aip2500705