Clinical Alert

Clinical Alert Rules, Accuracy and the Economics of Interrupting Care

A clinical alert is a message that clinical software raises when a patient's recorded data meet a programmed rule, so that a prescription, test order or treatment decision can change before harm occurs. For health economists it is an intervention like any other. It costs money to build and maintain, it uses clinician time whenever it fires, and it improves outcomes only when it is right and someone acts on it. Override rates and declining responses to repeated warnings are discussed under alert fatigue.

How an alert is built from data, a rule and a message

An alert is the visible end of a clinical decision support rule. Sutton and colleagues describe knowledge-based decision support as IF-THEN rules, with the system retrieving patient data to evaluate each rule and producing an action or output; non-knowledge-based systems draw the decision from machine learning or statistical pattern recognition instead. The output reaches the user through a communication mechanism such as the front end of the electronic health record, and these systems are often integrated with records and computerised provider order entry.

Computerised alerts and reminders are one of several forms decision support takes, alongside order sets, computerised guidelines, documentation templates and patient data reports. Order entry systems carry drug safety software that checks dosing, duplicate therapy and drug-drug interactions, and Kwan and colleagues give pop-up alerts about serious drug allergies as a typical example. Sutton describes alerts for abnormal laboratory results as simple and ubiquitous. Predictive alerts fire when a model's risk score crosses a threshold, as in sepsis warning systems.

Delivery through passive, interruptive and hard-stop alerts

Decision support is classified by its timing and by whether delivery is active or passive. Kwan and colleagues define an interruptive system as one that appears on screen and stops the user from performing any further tasks until it is acknowledged or a button is pressed; a passive alert presents information without stopping work. Sutton and colleagues advise limiting disruptive alerts to life-threatening or consequential contraindications, such as serious allergies.

Powers and colleagues define hard-stop alerts as those in which the user is either prevented from taking an action altogether or allowed to proceed only with the external override of a third party. Of the 32 studies, 79% of those measuring health outcomes and 88% of those measuring process outcomes showed improvement. Eleven studies reported unintended consequences, including avoidance of the hard-stopped workflow, increased alert frequency and delay to care, and hard stops outperformed soft stops in 3 of the 4 studies comparing them.

Design features linked to effect

Systematic reviews have tried to identify which features predict success. Kawamoto and colleagues reviewed 70 randomised trials and found that decision support significantly improved clinical practice in 68%. Four features independently predicted improvement. These were automatic provision as part of clinician workflow, recommendations rather than just assessments, provision at the time and location of decision making, and computer-based delivery. Of 32 systems with all four, 30 (94%) improved practice.

Kwan and colleagues found that systems requiring acknowledgement and a documented reason for not following the recommendation achieved improvements 4.8 percentage points larger (95% confidence interval 0.1 to 9.6), and systems letting the clinician carry out the recommended action directly achieved 4.4 points more (0.9 to 7.9). Interruptive delivery was associated with a 3.2-point difference whose interval included zero (-0.4 to 6.7).

Accuracy of an alert rule as a screening test

An alert rule sorts patients or orders into those that trigger a warning and those that do not. Its accuracy is therefore summarised with the two-by-two table used for screening tests, which Trevethan sets out. Written as proportions, the three measures used most for alerts are set out below.

$$Se = \frac{TP}{TP + FN}, \quad Sp = \frac{TN}{TN + FP}, \quad PPV = \frac{TP}{TP + FP}$$

where $Se$ is sensitivity, $Sp$ is specificity, $PPV$ is positive predictive value, $TP$ is the number of true alerts, $FP$ false alerts, $FN$ target situations that produced no alert and $TN$ non-target situations correctly left alone. Writing the cells in terms of prevalence gives the following.

$$PPV = \frac{Se \times p}{Se \times p + (1 - Sp) \times (1 - p)}$$

where $p$ is the prevalence of the target condition among the patients or orders screened.

Trevethan stresses that these values are not fixed attributes of a test, because they depend on its stringency and on prevalence. A rule screening thousands of orders for a rare hazard can have high specificity and still a low PPV, so most of what clinicians see is false. Sensitivity cannot be estimated from fired alerts alone, because missed cases sit among patients who never triggered the rule.

Predictive alerts add the choice of a cut-off on a risk score. Wong and colleagues externally validated the proprietary Epic Sepsis Model in 38,455 hospitalisations, with sepsis in 7%. The area under the ROC curve was 0.63, substantially worse than its developer reported. At a score of 6 or higher it alerted for 6,971 patients (18%), yet missed 1,709 of the 2,552 with sepsis (67%). Parsons and colleagues note that cut-off selection usually optimises sensitivity and specificity while overlooking the consequences of classification, and propose choosing it by expected net monetary benefit.

Trials show process change but sparse outcome evidence

Most alert trials measure whether clinicians did the recommended thing, not whether patients did better. Bright and colleagues reviewed 148 randomised trials of decision support. Of these, 86% assessed process measures, 20% clinical outcomes and 15% costs, and evidence for clinical, economic, workload and efficiency outcomes remained sparse.

Kwan and colleagues pooled 122 trials with 1,203,053 patients. Decision support increased the proportion receiving the desired care by 5.8 percentage points (95% confidence interval 4.0 to 7.6), with substantial heterogeneity (I² = 76%). In 30 trials reporting clinical endpoints, the median improvement in patients reaching guideline-based targets was 0.3 points.

Outcome effects are possible when the alert targets an action with proven benefit. Kucher and colleagues randomised 2,506 hospitalised patients at risk of deep-vein thrombosis and not receiving prophylaxis; for 1,255 the physician received an alert that had to be acknowledged. Pharmacological prophylaxis was 23.6% with the alert against 13.0% without, and deep-vein thrombosis or pulmonary embolism at 90 days was 4.9% against 8.2% (hazard ratio 0.59, 95% confidence interval 0.43 to 0.81).

Economic evaluation of a clinical alert

An alert is evaluated against a comparator, usually no alert or a different design such as a passive display, a narrower rule or another cut-off. Costs include development and integration, maintenance as medicines and guidance change, clinician time per alert, the downstream actions prompted and any costs avoided. Time spent on false alerts is an opportunity cost drawn from other patient care.

Lewkowicz and colleagues reviewed 27 studies of the economic impact of decision support in electronic records. Twenty-two reported reduced expenditure, especially in unnecessary laboratory testing, duplicate orders, transfusion and antibiotic prescribing, but most did not consider implementation cost, and the four model-based economic evaluations reported mixed results. Because trials mostly report process measures, the link to adverse events avoided and QALYs usually has to be modelled, for example with a decision tree, before feeding a cost-effectiveness analysis or budget impact analysis.

Worked example of cost per complication avoided (illustrative)

All numbers are illustrative. A hospital considers an alert identifying admitted patients at risk of a preventable complication who are not receiving preventive treatment.

1. Screening. Of 20,000 admissions a year, 5% (1,000) are at risk. With sensitivity 0.80 and specificity 0.90 the rule fires 800 true alerts and misses 200; it flags 10% of the 19,000 not at risk, giving 1,900 false alerts and 2,700 in total.
2. Positive predictive value. 800 / 2,700 = 0.296. The prevalence formula agrees, since the numerator is 0.80 x 0.05 = 0.04, the denominator 0.04 + 0.10 x 0.95 = 0.135, and 0.04 / 0.135 = 0.296.
3. Response and effect. Clinicians start treatment after 60% of true alerts, so 480 patients are treated who otherwise would not be. Treatment lowers the risk from 10% to 6%, so 480 x 0.04 = 19.2 complications are avoided.
4. Costs. Build and maintenance cost GBP 30,000 a year. Review costs GBP 1.50 per alert, so GBP 4,050, of which GBP 2,850 goes on false alerts. Treatment costs GBP 40, so GBP 19,200. Each complication avoided saves GBP 2,000, so savings are GBP 38,400.
5. Net cost per complication avoided.

$$\frac{\Delta C}{E} = \frac{F + c_r N + c_t A - s E}{E} = \frac{30000 + 4050 + 19200 - 38400}{19.2} = \frac{14850}{19.2} = 773.4$$

where $\Delta C$ is the net annual cost, $E$ the complications avoided, $F$ the fixed annual cost, $c_r$ the review cost per alert, $N$ the alerts fired, $c_t$ the treatment cost per patient, $A$ the patients treated and $s$ the saving per complication avoided.

1. Sensitivity check. If clinicians act on 40% of true alerts, 320 are treated and 12.8 complications avoided; costs of GBP 46,850 less savings of GBP 25,600 leave GBP 21,250, or GBP 1,660 per complication avoided.

The rule costs about GBP 773 per complication avoided and fires about 141 alerts (2,700 / 19.2) for each one. Whether that is good value depends on the health lost per complication, expressed in QALYs and compared with a threshold. The ratio more than doubles when the response falls to 40%, because fixed and review costs stay while the benefit shrinks. The example assumes nobody is treated after a false alert.

Clinical alerts beside reminders, alarms and other uses of the word

A clinical alert is one component of clinical decision support. Reminders prompt a recommended action such as a vaccination, whereas safety alerts warn of a risk in an action under way; the accuracy and costing logic applies to both. Alarms from bedside monitors raise a parallel false-alarm problem but come from equipment, not rules applied to the record. An evidence alert, a notice that new research may affect a guideline, is a different idea.

Limitations and common misreadings

Most errors treat what the software records as the effect on patients. Six recur.

1. Alerts fired as benefit. Alert counts measure burden; benefit comes only from true alerts that change care.
2. Changed orders as better outcomes. Process gains in trials are larger than effects on clinical endpoints.
3. Sensitivity from fired alerts. Reviewing fired alerts yields PPV only; missed cases need a search of patients who did not trigger the rule.
4. Accuracy as portable. The Epic Sepsis Model performed far worse externally than reported, so performance needs checking where the alert is used.
5. Implementation cost left out. Omitting build and maintenance costs flatters the result.
6. A hard stop as the safest choice. Hard stops can delay appropriate care and prompt workarounds.

Sources

• Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a systematic review. Annals of Internal Medicine. 2012;157(1):29-43. https://doi.org/10.7326/0003-4819-157-1-201207030-00450
• Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. https://doi.org/10.1136/bmj.38398.500764.8F
• Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. New England Journal of Medicine. 2005;352(10):969-977. https://doi.org/10.1056/NEJMoa041533
• Kwan JL, Lo L, Ferguson J, et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ. 2020;370:m3216. https://doi.org/10.1136/bmj.m3216
• Lewkowicz D, Wohlbrandt A, Boettinger E. Economic impact of clinical decision support interventions based on electronic health records. BMC Health Services Research. 2020;20:871. https://doi.org/10.1186/s12913-020-05688-3
• Parsons R, Blythe R, Cramb SM, McPhail SM. Integrating economic considerations into cutpoint selection may help align clinical decision support toward value-based healthcare. Journal of the American Medical Informatics Association. 2023;30(6):1103-1113. https://doi.org/10.1093/jamia/ocad042
• Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. Journal of the American Medical Informatics Association. 2018;25(11):1556-1566. https://doi.org/10.1093/jamia/ocy112
• Sutton RT, Pincock D, Baumgart DC, et al. An overview of clinical decision support systems: benefits, risks, and strategies for success. npj Digital Medicine. 2020;3:17. https://doi.org/10.1038/s41746-020-0221-y
• Trevethan R. Sensitivity, specificity, and predictive values: foundations, pliabilities, and pitfalls in research and practice. Frontiers in Public Health. 2017;5:307. https://doi.org/10.3389/fpubh.2017.00307
• Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine. 2021;181(8):1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626