Alert Fatigue

Alert Fatigue, Override Rates and the Cost of Clinician Attention

Alert fatigue is what happens when clinicians receive so many electronic warnings from clinical decision support systems, most of little use for the patient in front of them, that they stop responding reliably to any alert, including the ones that matter. In health economics the alert rule behind it is analysed as a screening problem, in which every alert rule trades missed harms against false alarms, and every false alarm uses clinician time that has an opportunity cost. A 2006 review of 17 studies found drug safety alerts in computerised prescribing overridden in 49% to 96% of cases. Alert design and the accuracy formulas are on the clinical alert page.

An alert rule behaves like a screening test

An alert rule sorts prescribing events into those that trigger a warning and those that do not, so it has the same two-by-two table as a diagnostic test. Its sensitivity is the share of dangerous situations it catches, its specificity the share of safe situations it leaves alone, and its positive predictive value the share of fired alerts that are true.

The review by van der Sijs and colleagues names low specificity, low sensitivity, unclear content and unnecessary workflow disruption as error-producing conditions in alerting systems. It defines appropriate alerts as true positives, correct and current for the patient, and notes that an appropriate alert is not always perceived as useful. Two studies in the review judged 36.5% and 39% of alerts to be false positives.

The economic point is that an alert works through a second step, because the clinician has to act on it. The health gain from a rule depends on the number of true alerts multiplied by the probability that each is acted on, and alert fatigue lowers that probability for every alert in the system.

What override studies show

Three sources show how often alerts are set aside and why the raw rate misleads.

• The 2006 review. Except for serious overdose alerts, overridden in about one case in four, drug safety alerts were overridden in 49% to 96% of cases. Adverse events followed 2.3%, 2.5% and 6% of overridden alerts in the three studies that measured them, and were judged preventable in 0.8% and none of the overrides in two of them.
• Ambulatory care (Isaac and colleagues). Across 233,537 alerts from 2,872 clinicians, 9.2% of drug interaction alerts and 23.0% of allergy alerts were accepted. An interaction alert was far less likely to be accepted when the patient had already received the drug (odds ratio 0.03).
• Inpatient care (Nanji and colleagues). In a 793-bed teaching hospital, 73.3% of allergy, interaction and duplicate drug alerts were overridden. About 60% of overrides were appropriate on chart review, from 98% for duplicate drug and 96.5% for allergy alerts down to 26.4% for age-based and 2.2% for renal dose substitution alerts.

The Nanji study shows why the override rate is a poor outcome measure. Most allergy and duplicate overrides were correct decisions, so those alerts mainly consumed time, while most renal substitution overrides were judged inappropriate. The loss sits in inappropriate overrides of true alerts.

Why responses decline with volume, repetition and exposure

The causes of alert fatigue are less settled than its prevalence. The 2006 review found that in outpatient studies the most important reason given for overriding was alert fatigue caused by a poor signal-to-noise ratio, because alerts were not serious, were irrelevant or were repeated.

Ancker and colleagues studied 112 primary care clinicians over 2010 to 2013. About a quarter of drug alerts and a third of reminders were repeats for the same patient in the same year. The likelihood of accepting a reminder fell by 30% for each additional reminder per encounter and by 10% for each five percentage point rise in the share of repeats, while the amount of work showed no association. Newly deployed reminders showed no decline in response over time, so the study found no evidence of desensitisation. In a randomised study of clinical trial alerts, Embi and Leonard found that response among 178 physicians fell from about 50% by 2.7% per two-week period but was still 30% to 40% after 36 weeks.

The opportunity cost of clinician attention

Each alert takes time drawn from other patient care, so its cost is an opportunity cost. A costing counts staff time per alert fired plus the cost of building and maintaining the rules.

Fatigue adds a less visible cost, since a false alert also slightly lowers the response to every other alert, including true ones. In economic terms clinician attention is a shared resource, and whoever adds a rule bears little of the cost it imposes on the response to all the others, so an alert set can grow beyond the level that maximises health gain.

Cutting or hardening alerts has costs too. In a randomised trial of a nearly hard-stop alert against co-prescribing warfarin and trimethoprim-sulfamethoxazole, Strom and colleagues found the desired response in 57.2% of intervention alerts against 13.5% in controls, but the trial was stopped early after four patients had clinically important treatment delays.

Choosing an alert threshold as an economic decision

Most alert rules have an adjustable trigger, such as a dose limit or a severity tier, and moving it changes the mix of true and false alerts. Saiyed and colleagues found that drug-dose alerts, run silently with all vendor defaults on, would have fired on 12% of orders; system-level changes, including raising maximum dose thresholds to 125%, cut this to 5%, and drug-specific changes to 3%.

Decision curve analysis, set out by Vickers and Elkin, values a classification rule by its net benefit, defined as follows.

$$NB = \frac{TP}{n} - \frac{FP}{n} \times \frac{p_t}{1 - p_t}$$

where $TP$ and $FP$ are the counts of true and false positives, $n$ is the number assessed, and $p_t$ is the threshold probability at which acting and not acting are judged equally good.

Applied to alerts, the threshold states how many false alerts are worth one true one. At 5%, $p_t/(1-p_t) = 0.05/0.95 = 0.0526$, so up to 19 false alerts are tolerated per true alert. Here the threshold is a value judgement about false against true alerts, not the trigger setting of the rule. The formula treats every positive as acted on; alert fatigue breaks that assumption, so an evaluation of an alert threshold has to model the response rate as well as accuracy. Unlike the net benefit of cost-effectiveness analysis, this measure is in units of true positives, not money or QALYs.

Evidence on cutting low-value alerts

Most studies report process measures such as acceptance or alert volume.

• Selective interruption. Shah and colleagues made only critical and high-severity alerts interruptive in 31 practices. Of 18,115 alerts, 71% were non-interruptive, and 67% of interruptive alerts were accepted.
• Tiering by severity. Paterno and colleagues compared two hospitals using the same interaction checking service. Compliance was 29% at the tiered site against 10%, and the most severe alerts, shown as hard stops, were all accepted there against 34%.
• Governance. Van Dort and colleagues reviewed eight hospital reports, all using a multidisciplinary committee; five of the six that evaluated their changes reported fewer alerts, but strategies could not be compared.

Page and colleagues reviewed 23 studies of 32 interruptive prescribing alerts; 17 (53%) showed a significant beneficial effect on prescriber behaviour, and neither of the two studies measuring patient outcomes found a significant improvement. An economic evaluation therefore has to link response rates to adverse drug events through a model, and an interrupted time series estimates the change better than a single before-and-after comparison.

Worked example of a broad rule against a targeted rule

The numbers are illustrative, not from any study. A hospital sees 210 dangerous drug interactions a month; unaddressed, each causes an adverse drug event with probability 0.2, so 42 events occur with no alert. Acting on a true alert prevents the event. Reviewing an alert takes 18 seconds at GBP 60 an hour, or GBP 0.30, and an event avoided is valued at GBP 5,000, combining treatment costs and health loss valued at a threshold.

The broad rule fires five times as often, so fatigue is assumed to lower its response. Monetary net benefit against no alert is calculated as follows.

$$NB_r = V \times p \times a_r \times TP_r - c \times N_r$$

where $V$ is the value of an event avoided, $p$ the probability that an unaddressed interaction causes an event, $a_r$ the share of true alerts acted on, $TP_r$ the true alerts, $c$ the time cost per alert and $N_r$ the alerts fired.

$$NB_A = 5{,}000 \times 0.2 \times 0.4 \times 200 - 0.30 \times 10{,}000 = 77{,}000$$

Here rule A yields GBP 77,000 a month, since it prevents 16 events, worth GBP 80,000, at GBP 3,000 for 50 hours.

$$NB_B = 5{,}000 \times 0.2 \times 0.7 \times 180 - 0.30 \times 2{,}000 = 125{,}400$$

Here rule B yields GBP 125,400 a month, since it prevents 25.2 events, worth GBP 126,000, at GBP 600 for 10 hours. Rule B misses 30 interactions against 10 for rule A, yet leaves 16.8 events against 26. Holding rule B at 70%, rule A would win only if its acted-on share reached the level below.

$$a^{*} = \frac{125{,}400 + 3{,}000}{5{,}000 \times 0.2 \times 200} = 0.642$$

where $a^{*}$ is the break-even share. The ranking rests on the response parameter, which few studies measure, so a threshold analysis on it is the natural check.

Alarm fatigue, adverse drug events and other neighbours

Alert fatigue concerns software warnings in prescribing systems and the electronic health record. Alarm fatigue is the parallel problem with monitoring devices. Sendelbach and Funk describe sensory overload from excessive alarms and report research finding 72% to 99% of clinical alarms false.

A missed true alert usually matters through a preventable adverse drug event. The 2006 review noted that adverse drug events after overridden alerts are not always preventable, so not all harm after an override is due to fatigue. The adverse drug reaction and adverse event pages cover how such harms are defined and costed.

Misreadings that distort alert evaluations

Several errors recur in alert evaluations and models. Most come from treating a process measure as if it were the outcome.

1. Override rate as harm. A high rate can reflect correct judgements, as with allergy and duplicate alerts.
2. Fewer alerts as safer. A cut can remove true alerts; the gain depends on how far the response to the rest rises.
3. Acceptance as outcome. Changed orders do not prove fewer adverse drug events.
4. A fixed response rate. Holding the acted-on share constant across alert volumes assumes away the problem.
5. Fatigue as carelessness. Override patterns mainly reflect alert volume, repetition and design, not clinician diligence.

Sources

• Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making. 2017;17:36. https://doi.org/10.1186/s12911-017-0430-8
• Embi PJ, Leonard AC. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. Journal of the American Medical Informatics Association. 2012;19(e1):e145-e148. https://doi.org/10.1136/amiajnl-2011-000743
• Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Archives of Internal Medicine. 2009;169(3):305-311. https://doi.org/10.1001/archinternmed.2008.551
• Nanji KC, Seger DL, Slight SP, et al. Medication-related clinical decision support alert overrides in inpatients. Journal of the American Medical Informatics Association. 2018;25(5):476-481. https://doi.org/10.1093/jamia/ocx115
• Page N, Baysari MT, Westbrook JI. A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. International Journal of Medical Informatics. 2017;105:22-30. https://doi.org/10.1016/j.ijmedinf.2017.05.011
• Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. Journal of the American Medical Informatics Association. 2009;16(1):40-46. https://doi.org/10.1197/jamia.M2808
• Saiyed SM, Greco PJ, Fernandes G, Kaelber DC. Optimizing drug-dose alerts using commercial software throughout an integrated health care system. Journal of the American Medical Informatics Association. 2017;24(6):1149-1154. https://doi.org/10.1093/jamia/ocx031
• Sendelbach S, Funk M. Alarm fatigue: a patient safety concern. AACN Advanced Critical Care. 2013;24(4):378-386. https://doi.org/10.1097/NCI.0b013e3182a903f9
• Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. Journal of the American Medical Informatics Association. 2006;13(1):5-11. https://doi.org/10.1197/jamia.M1868
• Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Archives of Internal Medicine. 2010;170(17):1578-1583. https://doi.org/10.1001/archinternmed.2010.324
• van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. Journal of the American Medical Informatics Association. 2006;13(2):138-147. https://doi.org/10.1197/jamia.M1809
• Van Dort BA, Zheng WY, Sundar V, Baysari MT. Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals. Journal of the American Medical Informatics Association. 2021;28(1):177-183. https://doi.org/10.1093/jamia/ocaa279
• Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. https://doi.org/10.1177/0272989X06295361