To examine the main effect of message rate on alert fatigue, we analyzed the effects of message rate on the probability of correctly recalling content of study messages among those who remembered receiving messages.
That unreliability is likely to be behind the initial high rate of alerts with the regression-based methods.
There was much variation in the rate of alerts based on the exceptions generated through their DUR software.
The estimated rate of alerts provides the estimated number of alerts (workups needed) per unit of time per number of inpatients monitored.
Their clinical outcomes assessment is based on the rate of exceptions, and not the rate of alerts, sent to the physician.
We recommend using the PPV, the NNE and/or the estimated rate of alerts combined with sensitivity to evaluate each of the plausible score cutoff values.
There was no difference in the perceived acceptance rate of alerts based on their type (i.e., cost, safety, health maintenance).
These metrics include the PPV, the number needed to evaluate (NNE), also known as the workup to detection (WTD) ratio, and the estimated rate of alerts.
We think that, given the rate of alerts we found, an SR expert using a live alert system could quickly review the alerts, identify from this set the important publications, and use this information in planning and scheduling review updates.
While the number of update-motivating publications annotated for each topic varies quite a bit, the overall rate of alerts that need to be monitored is small, with most of the motivating publications recognized and leading to a correct alert.
We suggest that pre-implementation evaluation of EWSs should include at least two metrics: sensitivity; and either the positive predictive value, number needed to evaluate, or estimated rate of alerts.
