Factors Driving Provider Adoption Of The TREWS Machine Learning-Based Early Warning System And Its Effects On Sepsis Treatment Timing

07-21-22 – Nature Medicine – Factors Driving Provider Adoption Of The TREWS Machine Learning-Based Early Warning System And Its Effects On Sepsis Treatment Timing

Nature Medicine published three peer-reviewed articles showing how, for the first time, AI has been shown to reduce mortality in a hospital setting using Bayesian Health’s Adaptive AI solution.

The first of three studies, centered on characterizing system accuracy, provider adoption, and impact of adoption of treatment timing. It was conducted over a two-year period at five hospitals from both academic and community-based hospital settings. The study looked at front-line usage by 2,000+ providers.

LINKS

Read the Study

Read the Press Release

NOTE – At the core of the research was an AI system referenced as Targeted Real-time Early Warning System (TREWS). Initially developed at Johns Hopkins, Bayesian Health has commercialized and advanced the methodology, integrating it into a broader adaptive AI platform that enables integration, monitoring, and tuning to account for real-world variations in populations and workflows and scaling to multiple condition areas. Bayesian led and managed the deployment across all five emergency departments and hospitals in the study.

 

 

Bias Checklist – JAMIA: Exploring Deployment of AI to Improve Patient Outcomes

As a user exploring deployment of healthcare AI, a key challenge has been the lack of a comprehensive assessment for measuring bias within your solution. Further complicating matters; most scientific papers focus on one or two aspects of bias while meta-reviews or industry tool-kits simply surveil or summarize existing quantitative measures.

In a first-of-its-kind research paper by JAMIA (a leading informatics journal) our very own Suchi Saria brings together a team of experts in health disparities, health services, machine learning and informatics, providing a rare, end-to-end perspective of bias.

If you are exploring deployment of AI to improve patient outcomes, lower readmissions and decrease alert fatigue, this checklist provides a solid foundation for identifying and overcoming sources of bias.

Download the full checklist here to better identify and understand bias in healthcare AI solutions.

 

Delivering 1.85hr Earlier Treatment for Sepsis at Johns Hopkins

Bayesian Health’s platform drove faster treatment for sepsis, with high provider adoption.

A large, five site study analyzing use and practice impact over two years for Bayesian Health’s sepsis module showed high sensitivity (80%+) with high precision (1 in 3 alerts were provider confirmed). Sepsis is a needle in a haystack problem, so it’s hard to achieve high precision at high sensitivity, and especially harder where you’re also optimizing for earlier detection times. But all three of these things are critical to cracking the code on earlier sepsis recognition and prevention.

Bayesian’s high quality, timely clinical signals resulted in 1.85 hour faster life-saving patient treatment driven by high provider adoption (89%).

With research showing the average adoption of clinical decision support tools to be in the low double-digits, our high adoption is exciting news, and shows physician confidence and trust in the Bayesian insights, leading to proactive, earlier care. This is great news for patient outcomes, where for sepsis every hour of delayed treatment directly impacts mortality rate.

Read the full prospective outcome study on Bayesian Health’s platform targeting sepsis, “Evaluating Adoption, Impact, and Factors Driving Adoption for TREWS, a Machine Learning-Based Sepsis Alerting System,” here.

How to safely deploy predictive tools in the healthcare setting 

Though predictive tools are widely used in the healthcare setting, there are no official rubrics or guidelines for evaluation or requirements for consistent regulatory oversight. But, what can often happen during deployment is something called “dataset shift.” Dataset shift occurs when a tool “underperforms because of a mismatch between the data set with which it was developed and the data on which it is deployed.” When dataset shift happens, the model can be considered “unsafe.”

As a result, it’s up to the predictive tool’s developers and implementers to monitor and evaluate the solution in order to ensure model safety, sensitivity and precision. Additionally, the tool’s users—clinicians—are a critical component of evaluation, as they are the individuals who can detect or see any dataset shift issues first.

Though evaluating for dataset shift can prove challenging, as it can happen for a variety of reasons (i.e. there can be population changes, physician behavior changes, etc.). It is essential to evaluate any data-driven predictive solution for dataset shift, in order to have safe and robust predictive tools at the bedside.

Bayesian Health’s CEO, along with other experts in the field, just published a letter in the New England Journal of Medicine (NEJM) outlining the causes of dataset shift and how to successfully recognize and mitigate dataset shift through both end-users and AI governance teams. 

Bayesian Health’s predictive AI platform integrates the guidelines and rubric laid out in the NEJM, to monitor for dataset shift and ensure model safety. We believe this type of evaluation is an essential part of any AI tool, and this type of ongoing monitoring is how Bayesian’s platform is able to be deployed in a variety of populations/demographics, and adapt to differing or new physician and care team behaviors and practice patterns.

You can read the full NEJM article here, and you can learn more about the AI/machine learning strategies we use to overcome common hurdles faced by many in the field, here.

Bayesian’s platform accurately anticipates acute respiratory failure in COVID-19 patients

Development and validation of ARC, a model for Anticipating Acute Respiratory Failure in COVID-19 patients

Many hospitalized COVID patients experience acute respiratory failure (ARF), but it has been difficult to predict which patients will need advanced respiratory support (e.g., mechanical ventilation) and which will not. Many predictive models have been reported for COVID patients, but most predict only the most severe outcomes (ICU admission, death, etc.) and only make a single prediction at the time a patient is admitted to the hospital. In an article published this week in Critical Care Explorations, we describe ARC (Anticipating Respiratory failure in COVID), a model that continuously monitors patients throughout a hospital encounter and identifies patients at risk of experiencing ARF. We collaborated with researchers at the University of Washington to validate the model on data collected from eight hospitals across two geographically distinct regions. At 75% specificity, the model achieved 77% sensitivity and predicted ARF at a median time of 32 hours prior to onset.

In addition to ARC, Bayesian has also trained models that predict ARF in non-COVID patients. Early identification of these patients would facilitate proactive care, leading to better patient outcomes and lower costs for health systems. Even prior to COVID, respiratory failure accounted for ~30% of unplanned transfers to intensive care units.

 

Read the full paper here.

Strategies Bayesian applies to Optimize Performance in its Predictive Models

Predicting events using clinical data is challenging because individual data points are often missing and/or noisy. For a given patient, input data are often collected at different times and with varying frequency. For any given measurement, the frequency can also vary between patients, often based on a clinician’s suspicion of potential complications. Learning strategies that do not account for this variation will be much less accurate. This study published in IEEE Transactions on Pattern Analysis and Machine Intelligence describes two solutions to this issue. The first is applying smart interpolation techniques that account for missing data based on patient context. The second is introducing a “Wait and Watch” policy that trades off delays in making predictions against the cost of false alerts. Compared to the standard approaches (e.g., logistic regression), these techniques achieve 3-4X higher sensitivity at comparable positive predictive value.

Bayesian Health applies these and many other leading-edge learning techniques to optimize performance and clinical actionability in our predictive models.

Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction

Read the full paper here.

How machine learning can identify sepsis early? A targeted real-time early warning score (TREWScore) for septic shock

A targeted real-time early warning score (TREWScore) for septic shock

Sepsis is a leading cause of death, contributing to 1 in every 2 or 3 hospital deaths. Early identification and treatment has a dramatic impact on morbidity and mortality. In a cover article published in Science Translational Medicine, Dr. Saria and colleagues showed that routinely available vital signs and lab results could be used to predict which patients would experience septic shock. TREWScore (Targeted Real-Time Early Warning Score) was more accurate than a routine screening protocol and another score used clinically for predicting septic shock (MEWS). TREWScore identified patients 28.2 hours (median) before onset with ⅔ of cases identified before any sepsis-related organ dysfunction.

This was the first study to show that applying machine learning techniques to clinical data could be used to proactively identify patients at risk of sepsis. Since then, over 200 related papers have been published.

Read the full research paper here.

Understanding how Bayesian’s machine learning models achieve high sensitivity and precision

Comparison of Automated Sepsis Identification Methods and Electronic Health Record–based Sepsis Phenotyping: Improving Case Identification Accuracy by Accounting for Confounding Comorbid Conditions

In order to train high-quality machine learning models, it is essential to be able to determine which samples in the training dataset did (and did not) experience the targeted outcome of interest. Accurate identification of positives and negatives (referred to as “phenotyping”) is particularly challenging in healthcare because there are many confounding data points that can require clinician judgement to interpret, and clinical review is impractical for large-scale datasets. In sepsis, a common method to identify sepsis cases in retrospective datasets is the presence of ICD billing codes. Although billing codes have high precision (low false positive rate), they suffer from low sensitivity (miss many positive cases) and cannot be used to determine sepsis onset time. This study published in Critical Care Explorations describes a new method for sepsis phenotyping that outperforms other automated tools because it accounts for comorbidities that confound other automated tools.

Rigorous definition of targets is one of many strategies Bayesian Health uses to develop best-in-class machine learning models that achieve high sensitivity (80-95%) with 300%-700%+ better precision than many other solutions.

Read the full research paper here.

Let’s Talk

CONTACT US

General Email

[email protected]

Careers

[email protected]

Stay up to date on the latest in machine learning and healthcare