New Study Validates EHMRG’s Accuracy and Fairness Across Varying Heart Failure Patients

A person walks into an emergency department with heart failure. Within hours, a doctor must make a consequential call to send them home or admit them to the hospital. Both options run their own risks – admitting the patient can lead to an unnecessary hospitalization and sending them home could cause a preventable death. For over a decade, an Ontario-made tool has helped guide that decision, and now a new study aims to answer a critical question: does it work as well for everyone? 

The tool is the Emergency Heart Failure Mortality Risk Grade (EHMRG), developed by Dr. Douglas Lee and colleagues, using a series of models derived in 2004-2007. EHMRG estimates a patient’s risk of dying within 7 or 30 days of an emergency department visit for acute heart failure (AHF) and has since become a trusted part of emergency care in Ontario. However, tools like EHMRG are built using patient data available at the time they’re developed, and health systems, populations, and care practices are ever-changing. That conundrum is what Dr. Lee, Karem Abdul-Samad and their collaborators aim to address in their recent publication in ESC Heart Failure – do the EHMRG models work accurately today and are they consistent across varying social and demographic backgrounds.  

A Multi-Institutional Effort, Rooted in Ontario

Led by Dr. Douglas Lee, with collaborators including Karem Abdul-Samad, Dr. Husam Abdel-Qadir, Dr. Slava Epelman, and Dr. Heather Ross, the study team drew on data at ICES, the Peter Munk Cardiac Centre at University Health Network (UHN), and the University of Toronto (U of T). The research was funded by the Ted Rogers Centre for Heart Research (TRCHR). 

The team examined a group of 7,537 patients who presented with AHF at 10 emergency departments across Ontario between 2017 and 2019. They linked patient records to provincial and national health databases and indices, allowing them to test whether the model’s predictions held up consistently across differences in sex, income, marital status, and marginalization. 

What the Data Showed

The results were reassuring, with one important caveat. EHMRG continued to accurately distinguish higher-risk from lower-risk patients, performing as well as it had in earlier studies, though it did mildly overestimate the 30-day death risk in the higher risk groups – a bias that would tend to push recommendations toward hospitalization.  

Its predictions also held steady across the tested socio-demographic subgroups, meaning the tool doesn’t appear to systematically disadvantage patients based on these factors. The one exception was patients living in long-term care or retirement homes, where the model’s ability to distinguish risk was somewhat weaker. Researchers suggest this may be a reflection of how uniform that population already is in terms of age and frailty, rather than any flaw in the model itself. A smaller dip in performance was also seen for patients with limited English fluency. 

For Karem Abdul-Samad, the results point to a larger responsibility researchers carry, “Clinical prediction tools have the potential to transform heart failure care, but that potential depends on their ability to perform consistently across diverse patient populations. Ensuring these tools are fair, reliable, and trustworthy is essential to delivering equitable care.”

This is believed to be the first study to examine the sociodemographic consistency of an AHF risk model – not just whether it works, but whether it works equally well for the people who rely on it. 

Looking Ahead 

The findings support EHMRG’s continued use in Ontario emergency departments, while pointing to where possible refinement may be warranted, particularly for patients in long-term care. More broadly, the study offers a template for how prediction tools of all kinds, including newer AI-driven ones, should be tested: not only for accuracy, but for fairness across the populations they’re meant to serve. 

For Dr. Lee, the study’s main takeaway was one of continued confidence in the tool, “When you build an algorithm, you want to know that it still works even after so many advances in heart failure research. The treatments are different than they were 15 years ago when EHMRG was first developed. This study helps reassure us that using the model now, without any adjustment, is still going to be safe. Overall, we want a model that works for everyone.”