“Okay, but will it work for MY patients?” This is one of the burning questions on every medical practitioner’s mind when they hear about new Machine Learning (ML) algorithms promising to improve their field. While ML has shown transformative potential, its real-world adoption remains slow. The key to broader implementation lies in demonstrating that these algorithms can perform consistently across diverse populations without losing their edge. Our research showcases an ML model that breaks boundaries – clinical, demographic and geographical. We have proven that our predictive algorithm for mental health crises can be successfully transferred and replicated across diverse healthcare settings, from a public healthcare provider in the UK to a private one in the US.
A mental health crisis refers to any situation in which a person’s actions, feelings, or behaviors present a risk to themselves or others, or prevent them from functioning within their community (e.g., suicide attempts, mental breakdowns, or self-harm events). In recent decades there has been a steady increase in mental health related visits to emergency departments. Most of these visits are from individuals suffering mental health crises. Accurate prediction of these crises would enable healthcare providers to manage their limited resources and intervene sooner, proactively reaching out to individuals at risk. This proactive approach can mitigate the effects of a mental health crisis or even prevent it leading to a healthier population while simultaneously reducing the burden on the healthcare system.
As with any tool used in clinical settings, it is vital that ML algorithms for mental health crisis prediction are rigorously tested across multiple settings and target populations before being adopted in practice. The success of these algorithms depends on their transferability—being trained on one dataset and effectively applied in another setting, and their replicability. Model development methods must be faithfully reproduced under new conditions. Consistent performance across these variations is crucial for successful implementations in the fields of medicine and psychiatry. The path to building such prediction algorithms presents unique challenges, including disparate data collection processes, data architectures, operational protocols, and patient demographics that vary considerably across systems.
In this study, we transferred our ML models developed on a dataset of about 17,000 patients from a National Health Service trust in England to a private hospital in Chicago, Illinois (US). The study cohort comprised about 3,000 patients hospitalized between 2018 and 2020 and the dataset contained electronic health records for those patients. The patients, aged 16 and older, had a history of mental health crises, with at least one crisis episode during the study period. We started with models that were originally trained on UK data, then adapted and retrained some of them using data from a private hospital in Chicago. We also developed new models tailored specifically to the US dataset, using the same methods that worked in the UK.
The results were remarkable — our models consistently performed well, aligning with what we observed in previous research. But what does this mean in practical terms? To put it in perspective, in our study, slightly less than 1% of patients each week, roughly 30 people out of the entire cohort, are at very high risk for a mental health crisis in the following four weeks. Our models accurately identified 7 or 8 of these patients from the top 100 patients flagged each week, ranked by riskiness score. Yearly, that adds up to over 360 mental health crises potentially mitigated or even prevented, just by implementing this system with a single healthcare provider!
The high performance achieved by our ML models makes them suitable for implementation in a number of different contexts. They can be used both in a setting where historical medical data is unavailable, by relying on a pretrained model and in a context where it’s not possible to transfer the model as is (e.g. due to contractual reasons). If both historical data and a pretrained model are available, they can be leveraged together to further fine-tune the model. Finally, our methods show that these algorithms can be used to create a fully custom solution for a particular healthcare provider, to further improve model performance, especially in those situations where clinical notes are available.
The findings from this work have significant clinical implications. Imagine a scenario where a patient is becoming more depressed, increasing their risk of suicide. The algorithm alerts the care management team, enabling them to reach out and intervene before a crisis takes place. A crisis prediction based tool can help flag such risks, thereby improving patient outcomes while simultaneously reducing healthcare costs due to better caseload management.