June 05, 2019
By Julie Gould
Jenny Lo-Ciganic, assistant professor at the Department of Pharmaceutical Outcomes and Policy in the College of Pharmacy at the University of Florida, explains why a machine learning algorithm is beneficial for risk prediction and stratification of opioid overdoses.
First, can you tell us a little bit about yourself and your research interests?
I'm Jenny Lo‑Ciganic, and I'm an assistant professor at the Department of Pharmaceutical Outcomes and Policy in the College of Pharmacy at the University of Florida.
I'm a pharmacoepidemiologist and my main research interests focus on improving drug safety, medication adherence and quality and the value of prescribing, and preventing prescription drug abuse, especially among vulnerable or minority populations like the Medicaid or Medicare population.
Right now, my main research work focus on applying innovative approaches, such as machine learning, like you saw in our study, to prevent opioid overdose.
What approaches are currently used to identify patients who are high‑risk for opioid overdose and how often are patients identified that are not truly high‑risk?
This is a great question. In the article we recently published, we applied five machine learning approaches and identified which approach has the best performance to predict patients who are at high risk of opioid overdose.
Among five approaches, deep learning or deep neural network and gradient boosting machine (or TreeNet), had the best performance to predict opioid overdose. Opioid overdose, is a really important public health issue and outcome, but it's a very rare outcome. Among Medicare beneficiaries, the occurrence of overdose is about 6 out of the 10,000. It is challenging to predict rare outcomes because false positives are likely to happen. In other words, if you want to capture 90% of patients with actual overdoses using a traditional statistical approach, then the number needed to evaluate in order to identify one overdose would be more than 1,700 patients. In contrast, the number needed to evaluate is approximately 500 using our deep neural network algorithm, which reducing more than 3 times of the effort. Another uniqueness of our work is that we conduct the risk stratification of patients based on our prediction scores. We grouped patients into low, medium, and high risk groups. If we targeted patients in the high‑risk group, we only need to screen 174 patients in order to identify one true overdose. Our work is valuable and promising to more accurately predict and identify opioid overdose.
Why would a machine learning algorithm be beneficial for the prediction of opioid overdoses?
Again, this is a great question. Machine learning algorithm is not always better than traditional statistical methods. Machine learning may be more valuable for more complicated scenarios. Using prescription drug abuse and opioid overdose in our work as an example, it is a multifacet issue. Complex interactions and hidden relationship may exists among predictors (e.g., prescription opioid use and mental health disorders, rural regional areas etc.) and among predictors and outcomes. Machine learning approaches have advantages of taking all these complex interactions and hidden relationship into account to improve prediction.
What is needed to successfully integrate a machine learning algorithm into everyday practice?
That's an interesting question, because that is what we want to do. When we developed the prediction algorithms, our ultimate goal is to incorporate our prediction algorithm into clinical practice or health care systems in order to improve the opioid safety. We will need to tailor our prediction tools into real-world practice settings, compatible to different platforms used by different health care systems, and make it user friendly. This requires some additional work, but it’s not impossible because Amazon and Netflix are great examples using their data to improve their services in our daily life.
In your opinion, how do you think the opioid epidemic will change following implementation of a machine learning algorithm?
The opioid epidemic is an important but complicated public health issue in the US. Our machine learning work focused on identifying those prescribed with at least 1 opioid prescription at high risk of overdose. But some individuals could get opioids on the streets and do not show up in the health care systems. Machine learning may be valuable to identify useful information (e.g., potential drug being abused) by mining social media data. From our work, machine learning approach may be promising and play an role to curb the opioid epidemic in the US.
Is there anything else you'd like to add?
Prior studies in this field were limited to accurately predict overdose in the real-world settings. Previous studies only used one or a few parameters (e.g., C-statistic) to evaluate prediction performance, but this could be misleading for non-rare outcomes such as overdose. To our knowledge, our work is the first one to thoroughly evaluate different aspects of prediction performance, and compare with the existing measures. Our risk stratification approach may better guide the payers or healthcare systems to allocate their resources, especially three‑quarters of the patients had no or minimal overdose risk. Depends on the type of interventions and resources, payers and health care systems can focus on those identified as high-risk and/or medium high of overdose.
Lo-Ciganic J, Huang JL, Zhang HH, et al. Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions [published online March 22, 2019]. JAMA Netw Open. 2019;2(3):e190968. doi:10.1001/jamanetworkopen.2019.0968