Abstract
Background: Sickle cell disease (SCD) is the most prevalent inherited blood disorder worldwide and associated with increased morbidity and mortality. Episodes of acute and severe pain known as vaso-occlusive crises (VOC) are frequent and the most common cause for hospital admission. Recently, mobile health (mHealth) has developed promising, patient-friendly, minimally invasive tools to monitor patients remotely. Our previous work has leveraged data from mHealth apps and wearable devices to evaluate several machine learning (ML) models which were able to predict pain scores in patients with SCD admitted for VOC to the SCD Day Hospital with an accuracy of 86%. We now extend our evaluation of using an mHealth app and wearable for patients admitted for VOC and for 30 days post discharge. We aim to evaluate the feasibility of extended monitoring and refine development of ML models to predict pain scores.
Methods: Patients with SCD aged 18 and above who were admitted for a VOC to the SCD Day Hospital or to Duke University Hospital were eligible for this study. Patients were followed for 30 days after discharge. Following informed consent, patients were provided: 1) the mobile app (Nanpar) on their own Apple device or provided with an Apple smartphone if needed, and 2) an Apple Watch if patients did not have their own. Patients were instructed to report their pain and other symptoms to nurses while inpatient as per standard of care monitoring and at least once daily in the Nanpar app. Patients were also asked to continuously wear the Apple Watch, removing only to charge. Pain scores were recorded on a visual analog scale ranging from 0 to 10, with 0 accounting for no pain, and 10 being most intense pain. Physiological data collected by the Apple Watch included measurements of heart rate, heart rate variability, oxygen saturation and activity (step count). These data were associated with self-reported pain scores collected via the app and the electronic health records (EHR) during Day Hospital or hospitalization to fit the machine learning Random Forest classification model. The association was done by considering the nearest neighbor of the time stamps for each pain record for each feature. The performance of the model is evaluated by the following metrics: accuracy, F1-score, root mean squared error (RMSE) and area under the ROC curve (AUC).
Results: Nineteen patients were included in this study from April through June 2022. The median age at time of inclusion was 30 years (IQR: 22-34). The majority of the patients had genotype HbSS (68%). All patients were Black or African American. The median time of follow-up was 31 days (IQR 30-33). Eleven patients (55%) were admitted to the SCD Day Hospital, while the remainder was admitted to the Duke University Hospital. This preliminary dataset consisted of 1480 data points. After micro-averaging due to the highly imbalanced dataset, the model resulted in the following metrics: micro-averaged accuracy: 0.89, micro-averaged F1-score: 0.49, RMSE: 1.64, AUC: 0.83. There was no correlation between any of the data elements recorded by the Apple Watch. Feature importance revealed step count as the most important feature in the predictive model. Our random forest model was able to accurately predict the pain scores 5-8 not only for patients who were admitted to the hospital, but also for patients after discharge from the hospital.
Discussion: Our model was able to predict pain using only data from the consumer wearable Apple Watch with 89% accuracy and a RMSE of 1.64. There was a class imbalance, however, consistent with our previously developed models with patients who were only monitored while at SCD Day Hospital. During this study, we included more patients and a higher variability of pain scores, which certainly improved our prediction model to accurately predict pain for all the pain scores. Interestingly, we found step count as the most important feature in our prediction model. This finding provides emphasis on further studying the relationship between pain and activity.
Conclusion: The consumer wearable Apple Watch was a feasible method to collect physiologic data and allowed us to accurately predict pain in patients with SCD through machine learning techniques. Future efforts will focus on larger numbers of patients and monitoring patients for longer periods of time to provide more accuracy of pain scores.
Disclosures
Stojancic:North Carolina State University: Current Employment, Other: Student; Duke University Hospital: Current Employment. Shah:CSL Behring: Consultancy; Bluebird Bio: Consultancy; Novartis: Research Funding, Speakers Bureau; Alexion: Speakers Bureau; Global Blood Therapeutics: Consultancy, Research Funding, Speakers Bureau.
