Abstract
Background: Prognostic judgment plays a critical role in the practice of oncology. These predictions help providers determine appropriate treatment options and advise patients about the future. There is, however, little known about how accurate oncology providers are in making such predictions. In this study, we aimed to determine the accuracy of oncology providers in predicting serious medical events in adults admitted to oncology services at the University of Washington.
Patients and Methods: Surveys were collected from oncology providers, including physicians, advanced practice providers (APPs), and nurses for patients within 24 hours of being admitted to our inpatient oncology services. Surveys assessed provider characteristics and their predictions about 30-day outcomes (death, ICU transfer, and transition to comfort care). Estimates of these outcomes were compared with actual outcomes by evaluating area under the receiver operating characteristic curve (AUC). For patients with newly diagnosed AML, the treatment-related mortality (TRM) score was calculated as previously described [Walter et al, J Clin Oncol 2011]. Analyses accounted for clustering within patient encounter, as more than one provider may have answered a survey (e.g., a physician and a nurse).
Results: Between May 2014 and April 2015, 727 surveys were collected representing 490 individual patient encounters. Among all surveys, median patient age was 54 years (range: 20-94). The primary diagnosis was leukemia in 49%, lymphoma in 22%, solid tumor in 25%, and benign hematologic problem in 4%. Respondents were 29% internal medicine house-staff, 26% oncology attendings or fellows, 21% APPs, 19% nurses, and 4% other physicians. Overall, providers overestimated the likelihood of ICU transfer (11% [9-13%] actual vs. 19% [17-20%] predicted, p<0.0001), were realistic regarding transition to comfort care (13% [11-15%] actual vs. 15% [13-16%] predicted, p=0.22), and minimally overestimated the likelihood of death (10% [8-12%] actual vs. 13% [11-14%] predicted, p=0.027). The AUCs for provider predictions of ICU transfer, transition to comfort care, and mortality were 0.60 (0.53-0.67), 0.80 (0.74-0.84), and 0.76 (0.70-0.82) respectively. We stratified responses based on provider and patient characteristics and describe our results in Table 1. Notably, providers appeared to prognosticate mortality more accurately for patients admitted for planned chemotherapy than for unplanned events (AUC 0.79 [0.71-0.86] vs 0.68 [0.59-0.78]), though the difference was not statistically significant. In the subset of patients with newly diagnosed AML receiving curative-intent induction chemotherapy, the 30-day mortality rate was 10%, and the AUC for provider prediction of mortality was 0.83 (0.71-0.95), which was very similar to the accuracy of predictions based on calculated TRM score for these patients with AUC 0.82 (0.68-0.95). Respondents reported basing their estimates on the TRM score in 5 of 63 cases.
Respondents reported making predictions based on many patient factors, but the three most commonly cited were the underlying malignancy, presence of comorbidities, and performance status; these reasons did not differ between respondents who were more versus less accurate in their ability to predict mortality.
Conclusions: The SPAR study suggests that providers predict early mortality with fair accuracy, ICU transfer with poor accuracy, and transition to comfort care with good accuracy. Inaccuracies in prediction of mortality and ICU transfer veer in the direction of pessimism. While sample size limits detailed subgroup analysis, our results do not suggest large differences in prediction ability by provider type, years of experience, or patient type. The TRM score, which has been shown to predict 28-day mortality in patients with newly diagnosed AML undergoing intensive induction chemotherapy, appears to be roughly as accurate as providers' predictions.
Walter:Pfizer, Inc.: Consultancy; AstraZeneca, Inc.: Consultancy; Covagen AG: Consultancy; Seattle Genetics, Inc.: Research Funding; Amphivena Therapeutics, Inc.: Consultancy, Research Funding; Amgen, Inc.: Research Funding.
