External validation and comparison of multiple prognostic scores in allogeneic hematopoietic stem cell transplantation

Given the potential benefits and perils associated with allogeneic hematopoietic stem cell transplantation (HSCT), informed risk estimation is an integral part of candidate evaluation. The past 20 years have seen the proliferation of risk indices for the prediction of HSCT outcomes. These models can be useful for patient counseling, treatment strategy optimization, and statistical analysis across cohorts.^1,2  Scores are based on a variety of different sets of parameters. The Hematopoietic Cell Transplantation–specific Comorbidity Index (HCT-CI)³  and its derivative Comorbidity-Age Index⁴  are based on the patient’s comorbidity profile. The score of the European Society for Blood and Marrow Transplantation (EBMT)⁵  includes characteristics of the patient (age), disease (status, time from transplantation), and donor (relation, donor-recipient HLA match, and sex match). The Comorbidity-EBMT index, proposed by Barba et al in 2014, combines the comorbidity-specific information from the HCT-CI with the broader range of values included in the EBMT score.⁶  In 2006, the Pretransplantation Assessment of Mortality (PAM) score⁷  was published with some features that overlap with the EBMT score (it also includes information on conditioning and laboratory markers of comorbidities); this score was simplified in 2015 (revised PAM [rPAM]),⁸  leaving only age, donor type, disease status, and pulmonary function, while adding donor and recipient cytomegalovirus (CMV) serostatus. Most recently, the Endothelial Activation and Stress Index (EASIx),⁹  a laboratory biomarker-based formula including serum creatinine, lactate dehydrogenase, and platelet count, was developed for the prediction of survival in patients developing acute graft-versus-host disease; this score has been extended into the general prediction of mortality when measured pretransplantation.¹⁰  Another prognostic tool commonly used for risk stratification is the revised Disease Risk Index (rDRI), which incorporates disease type and status at time of transplantation.¹¹  Supplemental Table 1 describes each score’s components. Data regarding the comparative performance of these scores in the same population are lacking. We aimed to externally validate and compare the performance of these 8 systems in a contemporary cohort of transplantation patients across several outcomes.

Clinical and laboratory data prior to transplantation were obtained from the electronic medical record for allogeneic transplantations performed between 2011 and 2015 at Chaim Sheba Medical Center at Tel HaShomer, Ramat Gan, Israel. Outcomes were cross-referenced with the national social security registry for survival of any patients lost to follow-up. We included adult patients (age ≥18 years) who underwent transplantation for any indication and who received grafts from matched sibling (MSD), matched unrelated (MUD; 10 of 10 HLA alleles), or mismatched unrelated (9 of 10 HLA alleles) donors. Patients missing the parameters necessary for calculation of all studied scores were excluded from the analysis, with the following exceptions: 38 patients with missing patient-donor CMV serostatus pair were included in the overall study but excluded from the analysis of the rPAM score; for 17 patients missing time from diagnosis to transplantation, a component of the EBMT score, this value was imputed using the median value of all patients with the same diagnosis and disease stage. The rDRI was inapplicable to 18 patients treated for nonmalignant conditions who were excluded from the assessment of that system. Conditioning regimens were deemed to be myeloablative or reduced-intensity based on the definitions of Bacigalupo et al,¹²  with treosulfan-based regimens also considered myeloablative.¹³  Comorbidities were measured by a qualified transplant physician using definitions provided by the HCT-CI.³  Human subject research was approved by the Chaim Sheba Medical Center Institutional Review Board, and all research was performed in accordance with the Declaration of Helsinki.

All outcomes were measured from the time of HSCT. Nonrelapse mortality (NRM) was defined as death without the competing event of relapse following HSCT. Time of relapse was determined by a clinical finding of recurrent disease.

Prognostic scores were calculated for each patient using the definitions provided in the publications of these prognostic indices: the HCT-CI,³  the Comorbidity-Age Index (Comorbidity-Age),⁴  the combined HCT-CI and EBMT (Comorbidity-EBMT),⁶  the EASIx,⁹  the PAM⁷  and rPAM⁸  scores, and the EBMT⁵  score (supplemental Table 1). For determination of disease status in calculation of the PAM score, we used the definition provided by the EBMT score similar to the example set by Barba et al.¹⁴  Disease status in the rPAM score was determined using the categories of the rDRI,¹¹  with nonmalignant diagnoses assigned an intermediate disease stage, as specified in the rPAM’s initial publication by Au et al.⁸  Scores were tested for normal distribution in the population using the Shapiro-Wilk test, and Pearson product-moment correlation coefficients were calculated between each pair of scores. For a subanalysis including only patients who received transplants for acute leukemia, an additional score, the Acute Leukemia–EBMT (AL-EBMT; developed using a machine learning technique), was calculated as well, following the schema outlined by Shouval et al.¹⁵ 

The scores were grouped into 3 to 6 levels each for estimating overall survival (OS), NRM, and relapse incidence using the Kaplan-Meier and cumulative incidence methods and compared using the log-rank and Gray tests. Multivariable regression, adjusted for age, donor type, conditioning intensity, and year, was performed only for the HCT-CI and EASIx scores because these covariates were themselves components of the remaining scores. Additionally, a multivariable model was built separately for the rDRI, adjusting for the same covariates other than disease risk.

Score discrimination was measured using the area under the receiver operating characteristic curve (AUC). Discrimination reflects the ability of a prediction model to differentiate between those who do and do not experience the studied outcome. Perfect discrimination corresponds with an AUC of 1.0, meaning that the predicted risk for all individuals who developed the outcome is higher than that for all individuals who did not experience the outcome. An AUC of 0.5 is indicative of a random predictor, that is, a coin toss.¹⁶  AUCs were calculated across the entire cohort for the prediction of OS, NRM, and relapse incidence at 100-day and 1-, 2-, and 3-year time points in each of the scores independently. AUCs were further validated using a bootstrapping technique with 100 samples, with median and interquartile range (IQR) reported in the supplemental Appendix. Calibration, the agreement between prediction and observed outcome, was assessed graphically by plotting predicted vs observed outcomes for score quartiles at the 100-day and 1- and 2-year time points. A model is considered well calibrated if, for example, among a group of 100 patients with a mean predicted risk of 20%, ∼20 patients develop the outcome.¹⁷  Finally, within a variety of subpopulations, discrimination was assessed to determine whether each score performed better or worse under a given set of conditions.

Statistical analyses were performed using R version 3.4.3 (R Foundation for Statistical Computing) and the packages “survival,” “cmprsk,” “prodlim,” “pec,” “rms,” and “ggplot2.”

Population characteristics are provided in Table 1. A total of 528 patients was included. The median age was 55 years (IQR, 40-64 years). Patients were treated for a variety of malignant and benign conditions. Acute myelogenous leukemia was most prevalent, at 44% of patients, followed by the myelodysplastic syndrome (14%) and non-Hodgkin lymphoma (12%). Fifty-six percent of patients had intermediate-risk disease per the rDRI at time of transplantation. The majority of patients received pretransplantation conditioning with a myeloablative regimen (74%), and 10% of patients were conditioned with a regimen including total body irradiation at any dose. HLA-matched sibling donors were used for 46% of patients treated. The median follow-up was 2.5 years (IQR, 1.7-3.9 years). Supplemental Table 2 compares our cohort to the derivation cohorts of each of the original scores.

Each score was calculated for all patients in the cohort, with the exception of the rPAM, which could not be calculated for 38 patients (7%) due to missing donor CMV serostatus, and the rDRI, which was inapplicable to 19 patients (4%). The distribution for each score is shown in Figure 1 and supplemental Figure 1. Scores were positively correlated (supplemental Figure 2). The Pearson correlation between scores was generally below 0.50, except for scores whose components substantially overlap, such as Comorbidity-EBMT, Comorbidity-Age, and HCT-CI; the rPAM, which includes the rDRI, or the EBMT; and PAM, which shares a definition of disease risk. Scores were nonnormally distributed (P < .001 in all cases), with more patients having low (favorable) than high (adverse) values. EASIx was notable for its distant outliers, with 75% of values between 0 and 3.76, and the remaining quartile extending to 212 (Figure 1F inset).

The highest-risk stratum was associated with increased risk for overall mortality in the Comorbidity-Age, Comorbidity-EBMT, EBMT, PAM, rPAM, and EASIx scores and the rDRI in the univariable setting (Table 2; supplemental Table 3; Figure 2). However, a monotonic increase (ie, increasing risk with each score stratum) was best observed in rPAM (hazard ratio, 1.5, 2.5, 3.3), corresponding with decreasing OS probability. Similar results were observed for NRM, with hazard ratios ≥ 3.0 in the Comorbidity-Age, PAM, rPAM, and EASIx for the highest-risk strata. Relapse was not predicted by the HCT-CI or EASIx scores. A multivariable analysis, adjusted for age, donor type, conditioning intensity disease status, and year, was conducted only for HCT-CI and EASIx scores, as these indices do not include disease- and transplant-related features (supplemental Table 4); the highest-risk stratum of EASIx remained an independent predictor of overall and NRM. A similar multivariable analysis for the rDRI, incorporating age, donor type, conditioning intensity, and year of transplantation, demonstrated that the rDRI remained a predictor of overall and NRM as well as relapse. Additionally, higher rDRI levels were associated with increasing risk in the multivariable models for both HCT-CI and EASIx, further supporting these indices’ potentially additive role.

Discrimination for OS, NRM, and relapse at 100 days, 1 year, 2 years, and 3 years posttransplantation is described in Figure 3, with further validation using 100 bootstraps presented in supplemental Table 5. For OS, AUCs ranged from 0.55 to 0.67. Values were highest for the PAM and rPAM scores across all time points, ranging from 0.62 to 0.66 for PAM and 0.63 to 0.67 for rPAM. The EASIx score showed comparable discrimination at day 100 (0.64), subsequently decreasing to as low as 0.58 at 3 years. The EBMT, HCT-CI, and Comorbidity-Age scores had AUCs ranging from 0.56 to 0.60 across all time points. Through the 2-year time point, PAM, rPAM, and EASIx had closely aligned AUCs for NRM (ranging from 0.63 to 0.67), though EASIx decreased at 3 years while PAM and rPAM remained stable. AUCs were lower overall for the prediction of relapse, with the highest AUC associated with the rPAM score and rDRI at 100 days and 1 year (ranging from 0.63 to 0.65); other AUCs for relapse were mostly in the 0.5 to 0.6 range. EASIx had the lowest AUCs for relapse at all time points. All scores were well calibrated for OS (supplemental Figure 3A-C).

Score performance was further studied by age (<55 years, ≥55 years), donor type (MSD, MUD), and conditioning intensity (myeloablative conditioning [MAC], reduced-intensity conditioning [RIC]). PAM and rPAM had higher AUCs in the younger age group (0.65 vs 0.59, 0.69 vs 0.61, respectively; supplemental Figure 4). In contrast, EASIx performed better among older patients (0.61 vs 0.56). Most prognostic indices had similar discrimination irrespective of donor, with 2 exceptions. The PAM score demonstrated greater discrimination in the MSD setting (MSD, 0.68 vs MUD, 0.59) whereas the EASIx score had greater discrimination in the MUD setting (MSD, 0.56 vs MUD, 0.62). Higher AUCs were observed for the myeloablative subgroup for the rPAM, HCT-CI, Comorbidity-Age, and EASIX scores (MAC, 0.70 vs RIC, 0.61; 0.57 vs 0.51; 0.59 vs 0.53; and 0.63 vs 0.53, respectively).

Additionally, the acute leukemias, representing the most common indication for allogeneic transplantation, were studied separately. The rDRI, rPAM, EBMT, PAM, Comorbidity-EBMT, and EASIx scores all had AUCs in the low 0.6 range for 2-year OS. An additional score, the AL-EBMT, which is applicable only to the acute leukemias, was also included and had a similar AUC (0.63).

In this retrospective analysis, we compared 8 prognostic models in a cohort of allogeneic transplant recipients. Score prediction performance, in terms of risk stratification and discrimination, varied considerably, both across outcomes and subgroups. The majority of models, most notably rPAM, successfully grouped patients into lower- and higher-risk strata, supporting their use for risk classification. However, accurate individualized prediction remains suboptimal. Similar to previous studies, the best score performances approached an AUC of 0.70 on a scale of 0.50 to 1.00, necessitating caution when making individual clinical decisions based on these tools. Score performance varied based on the outcome being measured, an effect observed most strikingly in the EASIx score, which was among the strongest predictors of NRM but had little or no information regarding relapse. Intriguingly, rPAM was roughly consistent in its prognostic capacity across all 3 outcomes studied.

Death following transplantation is typically understood as the tension between 2 competing events: transplantation-related mortality and relapse. Naturally, pretransplantation disease features tend to predict relapse whereas patient-specific characteristics are more indicative of transplantation-related mortality. Prognostic models in HSCT could be viewed as global scores, (eg, rPAM, PAM, EBMT, and Conditioning-EBMT), which incorporate variables from several domains to provide an estimate of the expected OS, vs domain-specific scores that include specific patient-related or disease-related features (eg, HCT-CI, Comorbidity-Age, EASIx, and rDRI). Depending on their components, the latter group may be informative of the risk of relapse or NRM. Indeed, rDRI was predictive of relapse, whereas EASIx was among the top predictors of NRM (Figure 3B). A clinician may use information from domain-based scores, but it is not clear how these should be combined with respect to balancing the benefit-risk ratio of transplantation. Therefore, a point could be made in favor of global scores integrating components predictive of relapse and NRM. The poor correlation between the EASIx or HCT-CI scores and all other scores (supplemental Figure 2) suggests that they may be additive. However, combination of the HCT-CI and EBMT (Comorbidity-EBMT) did not result in a meaningful improvement of prediction. Each of the global scores incorporates a variable for the risk inherent to the diagnosis and stage. The higher discrimination with rPAM may be attributed to the incorporation of the rDRI,^11,18  which is a more contemporary disease-risk scheme than the EBMT score’s embedded disease-risk criteria. Overall, the disease-risk variable is perhaps the single greatest predictor of transplantation outcomes.^15,18 

Determining the generalizability of prognostic scores and avoiding overoptimistic performance assessment requires external validation. Validation studies of each of these scores have been published,^19-23  however, few direct comparisons on the same population have rarely been performed, and mostly include 2 or 3 scores (HCT-CI and EBMT or PAM).^6,14,24-27  Furthermore, methodologies have varied; Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines,^17,28  which outline best practices for model development and validation, are inconsistently followed. In accordance with the TRIPOD recommendations, we report both the calibration and discrimination of the models. Overall, the models studied are well calibrated, indicating that predicted outcomes are aligned with observations. However, in the HCT-CI score, there is poor calibration for lower scores due to overestimation of survival for patients in the lowest score quartile. This is corrected in the Comorbidity-Age, implying that advanced age reduces overoptimism in the lower HCT-CI risk groups.

Score performances varied across subpopulations (supplemental Figure 4). In the rPAM, HCT-CI, and Comorbidity-Age scores, discriminative capacity was higher in the MAC setting compared with RIC. This may reflect a predominance of MAC among development cohorts (supplemental Table 2). EASIx also demonstrated higher AUCs in the myeloablative cohort; although EASIx was developed with a large number of RIC patients, it could be argued that MAC patients are more susceptible to the endothelial dysfunction that the score was initially developed to predict. Because acute leukemia is the leading indication for allogeneic transplantation, we performed a subanalysis restricted to this population (supplemental Figure 4D). The AL-EBMT score, an acute leukemia-specific score that was the first machine-learning–based predictive model developed in allogeneic transplantation, was also incorporated.¹⁵  The consistency of AUCs (range, 0.60-0.64) across all of the scores (except the comorbidity indices, which were lower), irrespective of different modeling approaches, suggests that databases comprising traditional parameters have been exhausted. Improvement in prediction will likely require the incorporation of novel biomarkers.

Models relying on robust and objective biomarkers may improve our ability to predict. We have previously shown that pretransplantation hypoalbuminemia and renal function abnormalities are among the strongest risk factors for poor outcomes in HSCT recipients.²⁹  EASIx stands out for incorporating only laboratory-based markers, while performing similarly to other clinically oriented scores (rPAM, PAM) in the initial time points. Furthermore, when EASIx is studied in a multivariable analysis adjusting for key clinical features, the highest score strata maintained a strong association with increased mortality. Meeting its authors’ underlying hypothesis, the score’s accuracy is driven by the ability to predict NRM, a toxicity-based measure, whereas relapse is not predicted. The continuing identification of genetic and microbiome markers of both relapse and nonrelapse risk should motivate the field to pursue further biologically driven risk-prediction schemes.^30-34 

Differences in the scores’ predictive performance between our cohort and the derivation cohorts may stem from differences in the populations (supplemental Table 2). The current cohort represents a more recent transplant era compared with the original studies, which may partially account for discrepancy in the models’ performance. Also, previous studies have suggested that the utility of risk indices may be center-specific.³⁵  Despite being a single-center cohort, these results recapitulate findings by independent validation studies. Alternative donors are not represented in our cohort and remain underrepresented across all validation studies. The generalizability of transplantation prognostic indices to the alternative donor setting has been studied in small cohorts with varying performance.^19,21  The development of donor-specific systems will likely contribute to more accurate predictions.³⁶  One must keep in mind that scores are limited to patients who received transplants and do not capture alternative treatments; therefore, a truly informed decision contemplating all potential therapeutic paths is not considered. In clinical practice, the scores’ greatest utility may be in identifying that subset of patients who are least likely to benefit from HSCT. In all scores, the highest stratum was associated with substantially increased risk of poor outcome. Given the limited correlation between scores driven by meaningfully different feature sets, each system may identify a different subset of these highest-risk patients. This approach simulates, and may augment, the clinical intuition that integrates a patient’s physiologic status (age, comorbidities) and procedural characteristics (donor, diagnosis, and stage). The highest risk represents a population with extremely limited alternative treatment options, however, these patients may be ideal candidates for clinical trials.

As novel therapeutic approaches emerge in hemato-oncology, the risk-benefit analysis for allogeneic transplantation becomes ever more important. In this retrospective comparison of the leading prognostic indices in HSCT, we show that most models can be used to stratify patients, but not to make individualized predictions. Barriers to improvement include, first and foremost, quantity and quality of source data as well as selection biases in development cohorts. Oversimplifications of the relationship between predictor and response, such as the use of categorized in place of continuous measures, and parametric assumptions on data behavior, may lead to the loss of prognostic information.³⁷  Also, aside from an inherent stochastic component, the risk of detrimental outcomes following transplantation evolves over time, and patients remain susceptible to events that cannot be anticipated (eg, infection, graft-versus-host disease, depression).¹⁶  The advent of electronic medical records and large international registries now permits a more granular exploration of transplantation outcomes. Personalization of transplantation procedure may be made possible by developing new and specific prediction schemes based on large, homogenous cohorts, while integrating novel modeling techniques.^15,36,38  Furthermore, big data analysis may identify modifiable features, which are predictive of clinical paths and therefore could be acted upon. We have previously shown that the risk of conditioning toxicity is dependent on the patient’s individual comorbidities rather than their cumulative burden, suggesting the potential for both the prediction and treatment optimization that such a granular approach allows.³⁹  A new generation of prediction models, integrating the newfound wealth of data and biological knowledge, is needed to truly inform individual decision-making in allogeneic transplantation.

This work was supported by The Varda and Boaz Dotan Research Center in Hemato-Oncology affiliated with the Cancer Biology Research Center of Tel Aviv University and The Shalvi Foundation for the Support of Medical Research.

Contribution: J.A.F., A. Shouval, A.N., and R.S. designed the study; J.A.F. and R.S. wrote the initial draft of the manuscript; and all authors were involved in the collection and interpretation of the data, edited the initial draft of the manuscript, and agreed to the final manuscript.

Conflict-of-interest disclosure: The authors declare no competing financial interests.

Correspondence: Joshua A. Fein, Sackler School of Medicine, Tel Aviv University, Ramat Aviv 69978, Israel; e-mail: joshuafein@gmail.com.