Validation of risk assessment models for venous thromboembolism in patients with cancer receiving systemic therapies Open Access

Patients with cancer face a ninefold to 12-fold higher risk of venous thromboembolism (VTE) than the general population.^1,2 The occurrence of VTE is associated with increased mortality, morbidity, health care costs, and psychological burden in patients with cancer.^3-6 The risk of VTE in patients with cancer is highly heterogeneous and influenced by many factors, which can be divided into tumor related (eg, site, stage, and histological grade), patient related (eg, age, gender, body mass index, and history of VTE), and treatment related (eg, chemotherapy and surgical procedures). Furthermore, biomarkers (blood count parameters, D-dimer, fibrinogen, or soluble P-selectin) can improve VTE risk prediction.^7-10 

The management of VTE in patients with cancer is challenging, due to the concurrent high risk of bleeding, which is only aggravated by anticoagulation therapy.^11,12 Unselected thromboprophylaxis is associated with an unfavorable risk-benefit profile, primarily due to an increased risk of bleeding. Therefore, accurate risk stratification is needed to identify patients who would benefit most from prophylactic anticoagulation. To enable individualized VTE risk assessment, several risk assessment models (RAMs) have been developed and externally validated in the last years,^8,13-17 and multiple guidelines recommend prophylactic anticoagulation in patients with cancer with a high VTE risk, as predicted using a validated RAM.^18-22 However, these validated RAMs were developed over the past 2 decades, and the landscape of anticancer treatment has dramatically changed since then.^23,24 Therefore, these RAMs and included risk items might not be suitable to reflect the VTE risk profiles in a contemporary cancer population treated with newer anticancer therapies, because the general risk patterns and specific risk factors for VTE might have changed. For instance, immune checkpoint inhibitors (ICIs) have been implied to carry a risk for cancer-associated VTE.²⁵ However, previous studies reported conflicting results,^26-28 and established RAMs performed poorly in ICI-treated patients in some studies.^28-30 Whether previously established RAMs can adequately discriminate the individual VTE risk in cancer patient cohorts receiving contemporary anticancer therapies needs to be clarified.

Therefore, we aimed to evaluate the performance of 6 externally validated VTE RAMs (Khorana score,⁸ PROTECHT,¹³ CONKO,¹⁵ COMPASS-CAT,¹⁴ CAT Score,¹⁶ and EHR-CAT¹⁷) in a prospective observational cohort study including patients with cancer initiating contemporary anticancer therapies.

The study was performed within the framework of the Vienna Cancer, Thrombosis and Bleeding (CAT-BLED) study, an ongoing, prospective, observational, single-center, cohort study, which initiated patient recruitment in July 2019. For this specific analysis, patients enrolled between July 2019 and April 2024 were included. Ambulatory patients with histologically confirmed, newly diagnosed, or recurrent/progressive cancer, initiating systemic anticancer therapy, who were referred to the oncological day clinic of the Vienna General Hospital, were eligible for inclusion and provided a written informed consent for study participation. Exclusion criteria include age <18 years and inability or refusal of an informed consent; in addition, for this analysis, patients receiving anticoagulation therapy were excluded. The study has been approved by the local ethics committee (EC number: 1533/2019) and was performed according to the Declaration of Helsinki and its later amendments, and all participants gave a written informed consent. Detailed information about the study has been described in a previous publication.¹¹ 

At study inclusion, patients underwent a clinical interview and baseline patient-, cancer-, and treatment-related data were gathered. Biobanking was performed within the Translational Research Unit Biobanking Program for Personalized Immunotherapy (EC 1164/2019) of the Division of Oncology at the Medical University of Vienna, Vienna, Austria. Patients were followed via regular in-person visits for a maximum of 2 years, and during follow-up visits, patients were asked to fill out questionnaires (in the case of refusal, a personal interview was conducted) to report on any events of interest. All events were then cross-referenced with the electronic medical records and were adjudicated by an independent adjudication committee comprising specialists in radiology, dermatology, or vascular medicine (angiology).

The primary outcome of interest was VTE, which represents a composite outcome of objectively confirmed symptomatic and incidental pulmonary embolism (PE), deep vein thrombosis (DVT) (in both upper and lower extremities), catheter-related thrombosis, superficial vein thrombosis extending at least 5 cm, and splanchnic vein thrombosis.

Statistical analyses were conducted using R Studio (version 4.4.1; R Core Team, Vienna, Austria). Standard descriptive statistics were used to present patient baseline characteristics, including absolute frequencies, percentages, median values, and interquartile range (IQR; representing the 25th and 75th percentiles). The α level was set at 0.05.

The cumulative VTE incidence was evaluated using a competing risk time-to-event analysis, in which all-cause mortality was considered as a competing event owing to the anticipated high risk of mortality in this patient population.³¹ Differences between the cumulative VTE incidences of ≥2 risk subgroups were analyzed with the Gray test.³² Regression modeling was conducted with proportional subdistribution hazards, according to Fine and Gray.³³ A Kaplan-Meier curve was used to plot 6-month overall survival. The discriminatory performance of 6 externally validated risk scores was assessed by receiver operating characteristic curve analysis and the area under the curve (AUC). AUCs were calculated based on the risk stratified cutoff for the Khorana score, PROTECHT, CONKO, COMPASS-CAT and EHR-CAT, whereas for the CAT Score, the continuous variable generated with the nomogram was used. To minimize bias from missing data, multiple imputation was performed using baseline and outcome variables. Five imputed data sets were generated and pooled using Rubin rules for AUC calculations. For cumulative VTE incidence stratified by the RAM risk, a multiple single-imputation approach was used, with the 5 data sets combined by taking the median values.³⁴ DeLong test was used to assess differences between the c-statistics of the complete-case and imputed data sets and to compare the discriminatory performance for the outcome of all VTEs versus only lower extremity DVTs and PEs.³⁵ 

A total of 806 patients with a median age of 61 years (IQR, 53-69) were included, of whom 399 (49.5%) were female. The median follow-up duration was 354 days (IQR, 207-443). The most common cancer types were lung (21.8%), breast (10.8%), and pancreatic cancer (10.3%), with 502 patients (62.3%) presenting with metastatic disease at study inclusion. Four hundred thirty-four patients (53.9%) had newly diagnosed cancer, whereas 372 (46.1%) had recurrent/progressive disease. Chemotherapy alone was initiated in 389 patients (48.3%). Furthermore, 271 patients (33.6%) received ICIs, including 134 patients (16.6%) treated with ICIs in combination with chemotherapy, 124 (15.4%) as monotherapy, and 13 (1.6%) in combination with targeted therapy. In total, 159 patients (19.7%) received targeted agents (Table 1).

During the 6-month follow-up period, a total of 91 VTE events occurred (supplemental Table 1). The 6-month cumulative incidence of VTE was 11.2% (95% confidence interval [CI], 9.0-13.3; Figure 1A). Overall, 149 patients (17.5%) died during the 6-month observation period, with a 6-month overall survival estimate of 82.7% (95% CI, 80.1-85.3; Figure 1B). The cumulative incidence of VTE was higher in patients with recurrent disease than patients with a newly diagnosed cancer (supplemental Figure 1). The parameters of the 6 evaluated RAMs, patient characteristics of the RAM-specific parameters, and a description of the imputed parameters are presented in the supplemental Tables 2 and 3.

The Khorana score yielded a c-statistic of 0.53 (95% CI, 0.50-0.56; Figure 2). Seventy-six patients (9.4%) were classified as high risk (score ≥ 3), 405 patients (50.3%) as intermediate risk (score of 2), and 325 patients (40.3%) as low risk (score of <2). The 6-month VTE incidence was 17.1% (95% CI, 8.6-25.6) in high-risk patients, 10.9% (95% CI, 7.8-13.9) in intermediate-risk patients, and 10.2% (95% CI, 6.9-13.4) in low-risk patients (Figure 3A). In a sensitivity analysis, when the cutoff for high risk was set to a score of ≥2 (as was done in the AVERT and CASSINI trials^36,37), the c-statistic changed slightly to 0.52 (95% CI, 0.48-0.55).

The c-statistic of the PROTECHT score was 0.58 (95% CI, 0.56-0.61; Figure 2). The high-risk group (score ≥3) included 208 patients (25.8%), whereas the low-risk group (score of <3) comprised 598 individuals (74.2%). The 6-month VTE incidence was 15.9% (95% CI, 10.9-20.8) in the high-risk group and 9.5% (95% CI, 7.2-11.9) in the low-risk group (Figure 3B).

The evaluation of the CONKO score revealed a pooled c-statistic of 0.54 (95% CI, 0.51-0.57; Figure 2). According to this score, 80 patients (9.9%) were in the high-risk category (score of ≥3), whereas 726 (90.1%) were in the low-risk category (score <3). The 6-month VTE incidence was 18.8% (95% CI, 10.2-27.3) for high-risk patients and 10.3% (95% CI, 8.1-12.5) for low-risk patients (Figure 3D).

The COMPASS-CAT score showed a c-statistic of 0.50 (95% CI, 0.47-0.53; Figure 2). This score identified 493 patients (61.2%) as high risk (score of ≥7) and 313 patients (38.8%) as low risk (score <7). The 6-month VTE incidence was 10.3% (95% CI, 7.7-13.0) in high-risk patients and 12.5% (95% CI, 8.8-16.1) in low-risk patients (Figure 3C).

The CAT Score had a c-statistic of 0.65 (95% CI, 0.62-0.67; Figure 2). Using the recommended cutoff of 8.0% estimated 6-month risk of VTE, the CAT Score categorized 147 patients (18.2%) as high risk and 659 patients (81.8%) as low risk of VTE. The 6-month VTE incidence was 19.7% (95% CI, 13.3-26.2) in the high-risk group and 9.3% (95% CI, 7.0-11.5) in the low-risk group (Figure 3E).

Finally, the EHR-CAT score yielded a c-statistic of 0.55 (95% CI, 0.52-0.57; Figure 2). This score categorized 505 patients (62.7%) as high risk (score of ≥3) and 301 patients (37.3%) as low risk (score of <3). The 6-month VTE incidence was 12.1% (95% CI, 9.2-14.9) in high-risk patients and 9.6% (95% CI, 6.3-13.0) in low-risk patients (Figure 3F).

Detailed performance characteristics of all 6 scores are presented in Table 2. In a sensitivity analysis, only considering PEs and lower extremity DVTs as outcomes of interest, the c-statistics for all 6 scores showed a slight improvement, yet these improvements were not statistically significant (supplemental Tables 4 and 5).

In addition, a complete-case analysis showed slight changes in the c-statistics of the 6 scores; however, these were not statistically significant (supplemental Tables 4 and 5). The CAT Score exhibited the largest decrease (AUC of 0.68, in the complete-case analysis to AUC of 0.65 after imputation), likely owing to the higher percentage of missing values (29.7%; supplemental Table 3).

Upon analyzing individual items of the RAMs, most of the traditional Khorana score parameters, including hemoglobin, platelet count, leukocytes, and body mass index, did not show a significant association with VTE in any of the scores that implement them (supplemental Table 6). In the PROTECHT score, treatment regimens including gemcitabine (subdistribution hazard ratio [SHR], 1.88; 95% CI, 1.07-3.32) or platinum-based therapies (SHR, 1.97; 95% CI, 1.28-3.02) were associated with an increased risk of VTE. Within the COMPASS-CAT score, only a history of VTE (SHR, 1.95; 95% CI, 1.06-3.42) and a recent cancer diagnosis (SHR, 2.19; 95% CI, 1.37-3.50) were associated with VTE risk. Cancer type stratification as a risk parameter was significantly associated with VTE in the Khorana score, CONKO, EHR-CAT, and CAT Score. Increasing D-dimer levels were also associated with the risk of VTE (SHR, 1.15; 95% CI, 1.09-1.23; per increase of 1 μg/mL) The results of the multivariable analyses are presented in the supplemental Table 6.

In this contemporary prospective cohort study of patients with cancer initiating anticancer therapies, we evaluated the performance of 6 previously developed and externally validated RAMs for cancer-associated VTE. Our analysis revealed substantial variation in the performance of the RAMs, with most showing a poor to moderate discriminatory ability. The CAT Score followed by the PROTECHT score had the highest AUCs, whereas the EHR-CAT and COMPASS-CAT could not discriminate between patients with high and low VTE risk. VTE incidence was identical in the low- and intermediate-risk groups of the Khorana score.

One of the main reasons for this variability in performance compared with the first publication of the scores could be the impact of anticancer therapies, in particular ICI therapy, on VTE risk. Systemic anticancer therapy is a well-established VTE risk factor, with particularly chemotherapy being associated with a high risk.⁹ This led to the implementation of chemotherapy (platinum based and gemcitabine) as a risk factor in the PROTECHT score, which likely contributed to its slightly better performance.¹³ Recently, several studies reported that patients treated with ICIs also have a high risk of VTE, although the extent of this risk remains under debate.^26-28 Approximately one-third of our cohort received ICI therapy, which reflects the current landscape of cancer treatment given that recent estimates suggested that >30% of patients with cancer may be eligible for ICI therapy, with even higher proportions among patients with non–small cell lung cancer.^23,24 Several models such as the Khorana score,⁸ PROTECHT,¹³ and CONKO¹⁵ were developed before the widespread implementation of ICI. Subsequent scores such as the COMPASS-CAT¹⁴ or the CAT Score¹⁶ did not capture this patient population in their derivation cohorts, and only the EHR-CAT score included a small proportion of patients receiving ICIs (6% on monotherapy and 3.6% in combination with chemotherapy).²⁰ The impact of ICI therapy on the performance of RAMs has also been noted in previous studies, particularly highlighting the Khorana score’s limited utility in ICI-treated patients.^28-30 

However, even though the derivation population of the CAT Score¹⁶ did not include patients treated with ICIs, we demonstrated in an external validation that the CAT Score performs well in contemporary oncologic cohorts.³⁸ It had the highest c-statistics in our present analysis, and we hypothesize that this is caused by the utilization of D-dimer as a risk parameter. No other RAM included a hemostatic biomarker as they relied on clinical parameters only. This underscores a potential limitation of these models, given that hemostatic biomarkers offer a more direct measure of the hypercoagulable state associated with cancer and its therapies, regardless of the underlying type of cancer and treatments. Biomarkers have proven utility in reflecting prothrombotic states, and emerging data suggest that inflammatory biomarkers could be relevant in ICI-treated patients.^10,28,39 In particular, elevated biomarkers of systemic inflammation may signal elevated VTE risk in patients on ICIs.^39,40 The Khorana score incorporates hemoglobin, platelets, and leukocytes but these were not predictive in our analyses, which is consistent with previous studies.^41,42 In contrast, several clinical parameters (cancer type, history of VTE, and a cancer diagnosis within the last 6 months) were predictive for future VTE in our analysis.

Differences in the distribution and included types of cancer across the original derivation cohorts and our study population might also contribute to differences in model performance. Notably, lung cancer was the most prevalent malignancy in our cohort, and an individual patient data meta-analysis demonstrated that the Khorana score lacks predictive value in patients with lung cancer.⁴³ In addition, the derivation cohort of the CONKO score included only patients with pancreatic cancer,¹⁵ COMPASS-CAT had a high proportion of patients with breast cancer (61.5%),¹⁴ the Khorana score derivation cohort included 12.1% of patients with lymphoma and had a much shorter observation time of 2.5 months,⁸ and the EHR-CAT study had a high rate of patients with colorectal cancer (11.8%).¹⁷ In comparison, our CAT-BLED cohort has a more heterogeneous distribution of cancer sites (with lung [21.8%], breast [10.8%], and pancreatic cancer [10.4%] being the most common). However, the CAT Score also included a relatively high proportion of lymphoma patients in its original derivation cohort (17%),¹⁶ and therefore, the variation in the distribution of cancer types might only partially explain the varying performance.

The poor performance of the EHR-CAT score is surprising and stands in contrast with a recent external validation in a Danish population.⁴⁴ However, this discrepancy might be explained by both the derivation cohort of the EHR-CAT and the Danish external validation cohort relying on electronic health records data and International Classification of Diseases, Tenth Revision, codes. Although these data provide a valuable resource for large-scale model derivation, they often fail to fully capture the complexity and diversity of real-world patient populations.⁴⁵ The high volume of data can result in overestimation of the importance of parameters that are easily recorded within electronic health records. However, such findings may not be well generalizable to other populations, particularly those in regions with less advanced health care infrastructures or institutions with more limited electronic health records capacities.⁴⁶ In addition, there are significant biases introduced by missing data and heterogeneity across different health care systems.⁴⁷ Finally, health care systems differ in how data are captured, stored, and coded, leading to further challenges in model reproducibility and external validation.⁴⁸ In contrast, the prospective CAT-BLED study was specifically designed to evaluate the risk of VTE and bleeding in patients with cancer initiating systemic anticancer therapies and relies on detailed patient data with adjudication of outcomes by an independent expert committee not involved in the design and conduction of the study.

Finally, we acknowledge several limitations of this study. First, as mentioned, the proportion and the types of cancer in our cohort differed from those in the original derivation studies of some of the RAMs. However, we are confident that our study is a representative cohort of patients treated with contemporary cancer therapies, including a large proportion of patients receiving ICI and targeted therapy. Second, we included cases of splanchnic vein thrombosis and catheter-associated thrombosis, types of VTE that are not consistently included in VTE RAMs. However, our sensitivity analysis evaluating only the prediction of PE and lower extremity DVT yielded similar results. Third, another important consideration is the imputation of missing data used for the evaluation of 6 RAMs. Although the performance of the scores slightly changed after imputation, this change was not statistically significant (supplemental Tables 4 and 5). Notably, the performance of the CAT Score decreased after imputation and is lower than the AUC reported in the external validation of the CAT Score in the CAT-BLED study.⁴⁹ This reduction is attributed to the rather high number of missing D-dimer levels, which were unavailable for 29.7% of patients. In addition, because the CAT Score uses only 2 parameters, imputing one of them has a greater impact on its predictive performance than the imputation of one parameter among several, as observed in the other RAMs.

In conclusion, the poor to modest and variable performance of all RAMs highlights the ongoing need for improvement in predicting VTE risk in patients with cancer. This is essential for accurately identifying high-risk individuals who would benefit from thromboprophylaxis. Advancing this field will require further research and studies to optimize existing models or establish new ones. Enhancements of VTE risk assessment could include the incorporation of promising hemostatic biomarkers, such as D-dimer,⁵⁰ exploring machine learning approaches that have demonstrated promising AUCs but require rigorous validation,⁵¹ and leveraging omics-based strategies.⁵² In addition, it is crucial to re-evaluate established RAMs in modern patient populations receiving novel treatment modalities, such as ICIs but also chimeric antigen receptor T-cell therapy and antibody-drug conjugates, given that these are rapidly being integrated into cancer care and may also carry distinct VTE risks.

The authors thank Daniel Kraemmer for his input with regard to the statistical analysis. The authors are also grateful to Lynn Gottmann, Josef Fürst, Martin Korpan, Markus Kleinberger, and Vincent Sunder-Plassmann for their help in enrolling patients into the CAT-BLED study.

This work was supported by the Early Career Research Grant of the “Gesellschaft für Thrombose und Hämostaseforschung” 2021 (Society for Thrombosis and Haemostasis Research) (F.M.).

Contribution: N.V., C.E., and C.A. conceptualized the study; N.V. and C.E. performed the data analysis; J.M.B., C.E., F.M., and N.V. conducted patient recruitment; C.A. and I.P. supervised the project; A.S.B. and M.P. provided vital input and contributed to the preparation of the manuscript; and all authors reviewed and approved the final manuscript.

Conflict-of-interest disclosure: F.M. has received travel/congress support from Novartis and honoraria for advisory boards from Servier and Bristol Myers Squibb (BMS). A.S.B. has received research support from Daiichi Sankyo and Roche; honoraria for lectures, consultation, or advisory board participation from Roche, BMS, Merck, Daiichi Sankyo, AstraZeneca, CeCaVa, and Seagen; and travel support from Roche, Amgen, and AbbVie. M.P. has received honoraria for lectures, consultation, or advisory board participation from for-profit companies such as Bayer, BMS, Novartis, Gerson Lehrman Group, CMC Contrast, GlaxoSmithKline, Mundipharma, Roche, BMJ Journals, MedMedia, AstraZeneca, AbbVie, Lilly, Medahead, Daiichi Sankyo, Sanofi, Merck Sharp & Dohme, Tocagen, Adastra, Gan & Lee Pharmaceuticals, Janssen, Servier, Miltenyi, Boehringer Ingelheim, Telix, and Medscape. I.P. has received personal fees for lectures and/or participation in advisory boards from BMS/Pfizer alliance, Rovi, and Sanofi. C.A. has received personal fees for lectures and/or participation in advisory boards from Bayer, Daiichi Sankyo, BMS/Pfizer alliance, and Sanofi. The remaining authors declare no competing financial interests.

Correspondence: Cihan Ay, Division of Hematology and Hemostaseology, Department of Medicine I, Medical University of Vienna, Waehringer Guertel 18-20, A-1090 Vienna, Austria; email: cihan.ay@meduniwien.ac.at.