https://orcid.org/0000-0002-1867-0526
TO THE EDITOR:
Most prediction models and prognostic tools in lymphoid malignancies, such as the International Prognostic Score (IPS),1 and the International Prognostic Index2 and its subtype variants,3 use simple binary variables (with cutoffs for continuous predictors) and simple sums to generate risk groups. These model features, although easy for a clinician to remember and calculate a score, sacrifice accuracy and model performance. Statistically, dichotomization of data variables is discouraged in prognostic models when it can be avoided because it significantly diminishes statistical power and may also conceal clinically meaningful nonlinear associations within a variable.4 Modern statistical methods, analytical software, and ubiquity of smartphones and electronic devices at the point of care have enabled the development and implementation of increasingly sophisticated models for prognosis and risk prediction in the clinical setting. These models may generate a personalized risk prediction and/or likelihood of an outcome for an individual patient. However, additional guidance is needed to translate individual predictions into routine clinical practice and to incorporate them into potential clinical trial designs and settings, in which particular “thresholds” or “risk groups” may be relevant.
The Advanced-stage Hodgkin lymphoma International Prognostic Index (A-HIPI) model in advanced-stage classic Hodgkin lymphoma (AS-HL), developed and validated by the global HoLISTIC Consortium5 uses age, albumin, bulk, gender, hemoglobin, lymphocyte count, and stage to generate individualized probability of progression-free survival (PFS) events or death overall survival (OS) within the first 5 years for newly diagnosed patients treated with standard chemotherapy (www.qxmd.com/calculate/calculator_869/a-hipi).6 The A-HIPI model was developed via TRIPOD guidelines7 on 4022 patients treated on 8 seminal international AS-HL clinical trials, and external validation was performed on a data set of 1431 patients from 4 global, prospective cancer registries. Discrimination of the A-HIPI model had an optimism corrected c statistic of 0.590 (95% confidence interval [CI], 0.557-0.622) for 5-year PFS (PFS5) and 0.730 (95% CI, 0.681-0.774) for 5-year OS (OS5) in external validation. Moreover, the A-HIPI model showed excellent calibration in external validation cohorts,6,8 which was also markedly superior to the IPS-7 and IPS-3 and similar to a recent machine learning model.8 Herein, we critically examined approaches to leverage the A-HIPI model to generate different risk groups, with analysis on the strengths and limitations from the HoLISTIC modeling team and clinical experts.
The PFS5 in the A-HIPI development data set was 77% (95% CI, 76-78), and the OS5 was 92% (95% CI, 91-93); this represents the “average outcome” for a patient with AS-HL. The distribution of PFS5 and OS5 predictions were not symmetrically distributed in either the A-HIPI discovery or validation data sets (Figure 1). Although not unexpected due to the excellent survival in this disease setting, this can present challenges in defining risk groups. Three approaches were examined for the generation of risk groups. Proposed cutoffs were defined using the distribution of A-HIPI risk scores and data from the model-building clinical trial–based cohort (clinical trials). Validation was done using the A-HIPI validation cohort (observational).
Online tool and alternative risk group analyses. (A) Screenshot of the A-HIPI risk group online tool. Users may select individualized percentile cutoffs for defining low- and high-risk disease. The application will provide dynamic outcomes based on the user-defined individualized risk groups. See interactive app at https://rtools.mayo.edu/holistic_ahipi/. (B) Risk groups based on deviation from average patient or clinical thresholds. The right-skewed distribution of A-HIPI risk scores greatly limited the applicability of these approaches. The standard-risk approach of the average patient was limited by the asymmetric distribution of patients because the majority of patients were considered average risk, and only ∼20% of patients were considered increased or high risk (approach 1). Proposed cutoffs based on clinical thresholds of PFS5 <70 and PFS5 >90 only identified 15% and <1% of the population, respectively (approach 2). 5y, 5-year; A-HIPI, advanced-stage Hodgkin lymphoma international prognostic index; PFS5, 5-year progression-free survival.
Online tool and alternative risk group analyses. (A) Screenshot of the A-HIPI risk group online tool. Users may select individualized percentile cutoffs for defining low- and high-risk disease. The application will provide dynamic outcomes based on the user-defined individualized risk groups. See interactive app at https://rtools.mayo.edu/holistic_ahipi/. (B) Risk groups based on deviation from average patient or clinical thresholds. The right-skewed distribution of A-HIPI risk scores greatly limited the applicability of these approaches. The standard-risk approach of the average patient was limited by the asymmetric distribution of patients because the majority of patients were considered average risk, and only ∼20% of patients were considered increased or high risk (approach 1). Proposed cutoffs based on clinical thresholds of PFS5 <70 and PFS5 >90 only identified 15% and <1% of the population, respectively (approach 2). 5y, 5-year; A-HIPI, advanced-stage Hodgkin lymphoma international prognostic index; PFS5, 5-year progression-free survival.
We ranked the A-HIPI risk scores of the 4022 patients with AS-HL in the model-building cohort and used the distribution of the risk scores as a benchmark. We defined the risk profile for a future patient compared with this distribution (eg, how do you rank compared with your peers?). Motivation for this approach was as follows: (1) continuous results are often presented this way already (eg, tertiles or quartiles); (2) it allows flexibility for the user to define the size of the risk groups and/or clinical threshold of interest; and (3) although therapies may change and overall outcomes improve, the patient population (including the distribution of clinical characteristics) used for the model remains consistent over time. This approach allows for flexibility in the trade-off between increasing the size of the risk group vs increasing the expected difference in outcomes between risk groups. Application of this approach showed strong alignment between the predicted model percentiles and the observed distribution of scores in the validation cohort (Table 1), further confirming the calibration reported in the initial A-HIPI publication.
Size of risk groups and outcomes using a rank-based approach
| Rank-based approach | ||||
|---|---|---|---|---|
| Model building | Validation cohort | |||
| High risk | ||||
| A-HIPI Patient rank (percentile) | A-HIPI cutoffs | Observed PFS5 <cutoff | % of pts <cutoff | Observed PFS5 <cutoff |
| 5% | <0.646 | 59.3 | 6% | 65.5 |
| 10% | <0.686 | 61.6 | 13% | 65.6 |
| 25% (quartile 1) | <0.737 | 66.1 | 27% | 71.4 |
| 33% (tertile) | <0.754 | 68.5 | 34% | 72.4 |
| 50% (median) | <0.781 | 71.5 | 49% | 74.6 |
| Rank-based approach | ||||
|---|---|---|---|---|
| Model building | Validation cohort | |||
| High risk | ||||
| A-HIPI Patient rank (percentile) | A-HIPI cutoffs | Observed PFS5 <cutoff | % of pts <cutoff | Observed PFS5 <cutoff |
| 5% | <0.646 | 59.3 | 6% | 65.5 |
| 10% | <0.686 | 61.6 | 13% | 65.6 |
| 25% (quartile 1) | <0.737 | 66.1 | 27% | 71.4 |
| 33% (tertile) | <0.754 | 68.5 | 34% | 72.4 |
| 50% (median) | <0.781 | 71.5 | 49% | 74.6 |
| Low risk | ||||
|---|---|---|---|---|
| Patient rank (percentile) | A-HIPI cutoffs | Observed PFS5 >cutoff | % of pts >cutoff | Observed PFS5 >cutoff |
| 50% (median) | >0.781 | 82.3 | 51% | 81.7 |
| 67% (tertile) | >0.806 | 83.3 | 35% | 83.2 |
| 75% (quartile 3) | >0.818 | 83.8 | 27% | 85.8 |
| 90% | >0.844 | 86.0 | 13% | 86.9 |
| 95% | >0.857 | 85.6 | 7% | 88.2 |
| Low risk | ||||
|---|---|---|---|---|
| Patient rank (percentile) | A-HIPI cutoffs | Observed PFS5 >cutoff | % of pts >cutoff | Observed PFS5 >cutoff |
| 50% (median) | >0.781 | 82.3 | 51% | 81.7 |
| 67% (tertile) | >0.806 | 83.3 | 35% | 83.2 |
| 75% (quartile 3) | >0.818 | 83.8 | 27% | 85.8 |
| 90% | >0.844 | 86.0 | 13% | 86.9 |
| 95% | >0.857 | 85.6 | 7% | 88.2 |
A-HIPI, advanced-stage Hodgkin lymphoma international prognostic index; PFS5, 5-year progression-free survival; pts, patients.
Defining a high-risk A-HIPI cutoff will depend on the needs of the study and/or user. Using the 90th percentile (risk of event >31%) identified a subset with the worst outcome (observed PFS5, 65.5%), albeit in a small number of patients meeting criteria in the validation cohort (13%), which would limit the pool of available patients for a clinical trial and may cause challenges for accrual. Relaxing the cutoff to the 75th percentile (risk of event >26%) doubles the percentage of patients labeled as having high-risk disease in the validation data set (27%) at the cost of the observed PFS5 in the high-risk group increasing to 71.4%. Using the highest tertile (risk of event >24%) showed little change in observed PFS5 in the validation cohort (72.4%) compared with the quartile and may represent the optimum cut point for defining a significant minority population of patients at increased risk for clinical trial designs. Further relaxing the definition of high risk using the median A-HIPI (risk of event >22%) would capture even more patients. However, this comes at the cost of a lower event rate in the study (observed PFS5, 74.6). Sample size (availability and cost), event rate (power), and study goals should be weighed when selecting an optimum cutoff for an individual clinical study.
In contrast, the A-HIPI model does not identify a population of patients with observed PFS5 >90% in the validation set. A cutoff of the 25th percentile (risk of event <18%) or 33rd percentile (risk of event <19%) provide reasonable thresholds for low-risk disease, with 27% and 35% of patients in the validation data set meeting these criteria with an observed PFS5 of 85.8% and 83.2%, respectively. An online R-Shiny application is publicly available, which allows users to dynamically define their own cutoffs to aid in discussion of risk estimates with patients or identify populations for clinical trial development (Figure 1A; https://rtools.mayo.edu/holistic_ahipi/).
The PFS5 across all patients was 77% (95% CI, 76-78). We explored defining “standard-risk” based on this CI as well as clinical boundaries, with patients above or below this classified as decreased or increased risk, respectively. The percentage of patients and observed outcomes within the identified risk groups were consistent between the 2 study cohorts. However, this approach was limited by the asymmetric distribution of patients because the majority of patients were considered average risk, and only ∼20% of patients were considered increased or high-risk (Figure 1B).
Expert AS-HL clinicians in HoLISTIC were queried regarding what estimates of PFS5 constitute high or low risk. Although this approach leverages expert clinical judgment from those treating AS-HL and primary users of the model, the right-skewed distribution of A-HIPI risk scores greatly limited the applicability of this approach. Proposed cutoffs of PFS5 <70 and PFS >90 only identified 15% and <1% of the population, respectively (Figure 1B).
In summary, although the A-HIPI model provides a rigorous, accurate, and individualized estimate of risk for patients with AS-HL, there are challenges with defining risk groups from individual risk predictions. Different applications and purposes, the asymmetric distribution of A-HIPI risk estimates, as well as varying individual clinical definitions of high risk, make it challenging to define consensus expert-based risk groupings for AS-HL. A flexible rank-based approach appeared to provide the most clinical utility and data granularity, which may be harnessed for future AS-HL clinical trial design and patient-stratification need. Further validation of these proposed risk groups in independent datasets is warranted, and additional analyses to refine high- and low-risk populations in AS-HL will be needed as the standard-of-care therapeutic options evolve in the frontline setting.9,10
Contribution: M.J.M., S.K.P., J.N.U., A.M.R., and A.M.E. designed the research, analyzed data, and wrote the manuscript; and R.M., S.R., J.W.F., A.G., M.F., E.H., D.H., P.J., B.K.L., E.M., K.J.S., and P.L.Z. analyzed data and wrote the manuscript.
Conflict-of-interest disclosure: S.K.P. reports consulting or advisory role with Seattle Genetics. J.W.F. is the Editor-in-Chief of Journal of Clinical Oncology, and the journal policy recused the author from having any role in the peer review of this manuscript; reports research funding from Enterome (institutional); and reports patents, royalties, and other intellectual property pertaining to bone marrow microenvironment signals (institutional). A.G. reports honoraria and consulting/advisory role with Takeda. E.H. reports consulting or advisory role with Merck Sharpe & Dohme (institutional), Roche/Genentech (institutional), AstraZeneca, Gilead Sciences (institutional), Bristol Myers Squibb (institutional), Servier, Novartis (institutional), BeiGene (institutional), Link Healthcare (institutional), and Antengene (institutional); speakers' bureau fees from Roche/Genentech, Regeneron, and AbbVie (institutional); and research funding from AstraZeneca (institutional), Celgene (institutional), Merck KGaA (institutional), Janssen-Cilag (institutional), Gilead Sciences (institutional), Mundipharma (institutional), Bristol Myers Squibb (institutional), and Roche/Genentech (institutional). P.J. reports honoraria from Genmab, Epizyme, and Incyte; consulting or advisory role with Epizyme; and patents, royalties, and other intellectual property regarding combined use of Fc gamma RIIb (CD32b) and CD20-specific antibodies (WO Patent, PCT/GB2011/051572; EU11760819.0). B.K.L. reports consulting or advisory role with Genentech/Roche, MEI Pharma, and Amgen; and research funding from Pharmacyclics/Janssen (institutional), Genentech/AbbVie (institutional), and Genmab (institutional). K.J.S. reports honoraria from Seattle Genetics, Roche, and Abbvie,; consulting or advisory role with Seattle Genetics, Roche and Abbvie; research funding from Bristol Myers Squibb, Seattle Genetics (institutional), Viracta (institutional), and Merck (institutional); and other relationship, including Data and Safety Monitoring Committee role, and uncompensated relationships with Regeneron. P.L.Z. reports consulting or advisory role with Celltrion, Gilead Sciences, Janssen-Cilag, Bristol Myers Squibb, Servier, Sandoz, MSD, Roche, EUSA Pharma, Kyowa Hakko Kirin, Takeda, Secura BIO, TG Therapeutics, Novartis, ADC Therapeutics, Incyte, and BeiGene; and speakers' bureau fees from MSD, EUSA Pharma, and Novartis. M.J.M. reports employment and stock/other ownership interests in Exact Sciences (immediate family member); consulting or advisory role with Bristol Myers Squibb (institutional); and research funding from Bristol Myers Squibb (institutional), Roche/Genentech (institutional), and Genmab (institutional). A.M.E. reports honoraria from Pfizer, Pharmacyclics, Research to Practice, Epizyme, Novartis, MorphoSys, Incyte, Targeted Oncology, AbbVie, Takeda, Patient Power, PER, OncLive Clinical Congress Consultants, HUTCHMED, Incyte, MorphoSys, and Daiichi Sankyo/AstraZeneca; consulting or advisory role with Pfizer, Novartis, Pharmacyclics, Incyte, Epizyme, MorphoSys, AbbVie, Incyte, and MorphoSys; and speakers' bureau fees from Research to Practice. The remaining authors declare no competing financial interests.
Correspondence: Andrew M. Evens, Rutgers Cancer Institute, 195 Little Albany St, New Brunswick, NJ 07871; email: andrew.evens@rutgers.edu.
