The predictive value of a positive phase 2 ASH abstract for peer-reviewed publication and progression to phase 3

TO THE EDITOR:

The American Society of Hematology (ASH) annual meeting is often considered the premier international conference in hematology. When “positive” results from single-arm phase 2 trials are presented at ASH, one would expect that these herald therapeutic advances, or at a minimum, follow into a larger, potentially confirmatory, randomized (phase 3) trial. However, a survey of ASH abstracts reporting positive results in acute myeloid leukemia (AML) indicated that these positive abstracts, accounting for the majority of abstracts, had limited predictive value for subsequent clinical utility.¹ As phase 3 trials are initiated to further investigate clinical benefit, proceeding (or not) from a positive abstract to a phase 3 study is a topic that warrants investigation. Across all hematological malignancies, it is currently unclear how often positive (or negative) abstracts lead to phase 3 trials, as are the reasons why or why not. Here, we hypothesized that most ASH abstracts reporting on phase 2 trials in oncological hematology are positive, and that the majority of drugs in these trials will not reach investigation in phase 3. Finally, because abstracts are necessarily preliminary, subsequent peer-reviewed publication is essential for correct dissemination of trial results. Therefore, we additionally assessed to what extent positive, rather than negative, studies are subsequently published as full-length peer-reviewed papers, potentially leading to “publication bias.”

We downloaded all 2013 through 2015 ASH abstracts (N = 17 251), using a web-scraper-based approach. This timeframe allowed at least 5 years between the abstract presentation and publication as full-length paper or advancement to phase 3. Abstracts (n = 1100) were selected based on presence of “phase 2” and all possible spellings thereof, in the title and/or abstract text. We excluded benign hematology, biomarker trials, phase 1/2 trials that reported only phase 1 data, and trials without efficacy results. If multiple abstracts reported or updated on the same study, only the most recent abstract was included. These processes yielded 368 abstracts, covering lymphomas (35%), multiple myeloma (21%), AML (12%), chronic lymphocytic leukemia (11%), myelodysplastic syndrome (8%), acute lymphoblastic leukemia (5%), chronic myeloid leukemia (3%), and other (5%). Abstracts were scored “positive,” “negative,” or “inconclusive.” Criteria for a positive abstract were words/phrases such as “encouraging,” “promising,” “could represent a novel therapeutic option,” and “warrants investigation in a randomized trial.” “Negative” abstracts included terms such as “does not support further research” and “demonstrates no clinical activity.” The remainder were scored as inconclusive. Uncertainty was resolved by discussion between the authors.

We scored 292/368 (79.3% 95% confidence interval [CI], 75-84) abstracts as positive, 37/368 (10.1% 95% CI, 7-14) as negative, and 39/368 (10.6%; 95% CI, 8-14) as inconclusive (Table 1).

We then searched papers indexed in PubMed using names of the primary or senior authors and 1 to 4 title keywords, with the most recent update on 1 August 2021. We found that 302/368 abstracts (82.1%; 95% CI, 78-86) were published as full-length papers in peer-reviewed journals (Table 1; Figure 1A). The abstract conclusion was not significantly associated with a full-length publication. Abstracts published as full-length papers had larger sample sizes than nonpublished abstracts (P = .001). Studies were most often published in British Journal of Haematology (39/302 [12.9%]) and Blood (34/302 [11.3%]). A total of 91.2% (31/34) of the studies published in Blood were positive. British Journal of Haematology published significantly more negative studies than Blood (Figure 1A; 26% vs 5.9%, Fisher exact P = .02). Positive studies were published more often in journals with higher impact factors (mean 2020 impact factor 18.2 for positive vs 9.2 for negative studies, P < .001; Figure 1B). Differences between the abstract conclusion and later peer-reviewed conclusion were infrequent: 6.2% of positive ASH abstracts changed to a negative conclusion in the peer-reviewed publication and vice versa.

Fifty-three percent of regimens described in positive abstracts did not proceed to phase 3 studies as reported in clinicaltrials.gov, and the corresponding figure for positive full-length papers was 47%. Progression to phase 3 was most common in chronic myeloid leukemia (86%) and lowest in lymphomas (46%). Twelve regimens described as negative and 9 as inconclusive in full-length papers proceeded to phase 3. Regimens proceeding to phase 3 trials were based on phase 2 trials with larger sample sizes as reported in the full-length paper (median 58; interquartile range, 40-87 vs 42, interquartile range, 28-69 for regimens that did not progress; Wilcoxon rank sum P < .001). To explore reasons for discontinuation, we sent questionnaires (supplemental Data) to the authors of positive studies (based on full-length papers) not prompting phase 3 trials. Three attempts were made to reach the authors; 51% (54 of 105 reachable authors) responded. The respondents were asked to confirm that there was no subsequent phase 3 trial and to select reasons why a randomized phase 3 trial was not undertaken. Failure of positive phase 2 trials to proceed to phase 3 was due to a decision by the pharmaceutical company (43%), lack of any intent to study the drug in phase III in the first place (41%), insufficient funding (35%), insufficient efficacy (despite the “positive” conclusion in the publication; 33%), and safety concerns (4%) (Figure 1C). Additional reasons were the availability of newer regimens, the rarity of the disease, or regulatory approval already obtained after phase 2.

Elaborating on the previous AML survey that inspired this research,¹ we assessed clinical utility of AML therapeutics in our dataset. Thirty-five abstracts reported on 24 individual drugs, of which midostaurin and venetoclax received regulatory approval, and quizartinib was positive in the phase 3 trial.² This suggests more favorable progression rates to phase 3 compared with previously reported data, in which only 1 of 37 drugs described in positive abstracts migrated into clinical practice.¹ 

Remarkably, one-third of regimens did not proceed to phase 3 at least in part because of insufficient efficacy, despite most recent presentation (as full-length paper) as promising. Reasons for discontinuation after a positive phase 2 trial is understandable in cases of funding constraints or safety concerns. However, numerical overestimation of initial results,³ or overly optimistic interpretation, not tempered through peer review, may be an additional explanation for these findings. These data therefore stress the importance of a critical reading of papers (and abstracts) with attention to issues such as selection bias and clinical relevance of reported “responses.” The discontinuation of promising novel regimens per decision of pharmaceutical companies may also be problematic: although some such decisions may have reflected (unreported) funding constraints, there also is an unfortunate implication that shareholder welfare can prevail over patients’ best interests. Similarly, patient participation in research never intended to continue to phase 3 is questionable given that outcomes in single-arm phase 2 trials are often poor predictors of outcomes in randomized phase 3 trials.^4,5 Because conducting phase 2 trials seems essential for career development of junior researchers, an ethical tension between the patient’s and researcher’s best interests arises.

These data confirm earlier findings that most abstracts presented at ASH are still represented as positive. It is reassuring that the large majority of ASH abstract are subsequently published as full-length papers, and overall, the conclusions are in line with each other, suggesting that early interpretation of data are reliable. Fortunately, positive and negative studies are represented equally often in peer-reviewed journals, not indicating any “direct” publication bias. However, positive studies are published more often in high-impact journals, suggesting an "indirect" publication bias. Because truly negative studies may be more rigorous and accurate than positive studies, high-impact journals should be encouraged to publish “negative” studies to ensure equal exposure.

Although further investigation is needed into why therapeutics in positive phase 2 trials do not progress to phase 3 trials, it is encouraging that nearly one-half of the regimens presented in positive ASH abstracts are steered toward phase 3, considering the resources needed for randomized studies.² Further improving research practice and ethics, combined with trial designs that smartly test multiple “promising” agents,^2,6 as well as synthesizing evidence from additional resources such as registries or expanded access programs,⁷ will further advance the development and implementation of novel therapeutic options across hematological malignancies.