Published abstracts at international meetings often over- or underestimate the initial response rate

To the editor:

Abstracts presented in meetings have a significant effect on decision-making and medical practice. Furthermore, these data may be incorporated in large meta-analyses and systematic reviews and influence therapeutic guidelines.

Considerable pressure is placed on rapid dissemination of response data in clinical trials. Early publication of data is fraught with hazards, as survival outcomes change with time and may not be mature before 3 years.¹  However, initial response rates may be expected to be final at abstract presentation, given the absence of time-dependent variables that can affect the outcome. A comprehensive retrospective analysis was performed to test this hypothesis.

Novel therapeutic agents covering diverse hematological malignancies between 2000 and 2016 were chosen. Then, clinical trials assessing these agents were searched using Medline, PubMed, and Google Scholar databases with the aim of selecting the articles most adjacent to the first clinical report of these drugs. The journals searched, which were determined to be the most likely to report most of the important clinical trials, were limited to the New England Journal of Medicine, Blood, the Journal of Clinical Oncology, Leukemia, the Lancet, and Haematologica. Manuscripts published in these journals were obtained and searched for the most initial abstract presentation at annual meetings of the American Society of Hematology, European Hematology Association, and American Society of Clinical Oncology. Searches were obtained by the website or with help from the journal staff for years before abstracts were digitized. Some articles addressed the specific conference meeting presented in the article, making the search more convenient. Other abstracts were searched in meetings, using the author name or keywords. When an abstract was presented more than once, the earliest described abstract was included in this analysis.

We searched for a discrepancy in response rates of, arbitrarily, at least 10% between abstracts and published manuscripts.

Similarly, we used an 18-month interval between abstract and publication as a cutoff point, on the assumption that a shorter interval would represent the “final” version submitted for publication.

Three thousand papers were reviewed. Of these 3000 articles, review papers, basic science, case reports, and letters were excluded. Three hundred clinical trials were identified and further searched for presentation at 1 of the aforementioned meetings. Ninety-nine studies were identified to have a corresponding abstract (supplemental Table 1, available on the Blood Web site). Among the diseases assessed, multiple myeloma comprised 27.3% of trials, followed by non-Hodgkin lymphoma and acute myeloid leukemia in 25.3% and 17.2%, respectively (Table 1).

Eighty-six percent of the abstracts were presented at the American Society of Hematology meetings. Fifty-one abstracts (52%) were eventually published as a full paper in Blood, 25% were published in the Journal of Clinical Oncology, and 9% were published in the Lancet (Table 1).

Fifty-nine studies had discrepant results. Forty-two studies (42.4%) had a discrepancy of 10% or more (Table 2), regardless of the time interval between paper and abstract. Twenty-three (55%) of 42 studies reported an increased response in the publication. Nineteen (45%) of 42 (19.2% of all identified studies) reported a decreased response in the publication. In 30 (71.4%) of 42 cases in which there was a discrepancy of 10% or more, the time interval between abstract and manuscript was 18 months or longer. Discrepancies were less common in phase 3 studies, being mainly demonstrated in phase 1 and 2 studies. In most studies, multiple centers were involved (69%), and among studies with discrepancy, 35 (59%) of 59 were multicenter. Large national and international study groups participated in 11% of all studies and 11.8% of studies with discrepancy. Pharma-sponsored studies were noted in 52% and 44% of all studies and among discrepant studies, respectively.

The most prominent reason for variation in data was additional patients collected by the time the article was published (49.2%).

Presentation of abstracts of clinical trials is crucial for disseminating information and for the planning of subsequent clinical investigations. A common assumption is that initial response data, which are neither time-dependent nor projected, provide reliable information, allowing for the further design of trials. The data provided herein suggest that in more than 40% of cases, a significant discrepancy exists between data reported at initial abstract and subsequent manuscript publication. If one excludes studies from large established trial groups, the discrepancy is still greater (data not shown). Although in some cases this was associated with an increasing patient number as the study progressed, discrepancies also frequently occur when the number of patients is not materially different. This may reflect a somewhat hasty presentation, possibly lacking rigorous review, either local or central. On other occasions, particularly where the initial abstract data were an underestimate, this may be associated with a subtle change in the eligibility criteria to enhance accrual.

Toma et al reviewed abstracts presenting randomized controlled trials of the American College of Cardiology scientific meetings and compared them with full-length publications.²  Changes in the estimate of primary outcome occurred commonly and were demonstrated in 41% (60/148) of published articles, with a mean change in effect of 0.44 SDs.

Similarly, differences were found in 30 of 51 pairs of abstracts presented in the Interscience Conference on Antimicrobial Agents and Chemotherapy vs the published articles.³  Indeed, time to publication was mostly associated with this inaccuracy. To the best of our knowledge, this description is in line with previous reports in other medical fields, as described earlier, and has not been looked at in the hematology field. Walter et al reviewed characteristics of phase 2 trials in AML.⁴  Major problems were identified, with special consideration of absence of control group, patient heterogeneity, and selection bias. This emphasizes the value of a final and complete analysis of data with final recruitment of patients.

There are several limitations to this work. First, several parameters were defined arbitrarily, such as 10% discrepancy and 18-month interval. Obviously, these variables could affect the results of these findings, yet it seems probable that these discrepancies could be significant when decisions regarding proceeding to advanced-phase clinical trials are made. Second, a comprehensive statistical analysis was not provided here, and we recognize the added value of this, as well as sensitivity analysis and a larger study cohort. Third, not all hematology-related journals were included in this analysis, with some highly prestigious and high-impact-factor journals missing. Fourth, older studies (before 2000) were not included because of difficulty in obtaining these abstracts using journal websites.

There is clearly a need for a broader and more extensive review of data to be followed to determine the approach to presented abstracts in meetings.

These data suggest that initial response data presented in abstracts at international meetings need to be cautiously interpreted, as the outcome may change with time, leading to either an over- or an underestimation of the outcome. Only peer-reviewed publications may be relied on to provide the definitive report of the outcome.