The American Society of Hematology Clinical Research Training Institute is associated with high retention in academic hematology

Training programs to develop clinician scientists with a focus on patient-oriented research are important to ensure that research findings affect patient outcomes in a positive and meaningful manner.^1,2  Traditionally, hematology training programs focused on basic and translational sciences. According to a survey of subspecialty training programs in adult and pediatric hematology and oncology, more than one third of programs devote at least 50% of the time to research efforts.³  Several National Institutes of Health working groups noted the lack of effective clinical research training programs and the lack of mentorship,⁴  and the American Society of Hematology (ASH) similarly identified this deficit more than a decade ago.³  In response to this need, ASH created the Clinical Research Training Institute (CRTI),^5,6  and the inaugural class was held in 2003.

Measuring the impact of such training programs is necessary yet challenging. Rigorous and ongoing evaluation of a training program is important to ensure that the program is meeting its goals and that it is of value to trainees and stakeholders. Evaluation may also suggest areas in which the program needs to change or improve. However, little work has been conducted to evaluate training programs in general or in hematology more specifically. In the few published studies evaluating hematology training programs, the primary goal was to evaluate clinical training, not research outcomes.^7,8 

CRTI has conducted several types of evaluations from program inception. However, a systematic evaluation of program outcomes has not been undertaken. The objective of this analysis was to evaluate the CRTI program through a single cross-sectional study of previous participants, 2 annual surveys focused on the mentorship experience and 2 annual evaluations conducted immediately before and after completion of the summer workshop. We then propose future directions to improve this evaluation process.

The CRTI program is a 1-year experience that begins with a weeklong workshop typically held in August. Eligibility criteria for entrance to the program include being a senior fellow or junior faculty member with an intended career in patient-oriented hematology research. Participants include clinicians focused on either malignant or benign hematology in pediatric or adult medicine. Participants primarily come from the United States or Canada, but more recently CRTI has included participants from Australia, Germany, the United Kingdom, and Singapore.

Each year, 20 participants are admitted to the program, although in some years, 1 or 2 additional participants have been included to encourage diversity. The summer workshop faculty includes ∼20 established clinical researchers, 5 or 6 biostatisticians, and representatives from key funding agencies including the National Heart, Lung, and Blood Institute and the National Cancer Institute. In recent years, a library scientist was invited.

Selection is a competitive process through a review panel of established hematology clinical researchers. Participants must submit a description of a research project with their application. The summer workshop begins and ends with a presentation of each participant’s proposal. The week consists of a mixture of didactic lecture and small group sessions that are focused on improving each proposal. Small groups are composed of 2 to 4 participants, 2 to 4 faculty members, and 1 biostatistician. Time is incorporated into the program to allow informal discussions to occur on daily hikes or evening sessions.⁵ 

The CRTI program also involves the small groups reconvening twice: first during the annual ASH meeting in December, and then the following May at ASH headquarters. Small group reunions focus on participant presentations of updates related to their projects and career development. Recognizing the importance of mentorship, CRTI instituted a formal mentorship program in 2011 in which each participant is matched with a CRTI faculty member. Contact is prescribed to occur a minimum of 4 times throughout the year but may occur more frequently.

Three sets of surveys formed the basis of this analysis. First was a cross-sectional survey conducted in 2014 that was designed to capture outcomes (number of grants, publications, and current involvement in research) within the previous year from the date the survey was disseminated. Participants in CRTI over 8 years (2005-2012) were included in this survey and were asked 16 questions that captured demographic characteristics and the perceived influence of CRTI participation on his or her career in addition to outcomes. For this survey, our objectives were to describe academic success by CRTI graduates, defined as the number of peer-reviewed articles published, the number of research grants obtained, and whether the graduate was still involved in clinical research over the previous year. We also wanted to determine whether success was associated with strong agreement that CRTI facilitated his or her career development as an independent researcher and strong agreement that CRTI was instrumental to retaining him or her in hematology research in addition to demographic variables.

The second set of analyses consisted of annual pre– and post–summer workshop surveys conducted from 2009 to 2015. Not all questions were asked each year, and the response scale was not consistent; therefore, this analysis focused on data from 2014 to 2015. Questions about the impact of the summer workshop were asked on a 5-point Likert scale. For this analysis, we sought to determine whether confidence in research conduct improved immediately after the summer workshop in a variety of domains compared with baseline responses obtained at the beginning of the workshop.

The last analysis included surveys distributed in 2012 and 2013 that sought to evaluate the mentorship program established in 2011. This survey was not distributed in 2014 or 2015. Our objective was to describe products resulting from interaction with the mentor from the participant’s perspective in terms of academic productivity or career opportunities.

The following demographic covariates were captured at the time of application to the program: position during CRTI (fellow vs faculty), sex, adult vs pediatric, and benign vs malignant.

For the cross-sectional survey, descriptive statistics including percentages and medians were calculated for all demographic, predictor, and outcome variables. To describe the relationship between the outcome of number of grants (0 vs ≥1) and the dichotomous demographic and predictor covariates, a χ² test was performed. The Wilcoxon 2-sample test was used to examine the relationship between distribution of the number of articles and percent effort in research with the dichotomous demographic and predictor covariates. The pre/post questions assessing agreement or confidence in a variety of domains were dichotomized to strongly agree vs agree, neutral, disagree, and strongly disagree. Then the differences in the percentage of strongly agree between pre– and post–summer workshop were compared using a McNemar test. To describe the benefits of mentorship, the percentage that responded yes in each area was reported. P < .05 was used to define statistical significance. SAS version 9.4 was used for all statistical analysis (SAS Institute, Cary, NC).

The 2014 cross-sectional survey was distributed to 160 participants from 2005 to 2012, and 115 responded (72% response rate; 6 responded anonymously; therefore, no demographic data are available for these respondents). Of the 109 respondents, 42% had a benign hematology focus, 65% treated adults, 48% were male, 69% were Caucasian, 28% were Asian, and 3% were African American (Table 1). At the time of CRTI, 79% were fellows, and 57% participated in 2009 or later. Ninety percent described having an academic career at the time of survey completion.

Table 2 illustrates that 64% of respondents had some current collaborations as a result of CRTI, with most collaborations being with CRTI faculty (46%). Table 2 also illustrates that 63% of respondents strongly agreed that CRTI facilitated their career development as an independent researcher.

In terms of academic success in the previous year, 47% had at least 1 grant, and the median number of peer-reviewed articles was 2. Typically, respondents spent 50% of their time conducting research, and 95% were still involved in research.

Table 3 illustrates the relationship between having at least 1 grant in the previous year, the number of published articles in the previous year, and the percent effort in research with the participant’s reported impact of CRTI on his or her career as well as demographic variables. Respondents who strongly agreed that CRTI facilitated their career development as independent researchers were more likely to have had at least one grant in the previous year (55% vs 33%, P = .0264), more published articles (median 3 vs 2, P = .0170), and greater percent effort in research (median 50% vs 30%, P = .0002) compared with the other respondents (Table 3). Similarly, respondents who strongly agreed that CRTI was instrumental to retaining them in hematology research reported a higher median number of articles (4 vs 2, P = .0046) and a higher percent effort in research (median 60% vs 30%, P < .0001). Male respondents reported a higher median number of articles (3 vs 2, P = .0092) and a higher median percent effort in research (50% vs 40%, P = .0029) than females. Respondents who were junior faculty at the time of CRTI participation reported a higher median percent effort in research (60% vs 40%, P = .0270) than those who were fellows. Last, respondents who reported any CRTI collaborations reported a higher median number of articles (3 vs 1, P = .0006) and a higher median percent effort in research (50% vs 40%, P = .0439).

In 2014 and 2015, the same pre–post survey was distributed to 41 participants, and 38 responded (93% response rate). Table 4 illustrates that there were significant gains in confidence conducting clinical research immediately after the summer workshop compared with the baseline assessment in almost all measured domains. Areas in which there was a larger increase in confidence were: understanding the principals of research design and execution, preparing research for presentation, opportunities to collaborate, and ability to develop a sound research hypothesis.

The mentoring survey was distributed to 40 of the participants from the 2012 and 2013 cohorts, and 38 responded (95% response rate). Over the 12-month period after the initial week of training, almost three-fourths of the respondents felt that the CRTI mentor played a role in increasing their knowledge or skills in conduct of research. Some of the respondents reported that the mentor interaction resulted in posters, presentations, or manuscripts. Mentors also facilitated job changes and promotions (Table 5). In free-text comments describing other benefits of the CRTI mentor, respondents cited overcoming hurdles in patient enrollment, setting short-term goals, improving presentation skills, refining their curriculum vitae, networking, work–life balance, role-modeling, reviewing offer letters and suggesting requests, and increased awareness about funding and training opportunities.

In this comprehensive evaluation of ASH’s 1-year CRTI program, we found that, within a range of 2 to 9 years after the original CRTI experience, 90% of respondents remained in academic hematology and 95% were still involved in hematology research. We also found that most respondents continued to have collaborations owing to CRTI, and most respondents strongly agreed that CRTI facilitated career development and contributed to retention in hematology research. We also found that strong agreement that CRTI contributed positively to career development was significantly associated with greater academic success, and that males and faculty member respondents were more likely to have more time dedicated to research. In almost every measure of confidence in conducting clinical research, respondents significantly improved after the summer workshop.

Our finding that 90% of respondents remained in academic hematology is higher than a previous survey of medical oncology fellows, which reported that 60% remained in an academic setting.⁹  These high rates of retention in academic hematology and hematology research are reassuring, although it is difficult to know how much CRTI directly influenced these positive outcomes.

When exploring factors related to measures of academic productivity, we found that sex was a significant variable. Males published one more paper per year and tended to spend 10% more of their reported effort in research. This finding is consistent with previously published research in academic medicine. Among a European evaluation of fellows, more of the female scientists were married or had partners who were also in academics compared with male scientists who were married or had partners. The females who had children also typically took 2 to 3 months of leave, whereas the males did not take leave periods when their children were born.¹⁰  It is unknown whether this modest decrease in academic success among female respondents is solely related to factors associated with having and raising children. Nonetheless, this gender disparity is important to emphasize in program planning. Although we have addressed work–life balance during current CRTI sessions, we did not directly focus on gender differences. With 50% of faculty being female, many of whom are working mothers, we should encourage discussions about approaches that have been helpful to academic success from more senior female faculty.

Respondents who collaborated with either faculty or other participants from CRTI published more manuscripts and spent more time in research. It is uncertain whether this finding is confounded by the underlying attributes of those more likely to be academically successful or whether CRTI directly contributes to positive outcomes. Nonetheless, these findings suggest that we should more proactively encourage collaboration with CRTI faculty and non-CRTI investigators.

A strength of this evaluation was the multiple approaches we took to evaluate the CRTI program, including short- and medium-term outcomes. Another strength was the high response rate for all surveys.

As with any survey, this study had limitations. Respondents may have been biased to report positive feelings or gratitude for a program that was provided at no cost to them. However, given the duration of time between CRTI participation and the 2014 cross-sectional survey, we anticipate that most of the “honeymoon period” positivity would have subsided. Another limitation was our lack of consistently worded surveys, particularly for the pre–post summer workshop survey, limiting our ability to evaluate the program over a prolonged period. However, as CRTI matures, we plan on improving the standardization of assessments going forward. Third, although we reported that 95% of respondents are still involved in research, our definition of involvement was not well defined, and future surveys will more carefully define and measure how respondents are involved in research. Fourth, although we were encouraged by our response rate of 72% to the cross-sectional survey, nonresponders may have differed from responders and biased our results. Finally, we lacked an appropriate control group, and a common criticism of an analysis of CRTI benefits is that CRTI respondents were destined to be successful because they applied to the program and were selected as the most promising applicants to the program. Nonetheless, our comparison within our cohort to identify factors associated with academic success is an important and novel approach to program evaluation and improvement.

In the future, we recommend several changes to increase participation in repeated and longer-term follow-up evaluations of CRTI participants. We are creating a CRTI “community.” For the past 2 years, we have emphasized the planned repeated interactions of CRTI participants for not only evaluation but also celebrations of success. Plans for social media groups, a newsletter, and gatherings at the annual meeting will allow us to check contact information and encourage participation. We have edited the surveys for uniformity in rating scales and rewritten questions to capture objectives of the training program (grant writing, research design, etc.). Recognizing that a 1-year follow-up may be too short to capture the impact of CRTI, we recommend that the interval for repeated follow-up evaluation occur 3 years after attendance of the summer didactic session. Within this timeframe, we anticipate that participants will be able to submit career development awards and conduct some of their proposed projects.

The diversity of CRTI participants and respondents was low, and we have made efforts to increase the participation of underrepresented minorities to 15%. We have included representatives from ASH’s Committees on Promoting Diversity on the CRTI Oversight Committee to disseminate information about CRTI to more diverse members. We have presented to the Committee on Training to reach training program directors and asked diverse alumni to recommend CRTI to diverse fellows and junior faculty. We recommend that these efforts continue and encourage the course directors to continue to maintain a similar proportion of underrepresented minorities on the faculty as well.

In conclusion, the ASH CRTI is a well-received program that is associated with an increased retention of academic hematologists and increased continued participation in patient-oriented research compared with typical fellowship-trained hematologists. This evaluation, despite its limitations, supports the continued efforts to mentor and train junior academic hematologists in a structured approach. Efforts should focus on reducing gender disparity and enhancing CRTI collaborations. In the future, more uniform and longitudinal assessments will inform our methods of training.

The views expressed in written conference materials or publications and by speakers and moderators do not necessarily reflect the official policies of the Department of Health and Human Services, nor does mention of trade names, commercial practices, or organizations imply endorsement by the US Government.

Funding for this conference is made possible in part by National Institutes of Health, National Heart, Lung, and Blood Institute grant 1R13HL129488-01 and National Cancer Institute grant 5R25CA168526‐03.

Contribution: A.A.K. organized the manuscript, assisted in planning the analysis, and wrote the manuscript; S.K.V. performed the analysis, wrote the analysis methods, and reviewed the manuscript; J.E. and J.B. collected the data and reviewed the manuscript; K.C. assisted with organizing the paper and planning the analysis and reviewed the manuscript; and L.S. organized, wrote and reviewed the manuscript, performed the analysis, and interpreted the data.

Conflict-of-interest disclosure: The authors declare no competing financial interests.

Correspondence: Allison A. King, Washington University School of Medicine, 4444 Forest Park Blvd, CB8505, St. Louis, MO, 63108; e-mail: king_a@wustl.edu.