Context: The Health System Impact (HSI) Fellowship, an innovative training program developed by the Canadian Institutes of Health Research's Institute of Health Services and Policy Research, provides PhD-trained health researchers with an embedded, experiential learning opportunity within a health system organization.
Methods/Design: An electronic Delphi (eDelphi) study was conducted to: (1) identify the criteria used to define success in the program and (2) elucidate the main contributions fellows made to their organizations. Through an iterative, two-round eDelphi process, perspectives were elicited from three stakeholder groups in the inaugural cohort of the HSI Fellowship: HSI fellows, host supervisors and academic supervisors.
Discussion: A consensus was reached on many criteria of success for an embedded research fellowship and on several perceived contributions of the fellows to their host organization and academic institutions. This work begins to identify specific criteria for success in the fellowship that can be used to improve future iterations of the program.
Health systems of developed countries are facing complex and multifaceted challenges in meeting the needs of patients and providers. Sophisticated research and analytic expertise are critically needed to aid in the design, implementation and evaluation of evidence-based solutions (CIHR 2018c). A capacity-building initiative gaining international recognition is embedding PhD-trained researchers into health system organizations outside of traditional academic settings (Cheetham et al. 2018; Ghaffar et al. 2017; Marshall et al. 2014, 2016; Vindrola-Padros et al. 2017, 2019). According to previously published case studies, embedding academic researchers into health systems is a promising approach to facilitate the integration of evidence- and practice-informed research in policy implementation, build research capacity and meet stakeholder and patient needs (Cheetham et al. 2018; Ghaffar et al. 2017; Hamelin and Paradis 2018; Paradis et al. 2017; Vindrola-Padros et al. 2017). A more rigorous evaluation of the embedded research model is ongoing in the UK (National Institute for Health Research 2017).
In October 2017, the Canadian Institutes of Health Research's Institute of Health Services and Policy Research (CIHR-IHSPR) launched the Health System Impact (HSI) Fellowship. The HSI Fellowship is a $5.5 million national initiative to train the next generation of embedded researchers to promote health system transformation in Canada (Bornstein et al. 2018; Tamblyn et al. 2016). The HSI Fellowship aims to: (1) prepare PhD graduates for a wide variety of careers (both inside and outside of academia) by enriching post-doctoral training with opportunities for "real-world" experience within health system organizations and (2) demonstrate the value of adding PhD-trained researchers to organizations outside traditional academic settings. More information about this dynamic program is available in the CIHR funding opportunity (CIHR 2018c). A central facet to this program is the co-mentorship of fellows by high-level, senior leaders in both the academic and the health system setting.
Although embedded research training opportunities are gaining popularity, there is limited Canadian research evaluating the preliminary impacts of these fellows within the health system organization. To the best of our knowledge, only one narrative review has explored the role embedded researchers could play in quality improvement efforts in healthcare settings (Vindrola-Padros et al. 2017). Although the review included a wide array of organizations across the world, there was no Canadian representation, an exclusive focus on quality improvement and little emphasis on the training of early-career researchers. Therefore, more evidence is needed from the Canadian context to inform program development, assess the impact of embedded fellows on health systems and facilitate the success of future fellows.
This present study fills a gap in the literature by: (1) identifying the criteria of a successful fellowship as defined by HSI fellows, host supervisors and academic supervisors and (2) elucidating each group's perspectives on key contributions made by fellows to their respective academic and health system organizations after the first year of operation of the HSI Fellowship.
An eDelphi (Diamond et al. 2014; Hasson et al. 2000) study was conducted to elicit the perspectives of fellows, host supervisors and academic supervisors regarding: (1) the criteria that each group uses to define a successful fellowship and (2) key contributions fellows have made in their academic and health system organizations. The eDelphi method was chosen due to the distribution of fellows and their host organizations and academic institutions across Canada.
The entire 2017 cohort of fellows (n = 46), their host organization supervisors and academic supervisors were invited to complete the survey (CIHR 2018b). As the first cohort of this innovative program, participants were considered the experts of their experience. Data were collected through two rounds of online questions administered by CIHR-IHSPR. These were sent four weeks apart, with two weekly follow-up reminders sent to non-responders.
In Round 1, two open-ended questions were administered to each of the three stakeholder groups asking their views on the main contributions of the fellow to their host organization and the criteria they would use to define a successful fellowship. For each stakeholder group, a team of three researchers analyzed the qualitative data generated in Round 1 using content analysis to identify themes (Hsieh and Shannon 2005). Items to be included in Round 2 were developed by each team from the list of themes for their respective stakeholder group. Next, one researcher who was not part of any of the three qualitative analysis groups identified themes common to the three groups as well as unique themes in each group. Prior to distribution of the Round 2 survey, all the authors met to discuss the themes. From this discussion, it was decided that common themes would be presented to all three stakeholder groups in the Round 2 survey to facilitate comparison across the groups and themes unique to one group would be presented to that group only.
In Round 2, participants were asked to rank their agreement with the themes generated in Round 1 on a 5-point Likert-type scale (range: 1 to 5). Descriptive statistics (mean and standard deviation [SD]) were used to assess the level of agreement (item endorsement). Although there is no universally agreed upon proportion to define consensus, a conservative threshold of 70% (mean 3.5/5) was set a priori (Hasson et al. 2000). All quantitative analyses were performed using R software (R Core Team 2019). Ethical clearance for the study was approved by the University of Waterloo Ethics Review Board (#32105).
Of 134 potential participants, 113 completed the first survey, resulting in an overall 85% response rate for Round 1 (fellows, n = 42/46 [91%]; host supervisors, n = 34/43 [79%]; academic supervisors, n = 37/45 [82%]). The Round 2 survey was only sent to individuals who participated in Round 1. In Round 2, 78 of 113 participants completed the survey, resulting in an overall 69% response rate (fellows, n = 36/42 [86%]; host supervisors, n = 21/34 [62%]; academic supervisors, n = 21/37 [57%]).
Criteria of a successful fellowship experience
Seven common criteria of a successful fellowship were identified across two or more stakeholder groups in Round 1 (Figure 1). In addition to common criteria across groups, some criteria of success were identified by only one group of stakeholders. Fellows identified better understanding of their desired career trajectory (employment preparedness) as a success criterion. Host supervisors identified four additional criteria, including capacity of adaptation to different working environments, openness to new learning experiences, team cohesion (collegial interaction) and positive experience by the fellow. Finally, academic supervisors considered measurable impacts within the health system and building a foundation for fellow research success as criteria for defining success.
All common criteria of success reached consensus (Figure 2). Among the common success criteria, academic supervisors had the highest level of agreement on all items, followed by fellows and host supervisors, respectively, with the exception of better knowledge of health systems and the role of health policy. The sole criterion with higher agreement among host supervisors versus fellows was advancing the host organization's mandate or priorities.
Among the unique criteria for success, only difficult to define success (mean 2.09; SD 0.87) did not reach the level of consensus. Fellows rated better understanding of their desired career trajectory to an average score of 4.22 (SD 0.87). The average scores of the host supervisors' specific criteria were 4.05 (SD 0.83) for capacity of adaptation to different working environments; 4.15 (SD 0.75) for openness to new learning experiences; 4.15 (SD 0.88) for team cohesion; and 4.40 (SD 0.80) for positive experience by the fellow. The mean scores for the academic supervisors' specific criteria were 3.52 (SD 1.03) for measurable impacts within the health system and 4.24 (SD 0.89) for building the foundation for fellow research success.
Key contributions of the fellows – ROUND 1
Seven common themes emerged among all three groups highlighting the key contributions of fellows to their respective organizations (Figure 3). In addition to the common themes, the HSI Fellowship impact project and grantsmanship (applying for or obtaining funding) were contributions identified only by fellows. The strong work ethic of the fellow was a contribution only identified by the host supervisors. Academic supervisors identified two additional contributions: revealing the strengths and weaknesses of current academic training and identifying challenges and barriers to learning health systems.
The results of Round 2 are presented in Figure 3. All three stakeholder groups had the highest level of agreement on the same three common key contributions (although the order of rankings differed). These included: research/analytic skills and expertise, networking, building relationships and collaboration and new perspectives, interventions, processes, methods or strategies. In comparison with host supervisors, the fellows' level of agreement was higher across all items with respect to key contributions to their respective organization. Whereas both fellows and academic supervisors reached consensus on leadership and providing mentorship or education as key contributions made by the fellow, host supervisors did not. All three groups strongly disagreed that fellows provide limited to no contribution to their respective organizations. As this item was negatively worded, lack of consensus in this case actually represents agreement across the three groups that fellows made some contribution to their host organization.
Among the unique themes identified by respondents, only grantsmanship (mean 2.94; SD 1.59), identified by fellows, did not reach the level of consensus. All other unique themes reached consensus in their respective groups. Fellows rated the HSI Fellowship impact project to an average score of 4.14 (SD 1.09). The average score of host supervisors for strong work ethic was 4.29 (SD 0.56). The mean score for key contributions identified by academic supervisors were 3.95 (SD 0.80) for revealing the strength and weakness of the current academic training and 3.65 (SD 0.88) for identifying challenges and barriers to learning health systems.
The objectives of this study focused on identifying the criteria of success for an embedded research fellowship and the perceived contributions of the fellows to their host organization and academic institutions. To our knowledge, this study is the first to use an eDelphi approach to examine and compare the experiences of three stakeholder groups participating in an embedded research training opportunity. Insights gained through this research will be explored further and implications for each group, and the program as a whole, will be presented.
A high level of agreement was found across the participant groups on the common criteria of success as well as fellows' contributions to respective organizations. Although fellows and academic supervisors strongly agreed, the primary differences in the perspectives were found among the host organizations. This may be attributed to the heterogeneity of the host organizations as they have different objectives and more variability in their characteristics, structures and organizational cultures than academic institutions. Host organizations included in the inaugural cohort of the HSI Fellowship represented over 30 different public, private for-profit, not-for-profit and Indigenous health organizations (CIHR 2018b). The high level of agreement, particularly within fellows and academic supervisors, may be a result of their familiarity with a post-doctoral program design and program core competencies.
Criteria of a successful fellowship experience
Consensus was shared among fellows and host and academic supervisors pertaining to their criteria for HSI Fellowship program success; however, there was variability of agreement within groups for some criteria. Although reaching the threshold for consensus across all three groups, academic or research productivity as a criterion of success demonstrated less within-group agreement for fellows and host supervisors. This is not surprising given the HSI Fellowship's stated program objectives, which focus primarily on experiential learning and enriched competency development outside the traditional academic setting. Despite calls for new or additional metrics to assess real-world research impact, peer-reviewed publications, grants and teaching continue to be valued above demonstrable improvements in health and health systems (Cruz Rivera et al. 2017; Robert 2016). This misalignment of program goals and contemporary academic career path rewards may create tension between academic and host supervisors, leading to unmanageable workloads as fellows attempt to succeed in both worlds. In the UK, embedded researchers have highlighted similar challenges of defining their project scope and maintaining academic identity (Vindrola-Padros et al. 2017, 2019).
Contributions of the fellows
Academic supervisors and fellows had a high level of agreement on contributions of the fellow, reinforcing the fact that the design of the HSI Fellowship program (Tamblyn et al. 2016) aligns with the core competencies identified for the program (Bornstein et al. 2018). All three groups reached consensus that a key contribution of fellows included research and analytic skills/expertise, which is consistent with previous literature on the characteristic features of embedded research (Vindrola-Padros et al. 2017).
Fellows and academic supervisors reached consensus on all the common key contributions of fellows identified in Round 1; however, host supervisors did not reach consensus on fellows' contributions for leadership and mentorship. This may reflect the differences in culture between academic institutions and host organizations. For example, some host organizations may not perceive fellows as leaders or mentors because they have established hierarchies that are based on tenure within their or similar organizations rather than academic qualifications. In academic settings, however, post-doctoral fellows are in leadership and mentorship roles, including teaching, supervising theses and research projects and/or more general mentorship for undergraduate and graduate students (Åkerlind 2005, 2009; Chen et al. 2015). It may also reflect the program's design, which featured leadership and mentorship as a core competency for fellows to develop with the mentorship and support of their health system and academic supervisors.
Strengths and limitations
There are several strengths to the present study. First, this study included perspectives from the three key stakeholder groups collaborating in an innovative training initiative to better understand key features of success and fellows' contributions. By examining the program from these three different perspectives, we have been able to elucidate areas where there may be tensions or misalignment between stakeholders. This study also begins to establish specific criteria for success in the fellowship that can improve future iterations of the program.
There are also several limitations of the current study. First, it is unclear how findings may have been affected if we had presented a list of all themes, both common and unique, to all participant groups in Round 2 of the eDelphi. Second, although there are structural differences in the three programs (e.g., one-year, two-year, start-up grant) (CIHR 2018a) included in the inaugural cohort, the study design did not enable us to stratify results based on fellowship type. Furthermore, the application processes were different. Either the organization formed a relationship and pursued work with a fellow (CIHR-IHSPR 2016) or the fellow forged a relationship with the preferred organization (CIHR 2018c). It remains to be determined if both approaches are of similar benefit. Variations within and across the three participant groups and the degree of consensus must be interpreted with caution.
Third, there were limited options presented to respondents that may not have captured their full range of perspectives. Although this is a general limitation of Delphi studies (Keeney et al. 2001), in this study, it was not possible to disentangle contributions based on fellows' previous experience and skills versus those based on program elements. Fourth, this study was conducted after the first year of the program, while many of the fellowships were still in progress or recently completed. Therefore, it may have been too early to gain a complete understanding of the criteria for success or the value of the fellows' roles. Finally, there may also have been differences between respondents and non-respondents for which there is insufficient consideration.
Although findings confirm high agreement among the three groups on criteria for success and fellows' contributions, discrepancies between the three groups suggest opportunities for program improvement. First, there is an opportunity to strengthen communication regarding the intention of the fellowship and facilitate discussion of the criteria for success among members of each fellowship triad. This includes structured communication between the funder and all parties at the program outset and communication between fellows and their hosts and academic supervisors throughout the fellowship. Currently, the program requires that the three parties complete a professional development plan at baseline and at three and 12 months. This plan is an important guide for goal-setting (for the fellowship and long-term career), self-reflection and supervisor assessment. An expectation management activity might be an interesting complement to reflect on the tripartite relationship in addition to the fellow's career.
Our findings highlight the heterogeneity within groups and the potential for competing priorities of the host organizations and academic institutions to impact fellowship success. Existing program orientation and training provided for all three groups may be enhanced based on this new knowledge concerning the criteria for success and fellows' key contributions.
This study is a first step toward gaining an understanding of the experiences of a complex, pan-Canadian embedded researcher training program from the perspectives of three key stakeholder groups. A consensus was reached on many criteria of success for an embedded research fellowship and on several perceived contributions of the fellows to their host organization and academic institutions. Future research should include a qualitative, in-depth exploration of the experiences of HSI fellows, host supervisors and academic supervisors to expand on emerging definitions of success and the program as a contributor to learning health systems.
Contexte : Les bourses d'apprentissage en matière d'impact sur le système de santé (BAIS) – un programme de formation novateur mis au point par l'Institut des services et des politiques de santé des Instituts de recherche en santé du Canada – offrent aux chercheurs titulaires d'un doctorat des occasions d'apprentissage expérientiel enchâssées dans un organisme de santé.
Méthode/concept : Nous avons mené une enquête Delphi électronique pour : (1) identifier les critères employés pour définir le succès du programme; et (2) dégager les principales contributions apportées par les boursiers à leur organisation. Un processus Delphi itératif mené à deux reprises a permis de dégager les points de vue de trois principaux groupes liés à la première cohorte des BAIS : les boursiers, les superviseurs des organismes d'accueil et les superviseurs universitaires.
Discussion : Un consensus a été atteint sur plusieurs critères liés au succès des bourses de recherche enchâssées ainsi que sur la perception des contributions apportées par les boursiers à leur organisme d'accueil et à leur université. Ce travail permet de dégager les critères spécifiques au succès des bourses, lesquels pourront être utilisés pour améliorer d'éventuelles moutures du programme.
About the Author(s)
Margaret Saari, RN, PhD, SE Research Centre, SE Health and School of Public Health and Health Systems, University of Waterloo, Waterloo, ON
Katie Aubrecht, PhD, Nova Scotia Health Authority Continuing Care-Research, Mount Saint Vincent University, Halifax, NS
Chantelle Bailey, PhD, Canadian Nurses Association and the Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, Toronto, ON
Ivy Cheng, MD, MSc, PhD, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON
Mark Embrett, PHD, Canada Health Infoway, McMaster University, Toronto, ON
El Kebir Ghandour, MD, PhD, CISSS Chaudière-Appalaches, Université Laval, Lévis, QC
Jennie Haw, PhD, Canadian Blood Services, Western University, Ottawa, ON
Andriy Koval, PhD, Observatory for Population and Public Health, University of British Columbia, Vancouver, BC
Rebecca H. Liu, PhD, University of Ottawa, Region of Peel Public Health, Toronto, ON
Kiran Pohar Manhas, JD, PhD, Alberta Health Services, University of Alberta, Calgary, AB
Farah N. Mawani, MSc, PhD, MAP Centre for Urban Health Solutions St. Michael's Hospital, Faculty of Environmental Studies, York University, Toronto, ON
Jennifer Mcconnell-Nzunga, PhD, Child Health BC, University of Victoria, Vancouver, BC
Kadia Petricca, MSc, PhD, North York General Hospital, Department of Research and Innovation, University of Toronto, IHPME, Toronto, ON
Meaghan Sim, RD, PhD, Nova Scotia Health Authority; Healthy Populations Institute, Dalhousie University, Halifax, NS
Deepa Singal, PhD, Manitoba Centre for Health Policy, University of Manitoba, Winnipeg, MB
Ania Syrowatka, PhD, Canadian Foundation for Healthcare Improvement, University of Toronto, Institute of Health Policy, Management and Evaluation, Toronto, ON
Jonathan Lai, MSc, PhD, Centre for Innovation in Autism and Intellectual Disabilities, Miriam Foundation, co-funded by Mitacs, School of Physical and Occupational Therapy, McGill University, Montreal, QC
Correspondence may be directed to: Marc-André Blanchette, DC, PhD, Département de chiropratique, Université du Québec à Trois-Rivières; Epidemiology, Biostatistics and Occupational Health, McGill University; Institut national d'excellence en santé et services sociaux; e-mail: email@example.com.
The authors would like to thank Meghan McMahon for her invaluable help during our respective fellowships and for her logistical support, without which this study could not have been completed.
Åkerlind, G.S. 2005. "Postdoctoral Researchers: Roles, Functions and Career Prospects." Higher Education Research & Development 24(1): 21–40.
Åkerlind, G.S. 2009. "Postdoctoral Research Positions as Preparation for an Academic Career." International Journal for Researcher Development 1(1): 84–96.
Bornstein, S., M. Heritage, A. Chudak, R. Tamblyn, M. McMahon and A.D. Brown. 2018. "Development of Enriched Core Competencies for Health Services and Policy Research." Health Services Research 53(Suppl. 2): 4004–23. doi:10.1111/1475-6773.12847.
Canadian Institutes of Health Research (CIHR). 2018a. "Canadian Health Services and Policy Research Alliance – Modernizing Training in Health Services and Policy Research." Retrieved July 24, 2019. <http://www.cihr-irsc.gc.ca/e/50024.html>.
Canadian Institutes of Health Research (CIHR). 2018b. "2017 Health System Impact Fellow Profiles." Retrieved July 24, 2019. <http://www.cihr-irsc.gc.ca/e/50660.html>.
Canadian Institutes of Health Research (CIHR). 2018c. "The Health System Impact Fellowship." Retrieved July 24, 2019. <http://www.cihr-irsc.gc.ca/e/51211.html>.
Cheetham, M., A. Wiseman, B. Khazaeli, E. Gibson, P. Gray, P. Van der Graaf et al. 2018. "Embedded Research: A Promising Way to Create Evidence-Informed Impact in Public Health?" Journal of Public Health 40(Suppl. 1): i64–70. doi:10.1093/pubmed/fdx125.
Chen, S., L. McAlpine and C. Amundsen. 2015. "Postdoctoral Positions as Preparation for Desired Careers: A Narrative Approach to Understanding Postdoctoral Experience." Higher Education Research & Development 34(6): 1083–96.
CIHR Institute of Health Services and Policy Research (CIHR-IHSPR). 2016. "Training Modernization Start-Up Grant in Health Services and Policy Research." Retrieved July 24, 2019. <https://www.researchnet-recherchenet.ca/rnr16/vwOpprtntyDtls.do%3Fprog=2540%26view=currentOpps%26type=EXACT%26resultCount=25%26sort=program%26all=1%26masterList=true>.
Cruz Rivera, S., D.G. Kyte, O.L. Aiyegbusi, T.J. Keeley and M.J. Calvert. 2017. "Assessing the Impact of Healthcare Research: A Systematic Review of Methodological Frameworks." PLoS Medicine 14(8): e1002370. doi:10.1371/journal.pmed.1002370.
Diamond, I.R., R.C. Grant, B.M. Feldman, P.B. Pencharz, S.C. Ling, A.M. Moore et al. 2014. "Defining Consensus: A Systematic Review Recommends Methodologic Criteria for Reporting of Delphi Studies." Journal of Clinical Epidemiology 67(4): 401–09. doi:10.1016/j.jclinepi.2013.12.002.
Ghaffar, A., E.V. Langlois, K. Rasanathan, S. Peterson, L. Adedokun and N.T. Tran. 2017. "Strengthening Health Systems through Embedded Research." Bulletin of the World Health Organization 95(2): 87. doi:10.2471/BLT.16.189126.
Hamelin, A.-M. and G. Paradis. 2018. "Population Health Intervention Research Training: The Value of Public Health Internships and Mentorship." Public Health Reviews 39: 6. doi:10.1186/s40985-018-0084-9.
Hasson, F., S. Keeney and H. McKenna. 2000. "Research Guidelines for the Delphi Survey Technique." Journal of Advanced Nursing 32(4): 1008–15.
Hsieh, H.F. and S.E. Shannon. 2005. "Three Approaches to Qualitative Content Analysis." Qualitative Health Research 15(9): 1277–88. doi:10.1177/1049732305276687.
Keeney, S., F. Hasson and H.P. McKenna. 2001. "A Critical Review of the Delphi Technique as a Research Methodology for Nursing." International Journal of Nursing Studies 38(2): 195–200.
Marshall, M., L. Eyre, M. Lalani, S. Khan, S. Mann, D. de Silva and J. Shapiro. 2016. "Increasing the Impact of Health Services Research on Service Improvement: The Researcher-in-Residence Model." Journal of the Royal Society of Medicine 109(6): 220–25. doi:10.1177/0141076816634318.
Marshall, M., C. Pagel, C. French, M. Utley, D. Allwood, N. Fulop et al. 2014. "Moving Improvement Research Closer to Practice: The Researcher-in-Residence Model." BMJ Quality & Safety 23(10): 801–05.
National Institute for Health Research. 2017. System Reference Number 122620 / NETSCC ID 16/52/21. Retrieved July 24, 2019. <https://njl-admin.nihr.ac.uk/document/download/2014583>.
Paradis, G., A.-M. Hamelin, M. Malowany, J. Levy, M. Rossignol, P. Bergeron et al. 2017. "The University-Public Health Partnership for Public Health Research Training in Quebec, Canada." American Journal of Public Health 107(1): 100–04. doi:10.2105/ajph.2016.303529.
R Core Team. 2019. The R Project for Statistical Computing. Retrieved July 24, 2019. <https://www.r-project.org>.
Robert, R.J. 2016. "Embedding Research in the Learning Health System." HealthcarePapers 16(Special Issue): 30–35. doi:10.12927/hcpap.2016.24724.
Tamblyn, R., M. McMahon, J. Nadigel, B. Dunning and E. Drake with the Institute of Health Services and Policy Research Advisory Board. 2016. "Health System Transformation through Research Innovation." HealthcarePapers 16(Special Issue): 8–19. doi:10.12927/hcpap.2016.24719.
Vindrola-Padros, C., L. Eyre, H. Baxter, H. Cramer, B. George, L. Wye et al. 2019. "Addressing the Challenges of Knowledge Co-Production in Quality Improvement: Learning from the Implementation of the Researcher-in-Residence Model." BMJ Quality & Safety 28(1): 67–73. doi:10.1136/bmjqs-2017-007127.
Vindrola-Padros, C., T. Pape, M. Utley and N.J. Fulop. 2017. "The Role of Embedded Research in Quality Improvement: A Narrative Review." BMJ Quality & Safety 26(1): 70–80. doi:10.1136/bmjqs-2015-004877.
Be the first to comment on this!