Developing Competencies for Health System Impact: Early Lessons learned from the Health System Impact Fellows
Background: The Health System Impact (HSI) Fellowship program provides highly qualified post-doctoral fellows studying health services and policy research (HSPR) with opportunities for experiential learning, enriched core competency development and mentorship from senior-level leaders within health system organizations. Its overall aim is to prepare post-doctoral fellows with the research and professional skills, experiences and networks to make meaningful and impactful contributions in careers in academic and applied health system settings.
Objective: This study examined whether this HSI Fellowship program has contributed to the development of enriched core competencies in HSPR.
Methods: A competency assessment tool was developed and administered to the 46 fellows and their health system and academic supervisors from the inaugural HSI Fellowship cohort. Fellows' self-assessments at baseline, three months and 12 months were analyzed, along with supervisors' assessments at three and 12 months. Descriptive analyses were used to examine competency development over time. Differences by gender and between supervisor and fellow ratings were analyzed.
Results: HSI fellows' self-assessments indicate that they strengthened their skills in all 10 enriched core competencies. Supervisors' assessments of the fellows' competencies also improved from baseline to 12 months. Gender differences at baseline disappeared by the 12-month assessment.
Conclusion: The HSI Fellowship provides an opportunity to develop the full suite of enriched core competencies, particularly in competency domains that are not currently emphasized in HSPR doctoral curriculum.
An enriched core competency framework for health services and policy research (HSPR) doctoral and post-doctoral training was introduced in Canada in 2016 (Bornstein et al. 2018; CHSPRA TMWG 2015). Its goal was to align the competencies emphasized in doctoral curriculum with the skills needed to maximize the impact of health services and policy research (HSPR) graduates in the diverse range of employment sectors and roles they are entering based on employment trends (see McMahon et al. 2019b) and emerging career opportunities, such as embedded scientist roles within organizations seeking to be learning health systems (Reid 2016). The enriched core competency framework maintains emphasis on rigorous scholarly training and research and analytic skills but also introduces a core suite of professional skills that are demanded in the non-academic labour market and necessary to effect change and make an impact in applied health system settings (Figure 1).1
The inaugural cohort of 46 Health System Impact (HSI) fellows was the first to pilot a training program for the enriched core competencies. The HSI Fellowship program provides fellows with an experiential learning opportunity within a health system organization (defined as a public, private, or not-for-profit organization based in Canada with a mandate focused on health, such as a ministry of health, regional health authority, health service delivery organization, health quality council, public health unit, health charity, health professional association, consulting firm with a health-focused mandate, etc.) where they spend the majority of their time, as well as protected time for academic research at a Canadian university; co-supervision and mentorship from a health system leader within the health system organization and an academic supervisor with expertise in HSPR at the university; a professional development training allowance; and participation in a national cohort of fellows and health system and academic leaders that includes an annual in-person National Cohort Retreat and quarterly webinar training sessions in enriched core competencies (for a full description, see CIHR-IHSPR ; McMahon et al. [2019a]; and McMahon and Tamblyn [2019c]). Each of these program elements is deliberately aligned with the suite of enriched core competencies and provides opportunities for fellows to complement their research skills with professional competencies and enhanced skills.
At the outset of the fellowship, fellows identify three enriched core competencies from the suite of 10 (see Figure 1) to target for development. Fellows create a professional development plan (PDP) and meet with their health system and academic supervisors to discuss their mentorship and support. To track competency development over time, fellows use a standardized framework to self-assess their competencies at the start of their fellowship (baseline) and two subsequent time points. Each fellow's health system and academic supervisor also assesses the fellow's competency strength and development.
The competency assessments provide an opportunity to learn whether the HSI Fellowship is contributing to the development of enriched core competencies and preparing a new cadre of doctoral graduates equipped with the research and professional skills to make an impact in a broad range of employment sectors and roles.
The objective of the present study was to analyze whether the inaugural cohort of HSI fellows' competencies evolved over the first year of the fellowship, if development occurred primarily in the self-identified target areas, and to assess the degree of alignment between fellows' and supervisors' competency assessments.
To measure the fellows' development of the enriched core competencies over the first year of the HSI Fellowship, the CIHR HSI Fellowship Professional Development Plan and Competency Tracking Tool assessments were analyzed.
The competency tracking tool was created by the Canadian Institutes of Health Research Institute of Health Services and Policy Research (CIHR-IHSPR) in collaboration with the Training Modernization Working Group of the Canadian Health Services and Policy Research Alliance (CHSPRA) based on a review of the literature and an environmental scan of existing competency assessment tools used in relevant fields. 2 The draft tool was presented to a group of deans and directors of 10 of Canada's HSPR doctoral training programs and health system leaders from across the country3 for feedback and refinement prior to finalizing. It was introduced to the HSI Fellows and their supervisors via an orientation webinar in the first month of the fellowship prior to its implementation for baseline assessment. Completion of the competency assessments was a mandatory requirement of the HSI Fellowship.
Competency assessments were completed using an online survey platform at three defined time points: at baseline, three months and 12 months (see Figure 2 for a timeline overview). Questions and response scales were consistent in all assessments to allow for the analysis of change over time and comparisons between fellow and supervisor ratings. At baseline, fellows were asked to:
- identify their top three competencies for development over the course of their fellowship;
- rate the strength of their competence on a Likert-type scale (range: 1 to 5) in each of the 10 competencies relative to their perception of others' competence in their peer group; and
- discuss their PDP with their health system and academic supervisors, who were required to indicate how they intended to help the fellow achieve his or her goals.
At three months, fellows and their health system and academic supervisors completed the assessment (separately). Fellows were asked to indicate which competencies they had the opportunity to strengthen over the past three months and, for each competency, to rate how much they felt they learned (response options: a lot, a fair bit, a little bit, nothing). Similarly, health system and academic supervisors rated the extent of their fellow's development in each of the competency domains (response options: a lot, a fair bit, a little, not at all) and rated the overall strength of the fellow's competence in each domain using the 5-point Likert-type scale.
At 12 months, fellows and supervisors rated the fellows' competency strength in each domain using the 5-point Likert-type scale. Additionally, fellows were asked to rate the importance of nine program-specific enablers to their competency development. These included: (1) being embedded in a health system organization, (2) mentorship from a health system leader, (3) working on an impact-oriented program of work of high relevance to a health system organization, (4) mentorship from the academic supervisor, (5) co-mentorship (team-based approach) from the health system and academic supervisors, (6) fellow-to-fellow interactions, (7) protected time for academic research, (8) the professional development training allowance and (9) the National Cohort Retreat and quarterly webinar training sessions.
Descriptive analyses were used to assess competency development over time for all fellows, as well as by gender. Gender differences were assessed using a two-sample t-test, and differences between the three-month and 12-month assessments were assessed using a paired-sample t-test in the sample of fellows for whom both a baseline and a 12-month assessment were available (n = 38/46 [82%]). The magnitude of change over time was assessed by calculating the difference in self-rated ability at baseline and 12 months, expressed as a per cent change of the difference relative to baseline. The same methods were used to assess differences between health system and academic supervisor ratings and differences between the supervisors' three- and 12-month ratings. Analyses and figures were conducted using Excel.
Overall, all 46 fellows completed the baseline and three-month competency assessments and 38 completed the 12-month assessment. Of the nine non-responders, five had transitioned into employment positions or were on leave at the time of the 12-month assessment. Seventy-eight of 92 supervisors completed the three-month assessment (84.8%), and 69 completed the 12-month assessment (75%). For all but two fellows, at least one of the health system and academic supervisors completed a three- and a 12-month assessment.
At baseline, fellows were asked to identify their top three competencies to develop. Figure 3 illustrates some consistency in fellows' top two selections, which were leadership, mentorship and collaboration (identified by 60% of fellows) and change management and implementation (identified by 49% of fellows). There was heterogeneity thereafter in fellows' self-identified target competencies.
Figure 3 also shows that although fellows identified three competencies to develop at the outset of their fellowship, by the three-month mark, approximately 80% of fellows reported that they had the opportunity to strengthen their competence in all 10 domains, and by 12 months, 85% had. Even competencies that were not prevalent selections at the start of the fellowship witnessed considerable development over time. For example, at baseline, only 15% and 17% of fellows identified interdisciplinary work and networking, respectively, as a top three target competency for development, but by 12 months, more than 97% of fellows reported that they had the opportunity to strengthen their skills in these domains.
Table 1, available here, presents fellows' self-assessed ratings of their strength in each competency domain at baseline and 12 months. At baseline, the mean overall cohort rating was highest in the analysis of data, evidence and critical thinking and lowest in the change management and implementation competencies. By 12 months, the mean cohort rating in all 10 competencies had significantly increased (p < 0.01), ranging from a 9.5% increase in the analysis of data competency to a 44% increase in the change management and implementation competency. Next to change management, self-assessed competence in understanding health systems and the policy-making process exhibited the greatest increase over time (31.2% increase), followed closely by networking (28.8% increase) and dialogue and negotiation (29.2% increase).
At baseline, the mean competency assessment rating for female fellows (n = 38) was higher than the mean for male fellows (n = 9) in six of the 10 domains, although statistically significantly higher in only two: project management and networking (see Table 1). At 12 months, there were no statistically significant gender differences in competency assessment ratings.
Table 2, available here, presents the supervisor ratings of their fellows' competency strength in each domain at three and 12 months, overall and by supervisor type. At both time points, the mean overall supervisor ratings were highest in networking and analysis of data, evidence and critical thinking and lowest in change management. As with the fellows' self-assessments, the overall supervisor assessments of fellows' competency strength increased significantly over time in each domain (p < 0.01), ranging from a 12.2% increase in dialogue and negotiation to a 22% increase and in the analysis and evaluation of health policies and programs. When assessed separately by supervisor type, health system supervisor assessments increased significantly over time in all 10 competency domains and academic supervisor assessments increased significantly in all but two domains: project management and dialogue and negotiation.
Differences between health system and academic supervisor competency assessments were not statistically significant except for the three-month assessment of the leadership competency and the 12-month assessment of interdisciplinary work where, in both cases, the mean academic supervisor assessment was higher (see Table 2).
Table 3 compares the fellow and overall supervisor competency assessments at 12 months and shows that in seven of 10 competencies, the supervisors' assessment of the fellows' competency strength is higher than the fellows' self-assessment; however, the differences are not statistically significant.
|TABLE 3. Fellows' and supervisors' competency assessments at 12 months|
|Competency||Fellows Mean (SD) (n = 38)||Supervisors Mean (SD) (n = 69)||Δ Mean, baseline to 12 months (fellows)||Δ Mean, 3 to 12 months (supervisors)|
|Leadership, mentorship and collaboration||3.81 (0.61)||4.00 (0.73)||0.61||0.5|
|Change management and implementation||3.50 (0.95)||3.45 (0.93)||1.07||0.41|
|Project management||3.68 (0.93)||3.96 (0.93)||0.59||0.46|
|Interdisciplinary work||4.21 (0.70)||4.13 (0.82)||0.61||0.49|
|Networking||4.03 (0.72)||4.26 (0.72)||0.9||0.53|
|Dialogue and negotiation||3.76 (0.79)||3.67 (0.93)||0.85||0.4|
|KT, communication and brokerage||3.76 (0.71)||3.97 (0.82)||0.76||0.55|
|Analysis and evaluation of health policies/programs||3.82 (0.77)||3.94 (0.94)||0.33||0.71|
|Analysis of data, evidence and critical thinking||4.14 (0.66)||4.30 (0.77)||0.43||0.58|
|Understanding health systems and the policy-making process||3.74 (0.89)||3.87 (0.78)||0.89||0.68|
|KT = knowledge translation.|
Table 4 shows that of nine defined HSI Fellowship program enablers designed to support fellows' competency development, fellows rated being embedded in a health system organization (mean rating 4.72, SD 0.57) and having a dedicated professional development training allowance (4.73, SD 0.56) as the top two enablers, followed by mentorship from the health system supervisor (4.35, SD 0.98). All nine enablers were rated at least moderately important (mean rating of 3 or higher).
|TABLE 4. Fellow ratings of the importance of nine program elements designed to enhance competency development|
|Program element||Mean rating (SD)|
|The professional development training allowance||4.73 (0.56)|
|Being embedded in a health system organization||4.72 (0.57)|
|Mentorship from my health system supervisor||4.35 (0.98)|
|Working on an impact-oriented project of high importance to a health system organization||4.32 (0.94)|
|Mentorship from my academic supervisor||4.24 (0.95)|
|Co-mentorship (team-based approach) from my health system and academic supervisors||4.12 (1.19)|
|Protected time for my academic research||3.91 (1.26)|
|National Cohort Retreat and quarterly webinar training sessions||3.84 (1.09)|
|Fellow-to-fellow interactions||3.35 (1.25)|
|Note: Ratings are based on a 5-point Likert-type scale.|
Discussion and Conclusion
This study examined whether the HSI Fellowship contributed to the development of enriched core competencies in HSPR, if development occurred primarily in fellows' self-identified target competencies and the extent to which fellows' and supervisors' competency assessments align. The study's overall goal is to advance knowledge about the effectiveness of embedded fellowship training programs in fostering the development of competencies that are demanded in the labour market and needed to create a cadre of research-savvy health system leaders with the skills to drive health system innovation and transformation.
This study found that HSI fellows strengthened their skills in all 10 enriched core competencies over the first year of the fellowship. Fellows' baseline assessments were, in general, lower in the professional competency domains that are not yet emphasized in doctoral training programs (e.g., change management) and higher in the traditional research and analytic competencies that are the focus of doctoral curriculum (e.g., analysis of data). However, fellows' self-assessment of their competencies improved from baseline to 12 months in both the professional and research competencies, as did the supervisors' assessments of the fellows' competencies. Improvements were observed in fellows' top three targeted competencies and in less-targeted competencies. Gender differences at the baseline assessment disappeared by the 12-month assessment. Overall, these findings suggest that the HSI Fellowship provides an opportunity for exposure to and development of the full suite of enriched core competencies and, in particular, in competency domains that are not currently emphasized in HSPR doctoral curricula.
According to the fellows, being embedded in a health system organization, having a dedicated professional development training allowance, receiving mentorship from a health system leader and working on an impact-oriented project of high importance to a health system organization were key enablers to their development of the enriched core competencies. These program elements are unique to the HSI Fellowship program when compared to conventional fellowships and are not yet systematically offered within HSPR doctoral training programs. To ensure the long-term sustainability of training modernization that maximizes doctoral graduates' impact in a broad array of employment sectors and roles, it will be important for Canada's university-based HSPR doctoral training programs to institutionalize the enriched core competencies within their curriculum, provide trainees with opportunities for field immersion within health system organizations and consider greater involvement of health system leaders in graduate training and mentorship.
This study has strengths and limitations. A notable strength is the use of a competency assessment tool to study and learn whether and how the program is working. The tool was administered at baseline and two subsequent time points to allow for assessment of change over time and was administered to fellows and their supervisors to allow for analysis of alignment in ratings. A limitation is the lack of a control group, such as a cohort of fellows funded through a conventional research fellowship, to test for a causal relationship between the intervention (the fellowship) and the outcomes (the competency development). Additionally, the assessments are based on fellow self-report and supervisor report and not on a validated evaluation of competencies. Potential bias could arise if fellows or supervisors underassessed early on in the fellowship in order to demonstrate improvement over time or, similarly, if they inflated their assessments at the 12-month mark in an effort to show improvement and program success.
Moving forward, analyzing the career trajectories of HSI fellows will be critical to understanding whether the program is achieving its objective of preparing doctoral graduates for success and impact in a broader array of employment settings, both within and beyond the academy. Emerging evidence based on the career transitions of the first cohort of one-year fellows (n = 24) is promising. HSI fellows have secured academic roles within universities as well as research and managerial roles within health system organizations. Examples include applied research scientist within public sector organizations, managerial positions within not-for-profit and healthcare delivery organizations and renewed fellowship positions. Although it is in its early days, the HSI Fellowship program appears to be successfully fostering the development of the enriched core competencies in HSPR and preparing fellows for careers in academic and applied health system settings.
Développement des compétences pour un impact sur le système de santé : premières leçons tirées du Programme des bourses d'apprentissage en matière d'impact sur le système de santé
Contexte : Le Programme des bourses d'apprentissage en matière d'impact sur le système de santé (BAIS) permet, à des postdoctorants hautement qualifiés en RPSS, de profiter d'un apprentissage expérientiel, de développer des compétences fondamentales enrichies et de bénéficier d'un mentorat de la part de cadres au sein des organismes de santé. Le programme a comme objectif principal de doter les doctorants des compétences, de l'expérience et des réseaux qui leur permettront, au cours de leur carrière, d'apporter d'importantes contributions dans les milieux universitaires et les établissements de santé.
Objectif : Cette étude vise à connaître à quel point le programme des BAIS a contribué au développement des compétences fondamentales enrichies dans le domaine de la recherche sur les politiques et les services de santé.
Méthode : Un outil d'évaluation des compétences a été développé puis soumis aux 46 boursiers formés depuis la création du programme, ainsi qu'à leurs superviseurs universitaires et des organismes d'accueil. Les autoévaluations des boursiers au début, à trois mois puis à 12 mois ont été analysées parallèlement aux évaluations des superviseurs à 3 mois et à 12 mois. Des analyses descriptives ont été employées pour étudier au cours du temps le développement des compétences. Les différences en fonction du genre ainsi qu'entre les notations des superviseurs et celles des boursiers ont été analysées.
Résultats : Les autoévaluations des boursiers font voir qu'ils ont renforcé leurs capacités dans chacune des 10 compétences fondamentales enrichies. Les évaluations remises par les superviseurs au sujet des compétences des boursiers permettent aussi d'observer une amélioration entre le début et à 12 mois. Les différences selon le genre observées au début se sont estompées à l'évaluation effectuée à 12 mois.
Conclusion : Le programme des BAIS permet de développer l'ensemble des compétences fondamentales enrichies, particulièrement dans les domaines sur lesquels les programmes de doctorat en RPSS ne mettent pas encore l'accent.
About the Author
Meghan McMahon, MSc, PhD, Associate Director, CIHR Institute of Health Services and Policy Research, Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON
Adalsteinn Brown, D.Phil, Dean, Professor, Dalla Lana School of Public Health, University of Toronto, Toronto, ON
Stephen Bornstein, PhD, Professor, Division of Community Health and Humanities, Faculty of Medicine, Department of Political Science, Faculty of Arts, Memorial University, Director, Newfoundland and Labrador Centre for Applied Health Research, Co-Director, SafetyNet Centre for Occupational Health and Safety Research, St. John's, NL
Robyn Tamblyn, PhD, Professor, Department of Medicine and Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Scientific Director (former), CIHR Institute of Health Services and Policy Research, Montreal, QC
Correspondence may be directed to: Meghan McMahon, PhD; e-mail: firstname.lastname@example.org.
The authors would like to thank Dr. Gery Ryan at the Pardee RAND Graduate School for inspiring our commitment to track and understand the fellows' competency development in order to continuously refine and improve the program and for sharing his insight and advice based on the Pardee RAND experience.
Bornstein, S., M. Heritage, A. Chudak, R. Tamblyn, M. McMahon and A.D. Brown. 2018. "Development of Enriched Core Competencies for Health Services and Policy Research." Health Services Research 53(5 Suppl. 2): 4004–23. doi.org/10.1111/1475-6773.12847.
Canadian Health Services and Policy Research Training Modernization Working Group (CHSPRA TMWG). 2015. Modernizing Health Services and Policy Research Training: A Pan-Canadian Strategy. Ottawa, ON: Canadian Institutes of Health Research. Retrieved December 4, 2018. <https://docs.wixstatic.com/ugd/5adc92_4b4c942ad529449489953892703473cc.pdf>.
CIHR Institute of Health Services and Policy Research (CIHR-IHSPR). 2016. "Canadian Health Services and Policy Research Alliance – Modernizing Training in Health Services and Policy Research." Retrieved December 4, 2018. <http://www.cihr-irsc.gc.ca/e/50024.html>.
McMahon, M., S. Bornstein, A. Brown, L. Simpson, L. Savitz and R. Tamblyn. 2019a. "Training for Health System Improvement: Emerging Lessons from Canadian and US Approaches to Embedded Fellowships." Healthcare Policy 15(Special Issue): 34–48. doi:10.12927/hcpap.2019.25981.
McMahon, M., B. Habib and R. Tamblyn. 2019b. "The Career Outcomes of Health Services and Policy Research Doctoral Graduates." Healthcare Policy 15(Special Issue): 16–33. doi:10.12927/hcpap.2019.25982.
McMahon, M. and Tamblyn, R. 2019c. "The Health System Impact Fellowship: Perspectives from the Program Leads: Comment on 'CIHR Health System Impact Fellows: Reflections on "Driving Change" within the Health System.'" International Journal of Health Policy Management 8(10): 623–26. doi:10.15171/ijhpm.2019.59.
Pardee RAND Graduate School. 2014. Program Review Self Study. Retrieved August 27, 2019. <https://www.prgs.edu/content/dam/prgs/documents/program-review-self-study.pdf>.
Reid, R.J. 2016. "Embedding Research in Learning Health Systems." HealthcarePapers 16(Special Issue): 30–35. doi:10.12927/hcpap.2016.24724.
1For a complete description of the enriched core competencies for HSPR and the process used to develop the framework, see Bornstein et al.'s 2018 paper.
2The competency assessment tools that were reviewed and that informed the HSI Fellowship competency tracking tool include the Pardee RAND Program Review Self-Study (2014), the University of Alberta's Individual Development Plan Workbook, myIDP, the NCHL Health Leadership Competency Model, LEADS in a Caring Environment Framework and the Vitae Researcher Development Framework.
3The group of deans/directors and health system leaders were the co-leads of the 10 CIHR Training Modernization Start-Up Grants, who received funding in March 2016 to pave the path for training modernization. The 10 co-lead pairs (n = 20 individuals total) represented eastern, central and western Canada.
Be the first to comment on this!
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed