Research to Action: An Evaluation
The evaluation of the Research to Action project was conducted using an Outcome Mapping (OM) methodology (Earl et al. 2001) with a mixed-methods, repeat survey (before/after) study design. This design uses concurrent measurement of process and outcome indicators at baseline and follow-up.
The RTA project proved effective at improving work environments and thereby promoting the retention and recruitment of nurses. Nurses involved in the RTA initiatives had a higher perception of leadership and support in their units, improved job satisfaction, increased empowerment and occupational commitment, and a greater intention to stay on the job.
The pilot projects were most successful when there were clearly stated objectives, buy-in from nurses, support from the steering committee and management, and adequate communication among stakeholders. Committed coordination and leadership, both locally and nationally, were central to success.
Considerable evidence has documented the challenges facing Canada's nursing human resources and their workplaces, such as high levels of turnover, excessive use of overtime and persistent shortages. There is a growing imperative to translate this research into action, and much of the available evidence presents viable policy alternatives for consideration. For example, a recent national synthesis report (Maddalena and Crupi 2008) recommended that, in consultation with stakeholders, processes should be put in place to share knowledge and best practices in nursing management, practice, staffing models and innovations in workplace health and well-being.
Nurses across the country report a desire to be more involved in decisions affecting them and their patients (Wortsman and Janowitz 2006). A recent study on the shortage of registered nurses in Canada (Tomblin Murphy et al. 2009) highlighted the need for collaboration among governments, employers, unions and other stakeholders to improve working conditions for nurses. Another report notes the potential benefits of reduced turnover among nurses, the cost of which has been identified as a major burden on the Canadian healthcare system (O'Brien-Pallas et al. 2010). One of the goals of the pan-Canadian framework for health human resources (HHR) planning adopted by the Federal/Provincial/Territorial Advisory Committee on Health Delivery and Human Resources is to enhance all jurisdictions' capacity to build and maintain a sustainable workforce in healthy, safe work environments (ACHDHR 2005).
Within this context, Health Canada's Office of Nursing Policy provided funding to the Canadian Federation of Nurses Unions (CFNU) and partner agencies in October 2008 to develop pilot projects across the country aimed at improving nurse retention and recruitment through various workplace improvement schemes. Each of the provincial partners contributed funds, in-kind support or both to the projects. The initiative was entitled Research to Action: Applied Workplace Solutions for Nurses (RTA). A national steering committee including representation from unions, governments and employers, each pilot project, CFNU and its national partners – the Canadian Nurses Association, the Canadian Healthcare Association and the Dietitians of Canada – was formed to oversee the development of 10 pilot projects. There was one project in each of Newfoundland and Labrador, Nova Scotia, New Brunswick, Prince Edward Island, Ontario, Manitoba, Saskatchewan, Alberta, British Columbia and Nunavut. The pilot projects, led by their own steering committees, focused on various aspects of nursing practice identified as particularly relevant to each jurisdiction, with a specific emphasis on improving the work life of nurses and transforming research knowledge into practice. The goals of the RTA initiative were to promote high-quality workplace environments, improve the retention and recruitment of nurses (RNs and LPNs), enhance the quality of patient care and engage stakeholders in collaborative partnerships. The first project began in May 2009 and the last project was completed in March 2011.
Specific objectives of this evaluation were to examine the effectiveness of various strategies to achieve these goals in the provincial pilot projects and understand the factors associated with the goals. The major evaluation question was, "How effective was the national project and each of the pilot projects?" Other specific objectives of the national evaluation were to understand the factors associated with the sustainability of the strategies and the transfer of knowledge from the pilot projects to regions across Canada and to professions beyond nursing. The evaluation also examined the effectiveness of partnerships and collaboration among unions, employers and governments at the national, provincial and local levels, and the factors associated with the sustainability of these partnerships. Because of their particularly unique context and make-up, the projects conducted in Alberta and Nunavut were evaluated separately. This study is specific to the remaining eight provinces.1
The evaluation was conducted using an Outcome Mapping (OM) methodology (Earl et al. 2001) with a mixed-methods, repeat survey (before/after) study design. This design involved concurrent measurement of process and outcome indicators at baseline (as early as possible in 2009) and follow-up (in 2010) at the pilot sites. The timing of the respective activities was different for each of the projects, based on the implementation date and other factors affecting project start dates. The evaluation timing was varied accordingly for each project.
Informed by input from the steering committees of each pilot project, the RTA national and provincial project leads, the national evaluation subcommittee and the national steering committee, the evaluation team developed a set of evaluation instruments to measure, for each project, a set of process and outcome indicators. These are variables that measure, respectively, the degree to which the activities for each pilot project have been implemented and the effects of that implementation. The instruments were designed to capture the perspectives of relevant sources – the nurses (RNs and, for some projects, LPNs) and managers participating in each pilot project, as well as the individual project coordinators and their respective steering committees – on each of the indicators. Five instruments were developed:
- a Strategies Journal for the project coordinators;
- a web-based Nurses' Outcomes Journal (NOJ) for the nurses participating in the project;
- an electronic Employers' Outcomes Journal (EOJ) to capture administrative data;
- a Performance Journal for capturing information from focus groups conducted separately for nurses and the project steering committees; and
- a web-based survey for members of the national steering committee to capture their levels of engagement and assessments of the project's outcomes.
Where appropriate, the instruments were developed using established measurement tools. Questions on the NOJ pertaining to perceived leadership were adapted from Shortell and colleagues (1991). Occupational commitment was measured using the same statements and scale used by Meyer and colleagues (1993). Psychological empowerment was measured using all the statements included in the instrument developed and psychometrically assessed by Spreitzer (1995). Categories used to measure levels of collaboration among stakeholders were adapted from those used by Hogue (1993).
Each instrument was reviewed by the respective pilot project steering committees as well as the national steering committee in a content validation process. Once finalized, the instruments were submitted to research ethics boards for approval. Acceptable response rates to the questionnaires were obtained, and the focus groups were well attended. During the first set of data collection, the subscales used in the Nurses' Outcomes Journal were examined for construct validity and internal consistency using Cronbach's alpha (Bland and Altman 1997).
Implementation of the projects' activities generally met or exceeded expectations. The goal of introducing an 80/20 staffing model (Bournes and Ferguson-Paré 2007) to a smaller rural setting in British Columbia was clearly achieved. The observations in Saskatchewan indicate that the activities related to developing, validating and employing a patient scoring tool (Curley 2007) to adjust staffing ratios had indeed been carried out. It is clear from the uptake of the mentoring program and attendance at the clinical workshops in Manitoba that project implementation there met or exceeded expectations. On the whole, the Ontario project achieved its implementation goals but was challenged in doing so because of a wide array of both internal and external factors. By closing date for the national evaluation, New Brunswick had launched over 90% of its web-based orientation modules and had exceeded its expectations in mentorship training. The Nova Scotia project experienced some delays with regard to the initial project planning, and by the end of the national evaluation period (December 2010) two of the three main activities (HSPnet and enhanced orientation) had not been implemented to the point of engaging new graduates as planned.2 The implementation of emergency and critical care training programs in PEI clearly met expectations. And finally, in Newfoundland and Labrador the project activities unfolded as expected, with no significant issues in implementation.
Although most projects were implemented more or less as planned, each project faced a variety of challenges as it developed, such as staffing shortages, the H1N1 pandemic and other external complications.
If the national CFNU project was effective, one would expect to see a shift in outcomes from baseline to follow-up – up for some (such as job satisfaction and occupational commitment) and down for others (such as turnover and absenteeism). To ascertain whether or not such a pattern existed for the nurse outcomes (the administrative data will be presented next), the mean outcomes at baseline and follow-up were compared independently. Because the outcomes were continuous variables with skewed distributions, they were dichotomized at the median and logistic regression was used to determine the odds of nurses' reporting a high outcome (i.e., above the median) at follow-up compared to baseline.
The odds of nurses' reporting a higher-than-median outcome were greater at follow-up than baseline for all measures – perceived leadership and support, job satisfaction, empowerment, occupational commitment, satisfaction with nursing, intention to stay in the job and self-assessed clinical preparedness. One can rule out chance as an explanatory factor in two of these outcomes: the odds of nurses' reporting a high level of leadership and support in their units were 147% greater at follow-up than at baseline (p=.005), and the odds of reporting high job satisfaction were 97% greater (p=.032). The odds of reporting high empowerment, occupational commitment and intention to stay in their job were also greater at follow-up (38%, 37% and 28%, respectively). While chance cannot be ruled out for these latter three observations, the fact that all variables showed greater odds of high scores at follow-up relative to baseline suggests that chance is an unlikely explanation. Such a view is strengthened by further analysis of the same variables.
To determine whether these changes can be attributed to the RTA initiative, the association between nurse-level outcomes and nurses' time spent participating in the RTA projects (separated into quartiles) were measured. Figure 1 presents the mean nurse outcomes at follow-up by total hours of involvement in CFNU initiatives (controls were included in the zero hours of involvement group). The more time nurses put into the CFNU initiatives, the higher their mean outcome scores. Those nurses who put in the greatest number of hours had a 29% more positive perception of the leadership and support in their units than did those nurses who put no time into the CFNU initiatives, and this association was highly statistically significant (p=.003).
The corresponding effect size for job satisfaction was 22% (p=.001); for empowerment, 12% (p=.006); for occupational commitment, 15% (p=.000); for satisfaction with nursing, 14% (p=.004); and for intention to stay in their job, 4% (p=.042). It is important to note that this analysis used general linear models incorporating the nurses' province, age and level of education as covariates, meaning that the effects shown above are not due to differences in these variables between nurses.
Figure 2 depicts the rates for professional development, turnover, overtime and absenteeism at baseline and follow-up. The data from all participating units (excluding control units) across the country at baseline and follow-up were both aggregated in order to yield the most stable baseline rates. Professional development (PD), a key element of many of the CFNU initiatives, went up 66.7% from baseline to follow-up from about 1.5 hours of PD per 100 worked hours to about 2.5. Concomitantly, turnover, overtime and absenteeism went down about 10%. Note that these administrative data were drawn from all units that participated, and there is no intent to generalize to other units. Thus, they are not a sample of units; there is no sampling error to rule out, and therefore no statistical testing was required.
To determine whether or not these changes in the administrative data are attributable to the RTA projects, the participating units were separated into two groups according to the proportion of all their nurses that were involved in the RTA initiative; units were classified either as having a lower- or higher-than-median proportion of nurses participating. Though not shown graphically, the two groups were then compared in terms of their levels of professional development, turnover, overtime and absenteeism at baseline and follow-up.
In those units with a lower proportion of their nurses being involved in the RTA initiative, the proportion of total hours worked taken as professional development went down 0.6 of a percentage point from baseline to follow-up. In those units with a higher participation proportion, professional development went up 2.0%. In the lower participating units, turnover went up about 1.0%, while in the higher participating units it went down about 3.5%. While overtime barely changed in the low participation units, it went down 1.5% in the higher participation units. Absenteeism did not follow this pattern of more positive changes in the higher participation units; the opposite was apparent.3 In summary, improvements in most of these administrative outcomes are associated with a higher level of participation in the RTA initiative.
The quantitative findings are consistent with the findings of the focus groups, where it was reported that at the level of the staff nurse, the impact of the pilot projects exceeded initial expectations. In general, nurses were pleased (some said they felt "privileged") to participate in the projects. Nurses expressed many opinions that support the view that the projects enhanced the nursing experience, including, for example, increased autonomy in decision-making, enhanced collegial relationships with nurses and other health professionals, enhanced "incidental learning" from colleagues, increased confidence, a renewal of their interest in and commitment to nursing as a profession and enhanced understanding of issues faced by nurses, management and policy makers, among others. Barriers to successful implementation were reported as being exacerbated by other confounding factors, such as short staffing on the unit, perceptions of excessive paperwork and heavy workloads.
Members of the national steering committee were asked, through an online survey, to assess the level of collaboration with various groups during the project as being one of five categories, ranging from relatively low to relatively high levels of collaboration. The level of collaboration with employers, the provincial leads and the project coordinators were each assessed as being very high. Although not assessed as highly, the level of union and government collaboration was rated at least "cooperative" by the vast majority of respondents. The assessments of the level of collaboration with educational partners were quite diverse, an outcome that is to be expected given the variation in levels of involvement of educational partners across different projects.
Respondents to the national steering committee survey were also asked to rate their satisfaction with the overall level of collaboration among national partners on a scale from 1 (meaning very dissatisfied) to 7 (very satisfied). More than two-thirds of respondents rated their satisfaction as 6 or 7 out of 7, and the average level of reported satisfaction was 5.75 out of 7, or 82%. Along these same lines, 83% of respondents reported that partnerships between key stakeholders had been strengthened as a result of the RTA initiative, and 83% reported they had gained new insights into the different perspectives of project partners. These findings are consistent with those from focus groups with the provincial steering committees, where it was repeatedly reported that the project had improved the level of collaboration among key stakeholders by bringing them together to work towards a common goal.
Sustainability of the pilot projects was an ongoing topic of discussion at provincial steering committee meetings, in particular nearing the end-of-project implementation as reported within the provincial strategy journals. Some pilot projects reported some success in sustaining at least some components of their projects. For example, the PEI program is continuing, and the mentorship program in Nova Scotia will be sustained long term. In those provinces where the probability for sustainability was reported as being relatively low, some efforts have been directed at sustaining components of the projects in one way or another. For example, in Newfoundland and Labrador there has been some discussion of the possibility of offering a modified version of the 80/20 staffing model – that is, implementing an 85/15 or a 90/10 model.
Participants in nurse focus groups and the provincial steering committees identified a number of factors that were required to ensure the success of new models or strategies. These included, in no particular order, collaborative design and agreement on common objectives, joint ownership in the process and outcomes of the proposed initiative, adequate funding, government and organizational buy-in, and adequate input from staff nurses and unions. Participants strongly suggested that the projects' successes are sustainable if the necessary supports are present, specifically, financial and organizational support. In addition, members of the project steering committees cited the commitment and desire of front-line nurses to engage in innovation, ongoing learning and quality improvement as central to the success of the projects.
These comments are consistent with responses to the survey of national steering committee members, who identified the availability of funding or other resources, buy-in from and partnership with relevant stakeholders, and evidence of the projects' effectiveness as the three most important determinants of their sustainability. When asked whether the RTA initiative was sustainable without Health Canada funding, 26% indicated that it was, 30% indicated it was not and 44% were undecided.
Transfer of knowledge
On scales of 1 (not at all transferable) to 7 (very transferable), respondents to the national steering committee survey were asked to assess the transferability of the RTA projects to other similar healthcare facilities and other healthcare professions. The average ratings were 6.2 and 5.6, respectively, suggesting that the RTA initiative is very transferable. The three most important determinants of the transferability of projects were identified as the availability of funding or other resources, evidence of the projects' effectiveness and on-site leadership.
Participant nurses and the provincial steering committees identified several other factors as essential to the transferability of the projects. These included a persistence or willingness to proceed and overcome challenges, strong leadership in the form of a project coordinator and oversight by a steering committee, and the need to establish effective lines of communication and processes to resolve problems as they arise.
The importance of sharing knowledge across the pilot projects was repeatedly cited by members of the national steering committee. They reported that "[we] believe the sharing of information between the RTA pilot projects was very useful for project implementation because it allowed us to share information and obtain valuable feedback from other members. Sharing information gave greater insight to the projects and allowed us a chance [to] express thoughts and ask questions." Sharing of information between projects "was absolutely huge."
Most respondents to the national survey reported discussing the RTA initiative with colleagues not involved in it at least once per month; most also reported that they had discussed the project with colleagues outside nursing.
There was some evidence that additional communication and knowledge exchange about the RTA projects was necessary, including with nurses who were not directly involved with the initiatives but who may have been indirectly affected by them. For example, some nurses involved in Nova Scotia's 80/20 project reported being resented and looked down upon by other nurses who felt the participating nurses were receiving special treatment or "slacking off." Other nurses involved in Nova Scotia's 80/20 project reported substantial differences in how the project was implemented in different districts: very satisfactorily in some, not at all satisfactorily in others. Improved communication was also credited with overcoming challenges that some projects had with uptake and recruiting participants in the early stages; for example, in British Columbia and Prince Edward Island, it was reported that once non-participating nurses learned about the benefits of the projects, they became interested in participating themselves. The importance of knowledge exchange was further highlighted by the desire of some respondents for even more of it.
- Keep the scope of initiatives such as RTA manageable. The success of a project will be contingent on the development of a realistic proposal with a manageable scope, and adherence to that scope is important for its implementation. Underestimation of the time requirements and resource needs – both human and non-human – of a project due to deviation from the initial plan led to delays and limitations in the implementation of several provincial projects.
- Build contingency plans into the project in order to manage unforeseen external factors that can have an impact on project implementation. For projects that include front-line staff, success is contingent on their engagement; therefore, opportunities to ensure their full participation must be made available. Thus, when planning for such projects, it is important to consider contextual factors such as bed closures and organizational restructuring, which may affect nurses' ability to participate.
- Consider integrating capacity to sustain change into project plans. For example, a longer project timeline with a larger number of participants would have provided an opportunity for quantitative evaluation data at the provincial level that would assist in sustainability. Further, new projects should align with existing provincial priorities and initiatives – or at least consider these in their planning – to ensure optimal uptake and sustainability.
On the basis of the consistency of quantitative and qualitative findings, it is concluded that the RTA initiative as a whole has been successful. More specifically, it was effective at improving work environments in support of the retention and recruitment of nurses, and it has significantly strengthened the level of collaboration between key stakeholders at the national, provincial and local levels. Further, the RTA projects are viewed as being sustainable and transferable with the appropriate resources and buy-in from key stakeholders needed to maintain them.
Overall, the RTA projects were successful when there were clearly stated project objectives, sufficient buy-in from nurses, support from the steering committee and management – aligned with common priorities – as well as adequate communication among stakeholders. If one of these factors was weak, then project implementation was affected. Committed coordination and leadership – locally and nationally – have been essential to the initiative's success.
The RTA initiative has allowed a number of potential means of improving working conditions for nurses to be identified and tested, thereby contributing substantially to the available evidence base in this field. Critical determinants of the sustainability and transferability of such interventions have been established. Significant challenges to implementing these various initiatives have been identified and largely overcome by those involved in the project through committed leadership, mutual respect and collaboration among key stakeholders based on shared goals and dedicated coordination.
About the Author
Gail Tomblin Murphy, RN, PhD Professor, School of Nursing Director, WHO/PAHO Collaborating Centre on Health Workforce Planning and Research , Dalhousie University, Halifax, NS
Rob AdlerMMedSc, PhD Adjunct Associate Professo, Associate Director, Research Global Health Office, University of Western Ontario
Adrian MacKenzieBSc(H)Senior Analyst , WHO/PAHO Collaborating Centre on Health Workforce Planning and Research Dalhousie University
Amanda CookMES, Project Manager, Research to Action Evaluation, Tomblin Murphy Consulting Inc.
Victor MaddalenaBN, MHSA, PhD Assistant Professor Faculty of Medicine Memorial University of Newfoundland
Advisory Committee on Health Delivery and Human Resources (ACHDHR). 2005. A Framework for Collaborative Pan-Canadian Health Human Resources Planning. Ottawa: Author. Retrieved November 6, 2011. <http://www.hc-sc.gc.ca/hcs-sss/pubs/hhrhs/2007-frame-cadre/index-eng.php>.
Bland, J.M. and D.G. Altman. 1997. "Statistics Notes: Cronbach's Alpha." British Medical Journal 314: 572.
Bournes, D.A. and M. Ferguson-Paré. 2007. "Human Becoming and 80/20: An Innovative Professional Development Model for Nurses." Nurse Science Quarterly 20(3): 237–53.
Curley, M.A.Q. 2007. Synergy: The Unique Relationship Between Nurses and Patients. Indianapolis: Sigma Theta Tau International.
Earl, S., F. Carden and T. Smutylo. 2001. Outcome Mapping: Building Learning and Reflection into Development Programs. Retrieved November 6, 2011. <http://www.idrc.ca/EN/Resources/Publications/Pages/IDRCBookDetails.aspx?PublicationID=121>.
Hogue, T. 1993. Community Based Collaboration: Community Wellness Multiplied. Bend, OR: Chandler Center for Community Leadership. Retrieved November 6, 2011. <http://crs.uvm.edu/nnco/collab/wellness.html>.
Maddalena, V. and A. Crupi. 2008. A Renewed Call for Action: A Synthesis Report on the Nursing Shortage in Canada. Ottawa: Canadian Federation of Nurses Unions.
Meyer, J.P., N.J. Allen and C.A. Smith. 1993. "Commitment to Organizations and Occupations: Extension and Test of a Three-Component Conceptualization." Journal of Applied Psychology 78(4): 538–51.
O'Brien-Pallas, L., G. Tomblin Murphy, J. Shamian, X. Li and L. Hayes. 2010. "Impact and Determinants of Nurse Turnover: A Pan-Canadian Study." Journal of Nursing Management 18(8): 1073–86.
Shortell, S.M., D.M. Rousseau, R.R. Gillies, K.J. Devers and T.L. Simons. 1991. "Organizational Assessment in Intensive Care Units (ICUs): Construct Development, Reliability, and Validity of the ICU Nurse–Physician Questionnaire." Medical Care 29(8): 707–26.
Spreitzer, G.M. 1995. "Psychological Empowerment in the Workplace: Dimensions, Measurement, and Validation." Academy of Management Journal 38(5): 1442–65.
Tomblin Murphy, G., S. Birch, R. Alder, A. MacKenzie, L. Lethbridge, L. Little and A. Cook. 2009. Tested Solutions for Eliminating Canada's Registered Nurse Shortage. Ottawa: Canadian Nurses Association.
Wortsman, A. and S. Janowitz. 2006. Taking Steps Forward: Retaining and Valuing Experienced Nurses. Ottawa: Canadian Federation of Nurses Unions. Retrieved November 6, 2011. <http://www.nursesunions.ca/sites/default/files/2006-Taking_Steps_Forward-En.pdf>.
1. The evaluation of the Alberta RTA project was conducted by Intergage Consulting Group (formerly JLS Management Consulting). The evaluation of the Nunavut RTA project was conducted by Karen Fingas Consulting.
2. For this reason, the quantitative analyses of the national evaluation focused on involvement in the 80/20 mentorship program in Nova Scotia.
3. It does not seem plausible that participation in the CFNU initiatives would increase absenteeism. It is more plausible that turnover, overtime and absenteeism would co-vary such that a decrease in turnover and overtime use would reduce absenteeism, or that a decrease in turnover together with a decrease in absenteeism would reduce overtime use. It is also noteworthy that the changes in absenteeism are the smallest – less than 1% in both participation groups – and therefore more likely to be due to chance (caused by random occurrence of rare events). In other words, this observation may be an artefact.
Be the first to comment on this!
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed