The Ontario Nursing Workload Demonstration Projects: Rethinking How We Measure, Cost and Plan the Work of Nurses
Background: In 2008 the Nursing Secretariat of Ontario's Ministry of Health and Long-Term Care formed a Nursing Workload Steering Committee to oversee the implementation of three demonstration projects with the objectives: to assess the feasibility of Health Outcomes for Better Information and Care (HOBIC) data as a measure of nursing workload, determine the indicators that best support nurse leaders to measure nursing work and make informed staffing decisions, and develop a model that predicts acute care nursing costs. Results: Three HOBIC scales – activities of daily living (ADLs), continence and fatigue – explained a small amount of the variance in nurse judgment of the amount of nursing time patients require in the first 24 hours of care. Nurses in the study appreciated providing their professional judgment to help estimate the nursing work requirements of patients. The priority and secondary indicators most important for decision-making included medical severity of patients, environmental complexity, nurse experience, patient turnover, nurse-to-patient ratio, cognitive status, infection control, nurse vacancy, predictability of patient types, nursing interventions, patient volumes, co-morbidities, patient self-care abilities, physical and psychosocial functioning, unit type and medical diagnosis. A fairly robust model was developed using existing data sources to estimate nursing input into a patient's costs. The model explained between 69% and 80% of the variation in nursing costs for each patient. Conclusion: In order to effectively measure, plan and cost nursing, we need to determine what nursing is. In the future, recognition of nurses as knowledge workers will require us to consider the many patient and environmental factors that affect the ability of nurses to apply their professional judgment to care for patients.
Over the years, numerous concerns have been raised regarding the validity and reliability of nursing workload data (Duffield et al. 2006; McGillis Hall et al. 2006; Hernandez and O'Brien-Pallas 1996; Hughes 1999). In response to these concerns, the Ontario Ministry of Health and Long-Term Care (MOHLTC) Health Results Team for Information Management and the Nursing Secretariat jointly commissioned a study in 2005 to assess the quality and value of nursing workload data collection. The study included a comprehensive consultation process with users, academics, researchers and front-line nursing staff. Results of the study led to the ministry recommendation in 2006 that only case-costing hospitals would be required to report nursing workload data to the MOHLTC (Government of Ontario 2006b). In response, in 2008 the Nursing Secretariat formed a Nursing Workload Steering Committee to oversee the implementation of three specific demonstration projects with an overall mandate to investigate "new generation" workload measures.
The HOBIC Project
Assess the feasibility of Health Outcomes for Better Information and Care (HOBIC) data as a measure of nursing workload.
In Ontario, nursing workload data have been collected using a variety of different systems, thus producing a variety of different data sets that were often inconsistent and not comparable from a provincial perspective. HOBIC is currently being implemented province-wide, as a standardized collection of patient health outcomes information that is valid, reliable, patient centred, evidence based, outcome focused and comparable across all sectors (Government of Ontario 2006a; Pringle and White 2000, 2002, 2003; White and Pringle 2005). Since the HOBIC data set includes standardized variables that are comparable across different organizations, there was optimism that HOBIC could be the consistent and comparable provincial data set for nursing that could potentially also supply new generation workload measures. The goal of this project was to determine the relationship between HOBIC and nursing workload.
HOBIC Project Demonstration Sites
Five demonstration sites across Ontario participated in the project.
Headwaters Health Care Centre, Orangeville
The Dashboard Project
Determine the indicators that best support nurse leaders to measure nursing work and make informed staffing decisions in order to populate a "nursing dashboard" inclusive of these indicators that could become a meaningful tool for decision-making.
Nursing workload data has not consistently been used to predict staffing requirements and make staffing decisions (McGillis Hall et al. 2006). Too often, little time is paid to designing and monitoring a workload measurement system that accurately reflects the practice environment, leading to distrust that workload data reflects actual workload (Hernandez and O'Brien, 1996). As a result, there are currently very few tools available to nurses to help them make staffing decisions. Researchers have called for the development of meaningful tools that are based on knowledge that experienced nurses use to make best practice staffing decisions (McGillis Hall et al. 2006). This project aimed to develop a decision-making tool that would value the experience of nurses.
The Nursing Cost Project
Develop a model that predicts acute care nursing costs.
Nursing workload measurement systems were originally developed to enable nurses to predict nurse staffing requirements (Edwardson and Giovanetti 1994). Since nursing is the largest component of the cost of a patient's stay in hospital (Cockerill et al. 1993), later, when the Ontario Case Costing Initiative was developed, it was agreed that determining the cost of nursing would be fundamental to accurately determining a patient's cost. To estimate the cost of nursing, experts looked to pre-existing data sources for potential predictors. At the time, nursing workload data was easily available and was the most reliable data source thought to estimate the amount of time a nurse spent with a patient and thus determine the cost of nursing. However, the data set is much more detailed than necessary; the only requirement of case costing is the total amount of nursing time a patient receives, not a detailed description of each task a nurse completes in a shift. The goal of this project was to determine if there is an alternate way to predict the cost of nursing care without nursing workload data.
Methods and Results
The HOBIC Project
HOBIC admission assessments, nurse judgment scales and nursing workload data were collected prospectively for 4750 patients within 14 acute care units between August 2008 and January 2009. HOBIC admission assessments were completed within four hours of a patient's admission. The HOBIC scales that were included in the analysis were functional status (ADLs, or activities of daily living), pain, nausea, fatigue, dyspnea, falls and pressure ulcers. Nurses also completed three judgment scales along with the HOBIC admission assessment:
- The patient's need for nursing care in the first 24 hours will be: (very low, low, average, high, very high)
- Compared with similar patients admitted to this unit, the amount of nursing time this patient will need in the first 24 hours of admission will be: (very low, low, average, high, very high)
- Compared with similar patients admitted to this unit, the level of nursing care this patient will need in the first 24 hours of admission will be: (very low, low, average, high, very high)
Nursing workload data was collected at three of the five demonstration sites. Each site had a "home-grown," task-oriented, nursing workload measurement system that enabled the capture of workload data through electronic documentation; thus, complete documentation would result in complete workload data. However, the mean workload values varied considerably between sites (see Table 1), confirming that workload was captured and measured differently between sites. Disregarding potential differences in patient populations, it is unrealistic that out of a possible 1,440 minutes in a 24-hour day, Site A could be giving approximately 190 minutes of care and Site B could be giving 1,200 minutes. Since it was not possible to test the reliability of the data, it was determined that the nursing workload data were not of sufficient quality and consistency to contribute to the project.
|Table 1. Nursing workload minutes within the first 24 hours of admission by site and unit|
A Spearman correlation coefficient was calculated to assess the correlation between HOBIC admission scales and nurse judgment of patient care requirements. The degree of correlation between the three judgment scales was very high (need, 1.000; time, 0.895; level, 0.893). The highest correlations were between the need for nursing care and ADLs (0.467), continence (0.377) and fatigue (0.315) scales. When combined in a multivariate model, ADLs, continence and fatigue together explained over 25% of the variance in the nurses' judgment of the amount of nursing time a patient needed in the first 24 hours. ADLs alone explained 23% of the variance in judgment, while continence explained 12% and fatigue explained 10%.
A questionnaire was conducted with a convenience sample of 45 nurses from the five demonstration sites to understand how nurses answered the questions and how they felt about providing their judgment about the nursing care requirements of patients. Overall, 84% of the sample of surveyed nurses indicated that they would like to continue to complete the judgment scales; 49% strongly agreed and 36% somewhat agreed that the scales captured their opinion of the patient's workload requirements. Almost all (98%) used their assessment and observation skills to complete the scales. Thirty-eight percent were willing to complete a short set of judgment scales daily or more often, and another 38% were willing to complete the scales either at admission only or on admission and discharge, indicating that the nurses surveyed were willing to share their professional judgment on an even more regular basis than what was required for this study.
The Dashboard Project
Approximately 30 nurse leaders, including managers, directors and vice presidents from the demonstration sites (see sidebar), were involved in the four-step process to determine the indicators that best support nurse leaders to make staffing decisions. After the indicators were determined, demonstration sites created dashboards to display the indicators.
1. Gather and study attributes that affect staffing requirements.
In 2007, a Nursing Workload Task Group reviewed the literature to develop a comprehensive list of over 50 attributes where evidence exists to suggest an impact on nursing work. The Patient Care Delivery Model (O'Brien-Pallas et al. 2001) provided the underlying theoretical framework for the selection of each evidence-based attribute. The model suggests a logical, systems approach to understanding the various elements that dynamically combine to describe the context for and outcomes of nursing care. The model identifies an iterative phenomenon among four phases: inputs, throughputs, outputs and the feedback loop. The phases of the model have key concepts, each of which has a variety of attributes. For example, the inputs to the system include patient, nurse and system characteristics (concepts). Depending on the service or program, patients would be characterized by age, gender, medical diagnosis and health status (attributes). The attributes characterizing nurses, such as experience, skills and knowledge, should meet the needs of the patients. The system for that care would be described using attributes such as the type and size of the unit. Working through the model adds the context, processes and complexities of care and the ensuing outcomes for patients, nurses and the system (unit, organization and beyond), which are then fed back for explanatory evaluation, education or program planning.
Dashboard Project Demonstration Sites
Seven sites from across Ontario participated in the project. Demonstration sites were selected to represent a cross-section of small, community, large and academic healthcare settings, as well as urban, rural and remote geographic locations.
Hamilton Health Sciences Centre, Hamilton
In June 2008, 23 nurses and other stakeholders from the demonstration sites attended a retreat where an open discussion was held about the impact each evidence-based attribute had on nursing work and how each attribute may or may not affect staffing requirements on a unit. The sites worked in groups to consider what indicators could be used to describe each attribute and discussed the relative importance of each attribute.
2. Determine preferred attributes and data availability.
Between June and August 2008, using the list of evidence-based attributes as a starting point, each demonstration site began to develop a preferred "wish list" of attributes for a dashboard that were considered the most important and relevant for making staffing decisions. Each site had different stakeholders present during the discussions – some had stakeholders with knowledge of hospital information systems present, and some did not. As a result, the process used at each site to determine preferred attributes varied, and it is unclear whether attributes were selected on the basis of importance or availability of potential indicators for the attribute. Demonstration sites then began to build their own unique "nursing dashboards." At this point, it became evident that some information was unavailable and that new data sources would need to be created to gather the necessary information.
3. Prioritize most useful attributes for decision-making.
In August 2008, before use of the nursing dashboard began, a baseline survey was completed by 30 nurse leaders from the demonstration sites to evaluate the usefulness of the full list of attributes for making staffing decisions. The top 12 attributes are presented in Table 2 in order of the percentage of stakeholders that agreed that the attribute is "very useful" (on a scale of "somewhat useful, "not very useful" or "not at all useful) for making staffing decisions.
|Table 2. Top 12 "very useful" attributes for making staffing decisions|
|Predictability of patient types||72%|
4. Rank and validate the key priority and secondary attributes.
In December 2008, a second retreat was held, where 30 nurse leaders involved in the project participated in a consensus exercise to rank by importance the "very useful" attributes from the baseline survey. Taking into consideration both the baseline survey results and the consensus exercise, participants classified each attribute subjectively as either "priority" or "secondary." In February 2009, a final survey was completed by 28 nurse leaders to further validate the priority and secondary rankings from the baseline survey and consensus exercise. Table 3 shows the results. There was a high level of agreement among respondents that each previously selected priority and secondary attribute was important for making staffing decisions.
|Table 3. "Priority" and "secondary" attributes for making staffing decisions|
|Percent agree attribute is priority for making staffing decisions|
|Predictability of patient types||84.6%|
|Percent agree attribute is secondary for making staffing decisions|
|Patient self-care abilities||80.8%|
|Physical and psychosocial functioning||80.8%|
The Nursing Cost Project
To gain a "gold standard" data set for the total amount of nursing time received by a sample of patients, time and motion studies were conducted on three units in August 2008. Each staff nurse on the unit was shadowed by a nursing student 24 hours a day for seven days, allowing all patient activity to be captured. Students used custom software on a portable computer to capture all the time spent by nurses on 12 activities originally identified by Hendrich et al. (2008): patient care activities, assessment/vital signs, care coordination, unit-related functions, documentation, medication administration, waiting, retrieving/delivering, patient/family care, administration/teaching and personal time. For patient-specific activities, the patient's unique visit number was captured so that it could later be linked to other administrative data sets. Total nursing time for the full length of stay of 57 patients was captured.
To determine if it was possible to increase the sample size by using the case costing data set, analyses were performed to investigate the amount of variation explained by the total nursing time spent in the time and motion study on the nursing costs reported in the case costing data set. The investigation found that 78.9% of the observed variance in direct nursing costs is explained by total nursing time spent (p < .0001). Since the nursing time received by patients in the time and motion study was highly correlated to the direct nursing costs reported in the case costing data set, sample size was increased by using the direct nursing cost data from case costing to correlate potential variables that might explain the variation in nursing costs for a particular patient.
Table 4 shows that a multiple regression model of the relationship between direct nursing costs from case costing and length of stay, and therapy, laboratory, diagnostic, ICU/CCU and pharmacy costs explained between 69% and 80% of the variation in nursing costs (p < .001). Other variables, such as costs of intervention, emergency department, clinics and operating room were significant but increased the R-square very little.
|Table 4. Multiple regression models of the relationship between direct nursing costs and length of stay, therapy costs, laboratory costs, diagnostic costs, ICU/CCU costs and pharmacy costs|
|Unit||Number of patients||R-square||Coefficient of variation||Root mean square error|
Three HOBIC scales, ADLs, continence and fatigue, explained a small amount of the variance in nurse judgment of the amount of nursing time patients require in the first 24 hours of care. However, much of the variance was still unexplained, suggesting there are factors other than those captured in the HOBIC admission assessment that contribute to the amount of nursing time patients require.
The variation in workload data between demonstration sites in the HOBIC project was consistent with studies by Cockerill et al. (1993) and O'Brien-Pallas et al. (1993), which found that when different workload measurement tools are applied to the same patients, clinical and statistical differences are witnessed in estimated hours of care. As a result, workload data was not used to correlate HOBIC scales. Instead, nurse judgment scales were developed to conduct the correlation analyses. The survey results showed that nurses in the study appreciated providing their professional judgment to help estimate the nursing work requirements of patients: 84% of the surveyed nurses indicated that they would like to continue to complete the judgment scales after the study was over. In future, an inter-rater reliability study of the nurse judgment scales will help determine whether there is value in using nurses' professional judgment to predict patient workload requirements. A study by Gran-Moravec and Hughes (2005) demonstrated that a tool utilizing RN professional judgment maximized appropriate staffing allocation to meet the needs of patients, staff and administration. The authors suggest a study to test the value of collecting nurse judgment variables on a daily basis, including exploring the possibility of adding nurse judgment scales to the HOBIC data collection.
A small group of nurse leaders representing demonstration sites from across Ontario came together and determined several attributes that were most important for making staffing decisions. These demonstration sites faced many barriers to successfully developing and implementing a dashboard. Many of the indicators selected and developed by the sites were not available from currently available data sources. In fact, most available data from hospital data collection systems have been designed for administrative purposes at an organizational level; whereas, nurse staffing decisions are made at the unit level. To create indicators, administrative data, including data from human resources, finance and ministry-mandated data sources, had to be re-purposed, or substitute measures were used. This is similar to the Australian experience, where it has been difficult to measure demand for nursing services because the routinely collected management information does not include the detailed patient-level information necessary to do so Twigg and Duffield 2008). The authors suggest taking advantage of the current evolution of the work to design and define the parameters for a "nursing data set" based on the most important attributes for decision-making, rather than what is currently available via administrative data sets. A web-based nursing dashboard template for the "nursing data set" that can send data to the MOHLTC for provincial roll-ups but can also be retrieved on demand by nurse managers would give each and every organization in the province the opportunity to begin to use this information in their decision-making.
Currently, standard indicator definitions for important attributes do not exist. As a result, each demonstration site constructed their own indicator definitions for their nursing dashboard, based on the data that was available. However, demonstration site participants consistently mentioned the value of standard indicator definitions for enabling clear unit, program, regional and provincial comparisons as well as consistent interpretation between organizations. To continue this work, the development and validation of clear and consistent indicator definitions for important nursing attributes will be crucial.
Effective use of a dashboard requires that users understand how to interpret indicator results. Many nurse leaders have been making staffing decisions using intuition based on years of experience, a method Arthur and James (1994) referred to as the most widely used, despite explosions in more sophisticated technologies. During the project it was clear that a change in decision-making will be required if dashboards are to be used effectively; as a result, education will need to be a key part of any dashboard project.
A fairly robust model was developed using existing data sources to estimate nursing input into a patient's costs. The model was able to explain between 69% and 80% of the variation in nursing costs for each patient, demonstrating that the cost of nursing care can be predicted without using a time-intensive system that requires the measurement of each and every task a nurse performs on a shift. Next steps for this project should include testing the model against several years of data in other organizations where the current workload measurement system is highly reliable to determine if this model is robust enough to use province-wide. Additional patient-level variables that can be easily extracted from an electronic record (e.g., operating room times, discharge disposition, admission source, HOBIC scales) should be investigated to determine if they add power to a model that predicts the cost of nursing care.
In order to effectively measure, plan and cost nursing, we need to determine what nursing is and what nurses do. Organizations need to engage nurses in a discussion to understand the variation as well as the uniqueness of the nursing role in different practice environments. In the future, recognition of nurses as knowledge workers will require us to consider the many patient and environmental factors that affect the ability of nurses to apply their professional judgment to care for patients. Only then will we be able to build the measurement tools that can capture what Duffield et al. (2006) so aptly refer to as the "invisible nature" of nursing.
About the Author
Mary Ferguson-Paré, RN, PhD, CHE, Vice President Professional Affairs and Chief Nurse Executive, University Health Network, Toronto, ON
Annabelle Bandurchin, MHSc, Innovation Project Manager, University Health Network, Toronto, ON
Correspondence may be directed to: Annabelle Bandurchin, firstname.lastname@example.org.
The authors would like to thank several individuals for enabling the successful completion of these projects: Gloria Johnson, Jan Walker, Nan Brooks, Imtiaz Daniel, Ramandip Grewal, Carolyn Plummer, Janet Rush, all members of the Nursing Cost Committee, and all members of the Nursing Workload Steering Committee. A special thanks to the Nursing Secretariat of Ontario's Ministry of Health and Long-Term Care, for funding this project.
Arthur, T. and N. James. 1994. "Determining Nurse Staffing Levels: A Critical Review of the Literature." Journal of Advanced Nursing 19: 558–65.
Cockerill, R., L. O'Brien-Pallas, H. Bolley and G. Pink. 1993. "Measuring Nursing Workload for Case Costing." Nursing Economics 11(6): 342–9.
College of Nurses of Ontario. 2009. Fact Sheet: What Is Nursing? Retrieved January 29, 2010. <http://www.cno.org/docs/general/StudentsIntro.pdf>.
Duffield, C., M. Roche and E. Merrick. 2006. "Methods of Measuring Nursing Workload in Australia." Collegian 13(1): 16–22.
Edwardson, S.R. and P.B. Giovannetti. 1994. "Nursing Workload Measurement Systems." In J.J. Fitzpatrick and J.S. Stevenson, eds., Annual Review of Nursing Research (pp. 95–124). New York: Springer Publishing Company, Inc.
Government of Ontario. 2006a. Health Outcomes for Better Information and Care (HOBIC). Retrieved January 29, 2010. <http://www.health.gov.on.ca/transformation/providers/information/data_initiatives/bg_hobic.pdf>.
Government of Ontario. 2006b. The Nursing Workload Measurement Project. Retrieved January 29, 2010. <http://www.health.gov.on.ca/transformation/providers/information/data_initiatives/bg_workload.pdf>.
Gran-Moravec, M.B. and C.M. Hughes. 2005. "Nursing Time Allocation and Other Considerations for Staffing." Nursing and Health Sciences 7(2): 126–33.
Hendrich A., M.P. Chow, B.A. Skierczynski and Z. Lu. 2008. "A 36-Hospital Time and Motion Study: How Do Medical–Surgical Nurses Spend Their Time?" The Permanente Journal 12(3): 27–34.
Hernandez, C.A. and L.L. O'Brien-Pallas. 1996. "Validity and Reliability of Nursing Workload Measurement Systems: Review of Validity and Reliability Theory." Canadian Journal of Nursing Leadership 9(3): 32–50.
Hughes, M. 1999. "Nursing Workload: An Unquantifiable Entity." Journal of Nursing Management 7(6): 317–22.
McGillis Hall, L., L. Pink, M. Lalonde, G. Tomblin Murphy, L. O'Brien-Pallas, H.K. Spence Laschinger, A. Tourangeau, J. Besner, D. White, D. Tregunno, D. Thomson, J. Peterson, L. Seto and J. Akeroyd. 2006. "Decision Making for Nurse Staffing: Canadian Perspectives." Policy, Politics, & Nursing Practice 7(4): 261–9.
O'Brien-Pallas, L., R. Cockerill and P. Leatt. 1993. "Different Systems, Different Costs? An Examination of the Comparability of Workload Measurement Systems." Journal of Nursing Administration 22(12): 17–22.
O'Brien-Pallas, L., D. Irvine Doran, M. Murray, R. Cockerill, S. Sidani, B. Laurie-Shaw and J. Lochhaas-Jerlack. 2001. "Evaluation of a Client Care Delivery Model, Part 1: Variability in Nursing Utilization in Community Home Nursing." Nursing Economics 19(6): 267–76.
Pringle, D. and P. White. 2000. "Nursing and Health Outcomes Project." RN Journal 12(3): 16.
Pringle, D. and P. White. 2002. "Nursing Matters: The Nursing and Health Outcomes Project of the Ontario Ministry of Health and Long-Term Care." Canadian Journal of Nursing Research 33(4): 115–21.
Pringle, D. and P. White. 2003. "Study Measures RNs Contribution to Patient Care." RN Journal 15(2): 22.
Twigg, D. and C. Duffield. 2009. "A Review of Workload Measures: A Context for a New Staffing Methodology in Western Australia." International Journal of Nursing Studies 46(1): 132-140.
White, P. and D. Pringle. 2005. "Collecting Patient Outcomes for Real: The Nursing and Health Outcomes Project. Canadian Journal of Nursing Leadership (1): 26–33.
Be the first to comment on this!
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed