This paper provides insights into the use of performance data by middle managerial staff in Ontario hospitals in 2019 and compares the results to a study conducted in Europe in the same year. A total of 236 managers working in 61 hospitals across Ontario provided responses to the survey. Compared to their European colleagues, Ontario respondents self-assessed using significantly more performance data for managerial decision making. The use of performance data in Ontario was mostly motivated by external accountability requirements, followed by internal quality improvement efforts. Ontario managers also reported accessibility, appropriateness and timeliness of data and human resources and engagement as the biggest barriers to further performance data utilization. Comparative studies, such as the one this paper is based on, provide the foundation for drawing lessons across jurisdictions. This paper also affirms the importance of hospital middle management in moving from quality assurance to quality improvement efforts and developing sustainable learning healthcare organizations and systems.

Introduction

Performance data are vital for steering improvement in healthcare (Baker and Axler 2015). Measuring and reporting on a range of indicators allows healthcare organizations, including hospitals, to take stock of current performance and guide efforts to improve care, while monitoring progress. In driving improvements, it is important not only to measure and report performance internally but also to compare and benchmark against peers and set ambitious but achievable targets (Enticott et al. 2021). While the impacts of public reporting of performance data, such as through report cards, have received considerable attention in the literature (Prang et al. 2021; Tu et al. 2009), little attention has been paid to the challenges and enablers of the effective use of performance data by hospital managers to drive quality improvements. The challenge of transforming data into actionable indicators, which are fit for purpose and use (Barbazza et al. 2021), is largely about tying measurement information to quality improvement efforts on the front lines and closing the gap between learning from the data and actually introducing changes (Dhalla and Tepper 2018). Using performance data to support quality assurance and improvement strategies contributes to building safer, more efficient and equitable organizations and healthcare systems (Busse et al. 2019; Smith 2010). The COVID-19 pandemic additionally emphasized the role of timely, relevant and linkable performance data across healthcare services for improving organizational and system response and resilience (Kringos et al. 2020; WHO 2017).

Our previous research showed that the amount, focus and quality of performance measurement work in European hospitals did allow for sufficiently detailed performance insights (Ivankovic et al. 2020). Whether hospitals are truly equipped to turn these data into information and use them to improve outcomes for individuals and populations remains a question of paramount importance. Despite all the data available to support improvements in healthcare delivery, using it to implement innovations, even seemingly simple ones, often presents a major challenge (Alexander and Hearld 2011). Hospital executives adapting their human resource policies in order to train and support new types of roles – such as "quality managers" and "linking pin data champions" – often occupying middle managerial positions, might facilitate the use of hospital performance data to their fuller potential (Botje et al. 2016).

Hospitals in Ontario are predominantly private, not-for-profit organizations that receive most of their funding from the provincial government (Kraetschmer et al. 2014). The province carries a rich history of working with performance data and creating a culture of accountability in its acute care hospital sector. These developments are notably linked to the Ontario Hospital Report Research Collaborative, a joint effort launched in the 1990s between the University of Toronto and the Ontario Hospital Association (OHA) to develop the first province-wide performance measurement and monitoring system based on the balanced scorecard format (Baker and Pink 1995; Kaplan and Norton 1992; Pink et al. 2001). The developments in performance measurement in the past two decades were marked by a series of reforms, including the Public Sector Accountability Act, 2001, the 2006 formation of Local Health Integration Networks (LHINs), the Excellent Care for All (ECFA) Act, 2010, the Connecting Care Act, 2019, and, most recently, the establishment of Ontario Health Teams (OHTs) (Embuldeniya et al. 2021; MacLeod 2015; Veillard et al. 2015). Importantly, the ECFA Act made the development of quality improvement plans and use of performance data with targets a requirement for hospital boards in Ontario, adding a strong performance-based compensation component for the executives. Despite challenges brought about by the COVID-19 pandemic, the introduction of OHTs aims to change the landscape in which the hospitals are operating, with increased focus on integration of care and population health management (Embuldeniya et al. 2021; Fahey-Walsh et al. 2020). These initiatives also illustrate the evolution of conceptualizing performance measurement and its use for accountability over time: from funding agreements based around service volumes through collecting, reporting and using quality of care and patient safety indicators to its current focus on patient-reported outcome and experience measures, care integration, population health management and value-based healthcare.

A study among mid-level hospital managers from 23 European countries was conducted in 2019 by the same core research team, exploring the use of performance data for managerial decision making (Ivankovic et al. 2020). Among other findings, this work also highlighted the unique role that hospital middle managers play in closing the gap between creating evidence and implementing changes to care delivery. Through adopting evidence-based managerial practices, and due to their unique position as a critical link between organizational accountability and day-to-day quality improvement work (Gutberg and Berta 2017), hospital middle managers hold a huge potential to facilitate the shift from organizational accountability toward the model of learning organizations (Ivankovic et al. 2020).

With the broad aim of gaining insights into the opportunities for strengthening performance measurement and its usefulness in driving improvement in Ontario, this study looked closely within Ontario and comparatively with Europe exploring (1) why hospital managers worked with performance data; (2) what kind of data fed into this work; (3) how performance data were used; and, additionally, (4) Ontario-specific barriers to the use of performance data.

Materials and Methods

A survey-based descriptive cross-sectional study was conducted in 2019 among mid-level managers working in Ontario hospitals. The survey elicited information on the use of performance data for managerial decision making. A similar survey, conducted by the same core research team in the European context (Ivankovic et al. 2020), provided the basis for the survey design and comparative analysis of results.

Survey design, questionnaire adaptation and piloting

The original survey questionnaire was developed, validated and used in a study among European hospital managers in 2019 as described in detail in a published scientific paper (Ivankovic et al. 2020). This study involved 125, mostly mid-level, hospital managers from 23 European countries, participants in the European Hospital and Healthcare Federation's exchange program (European Hospital and Healthcare Federation 2022). For use in Ontario, the previously developed questionnaire was amended and re-validated. To validate questions meeting their measurement goals, five individual face-to-face cognitive testing (Collins 2003) interviews were conducted between October 25 and 30, 2019, involving mid-level managers working in Greater Toronto Area–based hospitals. The final questionnaire consisted of 29 mandatory open- and closed-ended questions and was structured in four parts as shown in the questionnaire in Appendix 1, available online here. The questionnaire was set up and distributed through an online surveying platform.

Study population, questionnaire dissemination and data collection

The target population of the study were mid-level hospital managers working in Ontario hospitals. In collaboration with the OHA, all 141 OHA member organizations essentially representing all the hospitals in Ontario, were contacted. Hospitals were approached directly by the OHA with an e-mail containing a brief explanation of the aim, scope and timeline of the study and the link to the online questionnaire. Invitations included a request to disseminate the questionnaire throughout organizations, specifically targeting mid-level directors and managers. Participation was voluntary, and respondents had the option to either remain anonymous or to provide contact information – only available to the OHA – if they opted to be informed on the results and included in potential follow-up work. No data on sex or gender of the participants were collected. The questionnaire was disseminated on December 4, 2019, with weekly reminders sent until December 20, 2019. Data collection was finalized on January 6, 2020.

Data analysis and the comparison to the results of the European survey

Descriptive univariate analysis of data was conducted using R (v.3.6.1) on a full sample of respondents who were, for certain segments of the analysis, grouped by their reported managerial position and the type of hospital they worked for. The OHA criteria for grouping hospitals were used for the stratified analysis (OHA n.d.).

We analyzed the survey data by addressing the broad questions of "why, what and how" of performance data use. The "why" focused on motivation, benchmarking and confidence in data. The "what" looked at data domains and sources, and was anchored in the Performance Assessment Tool for Quality Improvement in Hospitals framework developed for the World Health Organization (Veillard et al. 2005). The "how" explored the use of managerial tools and differences in patterns of use between managerial roles. Finally, Ontario-specific barriers to a more impactful use of performance data among hospital managers were analyzed.

Responses from the European study (Ivankovic et al. 2020) were used to compare survey results between Ontario and Europe for questions that were identical in both. Likert scale responses were recoded, frequency distributions calculated and statistical significance identified by observing non-overlap between 95% confidence intervals (CIs) of the means between results from the two surveys. Additionally, to validate the internal consistency of responses, a reliability coefficient was calculated using data from organization-level questions originating from hospitals with the highest number of individual responses.

Although this was primarily a descriptive, quantitative study, throughout the paper, quantitative results are illustrated and contextualized with respondents' verbatim quotations (Thorne 2021), provided in the questionnaire's open-ended questions.

Results

Characteristics of the study population

Individual respondents, working in 61 different hospital organizations in Ontario, provided 236 full responses. This made up 43% (61/141) of all OHA member hospitals, from all five Ontario health regions (Government of Ontario n.d.). The majority of responses came from community (61%, 144/236), large acute teaching (16%, 37/236) and small (8%, 19/236) hospitals.

Most survey respondents self-identified as mid-level hospital managers (82%, 193/236). Their roles equally involved managing support of care processes, such as quality, human resources, information technologies and financing (50%, 97/193), and managing clinical care processes (50%, 96/193). Less than one-fifth of all respondents (18%, 43/236) managed plans and strategies for entire organizations. Demographic characteristics of study participants in Ontario were comparable to those in the European study as shown in detail in Appendix 2, available online here.

Respondents from seven hospitals, six community and one paediatric, provided 42% (98/236) of all responses. This uneven sample distribution across organizations was used to assess internal consistency of responses. Cronbach's alpha values of 0.781, 0.831 and 0.852 for the three hospitals providing most responses (two community hospitals with 840 and 560 data points and a paediatric hospital with 490 data points) confirmed high internal consistency of the responses received.

Why is performance data used? Motivation, benchmarking and confidence in data

Mid-level managers reported that the primary motivation of the hospital to collect and report performance data was external accountability, specifically to ensure that externally set standards and goals are achieved. However, mid-level managers noted that their own motivation as individuals was to drive internal improvement, regardless of achieving set goals. Both organizational- or hospital-level and individual motivation to work with performance data were self-assessed to be significantly higher among Ontario participants, compared to the European cohort of hospital managers. These results are summarized in Table 1.

Respondents provided examples of using performance data for managerial decision making in their routine work. Selected free-text responses, illustrative of different managerial, are presented.

Performance data are utilized to determine areas of focus for our annual quality improvement plan. For example, a fall causing harm was an issue for the organization and, as such, it became an important goal as well as an indicator. (Director of quality, patient experience and patient safety)

The preparation of the annual budget requires an understanding of the hospital's ability to overcome contractual economics and other cost pressures in balancing the budget's bottom line. This includes service volumes and understanding of the Ontario Cost Distribution Methodology results to determine if the hospital is an outlier in terms of performance and the delivery of care. In addition, an assessment of population demographics is necessary to understand the unique patient needs of our catchment area. Closing budget gaps include a review of cost per weight case by category and a comparison of areas of service to peers. (Director of finance)

I use performance data to justify request[s] for additional resources [such as staff]. (Human resources director)

[We use performance data] to identify program quality improvement [QI] goals and QI change initiative priorities as well as to validate that changes have led to improvements. For example, we identified the opportunity to participate in a National Baby Friendly Initiative QI Collaborative to improve our exclusive breastfeeding rates at hospital discharge for women who intend to exclusively breastfeed because our rates were 25% below benchmark. (Director of maternal and child health)

Survey results also revealed significantly more use of performance data for benchmarking purposes in Ontario, compared to the European cohort of hospital managers. More than half (57%, 130/228) of Ontario respondents replied that the performance data from their organizations get used "a great deal" or "considerably" to benchmark with other comparable organizations, while only around one-third of European respondents (36%, 37/104) replied the same way.

We are working on making improvements in our emergency department wait time for admitted beds. We use the provincial data to contact organizations that are doing well to see which ideas worked for them and if we can implement them here. Then we look at our data the following month to see if the changes made an improvement and whether it was sustained. (Quality and patient experience manager)

Although European respondents reported less use of benchmarking, they found it significantly more useful for guiding improvement in their daily work (means of the recoded Likert scale responses were 2.66 and 2.29 with 95% CIs [2.48, 2.84] and [2.14, 2.44], respectively).

Both European and Ontario respondents expressed moderate confidence in reliability of performance data collected and used in their organizations (means of the recoded Likert scale responses were 2.47 and 2.39 with 95% CIs [2.30, 2.64] and [2.26, 2.52], respectively). Both cohorts also felt that decision making based on performance data made it significantly easier for them as managers to explain and justify their decisions (means of the recoded Likert scale responses were 3.00 for European and 2.71 for Ontario respondents with 95% CIs [2.87, 3.13] and [2.58, 2.84], respectively).

What domains of performance data, and from which sources, feed into data-driven decision making?

For most dimensions of data, considerable organizational performance data collection, reporting and use were reported, with a distinct cascading pattern. Compared to their European counterparts, Ontario managers reported significantly more work with patient and staff safety-related data, as well as patient-reported data, as shown in detail in Figure 1.

Data sources used the most for performance data, in both Ontario and Europe, were administrative and electronic health records (EHRs). Accreditation and patient-reported data were used significantly more among Ontario respondents, while population-based registry data were the least used data source as shown in Figure 2.

Almost all the respondents in Ontario reported accessing data from internal sources (99%; 234/236), with more than half additionally accessing performance data from external sources (56%; 131/236). As external sources, the majority used data provided by the Ministry of Health (77%; 101/131), LHINs (72%; 94/131) and the Canadian Institute for Health Information (70%; 92/131).

I work as a senior analyst in decision support, so I regularly extract and present performance data from internal and external resources. Once I have extracted the data, I review them and make decisions based on the "story" that the data are telling me. I also manage the coding department, so I regularly review performance data to determine our hospital's results. If the indicators are showing negative results, I will then audit charts to determine if staff are coding according to national coding guidelines and standards, or if there is a need to reach out to physicians regarding documentation initiatives. (Decision support systems manager)

How does performance data get collected, reported and used and what are the use patterns and tools used?

Ontario managers reported moderate to considerable participation in collecting data, preparing reports and using performance data for decision making in their routine work (means of recoded Likert scale responses were 2.35, 2.39 and 3.00 with 95% CIs [2.20, 2.40], [2.24, 2.54] and [2.88, 3.12], respectively). This was significantly more than that reported by managers in Europe (means of recoded Likert scale responses were 2.09, 2.02 and 2.57 with 95% CIs [1.88, 2.30], [1.81, 2.23] and [2.38, 2.76], respectively), except for the collection dimension.

The extent of use of performance data for decision making were similar between clinical and non-clinical managerial staff in Ontario. However, non-clinical managers participated significantly more in preparing reports based on performance data (means of the recoded Likert scale responses for the reporting dimension for clinical and non-clinical managers were 1.88 and 2.74 with 95% CIs [1.67, 2.09] and [2.55, 2.93], respectively). Results did not show significant differences in the extent of use of performance data between more (over 10 years) and less (under 10 years) experienced hospital managers in Ontario (means of the recoded Likert scale responses were 3.02 and 2.98 with 95% CIs [2.83, 3.21], [2.84, 3.13], respectively).

A large majority of Ontario respondents reported using business intelligence tools, such as scorecards and dashboards, to report on performance data in their organizations, significantly more than those in Europe. The exception was the use of control charts, which was more common in Europe, as shown in Figure 3.

Barriers to the use of performance data in Ontario

Timeliness and accessibility of data were recognized as the most important barriers to using performance data as shown in Figure 4. Additional barriers emerged from the free-text replies: (1) appropriateness, relevance and usefulness of performance data; (2) physician engagement, user buy-in and "audience appetite"; (3) data "overload," excessive workload, lack of time and staff to work with the data; and (4) lack of confidence in systems that collect data, their accuracy and lack of consistency.

Poor performance on our dashboard prompted us to dive [deep] in[to] understanding what exactly was happening. We figured out that poor data quality and duplicate cases remaining open in the system were [some] of the bigger contributors. (Quality improvement specialist)

Discussion

This study aimed to provide insights into the use of performance data for managerial decision making in Ontario hospitals and to draw comparisons with Europe (Ivankovic et al. 2020). In gdeneral, the Ontario results showed considerable use of performance data among surveyed mid-level hospital managers, albeit with room for improvement. Compared to the European cohort, there seemed to be more use of performance data for managerial decision making in Ontario. Ontario managers reported accessibility, appropriateness and timeliness of data, as well as human resources and engagement to work with data, to be the most important barriers in performance data utilization.

Organization-level use of performance data was mostly motivated by external accountability and quality assurance, while the routine, daily work of mid-level hospital managers in Ontario favoured its use for driving quality improvements. Despite more extensive use of data for benchmarking among Ontario respondents, their European counterparts found benchmarking more valuable for guiding improvement efforts in their daily work. In light of the ongoing OHT Performance Measurement Framework's (Ontario Ministry of Health 2021) development and implementation in Ontario, these results indicate that summative (quality assurance) and formative (quality improvement) functions of performance data (Freeman 2002), as well as different approaches to benchmarking (Bevan et al. 2019; Ettorchi-Tardy et al. 2012; Klazinga et al. 2011), merit further research and discussion on the provincial level.

Both Ontario and European respondents showed similar cascading patterns of performance data collection, reporting and use – indicating more is collected than reported and finally used for managerial decision making. There seemed to be more activity with performance data use in Ontario in general, especially with data domains related to patient and staff safety and those reported by patients. Interestingly, the collect-report-use slope seems to be more pronounced for data categories that are mandatory to collect and report, such as patient and staff safety, as opposed to non-mandatory ones, such as patient-reported data.

As data sources, administrative records were used the most by both Ontario and European respondents, followed by EHRs. Interestingly, Ontario respondents reported using accreditation and patient-reported data almost as frequently as EHR data. These findings indicate that the already present, real-world and patient-reported nature of performance data sourced from different datasets needs to be carefully considered. This is especially true when discussing mechanisms to improve and appropriate indicators to capture the integration and coordination across providers and integrated care networks, such as the newly formed OHTs. Considering the care integration and population health management focus of these initiatives, a closer look into the use of population-based registries, which is more prevalent in Europe, and data linkage possibilities seems warranted. It is important to note a more homogenous nature of accreditation data across Ontario, whereas what accreditation means, if it is performed and how it is performed differs significantly between European countries (Araujo et al. 2020; Chuang et al. 2019).

Accessibility and timeliness of data emerged as the biggest barriers to performance data utilization in Ontario. Respondents also noted issues of appropriateness of existing metrics, human resources and engagement to work with data, as well as confidence in systems that collect data. These findings partially mirror the results of a recent study on barriers across European health information systems and enforce the need for sharing and learning from international experiences (Bogaert et al. 2021). In Ontario, as well as worldwide, the COVID-19 pandemic resulted in a more proactive use of performance data by hospitals, often using more timely data and predictive analytics, mostly to manage urgent capacity bottlenecks. Sustainability of these efforts, and their influence on readiness to use data across levels of care, in line with the OHT transformation, surely merit further research (CIHI n.d.; Krylova et al. 2022).

Research on the specific role of middle management in hospitals suggests that this managerial layer is vital to successful adoption of innovation, based on data-driven managerial decision making (van Beers et al. 2022). Despite various initiatives over time, including quality improvement program-based pay-for-performance schemes for executives and managers, alignment of external reporting and internal improvement remains, according to our study findings, a challenge. It is becoming increasingly clear that in order to streamline processes, improve outcomes and reduce variation, work of middle managers needs to be better understood and supported, to which this study adds. On the other hand, increasing requirements to measure and report significantly add to the "data burden" and are becoming a questionable investment of resources. Some evidence shows that more focus on collecting and reporting data might be harmful, resource-wise, to working on patient care and improvement initiatives. Still, without performance data, it is impossible to say whether improvements work in providing better care and outcomes (Kromm et al. 2014), calling attention to the delicate balance between collecting too much and using too little.

This study has several strengths. To our knowledge, this was the first comparative study exploring patterns of performance data use among middle managers in hospitals across Canada and Europe. Working with professional organizations – OHA in Ontario and HOPE in Europe – allowed for direct access to relevant cohorts of target study participants. We also recognize potential limitations of this research. Sampling approaches differed among the two cohorts, with the European study targeting potentially more proactive participants to the professional development program. It must be acknowledged that the contexts and ways in which healthcare systems work between, and among, European countries and Ontario also differ significantly. Ontario respondents present a more homogenous cohort compared to that of European managers, working in 23 different countries and health systems around the continent. We are aware this limits the generalizability of findings and hinders certain aspects of comparison between Ontario, Canada, in general, and Europe – as does the perception- and opinion-based nature of the survey used. Also, distribution of responses in the Ontario survey showed signs of unevenness across organizations, with almost half of all the responses coming from only seven hospitals – a fact that we tried using to our methodological advantage by calculating internal consistency of the results received, but one which surely limits generalizability of our findings.

Conclusion

Comparative studies on performance intelligence production and use, such as this one, can facilitate learning across jurisdictions. Given the decades of work in Ontario preceding this study, the differences to results in Europe are far from surprising, but they still do provide a pre-pandemic snapshot of the fast-changing performance data landscape. Our study findings also signal the importance of middle managerial staff in moving from assurance-based to improvement-based work and developing sustainable learning health systems and potentially encourage policy makers and system managers to further bridge existing gaps in use of data for continuous quality improvement. Implications for further positioning of Ontario hospitals in the patient-centred care and population health management era, emphasized by the OHT reform and challenged through the COVID-19 pandemic, remain to be seen.

Ethical considerations

Collection and analysis of data via this e-survey has been approved. Human participant ethics protocol (#38397) entitled "Survey of Hospital Managers in Ontario on the Use of Performance Data, Information and Intelligence" was submitted to the University of Toronto Office of Research Ethics and was granted approval (#17069) on November 22, 2019.

Funding

This work was carried out by the Marie Sk?odowska-Curie Innovative Training Network for HealthPros that has received funding from the European Union's Horizon 2020 research and innovation program (under grant agreement No. 765141). The funding organization had no role in the survey's design, implementation and analysis.

Declaration

At the time of conducting this study, Imtiaz Daniel, Sundeep Sodhi and Tessa Dundas were employed by the OHA. The views expressed in this publication are those of the authors. They do not purport to reflect the opinions or views of the OHA or the Canadian Institute for Health Information.

Utilisation des données sur le rendement par les gestionnaires d'hôpitaux de niveau intermédiaire en Ontario: résultats d'un sondage à l'échelle de la province et comparaison avec les gestionnaires d'hôpitaux en Europe

Résumé

Cet article donne un aperçu de l'utilisation des données sur le rendement par le personnel de gestion intermédiaire dans les hôpitaux de l'Ontario en 2019 et compare les résultats à une étude menée en Europe la même année. En tout, 236 gestionnaires œuvrant dans 61 hôpitaux ontariens ont répondu au sondage. Comparativement à leurs collègues européens, les répondants ontariens déclarent utiliser beaucoup plus de données sur le rendement pour la prise de décisions en matière de gestion. L'utilisation des données sur le rendement en Ontario est principalement motivée par les exigences externes en matière de reddition de comptes, suivies d'efforts internes d'amélioration de la qualité. Les gestionnaires ontariens indiquent également que l'accessibilité, la pertinence et l'actualité des données, des ressources humaines et de l'engagement étaient les principaux obstacles à une utilisation plus poussée des données sur le rendement. Des études comparatives, telles que celle sur laquelle se fonde le présent document, fournissent la base pour tirer des leçons entre les juridictions. Cet article affirme également l'importance de la gestion intermédiaire hospitalière dans le passage de l'assurance de qualité vers les efforts d'amélioration de la qualité ainsi que dans le développement d'organisations et de systèmes de santé d'apprentissage durables.

Comments

Be the first to comment on this!