Objectives: To determine the factors associated with the use and impact of performance data relevant to women's health.

Methods: We developed a survey on six levels of information use based on Knott and Wildavsky's (1980) policy utilization framework and used this survey to determine Ontario hospital administrators' use of women's health report indicators. We related responses to this survey to six potentially relevant organizational factors, such as women's health as a written hospital priority, a women's health program and hospital budget size, using correlation and multiple-regression analysis.

Results: Only women's health in a written hospital priority (p=0.01) and hospital budget (p=0.02, log transformed) were significantly associated with the highest level of use when all organizational factors were considered.

Conclusion: These findings suggest that the use of women's health performance indicators is strongly related to the size of the hospital budget and to organizational commitment to women's health.

Performance measures can be disseminated in the form of public reports for healthcare users and providers to assess available care, or as private reports for internal access by providers only (Hibbard et al. 2003). Different reports target different stakeholders and have different objectives that range from informing consumer choice, ensuring accountability in healthcare, supporting quality improvement in healthcare delivery and increasing efficiency of health services (Morris and Zelmer 2005). The publication of reports on healthcare to inform consumer choice is more prevalent in the United States, while Canadian efforts tend to promote accountability or improve performance in healthcare (Morris and Zelmer 2005).

A performance report can be utilized by healthcare providers (hospitals or individuals) to identify their level of performance and stimulate improvements in their quality of care (Hibbard et al. 2003; Brown et al. 2005); however, studies tend to show that performance reports have limited uptake and mixed effects on performance (Marshall et al. 2000). It is important for organizations to learn "how to link the performance measurement results to actions [for performance improvement], rather than having the performance measurement system simply keep records" (Adair et al. 2006: 67). In Canada, a study evaluating the usefulness of performance measures in the first acute myocardial infarction report card stated that 54% of responding hospitals made one or more changes as a result of the report (Tu and Cameron 2003). A systematic review of 11 studies found that public reporting of performance measures stimulated quality improvement activity, but there was mixed evidence for outcome improvement (Fung et al. 2008).

The current performance measurement literature focuses predominantly on public performance reporting, and thus research on the use of private performance reports is lacking. Canadian researchers studying effective knowledge transfer indicate that individualized feedback may help research organizations, such as the Hospital Report Research Collaborative (HRRC), improve their research dissemination and knowledge transfer activities (Lavis, Robertson et al. 2003). This paper assesses the use of a private report by hospitals that received Hospital Report 2003: Acute Care – Women's Health Private Report (WHPR). We hope to gain more insight into the factors that influence the use of privately reported performance measures so that we can facilitate performance improvement in women's health.

The report was published by the HRRC and provides hospital-specific results on women's health performance based on sex-specific (women only) and sex-sensitive (ratios of men vs. women) indicators of patient satisfaction, clinical outcomes and measures of management behaviour related to women's health. The WHPR was distributed to 96 Ontario acute care hospitals (80% of acute care hospitals) that voluntarily participated in the Hospital Report project; the methods underlying the project are described elsewhere (Magistretti et al. 2002; HRRC 2003).

Performance reports in women's health are more common now than they were a decade ago. However, there is relatively little evidence on the use of women's health performance reports by hospitals or individual providers. Nevertheless, there have been evaluations of hospital performance reports that may include components of women's health. A study by Hibbard and colleagues (2003) found that public disclosure of performance information for obstetrics care stimulated significantly more quality improvement activities in areas of low performance than private reports; but there was no significant difference for cardiac care. Providers may use broad, comparative public reports to stimulate improved performance if the policy context supports performance reporting and improvement (Brown et al. 2005), but will similar results be seen with private reports specific to women's health? There are many reasons why performance reports focusing on women's health may not have this effect, including, but not limited to, historic marginalization of women's health, the absence of a focus on women's health issues at a hospital and debate over what constitutes good performance in women's health.

Recent evidence indicates that organizational contextual issues are a factor in the effective use of performance measurement within the complex health system environment (Adair et al. 2006), and organizational characteristics have been associated with the perceived usefulness of performance measures in hospitals (Ginsburg 2003). This paper will describe the different levels of information use associated with the release of the WHPR and some of the organizational factors associated with its increasingly intensive use. Certain organizational factors have been associated with the uptake of innovations (Moch and Morse 1977; Kimberly and Evanisko 1981; Romano et al. 1999). For example, larger organizational size is positively related to the use of innovation in hospitals (Moch and Morse 1977; Kimberly and Evanisko 1981; Romano et al. 1999) potentially because larger size provides additional resources to support adoption activities (Moch and Morse 1977). Likewise, smaller hospital budgets may have fewer resources for research utilization and performance improvement. We anticipate that setting women's health as an organizational priority, providing women's health programs, or both strategies may also increase the use of the WHPR within hospitals because they suggest a focus on women's health in the "perceived improvement culture," defined as "the extent to which a respondent feels his or her hospital values performance data and supports using the data to bring about improvement" (Ginsburg 2003: 269). A study by Tung and Yang (2009) examined the factors that would improve performance in the Taiwan Healthcare Indicator Series and reported that the most important factors in performance improvement were senior management support, which signals a priority, and activities to apply the performance information effectively, such as benchmarking among hospitals.

Our study assessed the relationships between the intensity of information use of the WHPR and six organizational (independent) factors based on a literature review and consultation with researchers involved in studying quality improvement at Ontario hospitals. The following factors were either associated with an increased probability of improvement capacity or a focus on women's health:
 

• The size of the hospital budget was studied because it represents the available financial resources that could be dedicated to performance management, and is more granular than a categorical label of organizational size (Moch and Morse 1977; Kimberly and Evanisko 1981; Romano et al. 1999).
• The presence of a women's health champion among senior management was studied because women's health champions have been designated at hospitals to promote and facilitate women's health initiatives. Thus, women's health champions who are senior managers may strive to improve women's health performance measures by dedicating more resources to the area. In addition, the leadership and commitment of senior decision-makers may have an important "agenda-setting" role in an organizational response to performance data (Huberman 1994; Ginsburg 2003; Adair et al. 2006).
• Hospital prioritization of women's health in a written statement is an explicit commitment to women's health that may steer performance improvement towards women's health (Huberman 1994; Ginsburg 2003; Brown et al. 2005).
• Presence of a women's health program in a hospital was included as a factor because hospitals with such a program may perceive greater relevance in the women's health data, which could enhance data use for performance improvement (Huberman 1994; Ginsburg 2003). It was also expected that larger hospitals or hospitals with larger budgets have more programs and resources and therefore would be more likely to have a women's health program or be more interested in the report.
• Benchmark ranking of a hospital was also examined. This ranking is based on hospitals' achieving (and sustaining) good performance across several indicators to identify whether information use of the WHPR is associated with good performance (CIHI and HRRC 2005). Benchmark ranking was studied because hospitals with higher rankings may be more responsive to performance information in order to achieve and maintain good performance in areas that may include women's health.
• The hospital peer group (community, teaching or small hospital) was studied to identify a relationship between hospital type and information use of the WHPR because small hospitals have fewer financial, technological and human resources than the other types of hospitals (Joint Policy and Planning Committee 1997). Thus, we hypothesized that they would be less responsive to the WHPR.

In this study, we adapted Knott and Wildavsky's (1980) framework for seven stages of utilization to describe the intensity of use of the WHPR in Ontario hospitals. Table 1 shows our operationalization of Knott and Wildavsky's framework. The framework was originally designed to assess the uptake of information by decision-makers in terms of conceptual and instrumental use. Conceptual use was defined as the use that occurs when information influences one's perception of issues in general, and instrumental use occurs when a decision or action follows in part from the information (Rossi and Freeman 1985; Lavis, Ross et al. 2003). The seven stages of information use form a continuum from the least (reception level) to the most intensive use of information (impact level). One level builds on the previous level in a progressive manner such that each level must be completed before progressing to the next (Knott and Wildavsky 1980; Landry et al. 2001). The framework does not identify any independent factors that are associated with the achieved level of information use, but it has been used extensively to study the use of information by practitioners, professionals and decision-makers (Landry et al. 2001, 2003).

Methods

Questionnaire development

We constructed a survey to capture six of the seven levels of the conceptual framework as described in Table 1 and three hospital characteristics that could not be captured in routinely collected data: a hospital's prioritization of women's health in its written statements, presence of women's health programs and presence of a women's health champion among senior leadership: (1) "Is women's health explicitly articulated as a focus in any of your hospital's vision statements, business plans or other written statements of mission or support?" (2) "Does your hospital currently have at least one women's health program?" (Any hospital program in which the provision of care and/or services and/or research related to women and/or sex and/or gender is a central component, i.e., may or may not be designated or formally named "women's health" program.) (3) "Are you currently in a senior management position at the hospital? (i.e., vice-president, chief of staff, chief of finance or other senior executive)." The entire questionnaire is available from the authors (ADB) on request. We pilot-tested the draft questionnaire with three individuals and modified questions slightly afterwards to ensure the tool had face validity with hospital employees.

Questionnaire administration and other data collection

We sent the survey to three individuals within each hospital: chief executive officers (CEOs), women's health champions and Hospital Report Research Collaborative contacts from hospitals who participated in the Hospital Report project. CEOs were selected because only the CEO of each hospital participating in the Hospital Report 2003: Acute Care series directly received the WHPR. Thus, any dissemination of the report in a hospital would have started from the CEO, and his or her use of it would provide crucial information on the report's initial propagation within the hospital. Women's health champions were chosen because they were the CEO-nominated liaison for the hospital on women's health issues and identified as championing women's health activities in their organization. These champions range in their positions and titles from director of quality to vice-president of clinical services to chief financial officer. Hospital Report contacts were selected because they were the designated liaison between their hospital and the HRRC on any report card–related activities. All participants had a unique identifier code on their survey.

Hospitals were excluded if they stated that they did not want to participate in the study or had more than two turnovers in the year preceding the survey in two of the three positions studied; the latter exclusion maximizes accurate recall and avoids burdening any single remaining participant from a hospital. CEOs hired after December 2003 were also excluded because they did not directly receive the report. Individuals who held two or more of the positions under study were sent only one survey with one identifier code. A CEO was always coded as a CEO in order to examine the process of dissemination in the hospital. A person with the roles of both a women's health champion and HRRC contact was identified as a women's health champion. The effect of these exclusions is likely to bias results upwards in favour of greater utilization.

Ninety-six acute care hospitals in Ontario participated in Hospital Report 2003: Acute Care – Women's Health Private Report; however, only 80 hospitals were eligible for the study after the exclusion criteria were applied (discussed above). In November 2004, a total of 216 surveys were sent to the 80 hospitals eligible for the study (total: 70 CEOs, 78 women's health champions and 68 HRRC contact persons). Reminders were sent to non-respondents via e-mail or fax approximately two weeks after the deadline date, with the final deadline for response three weeks later (February 2005).

The data on hospital peer group, Hospital Report benchmark ranking and hospital budget (total operating revenue for fiscal year 2003/2004) were extracted from existing HRRC data sets designated for research activities. The study was approved by the ethics review boards at both the University of Waterloo and the University of Toronto.

Analysis

The unit of analysis consisted of two levels: individual and organizational. The questions in Table 1 were used to generate a score to determine the level of information use achieved by individuals. SAS 8.2 (Cary, North Carolina) was used to calculate the scores and to conduct the statistical analyses. Each achieved level from reception to impact was assigned a value of 1, where reception had to be reached before cognition was coded, cognition before effort, and so on for a maximum total score of 6. The highest organizational level of information use was based on the highest score from the individual responses for each hospital. Fisher's exact test was used to analyze the frequencies for peer group, women's health programs, women's health as a written hospital priority and women's health champions among senior management. A generalized linear model (GLM) approach was used to analyze the levels of organizational information use and their relationship with the organizational factors and interactions between them to control for potentially confounding factors. In addition, the generalized model framework handles continuous and categorical factors in a natural fashion. Collinearity, if present, was assessed through correlational analysis, examining the effect of omitting each variable in turn from the model and noting the change in the parameter estimates left in the model (data available on request).

A sensitivity analysis was conducted for two organizational characteristics because of inconsistencies in responses from respondents within organizations: the presence of women's health programs and women's health as a written hospital priority. The two separate assumptions were based on whether one or two of the respondents responded positively on an organizational characteristic. There were no differences in the results based on the sensitivity analysis.

Results

Fifty-eight of 80 hospitals responded with at least one survey (72.5%), and 18 responded with two or more surveys (22.5%). There was no statistically significant difference (p=0.61) in response rates between the three groups of individuals: 35.7% (25/70) of CEOs, 41.2% (28/68) of HRRC contacts and 33.3% (26/78) of women's health champions. However, there was a significant difference (p=0.0004) in response rates across hospital peer groups: 76.3% (45/59) of community, 100% (10/10) of teaching and 27.3% (3/11) of small hospitals responded. Table 2 shows that there was a significant difference (p=.0004) across the peer groups in respondents and non-respondents; respondents were more likely to be a community or teaching hospital, and had a mean hospital budget more than twice the size of non-respondent hospitals (p=0.0007).

Table 3 shows that the majority of respondents reported receiving and understanding the report (cognition level), but the use of the WHPR declined with subsequent levels of use for both units of analyses, with a greater drop-off from the cognition to effort level. Only 19.0% of individuals and 20.7% of hospitals reported reaching the impact level. Interestingly, every organization that implemented actions stimulated by the WHPR also reported a beneficial impact on its policies, programs or both; thus, impact was the most intense level of information use. For the survey respondents who did not use the WHPR (26.6% individuals; 20.7% organizations), HRRC contacts made up the greatest proportion (47.6%), despite their role in managing information coming from the HRRC. Among those who reported using the WHPR, there was no significant difference identified in the highest level of use among the three groups (p=0.1574).

Among responding hospitals, 86.2% (50/58) currently provide a women's health program, and 17.2% (10/58) have women's health as a written priority. Table 4 provides results from a univariate GLM analysis of relationship between the highest levels of information and each of the organizational factors of interest. In the single-factor model, only peer group, budget and women's health as a written hospital priority were significantly associated with the highest level of information use (p=0.03; p=0.0020; p=0.001, respectively). However, Table 5 shows that in the multivariate model, only the written hospital priority factor and hospital budget (log transformed) were significant, with p-values of 0.01 and 0.02, respectively.

Discussion

This study is the first to examine the use of a women's health private performance report among hospitals. Overall, the findings from this study show that reporting on women's health performance may help support improvement in women's health and that mainstreaming women's health into quality improvement in hospitals is possible by making this issue a documented priority for the organization. The findings also emphasize the importance of a strategic focus on women's health that may compensate for some of the historical and social disadvantages that have prevented women and girls from achieving equity with men and boys in health and healthcare (Health Canada 2000). "Equity" and "equality" may be used interchangeably in some organizational strategies. Although they do not mean the same thing, hospitals may apply them in a clinical context to the same ends. The decision to focus on women's health within the hospital may also reflect a more general business strategy to offer such services or programs.

The use of the WHPR was low, and the results suggest problems in sharing data. The WHPR successfully reached 79.3% of the hospitals that responded but only 36.6% of all the individual CEOs, women's health champions and HRRC contacts surveyed. This low level of dissemination among relevant individuals may be the result of poor communication between chief executives and managers or a low priority accorded to women's health. The use of the WHPR declined after reception for subsequent levels of use. The greatest drop-off in information use appears to be between the cognition and effort levels, thus identifying an area where greater efforts are needed to encourage and enable receivers of performance reports to adapt the information for decision-making. The low numbers of those who implemented change as a result of the WHPR (implementation level) is consistent with a study on the California Health Outcomes Project (CHOP) that found that two-thirds of respondents did not take specific actions or make an in-hospital response to the AMI mortality data in the CHOP report (Rainwater et al. 1998). As in that study, the low number of hospitals that acted on the WHPR may be due to a lack of relevant data in the report. Other factors may also account for the low information use, such as the awareness and understanding of the report, human or financial resources, leadership support and limited organizational focus on quality improvement. Hospitals and individuals who reported completing the implementation level of information use also reported experiencing a beneficial impact. This suggests that the WHPR has a part in stimulating action and producing at least some perceived benefits or improvements, a finding that is consistent with other studies on performance report data (Dziuban et al. 1994; Rainwater et al. 1998; Davies 2001; Fung et al. 2008).

Women's health as a written hospital priority is strongly correlated with the hospital budget and an increased use of the WHPR. The observed relationship suggests that larger budgets are associated with more resources for hospitals to express, focus and act explicitly on their formal priorities, and may allow priorities to be devoted to specific areas such as women's health. However, Brownell and colleagues (2001) provided evidence that healthcare reform as a result of budget reductions may result in more efficient healthcare services while maintaining equity.

It is important to note that the reported correlations do not, on their own, indicate causation. In fact, there may still be other, unexplored factors, such as organizational commitment to quality improvement, that could explain the observed relationship. Given the limits on our sample size – the number of hospitals in Ontario – further work may usefully pursue case studies or other qualitative techniques to draw out reasons for the association. Similarly, the association with peer group may mean that hospitals with highly educated teaching clinicians may be more likely to develop or implement women's health programs.

Interestingly, over 86% of responding hospitals had a women's health program, but it was not an influencing factor in the use of the WHPR. This finding implies that simply having a designated women's health program is not enough to drive the use of women's health performance data in hospitals when attempting to improve women's health. This finding also reinforces the fact that women's health as a program is not always the same as women's health as a priority. A program, particularly one with typical foci on gynaecology and obstetrics, does not necessarily mainstream women's health in the organization. In order to increase the use of women's health performance information to improve women's health, future work may usefully focus on assisting organizations in the development and integration of women's health as a formal corporate priority. However, it should be noted that sex-specific and sex-sensitive data may result in different patterns of use, depending on the presence of a women's health program in a hospital, but the study design did not reflect this. Additionally, hospital priorities are the purview of the board of directors, but there was no contact with this group of decision-makers to confirm women's health as a priority in each hospital.

Limitations

There were several limitations to this study. The voluntary nature of study participation has the potential for self-report bias among the 58 hospitals and 79 individuals who responded to the survey. The results of this study are also likely to overstate the findings for hospitals, meaning that we may have overstated the uptake of information. The low response rate may be attributed to the following factors: candidate respondents with competing priorities, survey timing (i.e., surveyed close to and during Christmas holidays), survey burden (i.e., hospital managers in Ontario responded to a large volume of surveys in 2004), and non-traditional survey method as surveys were only e-mailed (or faxed) to potential participants. As well, the request for completing the survey by multiple survey recipients in one organization required further clarification for some respondents. The multiple-respondent survey method may have lowered the response rate as a result of the misunderstanding that only one response was necessary when responses were expected from all survey recipients. All biases in the study are likely to inflate the results on intensity of information use, but these biases are unlikely to affect the observed associations between information use and organizational characteristics. The findings on the intensity of information use and on the associated organizational characteristics suggest that women's health performance data can be introduced or mainstreamed into typical corporate improvement initiatives, but that much effort is required to make women's health a typical corporate priority.

Conclusion

Of course, simply taking the steps described in this paper is not enough to increase the use of performance reports and improve outcomes. Research organizations that want to improve performance – such as those that publish performance reports in Canada – should collaborate with hospitals to provide relevant performance measurements and guide hospitals in the interpretation and use of the information to facilitate improvement in priority areas (Morris and Zelmer 2005). A study on cancer care indicator preferences in Ontario reported that different stakeholders had different preferences and that the "strategies for maximizing the relevance of performance reports might include technical process indicators, selection by multi-stakeholder deliberation, information that facilitates information application and customizable report interfaces" (Gagliardi et al. 2008: 175). A combination of strategies should be further examined and considered to promote the uptake of future performance reports. Actions to ensure that performance reports bring about awareness and change are important as we promote improvements in women's health.

Importance de la mesure du rendement dans la promotion de l'amélioration de la santé des femmes

Résumé

Objectifs : Déterminer les facteurs associés à l'utilisation et à l'impact des données sur le rendement pertinentes à la santé des femmes.

Méthodologie : Nous avons mis au point un sondage portant sur six niveaux d'utilisation de l'information, fondé sur le cadre d'utilisation de Knott et Wildavsky (1980). À l'aide de ce sondage, nous avons déterminé l'utilisation des indicateurs sur la santé des femmes par les administrateurs des hôpitaux ontariens. Au moyen de corrélations et d'analyses de régression multiple, nous avons établi le lien entre les réponses au questionnaire et six facteurs organisationnels potentiellement pertinents, tels que la santé des femmes inscrite comme priorité de l'hôpital, un programme pour la santé des femmes et la taille du budget de l'hôpital.

Résultats : Après avoir considéré tous les facteurs organisationnels, seuls la santé des femmes inscrite comme priorité de l'hôpital (p=0,01) et le budget de l'hôpital (p=0,02, transformation logarithmique) ont un lien significatif avec de hauts taux d'utilisation.

Conclusion : Ces résultats laissent croire que l'utilisation des indicateurs du rendement pour la santé des femmes est étroitement liée à la taille du budget et aux engagements de l'hôpital envers la santé des femmes.

Comments

Be the first to comment on this!