Healthcare Policy

Healthcare Policy 4(2) November 2008 : 70-74.doi:10.12927/hcpol.2008.20267
Knowledge, Translation, Linkage and Exchange

Performance Reporting to Help Organizations Promote Quality Improvement

Canadian Health Services Research Foundation

Abstract

In healthcare, a great deal of time, money and energy go into producing public reports for a wide range of audiences. Reporting strategies often target audiences like the general public, whose behaviour is not readily changed by the information in report cards. However, when it comes to effectively targeting groups that can actually use the data to achieve significant impacts, one audience stands out from the rest: health system managers and providers, who can interpret and apply performance data to improve the quality of care their organizations deliver. The evidence behind performance reports was recently summarized in Evidence Boost for Quality, a special subseries of Evidence Boost, produced by the Canadian Health Services Research Foundation to showcase healthcare issues where research indicates a preferred course of action in health services management and policy. To access archived issues of Evidence Boost, visit < http://www.chsrf.ca/mythbusters/eb_e.php > .

The problem: The greatest quality improvement opportunities happen at the organizational level, where performance reports are scarce.

In healthcare, a great deal of time, money and energy go into producing public reports for a wide range of audiences. In Canada, this type of performance reporting - often packaged as public "report cards" - is carried out not only by governments, but also by advocacy groups, independent agencies and, in some cases, arm's-length organizations established by governments (Wallace et al. 2007). All these organizations try to present their data in a way that suits the needs of a specific audience. However, when it comes to effectively targeting groups that can actually use the data to achieve significant impacts, one audience stands out from the rest: health system managers and providers, who can interpret and apply performance data to improve the quality of care their organizations deliver (Wallace et al. 2007; Brown et al. 2005).

Reporting strategies often target audiences like the general public, whose behaviour is not readily changed by the information in report cards (CHSRF 2006). As well, report cards often provide systems-level or aggregate data that are of little use to managers or providers wanting to make sustainable improvements in individual organizations or facilities (Robinowitz and Dudley 2006; Shekelle 2005). If the goal, then, is to spur quality improvement activities and enhance quality of care, performance reports are best targeted at hospitals and managed-care organizations (Wallace et al. 2007; Brown et al. 2005).

Strategy for Change

When it comes to using results to make improvements, process-of-care indicators are often more useful than outcomes indicators (Wallace et al. 2007). Take reporting on wait times, for example. Knowing how long people are waiting in some areas of the country compared to other areas may be helpful in gauging the state of healthcare, but it's not altogether helpful in identifying why wait times vary and where improvement efforts need to be focused (CHSPR 2004). In the same way, if the data are not accurately risk adjusted, the report doesn't allow a proper "apples-to-apples" comparison.

Benchmarks are also useful, particularly for identifying top- and bottom-performing facilities (Robinowitz and Dudley 2006). Reporting agencies can adopt and adapt the best practices from top performers, while working with low performers to improve care (Wallace et al. 2007). In fact, this is a common practice of agencies like Cancer Care Ontario, which feeds data on wait times back to organizations across the province and works with them to make quality improvements. To promote a culture of learning, however, reporting should be carried out in a way that celebrates improvement and doesn't lay blame or condemn individual providers for poor quality of care (Marshall et al. 2003).

It is important that reporting agencies regularly consult with their stakeholders to ensure the relevance and validity of the indicators on which they report (Wallace et al. 2007). This is common practice for the Canadian Institute for Health Information (CIHI) - an independent, not-for-profit organization that reports annually to managers, policy makers and others on Canada's health systems and the health of Canadians (CIHI 2007a,b).

What the Research Says

While report cards don't appear to influence the healthcare decisions of patients (CHSRF 2006), they do have some success with providers, particularly health system managers and groups of providers working in hospitals and other healthcare organizations (Wallace et al. 2007). It is generally agreed that most quality improvements happen at the organizational level (Marshall et al. 2003; Hibbard et al. 2003; Barr et al. 2006; Halm and Siu 2005), and healthcare organizations have been found to be more likely than individual providers to respond to public reports (Barr et al. 2006). Responsiveness, or the lack of it, may be due to organizational culture, which can sometimes lie at the root of quality issues (Marshall et al. 2004).

The evidence of the effectiveness of public reporting on healthcare quality comes mainly from the United States, with some evidence from the United Kingdom and Canada (Brown et al. 2005; Morris and Zelmer 2005; Wallace et al. 2007). Several US studies that have measured improvements of quality initiatives have demonstrated small but important effects (Hibbard et al. 2005; Castle et al. 2007; Lindenauer et al. 2007). For example, a 2005 US study measuring the effect of public reporting on hospital performance in Wisconsin noted that hospitals receiving a public or private report showed statistically significant quality improvement compared to the control group that received no report (Hibbard et al. 2005). One important caveat is that if the data and indicators being reported are limited, reporting can provide an incomplete picture of care and lead to "gaming" of the system - the phenomenon of "what's measured is what matters" (Bevan and Hood 2006).

Meanwhile, other US studies looking at whether reporting stimulates quality improvement activities have found reporting to be effective (Mannion et al. 2005; Barr et al. 2006; Hibbard et al. 2003). In a study of 13 hospitals in Rhode Island, researchers found that one of the results of releasing a public report was that the data were used to target new quality improvement activities, evaluate performance and monitor progress (Barr et al. 2006). A similar study found that more quality improvement activities were launched in hospitals that were reported on publicly and privately versus those that received no report at all (Hibbard et al. 2003).

Conclusion

Public reporting is about more than mere accountability. Policy makers and reporting agencies wanting to ensure that these reports have an impact should look at the facility or the regional level as their prime target audience. Performance reports can lead to quality improvement activities, and to overall improvement in health services and outcomes, when they are directed at people involved in the delivery of care at the organizational level.


Favoriser l'amélioration de la qualité par la diffusion d'information sur le rendement

Résumé

Dans les services de santé, beaucoup de temps, d'argent et d'énergie sont consacrés à la production de rapports publics destinés à des auditoires variés. Les stratégies de reddition de comptes ciblent souvent des auditoires tels que le grand public, dont le comportement est peu influencé par l'information rapportée. Toutefois, lorsqu'il est question de cibler efficacement des groupes qui peuvent réellement mettre cette information à profit pour changer des choses, un auditoire se démarque : les gestionnaires et les fournisseurs de services de santé qui sont en mesure d'interpréter les données obtenues sur le rendement et de s'en servir pour améliorer les soins dispensés. Les données probantes sur la diffusion d'information sur le rendement ont fait l'objet d'un numéro spécial de Données à l'appui, "Données à l'appui pour la qualité", produit récemment par la Fondation canadienne de la recherche sur les services de santé pour faire connaître les aspects des services de santé où la recherche indique un plan d'action prometteur pour la gestion et les politiques en matière de services de santé. Pour consulter les anciens numéros de Données à l'appui, veuillez visiter le < https://www.chsrf.ca/mythbusters/eb_f.php > .

References

Barr, J.K., T.E. Giannotti, S. Sofaer, C.E. Duquette, W.J. Waters and M.K. Petrillo. 2006. "Using Public Reports of Patient Satisfaction for Hospital Quality Improvement." Health Services Research 41(3 Pt. 1): 663-82.

Bevan, G. and C. Hood. 2006. "What's Measured Is What Matters: Targets and Gaming in the English Public Health Care System." Public Administration 84(3): 517-38.

Brown, A.D., H. Bhimani and H. MacLeod. 2005. "Making Performance Reports Work." Healthcare Papers 6(2): 8-22.

Canadian Health Services Research Foundation (CHSRF). 2006 (September). "Myth: People Use Health System Report Cards to Make Decisions about Their Healthcare." Mythbusters. Retrieved September 30, 2008. < http://www.chsrf.ca/mythbusters/pdf/myth23_e.pdf > .

Canadian Institute for Health Information (CIHI). 2007a. Health Care in Canada. Retrieved September 30, 2008. < http://secure.cihi.ca/cihiweb/dispPage.jsp?cw_page=AR_43_E > .

Canadian Institute for Health Information (CIHI). 2007b. Health Indicators. Retrieved September 30, 2008. < http://secure.cihi.ca/cihiweb/dispPage.jsp?cw_page=indicators_e > .

Castle, N.G., J. Engberg and D. Liu. 2007. "Have Nursing Home Compare Quality Measure Scores Changed Over Time in Response to Competition?" Quality and Safety in Health Care 16: 185-91.

Centre for Health Services and Policy Research (CHSPR). 2004. "What a Tangled Web We Weave: Improving Performance Reporting and Accountability in BC." Vancouver: University of British Columbia. Retrieved September 30, 2008. < http://www.chspr.ubc.ca/files/publications/2004/chspr04-20C.pdf > .

Halm, E.A. and A.L. Siu. 2005. "Are Quality Improvement Messages Registering?" Health Services Research 40(2): 311-15.

Hibbard, J.H., J. Stockard and M. Tusler. 2003. "Does Publicizing Hospital Performance Stimulate Quality Improvement Efforts?" Health Affairs 22(2): 84-94.

Hibbard, J.H., J. Stockard and M. Tusler. 2005. "It Isn't Just about Choice: The Potential of a Public Performance Report to Affect the Public Image of Hospitals." Medical Care Research and Review 62(3): 358-71.

Lindenauer, P.K., D. Remus, S. Roman, M.B. Rothberg, E.M. Benjamin, A. Ma and D.W. Bratzler. 2007. "Public Reporting and Pay for Performance in Hospital Quality Improvement." New England Journal of Medicine 356(5): 486-96.

Mannion, R., H. Davies and M. Marshall. 2005. "Impact of Star Performance Ratings in English Acute Hospital Trusts." Journal of Health Services Research and Policy 10(1): 18-24.

Marshall, M.N., P.S. Romano and H.T. Davies. 2004. "How Do We Maximize the Impact of the Public Reporting of Quality of Care?" International Journal for Quality in Health Care (Suppl. 1): 57-63.

Marshall, M.N., P.G. Shekelle, T.O. Huw and P.C. Smith. 2003. "Public Reporting on Quality in the United States and the United Kingdom." Health Affairs 22(3): 134-48.

Morris, K. and J. Zelmer. 2005. "Public Reporting of Performance Measures in Health Care." Report no. 4. Ottawa: Canadian Policy Research Networks. Retrieved September 30, 2008. < http://www.cprn.org/documents/34883_en.pdf > .

Robinowitz, D.L. and R.A. Dudley. 2006. "Public Reporting of Provider Performance: Can Its Impact Be Made Greater?" Annual Review of Public Health 27: 517-36.

Shekelle, P.G. 2005. "The English Star Rating System - Failure of Theory or Practice?" Journal of Health Services Research and Policy 10(1): 3-4.

Wallace, J., J.F. Teare, T. Verrall and B.T.B. Chan. 2007. "Public Reporting on the Quality of Healthcare: Emerging Evidence on Promising Practices for Effective Reporting." Ottawa: Canadian Health Services Research Foundation. Retrieved September 30, 2008. < http://www.chsrf.ca/pdf/Public_Reporting_E.pdf > .

Comments

Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed