Sarah Bowen and Sara Kreindler argue that indicators can be valuable, but are also often flawed. They suggest that performance indicators should at best serve as a flag for policy makers but should not drive decisions. We would argue that there is growing evidence of the positive impact of performance indicators. When performance indicators are selected based on sound strategies - and used as part of a clear performance management cycle that balances policy instruments (e.g., accountability agreements) and performance improvement processes (e.g., process redesign) - they can drive valuable performance improvements and help align strategies across all health system partners.

Bowen and Kreindler have crafted an engaging and well-written argument on some of the limitations of performance indicators in healthcare. In comparison to a number of other cautionary papers on indicators, they describe these limitations using practical examples but also acknowledge that all signs suggest indicators are here to stay. After documenting the challenges of working with decision-makers "who may lose sight of the real questions facing the healthcare system" and "who need to apply the results [but] may not be able to understand them," they conclude that "an indicator is just an indicator." It is meant to be a " 'tool, screen, or flag' (CCHSA 1996) to assist in decision-making, not a driver for decisions." This pronouncement leaves decision-makers in a rather dismal situation.

It might help Bowen and Kreindler's critique to put in context the ways in which indicators are being used across North America today and to relate this use to emerging evidence on indicator development and the most effective application of indicators in measuring and improving performance.

How Are Decision-Makers Using Indicators?  

Indicators are increasingly used as part of accountability agreements between those who fund care and those who organize or provide care. On their own, indicators can reflect increasingly important societal values such as transparency, accountability and fiscal responsibility. We can see clear examples of this in the United States, where pay-for-performance based on indicators has become a remarkably common tool for performance improvement and increased accountability. But we can also see examples of this usage in Canada and the United Kingdom, where reporting and improvement with reference to performance indicators are an important part of contracts for services. In this case, indicators are at the core of a performance management cycle that includes strategy development (What are we trying to achieve?), articulation of goals (How will we know when we are successful?), resource allocation based on priorities (How can we link budget and goals?), assignment of responsibility (Who will deliver?) and the constant evaluation and improvement against those goals and redefinition of strategy (What can we do better, and what have we learned?). This performance management process is exemplified in Ontario, where the agreements between the Ministry of Health and Long-Term Care and the Local Health Integration Networks (LHINs) reflect all these steps.

As governments and their agents move increasingly towards the role of purchasers of services (Salamon 2002), they will require indicators to assess whether or not these services have been provided according to standards. As all healthcare systems confront the challenge of sustainability, they will also need to use indicators to ensure that tax dollars are producing value for money and to ensure that governments can report to the public on this value. It is unclear whether Bowen and Kreindler's advice is compatible with this use of indicators. Indeed, in many cases indicators will drive decisions about whether or not to renew contracts, to increase or reduce capacity at an institution, or to investigate threats to the public interest.  

Where are indicators working well?  

Bowen and Kreindler make good points about what can go wrong with indicators, but they pay little attention to situations where indicators have had a strong impact. There are now multiple examples where attention to performance reporting has paid substantial dividends, even with imperfect indicators. Bevan and Hood's (2006) paper on how performance reporting in the United Kingdom improved health outcomes, Ken Kizer's experience at the US Veterans' Health Administration, where he relentlessly measured performance against key goals (Asch et al. 2004) and even the Institute for Healthcare Improvement's (2004) 100,000 Lives Campaign have used indicators to motivate health providers to better performance. By focusing on the potentially negative outcomes of indicators, Bowen and Kreindler appear to be using a version of the precautionary principle to suggest that we should go cautiously because the potential risks are too great. They should balance this view with some form of appreciative inquiry and review the successful uses of indicators, where their public reporting and use in determining payments have been linked to improved screening rates, shorter wait times for care or safer healthcare.  

How to Make Indicators Meaningful?  

Bowen and Kreindler are right to argue against "indicator mania." Simply producing numbers without any critical thought about what they mean or how they will be used is a recipe for wasting money at best and for making bad decisions at worst. However, over the past several years, attention has shifted away from questions about how to choose an indicator towards questions of how best to organize sets of indicators to reflect organizational goals and inform decision-making. Tools like the Balanced Scorecard are merely one reflection of a growing recognition that linking measurement to strategy is a critical element of getting measurement right. Here, we'd argue that Bowen and Kreindler don't go far enough in some of their critique. If we don't know what our strategy is, if we don't know what's most important in our healthcare system, then we shouldn't start measuring things. In contrast, if we can lay out a simple strategy and measure what we are trying to do and what we want to achieve - the essence of a strategy-based approach to measurement - then we have a firm foundation for choosing where to invest our dollars in better measurement and for using indicators to anticipate problems in health system reform. Indicators based on clear strategies then have the potential to focus the attention of multiple stakeholders operating in complex systems on what really matters for the health sector, so that they adjust their strategies and support the achievement of broader health reform objectives.  

Bowen and Kreindler are right - indicators are here to stay - but that is an argument for figuring out how to use them with all their warts and imperfections. Indicators will not improve until we start using them. All the advice in Bowen and Kreindler's paper means that we have to be purposeful and clear about which indicators are important and then ensure that we invest in supporting their collection, analysis and use towards the broader goals of health system reform.

Commentaire : Les indicateurs et leurs buts - mesure significative du rendement et importance d'une stratégie

Résumé

Sarah Bowen et Sara Kreindler affirment que les indicateurs peuvent être valables, mais qu'ils sont souvent défectueux. Elles laissent entendre que les indicateurs du rendement devraient, au mieux, servir à attirer l'attention des responsables de politiques, mais pas à orienter les décisions. Or, nous estimons qu'il existe de plus en plus de données démontrant que les indicateurs du rendement ont un impact positif. Si les indicateurs sont choisis en fonction d'une stratégie bien articulée - et s'ils sont utilisés dans le cadre d'un cycle clair de gestion du rendement qui équilibre les instruments de politiques (c'est-à-dire les accords de responsabilité) et les processus d'amélioration du rendement (c'est-à-dire une nouvelle conception du processus) - ils peuvent mener à une amélioration appréciable du rendement et peuvent aider à harmoniser les stratégies entre les divers intervenants du système de santé.

Comments

Be the first to comment on this!