Healthcare Quarterly

Healthcare Quarterly 16(1) January 2013 : 13-15.doi:10.12927/hcq.2013.23296
CIHI Survey

Making Sense of Health Rankings

Maria Hewitt and Michael C. Wolfson

In an era of increasingly complex medical care and escalating costs, healthcare decision-makers often rely on a broad range of indicators to gauge the health of a population, the quality of hospital care and the performance of healthcare systems. Reports that rank the health of Canadians and Canada's healthcare systems according to these indicators are widely cited in the media. These reports attempt to condense a complicated array of statistics into a relatively simple number, a rank that is used to make international and provincial comparisons. These reports have often been inconsistent. Unlike a familiar economic indicator – the gross domestic product (GDP), which represents a complex entity with a single number calculated according to an internationally agreed-upon methodology – rankings of health and healthcare are not yet standardized or well understood. This article aims to improve readers' understanding of ranking reports. It outlines the components and processes that underlie health rankings and explores why such rankings can be difficult to interpret.

Are Canadians healthier than residents of other countries? Have Canada's public health policies and healthcare systems succeeded in promoting the health of its citizens? The answers to these seemingly simple questions are important to the public and policy makers alike, but unfortunately they are surprisingly difficult to answer. To gauge how changes in Canadian policies or practices might affect the health of the population, comparisons are sometimes made between Canada and countries whose socio-economic characteristics are similar but whose healthcare financing or delivery mechanisms differ. Often, countries are rank ordered according to one or a number of indicators. Depending on the purpose of the comparisons, the indicators considered and methods used, the rank order of countries may be starkly different (Conference Board of Canada 2012; Health Consumer Powerhouse and Frontier Centre for Public Policy 2010).

Rankings are usually made to focus attention on the success of public health efforts, health system performance and quality of medical care (Conference Board of Canada 2006; Esmail and Hazel 2009; Health Consumer Powerhouse and Frontier Centre for Public Policy 2011). Although the media grab our attention with such rankings, how seriously can they be taken? The Canadian Institute for Health Information (CIHI) and Statistics Canada produced Making Sense of Health Rankings (CIHI 2008) to assist healthcare decision-makers with interpreting comparative analyses based on ranking methodologies. This article summarizes that report and addresses four key questions to consider carefully when evaluating health ranking reports: (1) Which aspects of health or healthcare are considered in the ranking? (2) How meaningful and valid are the indicators chosen to quantify these aspects of health and healthcare? (3) Are ranking indicators based on accurate, reliable and comparable information? and (4) Do sound methods underlie the ranking process?

Which Aspects of Health or Healthcare Are Considered in the Ranking?

To judge the value of a ranking report (or understand differences between two ranking reports), one must first evaluate whether the conceptual framework that underlies the ranking is sound and encompasses the domains of health that are important given the purpose of the ranking. It is then necessary to assess whether the indicators of health or healthcare that are included in the ranking methodology are consistent with the conceptual framework.

How Meaningful and Valid Are the Indicators Chosen to Quantify These Aspects of Health and Healthcare?

The meaningfulness and validity of the indicators chosen to quantify the aspects of health and healthcare included in ranking schemes are critical in determining the value of the analytical results. The indicators used should reflect important population health objectives or essential aspects of the health system. To better understand population health, indicators are needed to gauge morbidity and mortality, as well as health behaviours, living and working conditions and environmental factors affecting health and quality of life. To understand healthcare systems, indicators are needed that embody system aims, for example, healthcare that is accessible, appropriate, continuous, effective, efficient and safe. Indicators that are amenable to measurement and action are the ones most helpful.

Are Ranking Indicators Based on Accurate, Reliable and Comparable Information?

Consumers of ranking reports should judge whether the data used to support specific indicators are accurate, reliable and comparable. Data that are old, incomplete or otherwise not representative of the intended population or healthcare institutions should be viewed with caution. Similarly, any potential biases should be examined and ruled out in the sources of data. A recent study that examined international rankings of perinatal and infant mortality concluded that Canada's ranking would be substantially improved if adjustments were made to account for differences in how countries register births (Joseph et al. 2012).

Do Sound Methods Underlie the Ranking Process?

Confidence in any ranking scheme depends on adherence to sound methodological principles, explained in a well-documented and transparent fashion. This helps assure the reader that the analysis is free from bias. Specifically, readers of ranking reports can keep an eye out for the following:

  • The distribution of indicator values must be taken into account before cut points are established that distinguish "good" from "middling" or "bad" performance.
  • Rank scores are relative measures that can be misleading without examining the absolute values of the indicators that underlie the ranking method.
  • When comparing jurisdictions using ranking, adjustments must be made to account for underlying differences in the profile of the respective populations, including demographics, health status and other characteristics.
  • How each factor is weighted to achieve an overall ranking score needs to be carefully considered and made explicit. Ideally, there will be a set of principles underlying the weights and aggregation formula.
  • Numerous statistical issues should be considered in the ranking scheme, including, for example, taking into account any correlations among the set of indicators used as part of the ranking methodology; handling of extreme indicator values; and determining appropriate levels of precision in determining ties in rank order.

Efforts to rank the health status of the Canadian population and the performance of Canada's healthcare systems will inevitably increase as standard health indicators become well established and data become uniformly available across the country and internationally. The World Health Organization, the Organisation for Economic Co-operation and Development and other groups continue to work to develop measures of population health and health system performance that are applicable around the world (Klazinga 2010; Murray and Evans 2003; National Audit Office 2003).

Conclusion

Ranking reports are popular because they provide a quick high-level picture of different healthcare systems and point to "how we are doing" compared with other systems. When done well, these reports can point to differences that, upon more thorough examination, may assist in improving our own healthcare systems. However, as illustrated in this article, there are pitfalls that ranking reports have to avoid in order to provide an accurate and insightful perspective. Here we have tried to give readers the tools needed to critically assess the content of ranking reports. We hope this focus on critical assessment will be of assistance to both those who produce ranking reports and those who use them so that we are able to learn from these reports and their inter-jurisdictional comparisons.

About the Author(s)

Maria Hewitt, DrPH, a special projects lead, contributes to analytical work and participates in writing reports for CIHI’s Western Office, in Victoria, British Columbia.

Michael C. Wolfson, PhD, co-author of CIHI’s report Making Sense of Health Rankings, was, at the time of that report’s publication, assistant chief statistician of analysis and development for Statistics Canada. He is now Canada research chair in population health modelling/populomics, in the Faculty of Medicine, University of Ottawa, in Ottawa, Ontario.

References

Canadian Institute for Health Information. 2008. Making Sense of Health Rankings. Ottawa, ON: Author. Retrieved November 5, 2012. <https://secure.cihi.ca/estore/productSeries.htm?pc=PCC440>.

Conference Board of Canada. 2006. Healthy Provinces, Healthy Canadians: A Provincial Benchmarking Report. Ottawa, ON: Author. Retrieved November 5, 2012. <http://www.conferenceboard.ca/e-Library/abstract.aspx?did=1533>.

Conference Board of Canada. 2012. How Canada Performs: A Report Card on Canada. Ottawa, ON: Author. Retrieved November 5, 2012. <http://www.conferenceboard.ca/hcp/default.aspx>.

Esmail, N. and M. Hazel. 2009. Hospital Report Card: Ontario 2009. Vancouver, BC: The Fraser Institute. Retrieved July 9, 2008. <http://www.fraserinstitute.org/report-cards/hospital-performance/ontario.aspx>.

Health Consumer Powerhouse and Frontier Centre for Public Policy. 2010. Euro-Canada Health Consumer Index 2010. Winnipeg, MB: Author. Retrieved November 5, 2012. <http://www.fcpp.org/publication.php/3285>.

Health Consumer Powerhouse and Frontier Centre for Public Policy. 2011. Canada Health Consumer Index 2011. Winnipeg, MB: Author. Retrieved November 5, 2012. <http://www.fcpp.org/publication.php/4018>.

Joseph, K.S., L. Shiliang, J. Rouleau, S. Lisonkova, J.A. Hutcheon, R. Sauve et al. 2012. "Influence of Definition Based versus Pragmatic Birth Registration on International Comparisons of Perinatal and Infant Mortality: Population Based Retrospective Study." BMJ 344: e746. DOI: 10.1136/bmj.e746.

Klazinga, N. 2010. Improving Value in Health Care: Measuring Quality. Paris, France: Organisation for Economic Co-operation and Development. Retrieved November 5, 2012. <http://www.oecd.org/health/healthpoliciesanddata/oecdhealthpolicystudies-improvingvalueinhealthcaremeasuringquality.htm>.

Murray, C.J.L. and D.B. Evans, eds. 2003. Health Systems Performance Measurement: Debates, Methods and Empiricism. Geneva, Switzerland: World Health Organization. Retrieved November 5, 2012. <http://www.who.int/publications/2003/hspa/en/>.

National Audit Office. 2003. International Health Comparisons: A Compendium of Published Information on Healthcare Systems, the Provision of Healthcare and Health Achievement in 10 Countries. London, United Kingdom: Author. Retrieved November 5, 2012. <http://www.scribd.com/doc/334776/International-Health-Comparisons>.

Comments

Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed