Law & Governance

Abstract

[This article was originally published in HealthcarePapers 11(4)

Overall life expectancy in Canada is among the highest in the world and research evidence suggests that the healthcare system is part of the reason for this. However, patient waits, low international rankings and continued expenditure growth all provide a buttress against complacency. There can be little doubt that improvement can and must happen. Improvement depends on information, and more specifically information about outcomes of care. Without sound analysis of what works in the real world when applied to real patients, we have not done our jobs as stewards of the healthcare system. Current outcomes information in Canada is limited and tends to focus on measures of failure (e.g., hospital readmissions) rather than measures of success (e.g., improvement in functioning). Patient reported outcome measures (PROMs) must become part of regular data collection in the healthcare system. The importance of this is even more pronounced given that healthcare is now dominated by chronic conditions that need to be managed over long periods of time. We offer three recommendations for action: that we begin immediately to collect PROMs in elective surgery; that we start small-scale and coordinated experiments on the implementation of PROMs in care for chronic conditions; and that we convene a pan-Canadian working group to help coordinate and organize these activities. We recognize the challenges these issues raise, but our contention is that there are even greater challenges in continuing on as we are.

Canadian Healthcare: A Work in Progress

The Canadian healthcare system with its foundations of equity and universality forms part of our national identity and is rightly a source of pride for Canadians. Overall life expectancy in Canada is among the highest in the world and continues to increase, and what evidence there is suggests that an increasing proportion of those longer lives are spent without disabilities (Chen and Millar 2000). Studies comparing the United States and Canada suggest that our social safety net, including the healthcare system, likely contributes to the life expectancy gap between these two countries that has developed and increased since the 1970s (Siddiqi and Hertzman 2007). People who work in healthcare should be proud of their achievements.

However, there is no room for complacency. If the comparative lens is broadened, the picture begins to change. Canadian patients report longer waits for appointments in primary care, appointments with specialists and elective surgery (Schoen et al. 2010), and Canada is far behind on the adoption of an electronic health record (Schoen et al. 2009). Further, in recent international health and healthcare comparisons, Canada has not performed well:

• Canada ranked 11 of 24 Organisation for Economic Co-operation and Development (OECD) countries in terms of overall health performance (including life expectancy, disease rates and self-reported health) (Canadian Institute for Health Information [CIHI] 2008).
• According to a Euro-Canada Health Consumer Index (based on factors such as patient rights and information, waiting times, clinical outcomes and access to treatment), Canada ranked 23 out of 30 countries (CIHI 2008).

These ongoing challenges for Canadian healthcare cannot be blamed on fiscal constraints: currently we spend more than $190 billion annually on healthcare (CIHI 2010). As a percent of gross domestic product, our healthcare spending ranks fifth among OECD countries and continues to increase faster than might be predicted based on population growth, aging and inflation; an average 75-year-old in 2010 receives more (and more expensive) services (after adjusting for inflation) than his or her counterpart received in 2000. In British Columbia, in 2005–2006, more than $170 million in physician spending (about 8% of the total physician budget) would have been avoided if age-specific use rates from 1996–1997 had remained stable (McGrail et al. 2011).

It seems a safe assumption that these upward pressures for spending growth will continue – more is possible and more will be promoted – while downward pressures will continue from outside the system. The financial crisis has made this point a bit sharper than it might have been otherwise, but there is no escaping that 6% growth in healthcare expenditures, as promised in federal transfers in the 2004 Accord, cannot be sustained.

Steven Lewis sums up the Canadian healthcare situation in a stark but honest way: "We spend so much and achieve so little" (Baker et al. 2008: 291). There can be little doubt that improvement can and must happen, and this paper provides a discussion on improvement driven through the routine collection of outcome data.

Improvement Science: Learning from Berwick, Donabedian and Wennberg

The Institute for Healthcare Improvement (IHI), and its founder Don Berwick, promotes improvement through healthcare system redesign that simultaneously accomplishes three key objectives (the "Triple Aim"; Berwick et al. 2008):

• Improving population health
• Enhancing patient care experience (including quality, appropriateness and satisfaction)
• Reducing, or at least controlling, the cost of care

The big-picture goal is, of course, to improve all three, but this framework recognizes that some decision-making may be focused on one area. In that situation, the objective is to make improvements in the chosen area without causing backsliding in the others: for example, to curtail the growth in healthcare expenditures without harming care quality. This may sound simple, but the perspective is fundamentally important when we are seeking improvements in the performance of the healthcare system.

Making the Triple Aim operational in the system requires routine gathering of information on all three components. This is an agenda of "measurement to support improvement" in a dynamic healthcare system, where all actors in the system search for improvements in care delivery and opportunities for redesign to promote enhanced quality and better outcomes. Routine measurement is then to support formative and summative analyses of progress toward the Triple Aim: do the redesign initiatives indicate improvement or not? Our focus in this paper is on the population health outcomes piece of the Triple Aim approach.

The importance of outcomes is not a new concept. Avedis Donabedian, the father of the quality movement in healthcare, introduced us to the notion of "structure – process – outcome" for categorizing and understanding healthcare services. In his first article on the subject, in 1966, he argued strongly for outcome measurement: "Outcomes, by and large, remain the ultimate validators of the effectiveness and quality of medical care" (Donabedian 1966: 169). In other words, without information on outcomes and an investigation of what systems or structures are related to outcomes, we have not done our job as health service analysts. And without sound analysis of what works in the real world when applied to real patients, we have not done our jobs as stewards of the healthcare system.

More recently, the seminal work of Wennberg and his colleagues highlights the importance of outcome assessment. They have analyzed and mapped variations in Medicare spending per person across 306 health regions in the United States. One of their key findings is that more care does not produce better (and may even deliver worse) health outcomes (Fisher et al. 2003a, 2003b). Without collecting and analyzing information on outcomes, variations analyses can only show differences in quantity of service delivery and not variation in quality.

What Do We Mean by Health Outcomes?

Routine outcome data collection in healthcare systems has a long but far from glorious history. In the 1850s, during the Crimean War, Florence Nightingale used information she collected to assess the causes of increased mortality among soldiers. Subsequently, she focused less on mortality as an outcome but, rather, on whether, following a hospital episode, the sick were restored to full health. In the early 20th century, E.A. Codman, a Boston physician, collected before and after information from patients undergoing surgery. What unites these efforts are two important features: (1) they did not depend on sophisticated technology (five by eight inch pieces of paper in Codman's case) and (2) while they were driven to collect outcomes information to understand better the impact of healthcare practices on health, their efforts were not met with widespread support, and, in Codman's case, were met with derision (Neuhauser 2002).

Bringing the story up to date, in a Canadian context, we do in fact have some information on outcomes. Mortality statistics, the most hard-edged outcome, is well collected and routinely available. CIHI and Statistics Canada have developed a suite of indicators that characterize the Canadian population and the performance of the healthcare system. This is good information, but it is limited. These measures are either generally available only for small samples or at an ecological level (e.g., health status measures based on Canadian Community Health Survey data), or represent important but rare failures in the delivery of care (e.g., hospital readmissions, admissions for ambulatory care sensitive conditions).

Measurement of "success" in terms of improvements in patient health status or health-related quality of life is virtually non-existent in Canadian healthcare. The latter can be referred to as patient-reported outcome measures (PROMs), and our interest in particular is on such measures. The rationale for outcomes has been indicated earlier; our interest in outcomes reported by patients is driven by a belief that individual patients are the best judges of their own welfare. If you want to know whether an individual's health status has improved, you have to ask that individual!

Despite their absence in routine clinical practice in Canada, PROMs have been used widely in clinical trials and other research settings (Cella et al. 2007; Fayers and Machin 2007). Extensive work on the development of PROM-type survey tools has been undertaken in recent years, such that there now exist a large number of PROM instruments, many of which have been well validated, some using Canadian populations (Devlin et al. 2010). PROMs are designed to measure either "general" health status (i.e., generic PROMs [Brazier et al. 1998; Dolan 1997; Feeny et al. 2002]) or health status relating to a specific condition (i.e., condition-specific PROMs [Massof and Rubin 2001; Morey et al. 1998]). Examples of generic PROMs are given in Table 1, with an indication of the health domains and number of possible health states captured.

Examples of PROM Data Collection to Promote Improvement

Let us now consider implementation issues: how might PROM data be collected and used to promote improvement? This is not virgin territory, so it is important that lessons learned from previous and ongoing PROMs implementations are understood.

British Columbia

One of the earliest attempts to explore the benefits and costs of routine collection of PROMs comes from British Columbia: the Regional Evaluation of Surgical Indications and Outcomes (RESIO) Project (Wright et al. 2002). This work was undertaken in the late 1990s and involved 138 surgeons and 5,313 patients who underwent one of the following elective procedures: cataract replacement, cholecystectomy, hysterectomy, lumbar discectomy, prostatectomy or hip replacement. Data on health-related quality of life (using the Short Form [36] Health Survey [SF-36] and condition-specific measures) were gathered through patient self-report before surgery and then at either three months or 12 months post-surgery. The data were fed back to surgeons in real time to allow use of the data in clinical management. The project's main aim was "to determine the feasibility of routine evaluation of indications for and outcomes of elective surgery" (Wright et al. 2002: 461).

The most controversial finding of RESIO was that some patients who received surgery had had relatively minor symptoms and levels of disability. For example, 31% of patients who underwent cataract surgery had experienced a visual function score of 91or higher (on a scale where 100 indicates no visual impairment at all). The authors interpreted their data as evidence for the need to reconsider indications for elective surgery. Further, they found a wide variation in practice patterns and rates of surgical intervention.

The cost of the program, administered using postal distribution and manual data entry and analysis, was estimated to be $12 per patient. The conclusion by the investigators was very positive: "Evaluation of indications for and outcomes of elective surgery could be implemented systematically at reasonable cost and could be included in an accountability framework for health services" (Wright et al. 2002: 461).

The most concerning finding was that almost half of the surgeons indicated that the exercise was of "little value" and stated that they did not wish to continue receiving such information. The need to engage physicians and surgeons fully in such an initiative is clearly critical. RESIO teaches us that collaboration between clinicians and managers is required for success, with management taking shared accountability with clinicians for ensuring that the PROMs data are collected and used appropriately.

The United Kingdom

Other examples of routine PROMs data collection include the private hospital sector in the United Kingdom (Bupa) and the UK National Health Service (NHS) (Devlin and Appleby 2010). Bupa is an association of private UK hospitals. Much of their work is elective surgery undertaken in their facilities by surgeons, most of whom work in parallel in the public sector. The routine collection of PROMs data before and after surgery has been standard practice of care since the late 1990s, under the leadership of their senior medical team. The data are used to support clinical governance, audit and quality improvement, with data reported as control charts (Vallance-Owen et al. 2004).

The success of the Bupa model in part inspired a similar development in the NHS with the rollout of a routine PROM data collection process following pilot work by Browne et al. (2007). The pilot work collected data on five prospective cohorts of elective surgery patients: hip replacement, knee replacement, hernia repair, varicose vein surgery and cataract surgery. Patients were assessed before surgery and at three or six months post-surgery using the EuroQol (EQ-5D) and a condition-specific instrument relevant to their surgery. The feasibility results were very positive, with high response rates and a cost of approximately £6.50 per patient for the postal-based follow-up data collection – a very similar cost estimate from that reported in the RESIO study in Canada. The conclusion from the pilot was that "any future programme of routine PROMs administration is feasible," and this is borne out by the NHS uptake of the approach (Browne et al. 2007: 86). Routine collection of PROMs data commenced in 2010 and is now standard practice for selected elective surgery procedures in all English hospitals (Devlin and Appleby 2010).

Recommendation One: PROMs for Elective Surgery

It should be noted that all the examples cited here involve PROMs in the context of elective surgery. There is a clear case for PROMs in this area. PROMs data, gathered through routine collection in the context of elective surgery, offer opportunities to deliver benefits at a number of different levels in the healthcare system:

• Supporting patient choice and empowerment – The patient who has undergone surgery asks, "Is my recovery post-surgery similar to that of others or should I be worried?" More fundamentally, routine reporting on patient experience after surgery can help patients decide whether surgery at a particular point in time is right for them in the first place.
• Improving clinical management – The surgeon asks, "Which of my patients are experiencing ongoing health problems and might benefit from early clinical review?"
• Assessing performance and supporting quality improvement – The health sector manager asks, "Which are the high performing surgical teams and what lessons can they offer to other groups?"

The PROMs case in elective surgery is, in fact, so clear that it is difficult to understand why there has not been widespread adoption in Canada already.

Our first recommendation is that we rectify this immediately: all health authorities should plan now for the collection of PROMs data, with the aim being that by 2013 they are collected routinely for all elective surgeries nationally. Yes, there are implementation issues – Who collects the data? Using what tools, what collection methodology? Where do the data reside? and so on – but there is also readily available guidance from jurisdictions with experience, most notably the United Kingdom. A central coordinating body or information clearinghouse is clearly required to ensure a level of consistency of approach. This may be a relevant role for CIHI.

But What about PROMs for Chronic Conditions?

Elective surgeries, however, represent only a small part of healthcare services delivery – the real challenge is the management of chronic disease. It is here where there is little (or no) experience in the use of PROMs, and here where there is potentially a great deal to be gained.

More than nine million Canadians have chronic conditions. One third of these, or three million people, have more than one chronic condition (Health Council of Canada 2007). About 20% of people who are 65 years and older have two or more chronic conditions that are considered major, as defined by the associated expected use of healthcare services (McGrail et al. 2008). No matter how we slice it, it is clear that chronic conditions are a major feature of life for a significant proportion of Canadians and their families.

Chronic conditions are significant for the healthcare system as well. The top 5% of users of healthcare services are responsible for 30–40% of all expenditures (Deber 2009; Reid et al. 2003), and those users tend to be quite sick. At least at the high end, healthcare services provision appears to be in response to identified needs.

That healthcare services are provided in response to a need, however, is not the same as saying that they are always thoroughly and appropriately provided. There is an extensive literature on variations in healthcare services use, and the research evidence shows that the largest of those variations are found in areas where there is the least medical certainty. Where there is a lack of research evidence or clinical consensus about the right amount or course of care, there tends to be the greatest variation in care provided (Wennberg 2010).

Variations are a potential means to understand a great deal more about the effectiveness of healthcare delivery systems. It is possible with currently available data to identify variations in care delivery. The addition of information on outcomes would, in principle, allow differences in care – how many times and what specialists are consulted, what drugs or drug combinations are used, the organization and delivery of primary care and so on – to be linked to differences in outcomes, giving the opportunity to improve the quality of care for complex patients. This is crucial because complex patients tend not to be part of randomized trials, nor are pathways or patterns of care for people with multiple chronic conditions generally subject to rigorous testing or evaluation.

Patient-reported outcomes are particularly suited to helping improve care for patients with chronic conditions precisely because the conditions are chronic. Once diagnosed, people can expect to live many years with their conditions, and the efforts of the healthcare system are directed more to alleviating symptoms and forestalling progression in the disease than in effecting a cure. Mortality, in contrast, is an extremely poor outcome to guide us in identifying high-quality care since mortality is unexpected (at least in the short term) and is (thankfully) rare.

Given the potential utility of PROMs in chronic care, it is perhaps surprising that there is little evidence of their use in Canada or internationally in routine practice. This surprise may diminish somewhat once we start to contemplate how, exactly, they might be introduced for chronic conditions. Unlike surgical interventions, there is no obvious before and after because there is no obvious or discrete "intervention." Chronic conditions require longitudinal care, which is often multi-faceted and sometimes from multiple providers. Also unlike surgical interventions, the "condition" itself is difficult to define. A knee replacement is easy to understand. The care of someone who is over 75 years old and has four chronic conditions, two of which are considered major and one of which is unstable, is somewhat more difficult.

It is, nevertheless, precisely these more complex populations that consume a large share of healthcare services. Even marginal improvements in quality while controlling cost could have a large impact on the population's overall health. And so, while somewhat more conceptually challenging, there is a strong argument for movement toward the collection and use of PROMs in chronic care.

Recommendation Two: PROMs for Chronic Conditions

Our second recommendation is to begin work to incorporate PROMs into the provision of healthcare for chronic care management in Canada. Since we are talking about ongoing conditions, it can be inferred that we will need ongoing data collection. The lack of a specific intervention or point in time at which PROMs ought to be collected suggests that collection should occur at regular intervals, perhaps annually.

As a starting point, we suggest that coordinated pilot work be taken forward at across the country, at the health authority level, with pilots focusing on different patient groups (such as the frail elderly, or people with three or more chronic conditions) as well as perhaps different timings and modes of data collection.

What Will It Take to Get Canada to the PROM?

So, what will it take to get Canada to the PROM? First, we must acknowledge that Canada is not an outlier in this regard. Most countries are not collecting patient-reported outcomes in any broad-based and systematic way, and none that we know of are collecting these data for people with chronic conditions. In looking forward, perhaps the most productive first step is to understand the roadblocks to this point.

The discussion above indicates several possible reasons why PROMs are not currently collected in Canada. Clinical leadership is critical to the success of any PROMs initiative, as demonstrated in the RESIO study. While care providers are clearly not the only group that needs to be engaged, success is impossible to imagine without leaders from the provider community understanding the potential of PROMs, being willing to experiment with their collection and using the results to change their practice.

Another clear impediment is structural: at present, there are no incentives for institutions or providers to gather and use such information to improve quality of care. Payment incentives tied to patient outcomes might be an option to consider and would likely grab attention and deliver some momentum but, in truth, pay-for-performance initiatives in healthcare have mixed success (Gavagan et al. 2010; Van Herck et al. 2010; Werner et al. 2011).

Two other structural challenges may in the end be more significant. The first concerns the current fiscal climate, which makes this a particularly difficult time to promote new investments in healthcare that are not tied to direct patient care. The collection of PROMs would, we argue, improve care and help to control costs, but the payoffs will not be immediate; the impact will not be felt in next year's budget. On top of this, pursuing PROMs now would require international leadership from Canada to build evidence for the most efficient and effective ways, at a system level, to collect, analyze and use such data. This is a critically important development in healthcare, but not having a blueprint to follow makes it daunting.

Data collection is of course only the first step to promoting "improvement" in Canadian healthcare; if we stop there, our efforts are wasted. We should not underestimate what is involved in using PROMs to their best advantage. Any PROMs data we collect must be linkable (and linked) to administrative records of healthcare services use. This linkage is absolutely essential because it is only with a connection to services received that the outcomes data can truly take on their full meaning.

Recommendation Three: A Pan-Canadian Working Group on PROMs

At this point, our recommendation is to form a working group, ideally a cross-national group, that would create the needed plans for the pilot testing, implementation and analysis of PROMs data. A cross-national working group might be particularly useful because there are many implementation issues that would be common to each jurisdiction, and working together would lessen the implementation burden on each individual jurisdiction. Some of those most critical implementation issues include choosing instruments and timing for data collection; choosing the route of data collection (e.g., direct to patients, through general practitioners); how best to engage and encourage clinical leadership for the collection and use of these data; and identifying data stewardship issues such as housing, linking and providing access to data. Leadership from and a coordinating role for CIHI would seem appropriate.

We should be clear that this is not a call for a stringently controlled cross-national effort driven from the top down. Our call for a pan-Canadian group simply recognizes the scale of the common challenge facing all jurisdictions in collecting and using PROMs. It would be inefficient in the extreme for each health authority, or even each province, to develop independently everything outlined above. We have provided a sketch of the research and development that needs to be done around PROMs. Rapid progress requires lessons to be shared, pilot work to be coordinated and some decisions (such as regarding the generic instrument to be used) to be made jointly across jurisdictions. For example, some health authorities could proceed with working through implementation issues for PROMs in elective surgery, others could focus on collecting PROMs for chronic conditions and still others could deal with the privacy, information sharing and analytical issues. This approach would accelerate progress and could leapfrog Canada to become an international leader in the use of outcomes information to improve patient care and population health.

We recognize the challenges these issues raise, but our contention is that there are even greater challenges in continuing on as we are. There is general acceptance of patient-reported outcomes in the clinical trials that determine the safety, efficacy and cost-effectiveness of new drugs and devices that enter the healthcare system. The overall effectiveness of the system, and our ability to monitor our progress toward the Triple Aim of a healthy population, great quality of healthcare and controlled expenditures, depends on similar data in routine practice.

About the Author

Kimberlyn McGrail PhD Associate Director and Assistant Professor, Centre for Health Services and Policy Research and the School of Population and Public Health, The University of British Columbia

Stirling Bryan, PhD
Director and Professor, Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute and School of Population and Public Health, The University of British Columbia

Jennifer Davis, PhD
Post-Doctoral Fellow, Centre for Clinical Epidemiology and Evaluation,
Vancouver Coastal Health Research Institute

References

Baker, G.R., A. MacIntosh-Murray, C. Porcellato, L. Dionne, K. Stemacovich and K. Born. 2008. High Performing Healthcare Systems: Delivering Quality by Design. Toronto, ON: Longwoods Publishing Corporation.

Berwick, D.M., T.W. Nolan and J. Whittington. 2008. "The Triple Aim: Care, Health, and Cost." Health Affairs 27(3): 59. Retrieved June 7, 2011. <http://content.healthaffairs.org/content/27/3/759.short>.

Brazier, J., J. Roberts and M. Deverill. 2002. "The Estimation of a Preference-Based Measure of Health from the SF-36." Journal of Health Economics 21(2): 271–92. Retrieved November 15, 2011.

Brazier, J., T. Usherwood, R. Harper and K. Thomas. 1998. "Deriving a Preference-Based Single Index from the UK SF-36 Health Survey." Journal of Clinical Epidemiology 51(11): 1115–28. Retrieved November 15, 2011. <http://www.ncbi.nlm.nih.gov/pubmed/9817129>.

Browne, J., L. Jamieson, J. Lewsey, J. van der Meulen, N. Black, J. Cairns et al. 2007. Patient Reported Outcome Measures (PROMs) in Elective Surgery. London: Royal College of Surgeons of England, London School of Hygiene and Tropical Medicine and Royal College of Surgeons of England.

Canadian Institute for Health Information. 2008. Making Sense of Health Rankings. Ottawa, ON: Author.

Canadian Institute for Health Information. 2010. National Health Expenditure Trends, 1975–2010. Ottawa, ON: Author.

Cella, D., S. Yount, N. Rothrock, R. Gershon, K. Cook, B. Reeve et al. 2007. "The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH Roadmap Cooperative Group during Its First Two Years." Medical Care 45(5 Suppl. 1): S3–11. DOI: 10.1097/01.mlr.0000258615.42478.55.

Chen, J. and W.J. Millar. 2000. "Are Recent Cohorts Healthier Than Their Predecessors?" Health Reports 11(4): 9–23. Retrieved November 15, 2011. <http://www.statcan.gc.ca/studies-etudes/82-003/archive/2000/5064-eng.pdf>.

Deber, R. 2009. Speaking Truth to Power: The Adventures of a Policy Analyst in the Ivory Tower. Emmett Hall Memorial Lecture. Ottawa, ON: Face2Face. Retrieved June 7, 2011. <http://www.f2fe.com/CAHSPR/2009/docs/keynote/RaisaDeber.pdf>.

Devlin, N.J., J. Appleby, M. Buxton, A. Vallance-Owen, M. Emberton, K. Peters et al. 2010. Getting the Most Out of Proms: Putting Health Outcomes at the Heart of NHS Decision-Making. London: Office of Health Economics.

Dolan, P. 1997. "Modeling Valuations for EuroQol Health States." Medical Care 35(11): 1095–108.

Donabedian, A. 1966. "Evaluating the Quality of Medical Care." Milbank Memorial Fund Quarterly 44(3). Retrieved August 31, 2010. <http://www.jstor.org/stable/3348969>.

Fayers, P. and D. Machin. 2007. Quality of Life: The Assessment, Analysis and Interpretation of Patient-Reported Outcomes. West Sussex, England: John Wiley & Sons. Retrieved June 7, 2011. <http://www.amazon.com/Quality-Life-Assessment-Interpretation-Patient-reported/dp/047002450X>.

Feeny, D., W. Furlong, G.W. Torrance, C.H. Goldsmith, Z. Zhu, S. DePauw et al. 2002. "Multiattribute and Single-Attribute Utility Functions for the Health Utilities Index Mark 3 System." Medical Care 40(2): 113–28. Retrieved November 15, 2011. <http://www.ncbi.nlm.nih.gov/pubmed/11802084>.

Feeny, D., W. Furlong, M. Boyle and G.W. Torrance. 1995. "Multi-attribute Health Status Classification Systems. Health Utilities Index." PharmacoEconomics 7(6): 490–502. Retrieved November 15, 2011. <http://www.ncbi.nlm.nih.gov/pubmed/10155335/>.

Fisher, E., D. Wennberg and T. Stukel. 2003a. "The Implications of Regional Variations in Medicare Spending. Part 1: The Content, Quality, and Accessibility of Care." Annals of Internal Medicine 138(4): 273–311. Retrieved December 22, 2010, <http://www.annals.org/content/138/4/273.short>.

Fisher, E., D. Wennberg, T. Stukel, D.J. Gottlieb, F. Lucas and E.L. Pinder. 2003b. "The Implications of Regional Variations in Medicare Spending. Part 2: Health Outcomes and Satisfaction with Care." Annals of Internal Medicine 138(4): 288–322. Retrieved December 22, 2010. <http://www.annals.org/content/138/4/288.short>.

Gavagan, T.F., H. Du, B.G. Saver, G.J. Adams, D.M. Graham, R. McCray et al. 2010. "Effect of Financial Incentives on Improvement in Medical Quality Indicators for Primary Care." Journal of the American Board of Family Medicine 23(5): 622–31. DOI: 10.3122/jabfm.2010.05.070187.

Health Council of Canada. 2007. Why Health Care Renewal Matters: Learning from Canadians with Chronic Health Conditions. Ottawa, ON: Author. Retrieved June 5, 2011. <http://healthcouncilcanada.ca/en/index.php?page=shop.product_details&category_id=22&flypage=shop.flypage&product_id=41&option=com_virtuemart&Itemid=170>.

Horsman, J., W. Furlong, D. Feeny and G. Torrance. 2003. "The Health Utilities Index (HUI): Concepts, Measurement Properties and Applications." Health and Quality of Life Outcomes 1: 54. DOI: 10.1186/1477-7525-1-54.

Massof, R.W. and G.S. Rubin. 2001. "Public Health and the Eye Visual Function Assessment Questionnaires." Survey of Ophthalmology 45(6): 531–48.

McGrail, K.M., R.G. Evans, M.L. Barer, K.J. Kerluke and R. McKendry. 2011. "Diagnosing Senescence: Contributions to Physician Expenditure Increases in British Columbia, 1996/97 to 2005/06." Healthcare Policy 7(1): 41–54.

McGrail, K.M., A.-M. Broemeling, M.J. McGregor, K. Salomons, L.A. Ronald and R. McKendry. 2008. Home Health Services In British Columbia: A Portrait of Users and Trends over Time. Vancouver, BC: UBC Centre for Health Services and Policy Research.

Morey, A.F., J.W. McAninch, C.P. Duckett and R.S. Rogers. 1998. "American Urological Association Symptom Index in the Assessment of Urethroplasty Outcomes." The Journal of Urology 159(4): 119294. Retrieved November 15, 2011. <http://www.ncbi.nlm.nih.gov/pubmed/9507830>.

Neuhauser, D. 2002. "E.A. Codman." Quality and Safety in Health Care 11(1): 104–5. Retrieved February 14, 2011. <http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1743579&tool=pmcentrez&rendertype=abstract>.

Reid, R., R. Evans, M. Barer, S. Sheps, K. Kerluke, K. McGrail et al. 2003. "Conspicuous Consumption: Characterizing High Users of Physician Services in One Canadian Province." Journal of Health Services Research and Policy 8(4): 215–24. DOI: 10.1258/135581903322403281.

Schoen, C., R. Osborn, D. Squires, M.M. Doty, R. Pierson and S. Applebaum. 2010. "How Health Insurance Design Affects Access to Care and Costs, by Income, in Eleven Countries." Health Affairs DOI: 10.1377/hlthaff.2010.0862.

Schoen, C., R. Osborn, M.M. Doty, D. Squires, J. Peugh and S. Applebaum. 2009. "A Survey of Primary Care Physicians in Eleven Countries, 2009: Perspectives on Care, Costs, and Experiences." Health Affairs 28(6): w1171–83. DOI: 10.1377/hlthaff.28.6.w1171.

Siddiqi, A. and C. Hertzman. 2007. "Towards an Epidemiological Understanding of the Effects of Long-Term Institutional Changes on Population Health: A Case Study of Canada versus the USA." Social Science and Medicine (1982) 64(3): 589603. DOI: 10.1016/j.socscimed.2006.09.034.

Vallance-Owen, A., S. Cubbin, V. Warren and B. Matthews. 2004. "Outcome Monitoring to Facilitate Clinical Governance; Experience from a National Programme in the Independent Sector." Journal of Public Health 26(2): 187–92. DOI: 10.1093/pubmed/fdh127.

Van Herck, P., D. De Smedt, L. Annemans, R. Remmen, M.B. Rosenthal and W. Sermeus. 2010. "Systematic Review: Effects, Design Choices, and Context of Pay-for-Performance in Health Care." BMC Health Services Research 10: 247. DOI: 10.1186/1472-6963-10-247.

Wennberg, J.E. 2010. Tracking Medicine: A Researcher's Quest to Understand Health Care. Oxford, England: Oxford University Press. Retrieved May 3, 2011. <http://books.google.com/books?id=3riIUS-A4_gC&pgis=1>.

Werner, R.M., J.T. Kolstad, E.A. Stuart and D. Polsky. 2011. "The Effect of Pay-for-Performance in Hospitals: Lessons for Quality Improvement." Health Affairs 30(4): 690–98. DOI: 10.1377/hlthaff.2010.1277.

Wright, C.J., G.K. Chambers and Y. Robens-Paradise. 2002. "Evaluation of Indications for and Outcomes of Elective Surgery." Canadian Medical Association Journal 167(5): 461–66.