The Issue

The publication of the Institute of Medicine report To Err is Human nearly 15 years ago focused unprecedented attention on the safety of healthcare (Kohn et al. 2000). The numbers were shocking: 44,000 to 98,000 Americans died every year as a result of medical error, famously and vividly compared to three fully loaded jumbo jets crashing every other day in the United States. An energetic movement focused on improving the quality and safety of healthcare quickly gained momentum with the goal of not just measuring the dangers associated with healthcare but actually doing something about it.

Of the different types of healthcare, surgery has been the subject of particular scrutiny. There are many reasons why surgical safety has such a high profile, including the complexity of surgical care, the high profile of "never events" such as wrong-site surgery and retained surgical instruments, and the severity of complications following elective surgery. However, one of the major factors is the inherent danger of surgery; for every 100 surgical operations, one patient will die. Modern surgery is therefore hundreds of times more risky than activities people consider dangerous, such as skydiving or mountain climbing. In fact, the recreational activity most similar to surgery in terms of mortality risk is base jumping, an activity that involves parachuting from fixed structures such as buildings or cliffs.

It is not surprising, therefore, that a simple, inexpensive intervention that could substantially reduce the risk of surgical deaths and other adverse events would be rapidly embraced. A 19-item surgical safety checklist, developed by the World Health Organization (WHO) and tested in an international before-and-after study in eight hospitals, demonstrated a reduction in surgical mortality from 1.5% to 0.8% (Haynes et al. 2002), in effect preventing one of every two deaths after surgery. The WHO surgical safety checklist included a review at three time points – before induction of anaesthesia, prior to surgical incision and before leaving the operating room – listing manoeuvres such as verification of a patient's identity and surgical procedure, introduction of the members of the surgical team by name and role, and confirmation that medications such as antibiotics have been administered when appropriate.

The rationale for checklists included a large body of research regarding communication and teamwork in operating rooms (Lingard et al. 2002, 2004). In addition to clinical benefit, including a reduction in operative complications and death, surgical checklist use is also associated with improved team function, strengthening "team feeling," increasing the discussion of critical events, improving knowledge of team members' names and improving interprofessional coordination (Russ et al. 2013).

The Response

The international response to the publication of the WHO checklist study in the New England Journal of Medicine in January 2009 was swift and overwhelming. In the United Kingdom, a nationwide program was implemented by the National Health Service within weeks (NHS 2009). In Ontario, the Ministry of Health and Long-Term Care (MOHLTC) mandated public reporting of surgical safety checklist adherence for hospitals performing surgery beginning July 2010 (MOHLTC 2014), and use of surgical safety checklists became a Required Organizational Practice by Accreditation Canada in January 2011.

Checklist compliance is one of only 10 provincially mandated hospital-level patient safety quality indicators reported to the MOHLTC. The checklist quality indicator measures the number of times the three-phase surgical safety checklist was used during a surgical procedure over the total number of surgeries performed at the institution. A designated checklist coordinator, often the circulating nurse in the room, determines if the checklist was completed for each surgical procedure performed.

Compliance with surgical safety checklists is reported publicly, aggregated by six-month intervals. Reported compliance with checklists in Ontario is extraordinarily high. For example, among 97 large community hospitals in the period January to June 2013, almost all reported checklist compliance of 99% or 100%, with the lowest reported compliance for a hospital in this period being 91.6% (Health Quality Ontario 2014).

Despite the near-universal enthusiasm for surgical checklists among clinical leaders, hospital executives, patient safety organizations and provincial health officials, there remained lingering doubts about the potential of checklists to improve care in typical North American hospital settings. For one thing, the purported effectiveness of checklists – cutting the risk of operative death in half – was extremely large, larger even than the combined efficacy of the component parts of the checklist. Given that previous estimates of the preventability of operative deaths suggested that far fewer than 50% of deaths could be avoided with perfect medical care (Gawande et al. 1999), the real-world impact of checklist implementation was unlikely to have been as dramatic as the initial study. Many component parts of checklists, such as surgical site marking and verification of patient identity, had long been required operating procedures by hospital accreditation authorities, and other items, such as use of oxygen saturation monitoring during surgery, had been the standard of care in North American hospitals for decades. Could use of surgical checklists really enhance compliance with manoeuvres that had been standard practice for so long? Finally, many skeptics could not overlook the fact that the studies showing very large effects of safety checklists were non-randomized before-and-after studies, a study design notorious for overestimating treatment effectiveness due to selection bias, secular trends and other methodological flaws (Auerbach et al. 2007).

The Study

The rapid adoption of surgical safety checklists in Ontario provided a unique opportunity to study the effectiveness of checklists in typical practice settings. We took advantage of this natural experiment to essentially replicate earlier, smaller before-and-after studies, using the entire Ontario population (Urbach et al. 2014). We contacted all Ontario hospitals to determine the date each institution adopted safety checklists, and compared operative mortality and other surgical outcomes before and after.

Our findings were very different from those of the previous studies that showed large effects of surgical safety checklists. We studied 101 Ontario hospitals that performed 109,341 procedures in the three-month period before adoption of a surgical safety checklist and 106,370 after. None of the 101 hospitals had a statistically significant reduction in the risk of death after initiating the use of surgical checklists. The adjusted risk of death in hospital or within 30 days was 0.71% before implementation of a surgical checklist and 0.65% after. There was also no statistical change in the risk of postoperative complications (3.86% before implementation and 3.82% after). Even among subgroups of patients at higher risk of death and complications, such as those having emergency procedures or inpatient surgery, no significant benefit of the use of checklists was found.

Lessons Learned

Our study received substantial criticism; much of it assuming that poor implementation of surgical safety checklists in Ontario was to blame for the lack of an improvement in surgical outcomes (Leape 2014). Experts offered many reasons why surgical mortality did not improve as expected: Ontario hospitals over-reported actual compliance with safety checklists, Ontario teams embraced checklists and safety culture grudgingly rather than enthusiastically, and not enough time was given to detect the outcome benefits that surely must have occurred with proper implementation. Others criticized our study's methodology and design, even though the methods were identical to those of studies demonstrating large benefits of surgical checklists.

Interestingly, few believed our failure to find a benefit from the introduction of surgical safety checklists was because the checklists were not effective in the first place, despite the fact there is reason to believe that early studies substantially overestimated the benefits of surgical safety checklists. This has important implications for our interpretation of the results of other studies on patient safety interventions.

Although they are common study designs for the evaluation of patient safety interventions, before-and-after studies offer only weak evidence for the effectiveness of health interventions. Before-and-after studies are highly susceptible to selection bias, particularly if patients are selected differently before and after the implementation of the intervention. In the case of surgical safety checklists, for example, there is strong evidence that in hospitals where surgical safety checklists have been adopted, they are used most often in patients at low risk, and when they are omitted, it is usually in patients at high risk of operative death, such as those having emergency, unplanned procedures, and those requiring surgery on an immediate basis (van Klei et al. 2012). Before-and-after studies that include patients having complete surgical safety checklists will therefore systematically overestimate the benefit of checklists, as many high-risk patients who die after surgery would not have been included in the study in the period after the date of adoption of safety checklists.

Our unexpected finding, that surgical safety checklists were not effective in Ontario, offers an important lesson for the assessment of patient safety interventions: they deserve no less rigorous evaluation than any other type of healthcare. The standard for evidence of the clinical effectiveness of health interventions is randomized controlled trials. Heavily "context-dependent" interventions, such as patient safety manoeuvres or other policies that are adopted across an entire hospital or health system, can be studied using alternative clinical trial designs such as cluster-randomized trials (Campbell et al. 2007) and stepped wedge designs (Hussey et al. 2007) when it is not sensible or feasible to randomize individual patients to an intervention.

Moving forward, patient safety interventions should be evaluated with properly controlled prospective studies before their effectiveness can be assumed. Absent legitimate experimental evaluation, the benefit of patient safety manoeuvres – including surgical safety checklists – will remain under a cloud of skepticism.

About the Author

David R. Urbach, MD, MSc, is a surgeon and health services researcher at the University Health Network and the Institute for Clinical Evaluative Sciences, and Department of Surgery, University of Toronto.

Anand Govindarajan, MD, MSc, is a surgeon and health services researcher at Mount Sinai Hospital and the Institute for Clinical Evaluative Sciences, and Department of Surgery, University of Toronto.

Refik Saskin, MSc, is a staff scientist at the Institute for Clinical Evaluative Sciences.

Andrew S. Wilton, MSc, is an analyst at the Institute for Clinical Evaluative Sciences.

Nancy N. Baxter, MD, PhD, is a surgeon and health services researcher at St. Michael's Hospital and the Institute for Clinical Evaluative Sciences, and Department of Surgery, University of Toronto.

Acknowledgment

This study was supported by the Canadian Institutes of Health Research (CIHR MOP-89704) and by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the MOHLTC is intended or should be inferred.

References

Auerbach, A.D., C.S. Landefeld and K.G. Shojania. 2007. "The Tension Between Needing to Improve Care and Knowing How to Do It." New England Journal of Medicine 357(6): 608–13.

Campbell, M.J., A. Donner and N. Klar. 2007. "Developments in Cluster Randomized Trials and Statistics in Medicine." Statistics in Medicine 26(1): 2–19.

Gawande, A.A., E.J. Thomas, M.J. Zinner and T.A. Brennan. 1999. "The Incidence and Nature of Surgical Adverse Events in Colorado and Utah in 1992." Surgery 126(1): 66–75.

Haynes, A.B., T.G.Weiser, W.R. Berry, S.R. Lipsitz, A.H. Breizat, E.P. Dellinger et al. 2009. "A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population." New England Journal of Medicine 360(5): 491–99.

Health Quality Ontario. 2014. "Patient Safety. By Provincial Results: Percent of Surgeries in Which a Surgical Safety Checklist was Performed, April 2010–June 2014." Retrieved September 9, 2014. <http://www.hqontario.ca/public-reporting/patient-safety>.

Hussey, M.A. and J.P. Hughes. 2007. "Design and Analysis of Stepped Wedge Cluster Randomized Trials." Contemporary Clinical Trials 28(2): 182–91.

Kohn, L.T., J. Corrigan and M.S. Donaldson. 2009. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press.

Leape, L.L. 2014. "The Checklist Conundrum." New England Journal of Medicine 370(11): 1063–64.

Lingard, L, R. Reznick, S. Espin, G. Regehr and I. DeVito. 2002. "Team Communications in the Operating Room: Talk Patterns, Sites of Tension, and Implications for Novices." Academic Medicine 77(3): 232–37.

Lingard, L., S. Espin, S. Whyte, G. Regehr, G.R. Baker, R. Reznick et al. 2004. "Communication Failures in the Operating Room: An Observational Classification of Recurrent Types and Effects." Quality and Safety in Health Care 13(5): 330–34.

National Health Service. 2009. WHO Surgical Safety Checklist. [Patient Safety Alert] Retrieved September 9, 2014. <http://www.nrls.npsa.nhs.uk/resources/clinical-specialty/surgery/?entryid45=59860&p=3>.

Ontario Ministry of Health and Long-Term Care. 2014. Patient Safety. Retrieved September 9, 2014. <http://www.health.gov.on.ca/en/public/programs/patient_safety/>.

Russ, S., S. Rout, N. Sevdalis, K. Moorthy, A. Darzi and C. Vincent. 2013. "Do Safety Checklists Improve Teamwork and Communication in the Operating Room? A Systematic Review." Annals of Surgery 258(6): 856–71.

Urbach, D.R., A. Govindarajan, R. Saskin, A.S.Wilton and N.N. Baxter. 2014. "Introduction of Surgical Safety Checklists in Ontario, Canada." New England Journal of Medicine 370(11): 1029–38.

van Klei, W.A, R.G. Hoff, E.E. van Aarnhem, R.K. Simmermacher, L.P. Regli, T.H. Kappen et al. 2012. "Effects of the Introduction of the WHO "Surgical Safety Checklist" on In-Hospital Mortality: A Cohort Study." Annals of Surgery 255(1): 44–49.