I want the Safe Surgery Checklist to be used in my operation!
A recent publication by Canadian authors suggested that the Safe Surgery Checklist was ineffective in achieving its objectives of reducing complications and deaths, despite having been adopted by Accreditation Canada in 2011 as a “required organizational practice”, and previously proven to reduce complications and deaths in a number of studies.
We are troubled by the methodology and conclusions drawn by this retrospective study by Urbach et al, (Urbach et al. 2014) and are concerned that patients may suffer a disservice by its publication in the NEJM. During and after the publication of the World Health Organization study (Haynes et al. 2009) supporting the use of the Safe Surgery Checklist (SSC), one of us (BT) worked with the Canadian Patient Safety Institute, crossing Canada many times, assisting in the checklist implementation in all provinces. Two of us (BT and RR) were also the leads of the University of Toronto contribution to that 8-centre WHO study.
We believe there are a number of faulty assumptions that Urbach et al have made, leading to unreasonable conclusions. Firstly, the data were collected from June 2008-Sept 2010, with a third of hospitals’ data examined in the final six months, two thirds very early in the experience. The Veterans Administration hospitals in the US waited at least a year before they trained in the checklist (and found an 18% reduction in mortality with good controls) (Neily et al. 2010). The success of the checklist is directly related to time and effort in its implementation over a lengthy period of careful education and practice.
Secondly, the formal regular compliance reporting in the province of Ontario is not 100%. The data is suspect, as there is no form of auditing. While it is true that the process in Ontario has served to raise the profile of the checklist, there are no oversight processes in place to ensure compliance in each hospital (Baker, 2014).
Thirdly, using provincial-wide data assumes that the checklist is conducted in a consistent way every day in every operating room. This is simply not true. Even in Dr. Urbach’s own institution, which one of us led as surgeon-in-chief (BT), there were surgeons who needed to be repeatedly reminded by colleagues and nurses to conduct a thoughtful, meaningful checklist rather than adopt a superficial dismissive attitude towards this new two-minute exercise.
Fourthly, we are concerned about using population-based data of over 109,000 operations because of the acuity/complexity factor. The risk of complications and death are higher in more complex, higher risk surgery, and we’re concerned that with such a large number of cases surveyed (over 80% are day surgery and short stay surgery cases in Ontario), any “negative” findings in the complex group will be drowned out by the less risky or “silent majority”.
This Urbach study is uncontrolled. On the other hand, when controls in previous trials were in effect, the following points can be made:
- The WHO study ((Haynes et al. 2009) was controlled, and although difficult because of the 8 world-wide centres involved, the findings were unequivocal in supporting the use of the checklist.
- The subsequent Dutch study (de Vries et al. 2010) demonstrated exactly the same improvements after checklist use.
- In 2008, the Scottish NHS began implementing the checklist program through active clinical engagement with local champions at every hospital and ongoing implementation monitoring and support. Scotland, unlike the US and Canada, has active monitoring of its surgical death rates. Prior to implementation death rates had been flat for three years. In three years following full implementation, death rates decreased significantly, by an average of 0.06% per year. For the last two years (2011 and 2012), death rates were statistically significant low outliers, falling below 0.5% for their first time ever. The Scottish government has documented more than 9000 lives saved (Scottish National Health Service, 2013).
In addition, anecdotally, one of us (BT) was recently contacted by a Canadian University Surgical Chair who declared that the year prior to checklist implementation, his group of hospitals had reported 4 wrong-sided surgeries, and the year after had none. Should he report it? He was advised to keep on with the institution of the checklist, embed it in the culture of the organization, and if the wrong side surgery rate was still 0, consider reporting and publishing the finding. How many wrong-sided surgeries and how many deaths would make the conduct of a simple 2-minute tool such as the checklist worthwhile?
Regarding the mortality rate of the study in question: there was a 0.06% reduction in this uncontrolled study after the use of the checklist, with a p value of 0.07. That means that with a 93% confidence level, we can predict that during the course of the study, 70 people died in the province of Ontario because the checklist was not used. If we were patients whose surgeon ignored that fact (despite the confidence level not reaching 95%), and failed or refused to spend two minutes discussing in detail our respective cases with the other members of the operative teams, we would find another surgeon.
Finally, the Urbach study doesn’t address all the other beneficial effects of checklist use in the domain of improved team relationships, hospital staff satisfaction measures (Taylor et al., 2012) , and also its use in other more straightforward settings such as endoscopy, interventional radiology, cardiac cath lab (Taylor, 2012) and the like. We must not forget that one of the more startling results in recent years came from the Pronovost study (Pronovost, et al. 2007), proving that the use of a simple checklist could reduce mortality after central line insertion. The fact that some continue to disregard such practices in the ICU setting is a travesty, the same kind of travesty that might occur if the results of the above uncontrolled population study were to be considered evidence that the use of the surgical checklist should be abandoned .
Population data can be very valuable. However, without careful assessment and interpretation of that data, disturbing and potential harmful conclusions can be drawn.
About the Author(s)Bryce Taylor, MD, FRCSC, FACS, Professor of Surgery, University of Toronto
Jason Leitch, BDS, FDS, DDS, MPH, Clinical Director, The Quality Unit, Scottish Government
Richard Reznick, MD, FRCSC, FACS, Dean, Faculty of Health Sciences, Queen’s University, Kingston ON
Craig White, ClinPsyD, Phd, Professor, The Quality Unit, Scottish Government
Baker, M.A. Executive Lead for Patient Safety in Ontario 2010. Personal Communication. 2014
de Vries, E.N., H.A. Prins, R.M.P.H. Crolla, et al. “Effect of a Comprehensive Surgical Safety System on Patient Outcomes.” N Engl J Med 2010 :363:20 pp1928-1937
Haynes, A.B., T.G. Weiser, W.R. Berry, et al. “A surgical safety checklist to reduce morbidity and mortality in a global population.” N Engl J Med 2009; 360:491-9.
Neily J., P.D. Mills, Y.Young-Xu, et al. “Association between implementation of a medical team training program and surgical mortality.” JAMA 2010;304:1693-700.
Pronovost, P. D., Needham, S. Berenholtz, et al. “An intervention to decrease catheter-related bloodstream infections in the ICU.” N Engl J Med 2006;355:2725-32. [Erratum, N Engl J Med 2007;356:2660.]
Scottish National Health Service. “The Healthcare Quality Strategy for the NHS Scotland. Report at the International Society for Quality in Healthcare.” http://www.isqua.org/docs/edinburgh-powerpoint-presentations-2013/1545-afternoon-plenary-jason-leitch-pentland-mon-scotlands-quality-journey.pdf?sfvrsn=2. Retrieved October 24, 2013
Taylor, B.R. “Safe Surgery Saves Lives.” Qmentum Quarterly 4: 6-8. 2012
Taylor, B.R., A. Slater, R.R. Reznick. “The surgical checklist effects are sustained, and team culture is strengthened.” The Surgeon (Journal of the Royal Colleges of Surgeons of Edinburgh and Ireland), e-pub doi:10.1016/jsurge2009.11.012.
Urbach, D.R., A. Govindarajan, R. Saskin, A.S. Wilton, N.N. Baxter.”Introduction of surgical safety checklists in Ontario, Canada.” N Engl J Med 2014; 370:1029-38.
John McGurran wrote:
Posted 2014/05/20 at 11:35 AM EDT
The key point here is that a potentially valuable innovation such as the surgical checklist ought to be introduced similar to the Scottish experience (as described above): with clinical and administrative leadership, monitoring and support. As was clearly pointed out at Taming of the Queue 2014, Canadian health administration talks while the Scottish acts on matters of public system effectiveness.
Evaluation of checklist effectiveness (a subset of quality) in diverse surgical settings should be part of routine quality assurance in our publicly funded hospitals.
If the checklist was deemed important enough to be mandated then it follows that its use must be evaluated.
While the Urbach study has its flaws and detractors (including Dr. Atul Gawande) the health policy direction should be clear: evaluate the value across the broad surgical case mix. The question that remains is where does the authority lie to move in such a direction in all provinces and territories?
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed