A recent publication by Canadian authors suggested that the Safe Surgery Checklist was ineffective in achieving its objectives of reducing complications and deaths, despite having been adopted by Accreditation Canada in 2011 as a “required organizational practice”, and previously proven to reduce complications and deaths in a number of studies.

We are troubled by the methodology and conclusions drawn by this retrospective study by Urbach et al, (Urbach et al. 2014) and are concerned that patients may suffer a disservice by its publication in the NEJM. During and after the publication of the World Health Organization study (Haynes et al. 2009) supporting the use of the Safe Surgery Checklist (SSC), one of us (BT) worked with the Canadian Patient Safety Institute, crossing Canada many times, assisting in the checklist implementation in all provinces. Two of us (BT and RR) were also the leads of the University of Toronto contribution to that 8-centre WHO study.

We believe there are a number of faulty assumptions that Urbach et al have made, leading to unreasonable conclusions. Firstly, the data were collected from June 2008-Sept 2010, with a third of hospitals’ data examined in the final six months, two thirds very early in the experience. The Veterans Administration hospitals in the US waited at least a year before they trained in the checklist (and found an 18% reduction in mortality with good controls) (Neily et al. 2010). The success of the checklist is directly related to time and effort in its implementation over a lengthy period of careful education and practice.

Secondly, the formal regular compliance reporting in the province of Ontario is not 100%. The data is suspect, as there is no form of auditing. While it is true that the process in Ontario has served to raise the profile of the checklist, there are no oversight processes in place to ensure compliance in each hospital (Baker, 2014).

Thirdly, using provincial-wide data assumes that the checklist is conducted in a consistent way every day in every operating room. This is simply not true. Even in Dr. Urbach’s own institution, which one of us led as surgeon-in-chief (BT), there were surgeons who needed to be repeatedly reminded by colleagues and nurses to conduct a thoughtful, meaningful checklist rather than adopt a superficial dismissive attitude towards this new two-minute exercise.

Fourthly, we are concerned about using population-based data of over 109,000 operations because of the acuity/complexity factor. The risk of complications and death are higher in more complex, higher risk surgery, and we’re concerned that with such a large number of cases surveyed (over 80% are day surgery and short stay surgery cases in Ontario), any “negative” findings in the complex group will be drowned out by the less risky or “silent majority”.

This Urbach study is uncontrolled. On the other hand, when controls in previous trials were in effect, the following points can be made:

• The WHO study ((Haynes et al. 2009) was controlled, and although difficult because of the 8 world-wide centres involved, the findings were unequivocal in supporting the use of the checklist.
• The subsequent Dutch study (de Vries et al. 2010) demonstrated exactly the same improvements after checklist use.
• In 2008, the Scottish NHS began implementing the checklist program through active clinical engagement with local champions at every hospital and ongoing implementation monitoring and support. Scotland, unlike the US and Canada, has active monitoring of its surgical death rates. Prior to implementation death rates had been flat for three years. In three years following full implementation, death rates decreased significantly, by an average of 0.06% per year. For the last two years (2011 and 2012), death rates were statistically significant low outliers, falling below 0.5% for their first time ever. The Scottish government has documented more than 9000 lives saved (Scottish National Health Service, 2013).

In addition, anecdotally, one of us (BT) was recently contacted by a Canadian University Surgical Chair who declared that the year prior to checklist implementation, his group of hospitals had reported 4 wrong-sided surgeries, and the year after had none. Should he report it? He was advised to keep on with the institution of the checklist, embed it in the culture of the organization, and if the wrong side surgery rate was still 0, consider reporting and publishing the finding. How many wrong-sided surgeries and how many deaths would make the conduct of a simple 2-minute tool such as the checklist worthwhile?

Regarding the mortality rate of the study in question: there was a 0.06% reduction in this uncontrolled study after the use of the checklist, with a p value of 0.07. That means that with a 93% confidence level, we can predict that during the course of the study, 70 people died in the province of Ontario because the checklist was not used. If we were patients whose surgeon ignored that fact (despite the confidence level not reaching 95%), and failed or refused to spend two minutes discussing in detail our respective cases with the other members of the operative teams, we would find another surgeon.

Finally, the Urbach study doesn’t address all the other beneficial effects of checklist use in the domain of improved team relationships, hospital staff satisfaction measures (Taylor et al., 2012) , and also its use in other more straightforward settings such as endoscopy, interventional radiology, cardiac cath lab (Taylor, 2012) and the like. We must not forget that one of the more startling results in recent years came from the Pronovost study (Pronovost, et al. 2007), proving that the use of a simple checklist could reduce mortality after central line insertion. The fact that some continue to disregard such practices in the ICU setting is a travesty, the same kind of travesty that might occur if the results of the above uncontrolled population study were to be considered evidence that the use of the surgical checklist should be abandoned .

Population data can be very valuable. However, without careful assessment and interpretation of that data, disturbing and potential harmful conclusions can be drawn.

Comments

John McGurran wrote:

Posted 2014/05/20 at 11:35 AM EDT