The Canadian Institute for Health Information began publishing hospital standardized mortality ratio (HSMR) data for select Canadian hospitals in November 2007. This paper describes the experience of the Winnipeg Regional Health Authority in assessing the validity of the HSMR through statistical analysis, coding definitions and chart audits. We found a lack of empirical evidence supporting the use of the HSMR in measuring reductions in preventable deaths. We also found that limitations in standardization as well as differences in palliative care coding and place of death make inter-facility comparisons of HSMRs invalid. The results of our chart audit show that the HSMR is not a sensitive measure of adverse events as defined by "unexpected death" in the Canadian Adverse Events Study. It should not be viewed as an important indicator of patient safety or quality of care. We discuss the cumulative sum statistic as an alternative to the HSMR in monitoring in-hospital mortality.