In healthcare settings, indicators are useful tools to assess the structure, process and outcomes of care. Moreover, when used to report to the public, indicators ensure greater transparency for our healthcare system.

The purpose of this study was to identify in acute care settings three medication safety indicators that are suitable for public reporting in Ontario. A multi-phase process was developed that included a literature review, compilation and evaluation of possible indicators and a consensus-generation process involving a focus group (modified nominal group technique) with Ontario healthcare experts from various disciplines.

More than 300 potential medication safety indicators were identified through the literature review. Two analysts, working independently and using a defined set of selection criteria, narrowed the focus to 49 and subsequently 12 candidate indicators. A focus group of leading experts across the healthcare fields in Ontario was convened and reached consensus on three indicators. These three indicators focused on the areas of venous thromboembolism prevention, acute myocardial infarction discharge medications and medication reconciliation.

This report describes a multi-phase process undertaken by the Institute for Safe Medication Practices Canada to identify in acute care settings three medication safety indicators suitable for public reporting in Ontario. These indicators point to important areas in medication safety at which deficiencies can result in significant patient harm. There is a potential for these indicators to provide hospitals and healthcare providers with tangible and realistic mechanisms for measuring performance and, ultimately, improving the quality of care.

Indicators are measures that describe particular aspects of a system. They can be used to assess what happens to patients as a result of how well clinicians and organizational systems function to address the needs of patients. Monitoring performance over time, benchmarking and prioritization of activities are some of the ways that indicators allow for continuous quality improvement (Mainz 2003).

Indicators also serve as accountability tools to stakeholders; when used to report to the public, indicators can contribute to greater transparency in healthcare. Although indicators are critical to improving the quality and appropriateness of care, they are not direct measures of quality and are not meant to be definitive or diagnostic of a system. They do not necessarily encompass every aspect of the system they measure, which necessitates the need for investigation and analysis of the results in order to understand the context of the particular indicator within the institution's system. However, indicators can act as an initial step in improving quality of care by shedding light on general areas that warrant additional attention (Pencheon et al. 2008).

This article describes the process used by the Institute for Safe Medication Practices Canada (ISMP Canada) to identify medication safety indicators in acute care settings for public reporting to be recommended to the Ontario Ministry of Health and Long-Term Care (MOHLTC). In 2002, ISMP Canada and MOHLTC collaborated to create the Medication Safety Support Service (MSSS), a multidisciplinary advisory committee of representatives from the provincial professional colleges and association of medicine, nursing and pharmacy, as well as the Ontario Hospital Association. Since its formation, MSSS has undertaken a number of medication safety projects and has made recommendations for systems-based enhancements in the handling of concentrated electrolytes, opioids and anticoagulants. The development of medication safety indicators is therefore, in many respects, a natural outgrowth of the expertise and mandate of MSSS.

Medication Safety Indicators

Indicators of medication safety are an important subset of healthcare indicators. In the context of this project, medication safety refers to two aspects: The first is to ensure that patients are ordered the most appropriate pharmacological treatment plan based on the best available evidence. The second is to ensure that the treatment plan is carried out as ordered. This is consistent with the position that "achieving safer care has three agendas, all of which are necessary for success: identifying what works (efficacy), ensuring that the patient receives it (appropriate use), and delivering it flawlessly (no errors)" (Leape 2002: 504). Deficiencies in the first aspect of medication safety, such as the low rate of venous thromboembolism prophylaxis, have been the focus of both national and international patient safety initiatives and reports (Safer Healthcare Now! 2008; Shojania 2001). Likewise, deficiencies of the second aspect of medication safety, such as administration of a medication to the incorrect patient, are commonly known as medication errors and considered a key aspect of medication safety. The medication safety indicators selected in this project cover both aspects. They may be used to monitor and evaluate management, clinical and support functions that affect how safely and effectively medications are being used in our healthcare system (MacKinnon and McCaffrey 2004).

Like other aspects of healthcare, medication systems can be viewed as consisting of three factors: structures, processes and outcomes (Donabedian 2005). Monitoring these different aspects requires various types of indicators. Thus, the project focused upon developing the following:

• Structure indicators or measures of the environment – such as the hospital infrastructure or systems that impact medication use and safety. Such indicators are not directly linked to outcomes but can be helpful in guiding system improvements. They provide a snapshot of the organizational structure and the status of the organization's activities in a particular area of interest, such as whether or not an organization has a process for medication error reporting and analysis (New South Wales Therapeutic Advisory Group 2007).
• Process indicators or measures of compliance with processes of care – these have been shown to improve health outcomes. Process indicators may be directly linked to outcomes (e.g., pre-surgical antibiotic or anticoagulation prophylaxis) and can be helpful in guiding system-based improvements.
• Outcome indicators or data related to the outcomes of care or health system performance – such as the proportion of medication incidents that result in harm or death. Outcome indicators may be easy for the general public to understand but may not provide information that is sufficiently specific to guide system-based improvements.

Methods

To identify medication safety indicators, ISMP Canada undertook a multi-phase research and development process consistent with indicator development processes described by both Canadian and international bodies (Agency for Healthcare Research and Quality 2006; Canadian Institute for Health Information 2003; New South Wales Therapeutic Advisory Group 2007). Phases consisted of the following:

1. Literature review
2. Development of a set of indicator-selection criteria
3. Extraction of medication safety indicators from the literature
4. Use of the selection criteria to, through two screening rounds, narrow down the list to 12 candidate indicators
5. By means of a focus group of experts, reaching consensus on the three most appropriate indicators to be recommended for public reporting

The results of this process were then communicated to the Ontario MOHLTC and the participants by means of a final report.

Phase One: Literature Review

Using a set of search terms, Medline, Embase and Google databases were searched for national and international work on the subject of medication safety indicators. In addition, the reference sections of articles were manually reviewed and a number of healthcare and patient safety organizations (e.g., the Institute for Health Improvement, Accreditation Canada, the Canadian Institute for Health Information and the Canadian Patient Safety Institute) were consulted for reports and grey literature. Indicator manuals from other institutions were also included in the literature review, such as those from the New South Wales Therapeutic Advisory Group.

The search retrieved more than 100 domestic and international journal articles, studies and reports. All resources were printed and compiled for extraction of medication safety indicators.

Phase Two: Development of Selection Criteria

Selection criteria previously used in the development of medication safety indicators were consulted (Agency for Healthcare Research and Quality 2006; Canadian Institute for Health Information 2003; MOHLTC 2009; New South Wales Therapeutic Advisory Group 2007). Selection criteria that were developed were as follows:

• The indicator aligns with current or emerging medication and patient safety initiatives in Ontario and/or Canada (e.g., Accreditation Canada 2009; Safer Healthcare Now! 2007a, 2007b, 2007c, 2008).
• The data required for the indicator are readily available for the settings and time periods required, with no unreasonable obstacles or constraints on access, and the information can be used without restrictions.
• The indicator appears to measure what is intended (i.e., it has face validity), is accepted by the healthcare community, covers relevant content or domains and has predictive power.
• The information being collected can be used to inform and influence policy or funding or alter the behaviour of health services providers.
• The indicator can be readily interpreted, and the intended audience (in this case, the general public) can generally understand the implications if the value changes.
• There is evidence that the highlighted practice can result in improved outcomes (i.e., the indicator is evidence based).

Phase Three: Extraction of Indicators from the Literature

Two analysts independently extracted medication safety indicators from the retrieved literature; as well, a small number of indicators were created by the analysts to reflect important aspects of medication safety. More than 300 potential indicators were identified and, using the above selection criteria, submitted to two rounds of analysis and screening.

Phase Four: Narrowing Down to 12 Candidate Indicators

In the first round of screening, the goal was to reduce the list of indicators by quickly excluding those that clearly did not meet the selection criteria. The two analysts worked independently and, when finished, compared results and discussed and resolved discrepancies. Through this process, the list was reduced to 49 indicators. The 49 indicators were subjected to a second round of evaluation by the analysts, at the end of which 12 (four each for structure, process and outcome) indicators were identified as the most promising. Table 1 summarizes the 12 candidate indicators and shows the rationale for including them, how they align with other medication safety indicators or recommendations and their limitations.

The four structural candidate indicators looked at whether organizations had adopted policies or procedures to reduce the risk of harm from two classes of high-risk medications –concentrated electrolytes and narcotics; had a policy and process for reporting and analyzing medication incidents; and had conducted at least one medication safety-related analysis per year. All four of these indicators were essentially dichotomous (yes/no), although it was also possible to determine the percentage of units in a facility in which concentrated electrolyte (i.e., concentrated potassium) vials were available.

The four process indicators were as follows:

• Proportion of patients with acute myocardial infarction (AMI) discharged with appropriate (secondary prevention) medications
• Proportion of patients for whom medication reconciliation was conducted upon admission to hospital
• Proportion of selected surgical patients who were given antibiotic prophylaxis
• Proportion of selected surgical patients who were given prophylaxis anticoagulation to prevent venous thromboembolism (VTE)

The four outcome indicators were as follows:

• A list of the 10 medications most frequently associated with harm or death medication incidents (as previously reported by ISMP Canada [2006])
• A breakdown of the frequency of different types of medication incidents, such as incidents resulting in harm or in death (as previously reported by the Ontario Health Quality Council [2009])
• The proportion of medication incidents that result in harm or death per days of patient care
• The proportion of total deaths in Ontario associated with medication incidents, suggested by data from the Office of the Chief Coroner for Ontario

Phase Five: Generating Consensus on Three Indicators for Public Reporting

An expert focus group of 17 individuals was created consisting of representatives from MOHLTC, the Ontario Health Quality Council, hospitals from across the province and community pharmacy. The individuals of this group are familiar with the mandate of ISMP Canada and had attended at least one medication safety workshop or seminar held by ISMP Canada; as such, they were consulted for their participation in this endeavour. Table 2 provides a more detailed summary of the membership of this expert focus group. Using a modified nominal group technique (Jones and Hunter 1995), participants were provided with information about the 12 candidate indicators (detailed description, rationale, alignment with other indicators or measures and limitations) and then divided into seven small groups of two to three participants per group for discussion. Groups then voted for the three medication safety indicators of their choice, after which participants described the rationale of their selections; this was followed by further discussion and debate. A second round of voting was then held to make the final selection of three indicators. Focus group discussions were also recorded, transcribed and subjected to thematic analysis.

Results

By the end of the second round of voting, the indicators that received the most votes were all process indicators: AMI discharge medications and VTE prophylaxis were unanimously selected by all seven small groups, and medication reconciliation was selected by five groups. The expert panel also supported to a lesser extent the outcome indicator of the number of deaths associated with medication incidents, but due to the low level of support (two votes) it was not included in the final list of three medication safety indicators.

Thematic analysis of the focus group discussion notes revealed some of the issues that shaped the final selections. First, there was considerable discussion about the fundamental objective of the indicators: whether they should be designed to promote healthcare system accountability or to increase public awareness of medication safety. The group's decision was that indicators should be developed that primarily support healthcare accountability, although consideration should also be given to their suitability for sharing with the public (that is, public reporting).

In the case of the process indicators (AMI discharge medications, pre-surgical antibiotic and anticoagulant prophylaxis and medication reconciliation), it was clear from the comments of the panel members that considerable clinical evidence of effectiveness gave the indicators not only validity but also perceived potential to promote beneficial change. Moreover, as many institutions are already tracking some of these indicators (e.g., pre-surgical anticoagulant prophylaxis), gathering data for public reporting was seen as highly feasible. At the same time, one group felt that, at least in the case of surgical prophylaxis, the interventions were already largely integrated into standard practices and so the potential for change would be limited. This group argued that there might be greater benefit if indicators focused upon areas where there is less adaptation of best practices and therefore a greater need for improvement.

Medication reconciliation was recognized to be somewhat different from the other three process indicators in that it addresses overall system integration as opposed to a specific clinical practice. Its relationship to system integration was considered a significant challenge in healthcare by some participants. Other participants, however, felt that although medication reconciliation is important, it may not be as strongly linked to patient outcomes or impact compared with the other three process indicators (in the short list of 12 indicators).

There are also methodological challenges in creating a medication reconciliation indicator. Clear and feasible definitions must be created for both the numerator and denominator, and data need to be captured in a consistent manner. Ensuring comparability in medication reconciliation rates between hospitals could be difficult as different institutions may have varying criteria for determining which patients are appropriate candidates or how reconciliation is conducted. As a result, some participants suggested that medication reconciliation should be considered a "stretch goal" that healthcare could work toward and that could be used to dialogue with the public.

Although there was a general consensus in the group that the four candidate structure indicators (removal of concentrated electrolytes, narcotic safety, incident reporting system and prospective analysis) were important in terms of patient safety and accountability, participants were uncertain as to whether they would be appropriate for public reporting. The challenge for these indicators is that their significance may not be readily apparent to the public. For instance, the indicator of removing concentrated electrolytes would require explanations of what is meant by "concentrated electrolytes," what sort of risk they pose and how their removal from some settings can address patient safety.

A second issue raised by the structural indicators concerned the ability of the dichotomous structure indicators (yes/no) to track improvement in individual hospitals over time. In other words, if an institution was able to answer "yes" to an indicator, would there be benefit in repeating the question? One suggestion was to create a composite indicator so that the progress of individual hospitals in meeting all four indicators could be tracked over time.

The outcome indicators identified through the literature search and analysis (list of top 10 medications associated with medication incidents resulting in harm or death, types and rates of medication incidents and deaths associated with medication errors) were seen as having the advantage of being easy for the general public to understand. However, methodological and data limitations (see Table 1) were seen as potential challenges, particularly those limitations associated with voluntary medication incident reporting systems.

Discussion

Our review of literature identified more than 100 journal articles from which more than 300 potential medication safety indicators were extracted. This indicates a substantial body of work already done in this area. However, although most of the articles provided a final list of indicators, very few of them provided information regarding the rationale for their selection and the discussions involved in making these selections. By presenting the final indicators that were chosen as well as a thematic analysis of the focus group discussion, the results of this project provide insight to the rationale for each indicator selection, as well as some of the anticipated difficulties and challenges toward their implementation in healthcare organizations.

A limitation of the methodology used in this project expressed by a number of focus group members was that they were presented with only 12 candidate indicators (out of over 300) for consideration, and that there were no modifications to or addition of indicators after the first round of voting. Some members wondered if there were other suitable indicators beyond the 12 candidate indicators, especially from the 49 indicators after round one of screening. Some suggested that it would have been beneficial to have had an additional focus group meeting at an earlier stage of screening. To address this limitation, the list of 49 candidate indicators was subsequently provided to each of the focus group members after the meeting. Further feedback was then obtained, and it was clear that the final selections remained the same. Although the objective of this initiative was to identify three medication safety indicators for public reporting, the value of the 12 candidate indicators that were initially presented to the focus group should not be overlooked. Many of the experts within the focus group had recognized their role and importance within the healthcare system, and it was only after extensive deliberations that consensus on the three indicators was achieved. These additional indicators merit further analysis and may provide the basis for subsequent research opportunities.

Conclusion

This report describes a multi-phase process undertaken by ISMP Canada to identify a small number of indicators of medication safety for Ontario that would be informative, aligned with current patient safety initiatives, of acceptable quality (valid and reliable), actionable, understandable by the intended audience including the general public, evidence based and feasible for data collection. The indicators that were selected (AMI discharge medications, VTE prophylaxis and medication reconciliation) are evidence based and can be derived from existing and reliable hospital data. They point to important areas in the healthcare system in which deficiencies can result in significant patient harm, and they thus have the potential to provide hospitals and healthcare providers with tangible and realistic mechanisms for measuring performance and improving the quality of care. Moreover, if clearly defined and communicated with appropriate explanations, they should be understandable by the public, thereby increasing public awareness of the importance of medication safety.

Comments

Be the first to comment on this!