Once the decision was made to implement an electronic medication order entry and medication administration record (MOE/MAR) system at the University Health Network (UHN), a significant question soon emerged: How would UHN be able to determine if the project had indeed accomplished its stated objectives of improving patient safety and the medication ordering and processing cycle?

With this in mind, UHN enlisted the assistance of researchers who would conduct a quantitative study to measure the impact and benefits of the new system. While seemingly straightforward, this evaluation was made more challenging by the diverse expectations of MOE/MAR.

This article reports on the results of UHN's multi-year study looking at the impact of MOE/MAR. In our overview, we examine such elements as the methodology used as well as the challenges and constraints faced by the team. We also examine the following: the types of lessons learned during MOE/MAR's implementation; the effectiveness of teamwork; and the impact of external resources upon the project.

Why Study the Benefits?

The old saw, you can't manage what you can't measure, was the orienting principle behind the MOE/MAR evaluation from conception to implementation. The study's objective was to measure the impact of implementing MOE/MAR on patient safety, clinical workflow and the quality of patient care. We developed a number of specific performance indicators from which we expected definite results. We soon discovered there were very few studies, either at the UHN or anywhere else, that indicated consistent results regarding the impact of MOE/MAR. Some studies, in fact, indicated mixed or conflicting results. Up to this point, UHN had only conducted rudimentary benefits measurement evaluations on its IT projects as part of its overall project management function. Our experience suggested that this was common in the healthcare sector and provided us with yet another reason to move forward with our more comprehensive evaluation.

The need for a benefits/measurement study of MOE/MAR was made clear by UHN leadership. If the healthcare organization was committing significant time as well as human and financial resources toward implementing this major change initiative, it was essential to gather data and provide evidence-based research to justify to UHN decision-makers (and other healthcare organizations contemplating similar projects) that such investments were indeed worth the effort. Without such information it would not be possible to demonstrate MOE/MAR's effectiveness.

A comprehensive, multi-year study would provide ongoing information to the implementation team, users and administration that the project was proceeding according to plan. Study data would also be used to measure the implementation progress and any adjustments needed along the way to improve the process - allowing for the cycle of ongoing quality improvement at UHN.

Background and Overview

UHN's Shared Information Management Systems (SIMS) department had engaged an external healthcare consulting group to advise and support the overall MOE/MAR project. The lead consultant from the firm strongly advocated the need for a UHN leader-supported benefits/measurement study and engaged a faculty researcher at a local university to join him on the MOE/MAR study.

The researcher assembled a group of his undergraduate summer students and former graduates in industrial engineering to join the project team. From the outset, the students were integrated into the overall team structure rather than assigned tasks and sent off to work (and subsequently submitting reports of their findings). This was a successful strategy, as the students gained practical experience in their field and the organization (UHN) established a working relationship with the university that could be continued on future projects.

In addition, the team also convened a benefits realization steering committee that comprised the key stakeholders on the project to determine objectives and research questions.

The "Benefits" of Our Benefits Study

It was recognized that empirical evidence generated through the study could help validate (or challenge) the commitment to MOE/MAR. And, should MOE/MAR prove successful, it would positively affect attitudes and perceptions, supporting future implementations. Lastly, conducting the study reinforced a "culture of accountability" at the hospital so that the impact and benefits of future projects could be properly analyzed in a systematic and quantifiable way. By this time, SIMS staff had already incorporated a rigorous benefits/measurement framework into several other projects.

One of the key reasons we successfully secured support for the study was the promise of the ability to quickly show results. For example, as data were collected during the time spent on a unit with a particular group that had implemented MOE/MAR, we shared these data weekly (sometimes more often) with the groups so that each group could see what improvements there were on their specific units. Often, these included reductions in transcription errors in medication orders or change in time spent on recording medication orders into the MAR. Stakeholders were able to readily see changes that were both tangible and positive.

Methodology and Approach

A steering committee with diverse representation was established to oversee the study. To be consistent with the project management methodology in terms of the overall process, a project charter and detailed work-plan was devised and subsequently implemented. In this manner, the benefits measurement process could be separately tracked by the SIMS' Project Management Office to ensure it met timelines and project expectations relative to the MOE/MAR implementation.

Our approach to assessing the benefits of MOE/MAR proceeded in four stages:

1. Select Key Metrics and Indicators
   
   • Identify the right indicators to address the stated objectives of your MOE/MAR project. For example, is it patient safety or process improvements that the organization would most like to achieve?
   • Select indicators that are quantifiable, easy to collect and yield the most significant results. Recognize that clinical chart audits are very resource-intensive and require experienced clinician resources.
2. Identify Data Sources
   
   • For the UHN study, we developed our metrics around three different types of available data sources: Chart Audits, Time-Motion Study and Electronic Reporting (Data Warehouse).
3. Collect/Tabulate Data
   
   • Develop charts and tables that are accurate and easily interpreted so that immediate actions can be developed.
4. Communicate Results
   
   • Develop a communication mechanism to share results and feedback with stakeholders and users on a regular basis.

After documenting the medication ordering process (see Figure 1), we developed indicators that we felt would best demonstrate the impact of the MOE/MAR technology. The list of key indicators measured in the study is consolidated in Table 1. During this phase we were always mindful to choose indicators that would be meaningful and applicable for clinical and hospital executive decision-making.

[Figure 1]

[Table 1]

Results and Findings

The data for this study were collected over a two-year period (May 2004 to May 2006) for all eight clinical service clusters that were implemented at UHN. These include both medical and surgical patient care areas, which exhibit different detailed medication ordering cycle workflows (see Table 2).

[Table 2]

Technology Adoption

From the CPOE rate of 89%, we can conclude that the adoption of the technology is high; many institutions do not see this level over 50%. However, based on our surveys after a two-month implementation process, the clinicians (MD, RN, Rx) still had mixed feelings about the system, as they were still moving down the learning curve.

Quality of Care and Patient Safety

The ability to use clinical-decision support was a significant benefit of MOE/MAR that could not have been realized with a paper-based system. With an average of 4,000 interaction alerts "fired" per month, these numbers show that UHN is heading in the right direction vis-à-vis utilizing technology to further bolster patient safety. However, the mere fact that more alerts are being fired in itself is not necessarily a good thing. Over time the alerts could be considered a nuisance and eventually be ignored by a physician. This could not only reduce the usefulness of a potentially effective decision-support tool but could also lead to an increase in patient safety risks for patients. To ensure that this does not happen, SIMS and its clinical advisors continuously evaluate the alerts programmed into the system to ensure they are accurate and useful, as well as monitor the acknowledgment rate of the alert recipients.

The near complete reduction of transcription errors in both the chart audit study and the reported medication incidences has allowed us to establish the value of the MOE/MAR system and speak directly to patient safety. In addition, non-measured benefits (because there were no base-rate data) were realized from the system, namely, 100% complete and legible orders.

Clinical Workflow

The overall medication ordering process and turnaround time of the "Now" dose markedly decreased with the implementation of the MOE/MAR system. This creates an improved patient care environment due to fewer medication delays.

UHN has historically attempted to decrease the number of verbal and telephone orders. In this regard MOE/MAR has been able to facilitate a 75% decrease in these types of orders, thereby allowing the physicians and nurse practitioners to directly enter their own orders. This has noticeably reduced miscommunication while increasing accountability.

Finally, customized order sets - tailored to each practice area - and offsite accessibility have been valuable to address the challenges posed by residents and clinical fellows who rotate throughout an academic teaching hospital.

Policy Adherence

Throughout the project, we have seen a shift in accountability. The introduction of the electronic ordering and documentation of medication has created a shift in medication reporting practice. We have a seen a 150% increase in overall reported medication incidents. This is a positive step towards UHN's stated goal of "transparent" patient care practice, and is also good for the overall organization.

We have also seen an almost 97% co-signature rate for verbal and telephone orders. Thus, UHN has almost achieved its goal of 100% co-signed orders.

Process Improvements

Since the results of the study were shared and communicated on an ongoing basis after every cluster implementation, we were able to utilize the data to identify areas of improvement and our approach for the subsequent rollouts. This allowed us to provide an ongoing cycle of improvements, using a methodology similar to the PDSA cycle (Plan/Do/See/Act) created by the Institute for Healthcare Improvement.

There were three areas where we could use the results for improvement. The first area was the current product/application, as we had identified improvement opportunities in the customization of the application to better suit the clinician needs. We were also able to identify clinical work improvements by taking advantage of the electronic capabilities not available on paper. As well, we were able to use the data to improve on our project implementation processes by making mid-course corrections, our status report card (as mentioned in the Project Management section) and allowing us to identify unreported issues by the users and resolving them in a timely manner.

Challenges

The initial challenge for us was securing executive commitment to conduct a benefits/measurement study. While there was general interest, there were also many questions from organization leaders about what it would cost, how much time it would take to complete and how many people would need to be involved. In addition, there was some scepticism about how quickly results would be available. To put all these issues in context, we returned to our initial question: How will we know whether the decision to invest time, people and money in MOE/MAR was worth it? We, UHN's senior leadership, and MOE/MAR leaders all recognized that the best way to answer these questions was to proceed with the study, using available financial resources and developing a detailed and time-bound project plan.

During steering committee meetings, another issue arose. The committee was made up of researchers, clinicians, administrators, academics and project managers. This mix of people and perspectives naturally led to a diversity of opinions on the purpose of the study, the attainability of the data and the value of the results that could be achieved. Managing these differing agendas was a significant challenge. Once a scope and focus for the study was determined, it was important to create a smaller operational group, made up of key members with commitment to the same vision. This group was responsible for monitoring the project and ensuring it stayed on track. In addition they would report back to the study steering committee to review quarterly and bi-annual findings.

We initially concentrated more of our efforts on the physician order entry impacts, as we had originally separated our project rollout to the MOE component only to help manage the implementation workload. However, our study had quickly identified serious process issues that come with an electronic MOE and a manual paper MAR system. This identification led the project team to revise their rollout schedule after the initial cluster implementation decision to include both MOE and MAR functionalities simultaneously. As a result, this corrected the hybrid process issues and helped realize the benefits of using both modules together.

Another major challenge was extracting data from UHN and SIMS' electronic reporting systems from which we could make decisions about the indicators we would use as measures. Generating data took months. And when we did produce data, we questioned the usefulness and quality. The electronic reporting system simply was not designed for this purpose. Most hospital information systems do not support process improvement activities; rather they focus on clinical and finance reporting. Many of the reporting and data sources for this study had to be customized and required periods of time to be ready. One challenging example was the medication timing report, which included order, verification and administration time. This may also be an issue for other organizations interested in conducting a similar study.

Finally, it is also essential to recruit people with a solid knowledge of process as well as analytical and problem-solving skills (not merely data collection or data entry skills). We were fortunate that our team had these core competencies. A study like this requires individuals who know how to interpret data. In short, professionals with research experience - as well as systems analysis skills - are essential to the project's well-being.

After the Study - Should We Continue?

What began as a research study has now evolved into a framework for how people do their work on a daily basis. While the MOE/MAR implementation has been completed, UHN continues to internally study the results, submitting reports to senior administration and clinical leaders throughout the hospital network.

The plan over the next year is to continue to incorporate the key lessons from this project into a continuous cycle of planning for other information technology-based projects at UHN.

Comments

Be the first to comment on this!