Healthcare Quarterly
Toward an Integrated Simulation Approach for Predicting and Preventing Technology-Induced Errors in Healthcare: Implications for Healthcare Decision-Makers
Abstract
| Terminology at a
Glance Clinical simulation: Simulations of clinical activities (e.g., medication administration) involving human subjects (e.g., doctors and nurses) using HIS and devices. Health information systems (HIS): Information systems (such as electronic health records and medication order entry systems) designed to support healthcare activities, workflow and decision-making. "In the box" computer-based simulation: Simulations that are computer-based and embody mathematical models. Monte Carlo experiment: An experiment that obtains and displays a collection of simulation outputs for a stochastic model. Risk management: A structured approach to identifying, classifying, controlling and monitoring risk. Technological maturation: Advancement of an industry to the point where industry specific procedures and methods are developed for the design and evaluation of technology used in that industry. Technology-induced error: Error that inadvertently occurs as a result of using a technology (e.g., medication errors that result from using a system). User interface features: Aspects of a systems' human computer interface (e.g., screen layout, menu items, scroll bars). |
The reduction of medical errors has been a driving force behind many healthcare administrators' and researchers' efforts to implement healthcare information systems (HIS) (e.g., electronic health record systems, computerized physician order entry [CPOE]) and other health technologies in Canada and worldwide. Along these lines, in the 1990s a number of influential published papers demonstrated that the ability of HIS to reduce medical errors can be significant (Bates et al. 1999).
More recently, research has emerged clearly showing that poorly designed, implemented, customized and operated HIS have the potential to cause or facilitate new types of errors, often referred to as technology-induced errors, technology-facilitated errors or unintended consequences (Ash et al. 2007; Borycki and Kushniruk 2008; Koppel et al. 2005; Kushniruk et al. 2005). For example, Koppel et al. (2005) were able to observe and identify real-world clinical situations that may lead to technology-facilitated errors involving HIS and devices. Thus, it is important to be able to understand both the intended and unintended impacts of HIS. Although much of this research has demonstrated that technology-induced errors are a concern that warrants significant attention and caution when implementing such systems in healthcare organizations (Joint Commission 2008), most of this research has involved the use of methodologies (i.e., observational research, interviews, ethnography) that allow for the unintended consequences of technology use to be identified only after the technology has been implemented in a real-world setting (e.g., Ash et al. 2007; Koppel et al. 2005). Furthermore, some researchers have suggested that using qualitative approaches such as the ones outlined above to study technology-induced errors may lead to subject or researcher recall bias as well as researcher interpretation bias (Ericsson and Simon 1993; Miles and Huberman 1994).
Recently, other researchers have advocated the use of predictive approaches (i.e., clinical simulations applied to technology evaluation) that identify the unintended consequences of a technology prior to its implementation (i.e., during the software development life cycle and prior to the technology's implementation in a given organizational setting), when the costs associated with making changes to a technology are lower for the organization (Patton 2001). For example, Kushniruk and colleagues (2005, 2006) have demonstrated that clinical simulation can be used to show that the design and function of user interface features may lead to the entry of incorrect information, and that cumbersome workflows may lead to workarounds that can facilitate error (Kushniruk et al. 2005). These researchers have demonstrated the utility of clinical simulation in identifying potential intended and unintended consequences of using a HIS or device prior to its deployment in a real-world environment (Borycki et al. 2006; Kushniruk et al. 2005, 2006). A limitation of this approach is that it may not provide sufficient information to healthcare decision-makers (i.e., chief information officers, chief medical information officers and directors of clinical informatics) about the departmental- or organizational-level implications of these unintended consequences.
The purpose of this paper is to describe a new approach that links clinical simulations with computer-based simulations. This approach can be used to provide healthcare decision-makers with information about the departmental and organizational implications of unintended consequences associated with technology use over time. Such information can be used in making decisions about changes or customizations to a systems' interface design or workflows emerging from HIS use prior to its implementation in an organization. These data can also inform organizational training and optimization plans once the HIS has been implemented, thereby preventing unintended consequences arising from individuals using the system in ways that were not intended by the system's designers (Orlikowski 1992).
Simulations can be seen as a critical element in risk management involving HIS. Risk management involves several steps: (1) identifying the risks and assessing of their severity and frequency, (2) classifying the risks as manageable or non-manageable, (3) controlling the risks and (4) monitoring the effect of the controls (Schmuland 2005). Currently, risk management does not have widespread use in HIS except in the design of medical device software (for an example, see Jones et al. 2002). The risk management process can occur in the design phase (see Alexander 2003; Boehm and Turner 2003; Jones et al. 2002). In the risk management context, computer simulation (i.e., clinical and computer-based) can be seen as a risk control measure to reduce adverse events in healthcare and any subsequent claims against the organization. Computer simulation also introduces its own risks that would need to be examined in a risk management process.
Emergence of Technology-Induced Errors in Healthcare
Technology-induced errors arising from the use of HIS and devices is not a new phenomenon. Some of the earliest reports of such errors include those arising from the design of software and devices, software programming, and interactions between software, devices and their human operators (Borycki and Kushniruk 2008; Joint Commission 2008; Leveson 1995). Historically, the occurrence of these types of device and software errors was rare in the healthcare domain for a number of reasons, the most significant being the relatively low use of technology by healthcare professionals compared with other industries (e.g., business and manufacturing). With the advent of the 1980s, healthcare saw a significant (if not exponential) rise in HIS and device use (e.g., medication administration systems, bar-code scanners, computer workstations and handheld computers) (Shortliffe and Cimino 2006).
In the past 30 years, improvements in computing speed and power and the maturing of most HIS software (from simple programs that allow for some text-based entry of clinical data to the development of software and devices that provide advanced decision support functions) have increased HIS complexity and, at the same time, introduced new opportunities for technology-induced errors (Borycki and Kushniruk 2008; Koppel et al. 2005; Musen et al. 2006). These developments have resulted in increased reports of such errors and calls for caution and planning when implementing these systems (Joint Commission 2008).
Technological Maturation and Information Systems Safety
Information systems support workers in many differing industries, such as healthcare, aviation, business and nuclear power. According to the general computer science and information technology literature, information systems are iteratively improved upon or refined over time to obtain a better fit for the system's human operators, the tasks they perform and the organizations where work is done (Preece et al. 1994). Therefore, software and device companies are continually improving and refining their products. Technological maturation involves both refining information systems and technologies and reducing the likelihood of technology-induced errors occurring.
Technological maturation at an industry level often involves the development of industry-specific procedures and methods for testing information systems. For example, in the aviation industry, cockpit simulators were developed to test new aircraft software - such testing helps to determine the safety of the software prior to its use in general commercial air travel. In addition, when software failures occur (e.g., aircraft malfunction is caused by software during simulated flight), the causes of the malfunction are identified and addressed to prevent future failures. This often results in further refining of the software prior to its continued use.
Technological Maturation in Healthcare
HIS are undergoing similar maturational processes. Researchers are developing software-testing methodologies to ensure a good fit for the healthcare "industry" (Borycki et al. 2009) and to determine software safety. Researchers have demonstrated that technology-induced errors can be documented and observed in situ in a clinical setting (Ash et al. 2007; Koppel et al. 2005). Methodologies such as ethnography, observational research and qualitative interviews have been used to successfully document the occurrence of technology-induced errors in clinical settings (Ash et al. 2007; Koppel et al. 2005). However, many of these methodologies are used after a technology has been implemented. As a result, technology-induced errors are identified only after significant expenditures have been made by the healthcare organizations to select a system, redesign organizational work processes, customize a system and implement it (Borycki and Kushniruk 2008). As well, healthcare providers are already using these systems in real-world settings when the error-facilitating aspects are identified and safety issues are raised.
According to the software engineering literature, it is more costly to redesign and re-implement a technology after it has been implemented in an organizational setting (Patton 2001). This literature suggests that every effort should be made to identify possible technology-induced errors during software design and development to eliminate or reduce the costs of unintended consequences and the costs associated with HIS and device redesign, redevelopment, re-implementation, training and optimization (Patton 2001).
Industry-Specific Approaches to Identifying Technology-Induced Errors prior to HIS Implementation
In an effort to address this need for early detection of errors, researchers are attempting to develop methods that can be used to test software safety early in the HIS design and development process. Two methods, clinical simulation and computer-based simulation, are discussed below.
Clinical Simulation
One methodological approach that has demonstrated its value in providing information about HIS safety prior to systems implementation is clinical simulation. Clinical simulations in health informatics involve individuals (e.g., physicians, nurses, other health professionals and actors playing patients) interacting with HIS and devices in laboratory settings (e.g., a vacant hospital room), carrying out representative, realistic clinical tasks in response to scenarios typical of those found in real-world settings (e.g., hospitals and clinics) (Borycki et al. 2009). Clinical simulations involve video and audio recordings of the actions of health professionals as well as screen recordings of their interactions with a HIS and devices (Kushniruk et al. 2006).
Our initial work in the area of clinical simulations as applied to HIS identified a relationship between HIS attributes (i.e., interface design features and functions) and specific types of technology-induced errors. We were able to use our methodology to identify and predict potential technology-induced errors prior to systems implementation. We were also able to identify the probability with which a specific interface's attributes were associated with a technology-induced error (Kushniruk et al. 2005). Typically, this simulation involved video recording of subjects (e.g., physicians or nurses) interacting with HIS (e.g., electronic health records) as they carried out tasks such as reviewing patient data (Figure 1). The data collected included rich video and audio data that could be analyzed to identify specific sources of error (as described in Kushniruk et al. 2005). Such potential sources of error could then be corrected prior to the widespread release of the HIS in healthcare organizations.
We have also used clinical simulations to evaluate the impact of new HIS and devices upon health professionals' workflow. We were able to observe the impact of interactions between healthcare workers, HIS and devices (e.g., a bar-code scanners, medication carts and computer workstations). Unintended consequences of these implementations (i.e., workarounds) were observed, and feedback was provided to systems designers and implementers that was used to refine the technology and develop training for health professional users (see Kushniruk et al. 2006).
Extending Clinical Simulations to Computer-Based Simulations
Many healthcare administrators use computer-based simulations to provide decision support. Computer-based simulations are used by organizational decision-makers to assess the potential impact of proposed organizational changes upon institutional outcomes, especially in cases where there are ethical or economic considerations that may arise from the organizational change (Anderson 2005; Sobolev et al. 2008).
"Computer simulation provides an alternative method to quantifying the effects of proposed changes in healthcare delivery" (Sobolev et al. 2008: 128). For example, it has been used by the US and Canadian governments to simulate the spread of a communicable disease across a region and evaluate the impact of targeted interventions (e.g., vaccinations) upon the spread of that disease (e.g., Vardavas et al. 2007). Knowledge arising from the use of computer-based simulation has been used to develop pandemic preparedness plans by local, provincial and national governments. Sobolev et al. (2008) used a computer-based simulation to examine the impact of two differing methods of booking elective surgeries - pre-booking and booking from wait lists - upon the likelihood of a patient receiving a procedure based on priority. In health informatics, simulations have been used to estimate the costs of using two types of wide-area networks across a province for linking physicians and hospitals (McDaniel 1995), to estimate "the effects of increasing the percentage of medical orders that physicians enter directly into HIS upon medical errors arising from the processing of medical orders" (Anderson et al. 1988) and to assess the impact of implementing CPOE (Anderson 2005).
Use of Computer-Based Simulations in Predicting the Impact of HIS
In our work, we have used computer-based simulations to model the impact of HIS and to support healthcare manager decision-making. We have used data arising from clinical simulations involving technology (e.g., error rates obtained from an analysis of subjects' interactions with systems) as input parameters to computer-based simulation models. We have thus linked clinical simulations to computer-based simulations in studying technology-induced error involving systems such as electronic health records and medication administration systems. In our research, data obtained from clinical simulations involving the study of humans interacting with a HIS, including the probabilities of specific interface design features leading to technology-induced error, were used to populate a computer-based model. Forecasts were then made about the number of errors that would take place over time in a large healthcare institution if the features of the interface remained the same and the system were implemented. In addition, we were able to determine the impact of correcting or changing specific interface design features upon error rates (Kushniruk et al. 2008).
Decision-makers can use this information to make decisions about software design and development, selecting those features and functions they wish to address. Furthermore, in cases where there is little opportunity to modify the design of a system, decision-makers can allocate additional resources to staff training and optimization of staff use of the system after implementation. In this way, the simulation is used to identify and solve problems prior to their real-world occurrence.
Figure 2 illustrates a computer-based simulation model developed by the authors (using the software simulation package STELLA®, isee systems, New Hampshire) that is based on results from studies involving human subjects in clinical simulations and that predicts the number and pattern of errors that would arise from use of a handheld prescription writing device. (For a full description of the study, see Kushniruk et al. [2005].) The purpose of this model is to forecast the expected number of medication errors ("slips," which are errors caught by users, and "mistakes," errors not caught by users). The estimates used for the model variables are based on the empirical findings from Kushniruk et al. (2005).
The decision points of the model use Monte Carlo experiments to pass values of "1" or "0" through the model according to the outcome of each experiment. For example, the starting point of the model (shown on the left), called "new Rx," uses the empirical finding that physicians encountered a usability problem during the entry of a prescription 73 times over 89 prescriptions (82%). The model thus passes a value of "1" to the next module 82% of the time, representing a new usability problem. At the next module, the usability problem was classified as either an "interface problem" or a "content problem"; these were found to occur 84% and 16% of the time, respectively. Thus, this module conducted a Monte Carlo experiment, producing a "1" 84% of the time and a "0" 16% of the time. If a "0" is passed from the previous model, indicating no usability problem, then no further experiment is done at this module. These values work their way through the model, being acted upon by logic defined at each module, and result in the output as either "1" to represent the presence of a medication error or "0" to represent no medication error occurring.
Graphs and tables can be incorporated into the output of the simulation. A table, which can be exported into other applications such as Microsoft Excel for further analysis, is useful in tracking individual values through the model to validate that the logic was performed as intended. Graphs allow for a visual inspection of the output in real time. Figure 3 illustrates a graph over time of slips and mistakes for the prescription writing application modelled in Figure 2. Such information can be used to inform decision-makers who are planning the implementation of HIS and devices in their organization.
Discussion and Summary
The practical implications of combining clinical simulations with "in the box" computer-based simulation are significant for healthcare administrators, who are increasingly being asked to use evidenced-based approaches to solving healthcare problems. In cases where costs or other reasons prevent healthcare administrators from implementing or evaluating HIS, clinical and computer-based simulations offer an alternative and provide insights as to possible implications of HIS upon the organization. Implementing HIS is a costly endeavour - there are costs associated with the design, development, customization, implementation and optimization of HIS and devices. Health informatics researchers have developed a range of methods for evaluating HIS, but many of these methods (e.g., ethnography, observational research and interviews) have been used to evaluate the impact of unintended consequences of HIS after implementation. There is a need for a more proactive risk management-oriented approach.
Researchers have developed clinical simulations as a method of evaluating the effects of HIS on error rates. These methods have provided insights as to the potential causes of these types of technology-induced errors, but there is a need to provide additional information to healthcare administrators regarding the impact of such errors at the departmental and organizational levels prior to systems development, procurement and implementation. If this impact is understood, healthcare decision-makers can implement corrective actions before implementing the system, limiting or mitigating the risk of technology-induced errors. Actions could include allocating human and financial resources to the redesign of some interface features, re-engineering processes linked to HIS and simplifying them, tailoring training and optimizing technology to prevent unintended consequences associated with its use. This risk management approach determines a clear identification of risks of the system including adverse events, an assessment of their frequency, the cost impact for the organization and which risks are serious enough to control. Then controls can be designed and monitored for residual risks in HIS or a device.
Risk management is well established in healthcare as a tool for accountability. Risk management policy should be expanded to include simulation as a risk control for downstream errors involving technology. If the risks identified from simulation are included in planning and policy, this will improve the implementation of HIS and institute an accountability structure that is acceptable to decision-makers.
About the Author(s)
Elizabeth M. Borycki, RN, PhD, is an assistant professor at the School of Health Information Science, University of Victoria, Victoria, British Columbia. You can contact her by e-mail at emb@uvic.ca.
Andre Kushniruk, PhD, is a professor at the School of Health Information Science, University of Victoria. He can be reached by e-mail at andrek@uvic.ca.
Elizabeth Keay, MD, FRCPC, CRM, is a PhD student in the School of Health Information Science, University of Victoria. You can reach her at lizkeay@shaw.ca.
James Nicoll, BA, is a health informatics MSc student studying at the University of Victoria. He can be contacted at jnicoll@uvic.ca.
James Anderson, PhD, is a professor of medical sociology, a professor of health communication and a fellow of the American College of Medical Informatics. He works in the Department of Sociology and Anthropology at Purdue University, West Lafayette, Indiana. You can contact him by e-mail at Andersonj@purdue.edu.
Marilyn Anderson works for Anderson Consulting in West Lafayette, Indiana. You can contact her by e-mail at marilynmanderson@comcast.net.
References
Alexander, I. 2003. "Misuse Cases: Use Cases with Hostile Intent." IEEE Software 20(1): 58-66.
Anderson, J.G. and C.E. Aydin. 2005. Evaluating the Organizational Impact of Healthcare Information Systems. New York: Springer.
Anderson, J.G., S.J. Jay, S.J. Clevenger, D.R. Kassing, J. Perry and M. Anderson. 1988. "Physician Utilization of a Hospital Information System: A Computer Simulation Model." Proceedings of the 12th Annual Symposium on Computer Applications in Healthcare 858-61.
Ash, J.S., D.F. Sittig, R.H. Dykstra, K. Guappone, J.D. Carpenter and V. Seshadri. 2007. "Categorizing the Unintended Sociotechnical Consequences of Computerized Provider Order Entry." International Journal of Medical Informatics 76(1): 21-27.
Bates, D.W., J.M. Teich, J. Lee, D. Seger, G.J. Kuperman, N. Ma'Luf, D. Boyle and L. Leape. 1999. "The Impact of Computerized Physician Order Entry on Medication Error Prevention." Journal of the American Medical Informatics Association 6(4): 313-21.
Boehm, B. and R. Turner. 2003. "Using Risk to Balance Agile and Plan-Driven Methods." Computer 36(6): 57-66.
Borycki, E.M. and A.W. Kushniruk. 2008. "Where Do Technology-Induced Errors Come From? Towards a Model for Conceptualizing and Diagnosing Errors Caused by Technology." In A.W. Kushniruk and E.M. Borycki, eds., Human, Social and Organizational Aspects of Health Information Systems. Hershey, NY: Idea Group.
Borycki, E.M., A.W. Kushniruk, S. Kuwata and H. Watanabe. 2009. "Simulations to Assess Medication Administration Systems." In B. Staudinger, V. Hoess and H. Ostermann, eds., Nursing and Clinical Informatics. Hershey, PA: Idea Group.
Borycki, E.M., A.W. Kushniruk, S. Kuwata and J. Kannry. 2006. "Use of Simulation Approaches in the Study of Clinician Workflow." AMIA Conference Proceedings 61-65.
Ericsson, K.A. and H.A. Simon. 1993. Protocol Analysis: Verbal Reports as Data (2nd ed.). Cambridge, MA: MIT Press.
Joint Commission. 2008. "Safely Implementing Health Information and Converging Technologies." Sentinel Event Alert 42: 1-5.
Jones, P.L., J. Jorgens, A.R. Taylor and M. Weber. 2002. "Risk Management in the Design of Medical Device Software Systems." Biomedical Instrumentation and Technology 36: 237-66.
Koppel, R., J.P. Metlay, A. Cohen, B. Abaluck, R. Localio, S.E. Kimmel and B.L. Strom. 2005. "Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors." Journal of the American Medical Association 293(10): 1197-203.
Kushniruk, A., E.M. Borycki, J.G. Anderson and M.M. Anderson. 2008. "Combining Two Forms of Simulation to Predict Impact of Interface Design on Technology-Induced Error in Healthcare." Proceedings of the 2008 Spring Simulation Conference 497-504.
Kushniruk, A.W., E.M. Borycki, S. Kuwata and J. Kannry. 2006. "Predicting Changes in Workflow Resulting from Healthcare Information Systems: Ensuring the Safety of Healthcare." Healthcare Quarterly 9: 114-18.
Kushniruk, A.W., M.M. Triola, E.M. Borycki, B. Stein and J.L. Kannry. 2005. "Technology-Induced Error and Usability: The Relationship between Usability Problems and Prescription Error When Using a Handheld Application." International Journal of Medical Informatics 74: 519-26.
Levenson, N. 1995. Safeware: System Safety and Computers. Reading, MA: Addison-Wesley.
McDaniel, J.G. 1995. "Discrete-Event Simulation of a Wide-Area Healthcare Network." Journal of the American Medical Informatics Association 2(4): 220-37.
Miles, M.B. and A.M. Huberman. 1994. An Expanded Sourcebook: Qualitative Data Analysis (2nd ed.). Thousand Oaks, CA: Sage.
Musen, M.A., Y. Shahar and E.H. Shortliffe. 2006. "Clinical Decision Support Systems." In E.H. Shortliffe and J.J. Cimino, eds., Biomedical Informatics: Computer Applications in Health Care and Biomedicine. New York: Springer Verlag.
Orlikowski, W.J. 1992. "The Duality of Technology: Rethinking the Concept of Technology in Organizations." Organizational Science 3(3): 398-427.
Patton, R. 2001. Software Testing. Indianapolis, IN: Sams.
Preece, J., Y. Rogers, H. Sharp, D. Benyon, S. Holland and T. Carey. 1994. Human Computer Interaction. New York: Addison-Wesley.
Schmuland, C. 2005. "Value-Added Medical-Device Risk Management." IEEE Transactions on Device and Materials Reliability 5(3): 488-93.
Shortliffe, E.H. and J.J. Cimino. 2006. Biomedical Informatics: Computer Applications in Health care and Biomedicine. New York: Springer Verlag.
Sobolev, B.G., V. Sanchez, L. Kuramoto, A.R. Levy, M. Schechter and J.M. Fitzgerald. 2008. "Evaluation of Booking Systems for Elective Surgery Using Simulation Experiments." Healthcare Policy 3(4): 113-28.
Vardavas, R., R. Breban and S. Blower S. 2007. "Can Influenza Epidemics Be Prevented by Voluntary Vaccination?" PLoS Computers in Biology 3(5): e85.
Comments
Be the first to comment on this!
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed