A Proposed Benefits Evaluation Framework for Health Information Systems in Canada
This article describes a benefits evaluation framework for the health information systems currently being implemented across Canada through Canada Health Infoway with its jurisdictional partners and investment programs. This framework is based on the information systems success model by DeLone and McLean, the empirical analysis by van der Meijden on the use of this model in the health setting and our own review of evaluation studies and systematic review articles in health information systems. The current framework includes three dimensions of quality (system, information and service), two dimensions of system usage (use and user satisfaction) and three dimensions of net benefits (quality, access and productivity). Measures have been developed and work is under way to establish detailed evaluation plans and instruments for the individual investment programs to launch a series of benefits evaluation field studies across jurisdictions later this year.
Canada Health Infoway Inc. is an independent, not-for-profit organization whose members are Canada's 14 federal, provincial and territorial deputy ministers of health. Launched in 2001, Infoway and its public sector partners have over 200 projects, either completed or under way, delivering electronic health information system (HIS) solutions to Canadians. Infoway's mission is to foster and accelerate the development and adoption of HIS with compatible standards and communications technologies on a pan-Canadian basis, with tangible benefits to Canadians. The goal is to have an interoperable electronic health record in place for 50% of Canada's population by the end of 2009. Measure benefits and adjust is one of Infoway's business strategies. This is to be achieved with a robust benefits evaluation framework that allows Infoway to continually measure benefits achieved against those planned and make necessary adjustments.
The aim of the Infoway Benefits Evaluation Framework is to provide a high-level evidence-based model to guide subsequent field evaluation by the respective jurisdictions and investment programs. Individual Infoway investment programs (diagnostic imaging, laboratory information systems, drug information systems, telehealth, public health surveillance and interoperable electronic health record) currently have selected evaluation measures and will have designs and instruments that are specific and appropriate for their particular programs and settings.
The design of the framework, and the evaluation projects that flow from it, should allow us to evaluate the benefits of Infoway investments in terms of healthcare quality, productivity and access. To assess whether the conditions for benefits realization are met, the framework identifies key functionality and uses metrics, which will be tracked as an assessment tool to flag issues.
The Infoway Benefits Evaluation Framework draws on earlier work by DeLone and McLean (2003) in examining the measures for success of information systems (IS) in business settings, the review by van der Meijden et al. (2003) on determinants of success of in-patient clinical information systems (CIS) based on the IS success model and a literature scan on health information system (HIS) evaluation studies by Infoway. The conceptual foundations for our framework are briefly described below.
IS Success Model
The original DeLone and McLean IS success model published in 1992 was synthesized from 180 conceptual and empirical IS studies in different settings. In 2003, DeLone and McLean produced an updated version of their IS model, based on findings from 285 referred journal papers and proceedings published in 1992-2002 that examined, validated or cited the model. In the revised model, there are six conceptual dimensions of IS success. Three are quality dimensions in terms of the information, system and service involved. Information quality includes such metrics as completeness, ease of understanding and relevance. System quality includes adaptability, availability and response time. Service quality includes assurance and responsiveness. The quality of the information, system and service can affect the extent of system use, intention to use and user satisfaction. In turn, system usage and satisfaction can lead to positive and negative impacts at the individual and organizational levels, which are collectively viewed as net benefits.
CIS Success Model
In 2003, van der Meijden et al. published a literature review on the determinants of success for in-patient CIS that required data entry by healthcare professionals. The review adopted the 1992 version of the DeLone and McLean IS success model to test its ability to categorize the success factors under the original dimensions of information and system quality, system usage and user satisfaction, and individual and organizational impacts. The review included 33 relevant studies describing 29 CIS and identified 50 different attributes as success measures under the six dimensions. In addition, van der Meijden et al. found three types of contingent factors that could not be assigned to the six dimensions. The contingent factors were system development, implementation process, and culture and characteristics of the organization. These findings are consistent with the information technology (IT) interaction model introduced by Silver et al. in 1995 but still taught in business schools today. The IT interaction model emphasizes the importance of the implementation process, the organization and the external environment on the type of IS deployed.
Review of HIS Systematic Reviews
An extensive review of systematic review articles on the evaluation of HIS including telehealth and telemedicine applications is being conducted by one of the authors (FL). To date, 50 HIS-related systematic review articles have been identified and are being reviewed. Twenty-seven of these reviews are on different types of health-related information systems, tools and resources used by clinicians and patients in hospital, clinic and home settings. The other 23 articles are systematic reviews on the evaluation of telehealth and telemedicine applications in urban and rural settings. Thus far, we have found 14 systematic reviews of clinical information systems or tools (Ammenwerth and de Keizer 2004; Balas et al. 1996, 1998; Cramer et al. 2003; Delpierre et al. 2004; Garg et al. 2005; Kaushal et al. 2003; Kawamoto et al. 2005; Mitchell and Sullivan 2001; Montgomery and Fahey 1998, Sullivan and Mitchell 1995; van der Loo et al. 1995; van der Meijden et al. 2003; Walton et al. 1999) and seven telehealth and telemedicine evaluation studies (Hailey and Crowe 2003; Hailey et al. 2004; Hayward 2005; Jaatinen et al. 2002; Jennett et al. 2003; National Initiative for Telehealth 2003; Scott et al. 2003) to be particularly relevant and useful. For the six dimensions in the revised DeLone and McLean IS success model, we were able to define 20 categories of measures as success factors and approximately 60 empirical measures based on the attributes described in these 20 systematic review articles.
The Infoway Benefits Evaluation Framework
The current Infoway Benefits Evaluation Framework has adopted all six dimensions of the 2003 DeLone and McLean model for system, information and service quality; system use and user satisfaction; and net benefits (Figure 1). In each dimension, we have defined two or more categories of measures based on the success factors from the van der Meijden et al. (2003) model. These categories were confirmed with the quantifiable benefits from the Infoway literature scan and the selected measures identified from our review of HIS systematic reviews. These categories of measures are defined below and detailed in Table 1.
Three categories of system quality measures are considered - functionality, performance and security. Functionality covers the type of HIS features and level of decision support available. HIS features are the operational capabilities of the HIS, such as patient registration, order entry, results reporting, online query and integration and linkage with other systems. The level of support is the decision support capability embedded in the HIS. Six levels of decision support are suggested: report or view covers basic operating features only with no decision support; reference covers access to evidence-based or library information resources as defined in the Hayward (2005) clinical decision support systems review; reminder, alert, assist and guide are from Randolph et al.'s (2000) article on critical appraisal of clinical decision support systems as part of the Journal of the American Medical Association's "Users' Guide to Medical Literature." Performance refers to technical functioning of the HIS in terms of accessibility, reliability and response time. Accessibility is the availability of the system to local and remote settings. Reliability is the dependability of the system operation, including scheduled and unscheduled downtime. Response time refers to the length of time it takes the computer to reply to a user action such as data entry or a query. Security covers the technical capability of the system to protect the data being recorded, stored and accessed for subsequent use.
Two categories of information quality are being considered - content and availability. Content covers the completeness, accuracy, relevance and comprehension of the information. Availability covers the timeliness of the information, when and where needed, as well as the reliability and consistency of this information over time. Note that issues related to the privacy and confidentiality of patient information are being addressed in another Infoway program on privacy legislations and are thus not covered here.
The only category of measure being considered for service quality is responsiveness, which focuses on post-implementation user training, ongoing technical support and availability of such support. Note that the revised DeLone and McLean IS success model has assurance and empathy as two other measures of service quality, which are not covered here for pragmatic reasons as they are highly subjective in nature.
Three system usage categories are being considered - usage behaviour and pattern, self-reported use and intention to use. Usage behaviour and pattern covers the frequency, duration, location, type or nature and flexibility of actual usage that took place, while self-reported use covers the same subcategories but as they are perceived by the users. Type or nature and flexibility of use refers to the purpose of the session based on actual or perceived usage by the user. Intention to use is the proportion of and factors for current non-users of an implemented HIS to become users of that system.
There are three categories of measures under satisfaction - competency, user satisfaction and ease of use. Competency covers the knowledge, skills and experience of the users in the HIS. User satisfaction covers the subjective opinions of users in terms of their perceived expectations, information/system/service quality and use of the system. Ease of use refers to the user-friendliness of the HIS and its learnability, or how easy it is to learn to use the system.
Assessment of Quality and Use
The dimensions of system, information and service quality, and system use and user satisfaction will be assessed via indicators based on previous studies found in the literature reviews. Indicators that measure performance and reliability based on response times for standardized tasks, integration with workflow, user-friendliness and security will be used for the system quality component of the framework. Indicators that account for completeness and accuracy of information as well as timeliness and relevance of information will be used to describe the information quality component of the framework. The service quality component of the framework will be estimated through indicators of training and support.
The use component of the framework will be based on indicators that describe the frequency of system use by both the provider and the patient as well as intention to increase use. The framework's user satisfaction component will use indicators of perceived usefulness and value of the system, perceived impact on productivity and integration with workflow and perceived impact on quality of care.
Data for these indicators will be collected primarily through surveys of users.
There are three categories of net benefits based on previously established criteria from the Infoway business plan - quality, access and productivity. Within this framework quality is defined in terms of appropriateness and effectiveness, patient safety and health outcomes. The appropriateness aspect covers the adherence and compliance to benchmark, policy or practice standards, self-reported practice or practice captured in system. The effectiveness aspect covers the continuity of care that spans information, relational and management continuity, as related to individuals or local and dispersed teams and referral of services. Health outcomes covers both short-term clinical outcomes of patients and longer-term change in health status attributable to HIS interventions. Patient safety refers to adverse events, prevention, surveillance and risk management.
Access covers the availability of services, ability to access services (timeliness and economics) and extent of patient or caregiver participation in care provision.
Productivity covers efficiency, care coordination and net cost. Efficiency refers to resource use, output and care continuity improvement, and health systems management capability. Care coordination refers to care provision by teams and continuity of care across the continuum. Finally, net cost covers monetary avoidance, reduction and saving.
Assessment of Net Benefits
The quality category of the framework will be characterized by indicators that describe patient safety through data collected regarding drug interactions and therapy duplications, completeness and legibility of prescriptions, adverse drug events and provider response to alerts; changes in health system outcomes through data on hospital readmission rates, efficiency of recovery, patient transfers and chronic disease management; and changes in provider effectiveness and appropriateness of care through data regarding timeliness of service delivery and access to test results, vaccination rates and outbreak detection and intervention.
The access category of the framework will be described through indicators of change in access to services based on data collected regarding volume of service provision, access to previously unavailable services and timeliness of services. Indicators of change in patient and caregiver participation, based on data regarding patient awareness and adherence, will also be used to describe the access category.
The productivity category of the framework will be accounted for via indicators of change in provider efficiency through data regarding radiology technologist and radiologist efficiency, pharmacists and laboratory technician callbacks, time to take a medication history or assess a patient and clinician workflow; change in coordination of care through data regarding information sharing among different providers and management of outbreaks; and change in net costs through data regarding unnecessary duplicate tests and vaccinations.
These indicators will be based on data collected through chart reviews, system log reports, surveys of users, time-motion studies, qualitative interviews, focus groups, case studies and administrative data.
The advantage of adhering to a common frame of reference such as the Infoway Benefits Evaluation Framework is that it ensures consistency in measurement and meaning across field evaluation studies. It also allows cross-jurisdiction and cross-program comparison of evaluation findings from individual studies. The framework is considered rigorous as it was derived from an evidence base of published systematic reviews of HIS literature. The framework has the potential to be expanded as a more comprehensive HIS success model for different health settings. Key organizational and context factors such as strategy, culture and business process are considered out of scope at the moment. The framework is also selective in the measures included, focusing only on the most significant metrics and those that are practical and measurable by Infoway and its partners. Expansion of this framework is not within the Infoway mandate; however, further evolution of this framework would be welcomed.
About the Author
Francis Lau, PhD, is Associate Professor, School of Health Information Science at the University of Victoria, Victoria, British Columbia and also a member of the expert advisory panel.
Simon Hagens, MBA, works for Canada Health Infoway as Manager, Benefits Strategy and Analysis in Toronto, Ontario.
Sarah Muttitt, MD, is Canada Health Infoway's Vice President of Innovation and Adoption based in Toronto, Ontario.
Corresponding author: Simon Hagens, Manager, Benefits Strategy & Analysis, Canada Health Infoway/Inforoute Santé du Canada, 150 King Street West, Suite 1300, Toronto ON M5H 1J9, email@example.com.
AcknowledgmentWe wish to thank David Bates, Nikki Shaw, Robyn Tamblyn, Richard Scott, Michael Wolfson, Anne McFarlane and Doreen Neville as well as expert advisory panel members Mark Nenadovic and Nancy Kraetschmer and Gavin Giles of Infoway, for their feedback on earlier versions of the framework. The help of Lindsay Forland and Morgan Price in collating the literature for the DeLone and McLean IS success model and van der Meijden's adaptation of the IS model is also acknowledged.
Ammenwerth, E. and N. de Keizer. 2004. "An Inventory of Evaluation Studies of Information Technology in Health Care." Methods of Information in Medicine 44: 44-56.
Balas, E.A.. S.M. Austin, J.A. Mitchell, B.G. Ewigman, K.D. Bopp and G.D. Brown. 1996. "The Clinical Value of Computerized Information Services." Archives of Family Medicine 5: 271-8.
Balas, E.A., S.A. Boren and G. Griffing. 1998. "Computerized Management of Diabetes: A Synthesis of Controlled Trials." Proceedings of the 1998 AMIA Annual Symposium. Bethesda, MD: American Medical Informatics Association.
Cramer, K., L. Hartling, N. Wiebe K., Russell, E. Crumley, M. Pusic and T.P. Klassen. 2003. Computer-Based Delivery of Health Evidence - Final Report, Part I. Edmonton: Alberta Heritage Foundation for Medical Research.
DeLone, W.H. and E.R. McLean. 2003. "The DeLone and McLean Model of Information Systems Success: A Ten-Year Update." Journal of Management Information Systems 19(4): 9-30.
Delpierre, C., L. Cuzin, J. Fillaux, M. Alvarez, P. Massip and T. Lang. 2004. "A Systematic Review of Computer-Based Patient Record Systems and Quality of Care: More Randomized Clinical Trials or a Broader Approach?" International Journal for Quality in Health Care 15(5): 407-16.
Garg, A.X., N.K. Adhikari, H. McDonald, M.P. Rosas-Arellano, P.J. Devereaux, J. Beyene, J. Sam and R.B. Haynes. 2005. "Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review." Journal of the American Medical Association 293(10): 1223-38.
Hailey, D. and B. Crowe. 2003. "A Profile of Success and Failure in Telehealth - Evidence and Opinion from the Successes and Failures in Telehealth Conferences." Journal of Telemedicine and Telecare 9(Suppl. 2): 22-4.
Hailey, D., A. Ohinmaa and R. Roine. 2004. "Study Quality and Evidence of Benefit in Recent Assessments of Telemedicine." Journal of Telemedicine and Telecare 10(6): 318-24.
Hayward, R. 2005. "Clinical Decision Support Tools - Do They Support Clinicians?" Presentation at the University of Victoria health informatics seminar series.
Jaatinen, P.T., J. Forsstrom and P. Loula. 2002. "Teleconsultations: Who Uses Them and How?" Journal of Telemedicine and Telecare 8(6): 319-24.
Jennett, P., L.A. Hall, D. Hailey, A. Ohinmaa, C. Anderson, R. Thomas, B. Young, D. Lorenzetti and R.E. Scott. 2003. "The Socio-economic Impact of Telehealth: A Systematic Review." Journal of Telemedicine and Telecare 9(6): 311-20.
Kaushal, R., K.G. Shojania and D.W. Bates. 2003. "Effects of Computerized Physician Order Entry and Clinical Decision Support Systems on Medication Safety." Archives of Internal Medicine 163: 1409-16.
Kawamoto, K., C.A. Houlihan, E.A. Balas and D.F. Lobach. 2005. "Improving Clinical Practice Using Clinical Decision Support Systems: A Systematic Review of Trials to Identify Features Critical to Success." British Medical Journal 330: 765.
Mitchell, E. and F. Sullivan. 2001. "A Descriptive Feast but an Evaluative Famine: Systematic Review of Published Articles on Primary Care Computing during 1980-97." British Medical Journal 322: 279-82.
Montgomery, A.A. and T. Fahey. 1998. "A Systematic Review of the Use of Computers in the Management of Hypertension." Journal of Epidemiology and Community Health 52: 520-5.
National Initiative for Telehealth. 2003. NIFTE Framework of Guidelines. Ottawa: Author.
Randolph, A.G., B.R. Haynes, J.C. Wyatt, D.J. Cook and G.H. Guyatt. 2000. "How to Use an Article Evaluating the Clinical Impact of a Computer-Based Clinical Decision Support System." Journal of the American Medical Association 28(1): 67-74.
Scott, R.E., F. McCarthy, P. Jennett, T. Perverseff, M.F. Palacios and B. Rush. 2003. National Telehealth Outcome Indicators Project. A Canadian Consensus Approach to Identification and Definition of Outcome Indicators for Evaluation of Telehealth. Health Telematics Unit, University of Calgary: Calgary.
Silver, M.S., M.L. Markus and C.M. Beath. 1995. "The Information Technology Interaction Model: A Foundation for the MBA Core Course." Management Information Systems Quarterly19(3): 361-87.
Sullivan, F. and E. Mitchell. 1995. "Has General Practitioner Computing Made a Difference to Patient Care?" British Medical Journal 311: 848-52.
van der Loo, R.P., E.M.S.J. Gennip, A.R. Bakker, A. Hasman and F.F.H. Rutten. 1995. "Evaluation of Automated Information Systems in Healthcare: An Approach to Classify Evaluative Studies." Computer Methods and Programs in Biomedicine 48: 45-52.
van der Meijden, M.J., H.J. Tange, J. Troost and A. Hasman. 2003. "Determinants of Success of Inpatient Clinical Information Systems: A Literature Review." Journal of the American Medical Informatics Association 10(3): 235-43.
Walton, R., S. Dovey, E. Harvey and N. Freemantle. 1999. "Computer Support for Determining Drug Dose: Systematic Review and Meta Analysis." British Medical Journal 318: 984-90.
Andrea McLellan wrote:
Posted 2015/05/05 at 09:25 AM EDT
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed