This article describes a benefits evaluation framework for the health information systems currently being implemented across Canada through Canada Health Infoway with its jurisdictional partners and investment programs. This framework is based on the information systems success model by DeLone and McLean, the empirical analysis by van der Meijden on the use of this model in the health setting and our own review of evaluation studies and systematic review articles in health information systems. The current framework includes three dimensions of quality (system, information and service), two dimensions of system usage (use and user satisfaction) and three dimensions of net benefits (quality, access and productivity). Measures have been developed and work is under way to establish detailed evaluation plans and instruments for the individual investment programs to launch a series of benefits evaluation field studies across jurisdictions later this year.

Canada Health Infoway Inc. is an independent, not-for-profit organization whose members are Canada's 14 federal, provincial and territorial deputy ministers of health. Launched in 2001, Infoway and its public sector partners have over 200 projects, either completed or under way, delivering electronic health information system (HIS) solutions to Canadians. Infoway's mission is to foster and accelerate the development and adoption of HIS with compatible standards and communications technologies on a pan-Canadian basis, with tangible benefits to Canadians. The goal is to have an interoperable electronic health record in place for 50% of Canada's population by the end of 2009. Measure benefits and adjust is one of Infoway's business strategies. This is to be achieved with a robust benefits evaluation framework that allows Infoway to continually measure benefits achieved against those planned and make necessary adjustments.  

The aim of the Infoway Benefits Evaluation Framework is to provide a high-level evidence-based model to guide subsequent field evaluation by the respective jurisdictions and investment programs. Individual Infoway investment programs (diagnostic imaging, laboratory information systems, drug information systems, telehealth, public health surveillance and interoperable electronic health record) currently have selected evaluation measures and will have designs and instruments that are specific and appropriate for their particular programs and settings.

The design of the framework, and the evaluation projects that flow from it, should allow us to evaluate the benefits of Infoway investments in terms of healthcare quality, productivity and access. To assess whether the conditions for benefits realization are met, the framework identifies key functionality and uses metrics, which will be tracked as an assessment tool to flag issues.

Conceptual Foundations

The Infoway Benefits Evaluation Framework draws on earlier work by DeLone and McLean (2003) in examining the measures for success of information systems (IS) in business settings, the review by van der Meijden et al. (2003) on determinants of success of in-patient clinical information systems (CIS) based on the IS success model and a literature scan on health information system (HIS) evaluation studies by Infoway. The conceptual foundations for our framework are briefly described below.

IS Success Model

The original DeLone and McLean IS success model published in 1992 was synthesized from 180 conceptual and empirical IS studies in different settings. In 2003, DeLone and McLean produced an updated version of their IS model, based on findings from 285 referred journal papers and proceedings published in 1992-2002 that examined, validated or cited the model. In the revised model, there are six conceptual dimensions of IS success. Three are quality dimensions in terms of the information, system and service involved. Information quality includes such metrics as completeness, ease of understanding and relevance. System quality includes adaptability, availability and response time. Service quality includes assurance and responsiveness. The quality of the information, system and service can affect the extent of system use, intention to use and user satisfaction. In turn, system usage and satisfaction can lead to positive and negative impacts at the individual and organizational levels, which are collectively viewed as net benefits.

CIS Success Model

In 2003, van der Meijden et al. published a literature review on the determinants of success for in-patient CIS that required data entry by healthcare professionals. The review adopted the 1992 version of the DeLone and McLean IS success model to test its ability to categorize the success factors under the original dimensions of information and system quality, system usage and user satisfaction, and individual and organizational impacts. The review included 33 relevant studies describing 29 CIS and identified 50 different attributes as success measures under the six dimensions. In addition, van der Meijden et al. found three types of contingent factors that could not be assigned to the six dimensions. The contingent factors were system development, implementation process, and culture and characteristics of the organization. These findings are consistent with the information technology (IT) interaction model introduced by Silver et al. in 1995 but still taught in business schools today. The IT interaction model emphasizes the importance of the implementation process, the organization and the external environment on the type of IS deployed.

Review of HIS Systematic Reviews

An extensive review of systematic review articles on the evaluation of HIS including telehealth and telemedicine applications is being conducted by one of the authors (FL). To date, 50 HIS-related systematic review articles have been identified and are being reviewed. Twenty-seven of these reviews are on different types of health-related information systems, tools and resources used by clinicians and patients in hospital, clinic and home settings. The other 23 articles are systematic reviews on the evaluation of telehealth and telemedicine applications in urban and rural settings. Thus far, we have found 14 systematic reviews of clinical information systems or tools (Ammenwerth and de Keizer 2004; Balas et al. 1996, 1998; Cramer et al. 2003; Delpierre et al. 2004; Garg et al. 2005; Kaushal et al. 2003; Kawamoto et al. 2005; Mitchell and Sullivan 2001; Montgomery and Fahey 1998, Sullivan and Mitchell 1995; van der Loo et al. 1995; van der Meijden et al. 2003; Walton et al. 1999) and seven telehealth and telemedicine evaluation studies (Hailey and Crowe 2003; Hailey et al. 2004; Hayward 2005; Jaatinen et al. 2002; Jennett et al. 2003; National Initiative for Telehealth 2003; Scott et al. 2003) to be particularly relevant and useful. For the six dimensions in the revised DeLone and McLean IS success model, we were able to define 20 categories of measures as success factors and approximately 60 empirical measures based on the attributes described in these 20 systematic review articles.

[Figure 1]

The Infoway Benefits Evaluation Framework

The current Infoway Benefits Evaluation Framework has adopted all six dimensions of the 2003 DeLone and McLean model for system, information and service quality; system use and user satisfaction; and net benefits (Figure 1). In each dimension, we have defined two or more categories of measures based on the success factors from the van der Meijden et al. (2003) model. These categories were confirmed with the quantifiable benefits from the Infoway literature scan and the selected measures identified from our review of HIS systematic reviews. These categories of measures are defined below and detailed in Table 1.

[Table 1]

System Quality

Three categories of system quality measures are considered - functionality, performance and security. Functionality covers the type of HIS features and level of decision support available. HIS features are the operational capabilities of the HIS, such as patient registration, order entry, results reporting, online query and integration and linkage with other systems. The level of support is the decision support capability embedded in the HIS. Six levels of decision support are suggested: report or view covers basic operating features only with no decision support; reference covers access to evidence-based or library information resources as defined in the Hayward (2005) clinical decision support systems review; reminder, alert, assist and guide are from Randolph et al.'s (2000) article on critical appraisal of clinical decision support systems as part of the Journal of the American Medical Association's "Users' Guide to Medical Literature." Performance refers to technical functioning of the HIS in terms of accessibility, reliability and response time. Accessibility is the availability of the system to local and remote settings. Reliability is the dependability of the system operation, including scheduled and unscheduled downtime. Response time refers to the length of time it takes the computer to reply to a user action such as data entry or a query. Security covers the technical capability of the system to protect the data being recorded, stored and accessed for subsequent use.

Information Quality

Two categories of information quality are being considered - content and availability. Content covers the completeness, accuracy, relevance and comprehension of the information. Availability covers the timeliness of the information, when and where needed, as well as the reliability and consistency of this information over time. Note that issues related to the privacy and confidentiality of patient information are being addressed in another Infoway program on privacy legislations and are thus not covered here.

Service Quality

The only category of measure being considered for service quality is responsiveness, which focuses on post-implementation user training, ongoing technical support and availability of such support. Note that the revised DeLone and McLean IS success model has assurance and empathy as two other measures of service quality, which are not covered here for pragmatic reasons as they are highly subjective in nature.

System Usage

Three system usage categories are being considered - usage behaviour and pattern, self-reported use and intention to use. Usage behaviour and pattern covers the frequency, duration, location, type or nature and flexibility of actual usage that took place, while self-reported use covers the same subcategories but as they are perceived by the users. Type or nature and flexibility of use refers to the purpose of the session based on actual or perceived usage by the user. Intention to use is the proportion of and factors for current non-users of an implemented HIS to become users of that system.

User Satisfaction

There are three categories of measures under satisfaction - competency, user satisfaction and ease of use. Competency covers the knowledge, skills and experience of the users in the HIS. User satisfaction covers the subjective opinions of users in terms of their perceived expectations, information/system/service quality and use of the system. Ease of use refers to the user-friendliness of the HIS and its learnability, or how easy it is to learn to use the system.

Assessment of Quality and Use  

The dimensions of system, information and service quality, and system use and user satisfaction will be assessed via indicators based on previous studies found in the literature reviews. Indicators that measure performance and reliability based on response times for standardized tasks, integration with workflow, user-friendliness and security will be used for the system quality component of the framework. Indicators that account for completeness and accuracy of information as well as timeliness and relevance of information will be used to describe the information quality component of the framework. The service quality component of the framework will be estimated through indicators of training and support.

The use component of the framework will be based on indicators that describe the frequency of system use by both the provider and the patient as well as intention to increase use. The framework's user satisfaction component will use indicators of perceived usefulness and value of the system, perceived impact on productivity and integration with workflow and perceived impact on quality of care.

Data for these indicators will be collected primarily through   surveys of users.

Net Benefits

There are three categories of net benefits based on previously established criteria from the Infoway business plan - quality, access and productivity. Within this framework quality is defined in terms of appropriateness and effectiveness, patient safety and health outcomes. The appropriateness aspect covers the adherence and compliance to benchmark, policy or practice standards, self-reported practice or practice captured in system. The effectiveness aspect covers the continuity of care that spans information, relational and management continuity, as related to individuals or local and dispersed teams and referral of services. Health outcomes covers both short-term clinical outcomes of patients and longer-term change in health status attributable to HIS interventions. Patient safety refers to adverse events, prevention, surveillance and risk management.  

Access covers the availability of services, ability to access services (timeliness and economics) and extent of patient or caregiver participation in care provision.  

Productivity covers efficiency, care coordination and net cost. Efficiency refers to resource use, output and care continuity improvement, and health systems management capability. Care coordination refers to care provision by teams and continuity of care across the continuum. Finally, net cost covers monetary avoidance, reduction and saving.

Assessment of Net Benefits

The quality category of the framework will be characterized by indicators that describe patient safety through data collected regarding drug interactions and therapy duplications, completeness and legibility of prescriptions, adverse drug events and provider response to alerts; changes in health system outcomes through data on hospital readmission rates, efficiency of recovery, patient transfers and chronic disease management; and changes in provider effectiveness and appropriateness of care through data regarding timeliness of service delivery and access to test results, vaccination rates and outbreak detection and intervention.

The access category of the framework will be described through indicators of change in access to services based on data collected regarding volume of service provision, access to previously unavailable services and timeliness of services. Indicators of change in patient and caregiver participation, based on data regarding patient awareness and adherence, will also be used to describe the access category.

The productivity category of the framework will be accounted for via indicators of change in provider efficiency through data regarding radiology technologist and radiologist efficiency, pharmacists and laboratory technician callbacks, time to take a medication history or assess a patient and clinician workflow; change in coordination of care through data regarding information sharing among different providers and management of outbreaks; and change in net costs through data regarding unnecessary duplicate tests and vaccinations.

These indicators will be based on data collected through chart reviews, system log reports, surveys of users, time-motion studies, qualitative interviews, focus groups, case studies and administrative data.

Next Steps

The advantage of adhering to a common frame of reference such as the Infoway Benefits Evaluation Framework is that it ensures consistency in measurement and meaning across field evaluation studies. It also allows cross-jurisdiction and cross-program comparison of evaluation findings from individual studies. The framework is considered rigorous as it was derived from an evidence base of published systematic reviews of HIS literature. The framework has the potential to be expanded as a more comprehensive HIS success model for different health settings. Key organizational and context factors such as strategy, culture and business process are considered out of scope at the moment. The framework is also selective in the measures included, focusing only on the most significant metrics and those that are practical and measurable by Infoway and its partners. Expansion of this framework is not within the Infoway mandate; however, further evolution of this framework would be welcomed.

Comments

Andrea McLellan wrote:

Posted 2015/05/05 at 09:25 AM EDT