Introduction

Building Ontario's Wait Time Information System (WTIS) was one of the largest and most complex technology projects Cancer Care Ontario (CCO) had ever taken on. Increasing public concern about wait times and the lack of adequate tools to provide a clear or accurate picture of provincial wait times had led to a sense of urgency for the province to report wait time data. While healthcare providers and the Ministry of Health and Long-term Care (MOHLTC) sought to address timely access to care, the challenges to develop a suitable information management/information technology (IM/IT) solution within aggressive timelines were significant. For the WTIS project, success was defined by the ability to deliver a tool to capture wait time data that addressed business and clinical needs and by providing individuals with the ability to use the tool and its data to improve access to care.

Specifically, the WTIS was developed for use by hospitals and clinicians - already facing significant change and with little capacity to spare - working to support MOHLTC's broader transformation agenda and adjusting to the introduction of Local Health Integration Networks (LHINs). The WTIS had to integrate with numerous technically diverse hospital systems while supporting a province-wide e-Health strategy still in its infancy.

This article examines the approach that the project used to drive, develop and deliver an IM/IT solution that met business and clinical needs, and shares lessons learned in delivering a province-wide technology application.

Use the Business Imperative to Drive Technology

For an IM/IT solution to be effective, it must address a clearly identified business/clinical problem. The goal of the Wait Time Strategy was to reduce wait times and improve access to care. The wait time problem was largely seen as an information problem, and the WTIS was built to capture and report on wait time data. With that goal in mind, the project set out to develop a clear picture of how to effectively measure wait times and identify what information would best drive performance management.

With short time frames to develop and deploy the WTIS, requirements gathering had to be swift and thorough. Recognizing the value and importance of user insight, Clinical Expert Panels (CEPs) and hospital stakeholders were consulted early in the process, allowing the WTIS project team to better understand the clinical landscape and business requirements.

The WTIS project team held ongoing stakeholder sessions to collect and prioritize the business/clinical requirements. With stakeholder input, the project team then determined the technical and functional requirements that shaped the development scope. Through these sessions, the team identified that the WTIS would need to provide wait list management information at the clinician, service type and hospital levels. By answering the following questions, the WTIS would allow informed decision-making:

• Who is waiting for which surgeon for which procedure?
• How will urgency be determined?
• How many people are waiting?
• How long have people been waiting?
• How long have people been waiting compared to a target time?

The WTIS project team also worked with its partners in MOHLTC, the Institute for Clinical Evaluative Studies (ICES), the Canadian Institute for Health Information (CIHI) and the Ontario Hospital Association (OHA) to understand their data needs and ensure that the system would appropriately address data standardization and quality requirements.

Through this consultative process the project team was able to establish the requirements that would guide development of the WTIS. The team was able to identify the information and processes needed to measure wait times, as well as the specific technology to prioritize patients, and describe the key business processes and workflow scenarios to drive the system design process.

Lay the Foundation for Technology

With many priorities competing for attention and limited resources (people, time, equipment and funds) it was critical that hospitals and clinicians felt incented to deploy and begin using the WTIS. The following were essential to the project's foundation.

Make the Case for One System

Through Hospital Accountability Agreements, funding was married to wait time data collection and provided the impetus for hospitals to deploy the WTIS and work to ensure clinicians were all using one system to capture wait time data. To receive the initial incremental funding, hospitals were required to meet conditions of funding that included performing additional processes, meeting targets and collecting, reporting and using wait time data to govern their organization's access management strategy.

Standardize Data and Data Collection

A key challenge for the WTIS project team was the lack of national or provincial standards to define acceptable wait times. As a result, hospitals and clinicians were limited in their ability to use technology to effectively monitor and manage performance. Clinicians and hospital stakeholders, represented by Clinical Expert Panels, agreed that this information was critical to their practices and to addressing the wait time issue. The promise of wait time definitions, and a consistent standardized manner to collect data electronically and in near real-time, provided further incentive and established the foundation for the WTIS.

Demonstrate the Value of Data Early

While work was underway to develop the WTIS, the MOHLTC was committed to begin public reporting as soon as possible. As a result, in April 2005, CCO led an initiative to develop and deliver an interim upload tool for wait time data collection using its existing IT infrastructure. Based on their readiness and willingness to begin capturing data monthly, five hospitals were chosen as Beta sites to begin using this basic reporting system. Eventually, all wait time-funded hospitals would use the tool to capture their wait time data until the WTIS was developed and deployed.

To use the interim upload tool, dedicated resources, time commitment and training would be required. The project took a calculated risk: if hospitals found the process cumbersome or the extra work an imposition, support for the WTIS could be eroded. However, the initial data collection actually helped "prime the pump." Hospitals quickly began using the data and saw its potential to inform process, policy, priorities and wait list management. It was a turning point that helped smooth the way for future acceptance of the WTIS.

Establish Checkpoints to Ensure Data Quality

The interim upload tool also provided valuable insights about data integrity and the process of data capture in hospitals, helping shape the future requirements, development and evolution of the WTIS. Questions were raised about data quality checks and validation to ensure complete and accurate data, and decisions had to be made about how to best analyze outliers, deal with trending across time and address cases with extremely long wait times. This was also the beginning of a rigorous data quality process that would include the creation of a Data Certification Council to provide third-party validation of data quality. With such powerful information being reported publicly, the stakes were high in ensuring the accuracy of the data.

Secure e-Health Expertise

Given the importance and aggressive timelines of the project, CCO assembled best-in-class vendors who worked alongside CCO resources to create a team with broad expertise and knowledge. CCO's experience in complex technical healthcare initiatives and an understanding of hospital and clinician communities was critical to enable delivery of a technical solution within aggressive timelines. Because a key strategy was to align the WTIS with other relevant provincial e-Health strategies, e-Health expertise was extremely important.

The project resourcing strategy aligned vendor resources to work closely with the CCO technical and business leads and ensured that both CCO's technical expertise and the knowledge gained during the system design and development was transferred and shared. Rather than operating "virtually," the team was co-located at CCO's premises. A collaborative "one team" approach rapidly emerged, allowing CCO and vendors to work closely on requirements and have both the information and authority to make decisions quickly. The one team approach is further discussed in "Taking it to the Streets: Delivering on Deployment".

Manage and Mitigate Privacy and Security Concerns

From the outset, the team had to address issues of privacy and security relating to the collection, use and disclosure of patient health information. As a recognized and prescribed entity under Ontario's Personal Health Information Protection Act (PHIPA), CCO was known within the healthcare sector for its privacy program, experience and approach. A security-threat risk assessment was also completed. The project leveraged CCO's existing secure hosting environment to reduce the number of security issues that might arise.

A WTIS privacy impact assessment (PIA) was completed early on in the project. Recommendations were implemented to develop a comprehensive strategy to address and mitigate privacy concerns. CCO liaised with the Information and Privacy Commissioner/Ontario to ensure that the PIA results and the privacy governance structure put in place to protect the personal health information that would be collected, used and disclosed via the WTIS were aligned with provincial privacy best practices.

CCO's approach required all third-party vendors working on its behalf to act as "privacy" agents of CCO in their dealings with hospital stakeholders. Vendors undertook responsibility for understanding and complying with CCO's privacy requirements. Many privacy concerns were overcome through deployment activities such as teleconferences, ongoing communication and training materials the project team provided to support hospitals in bringing clinicians on board. Most importantly, privacy issues were not seen as obstacles, but rather as enablers to the proper sharing of information between systems and organizations.

Build the Systems

The following considerations were essential in building the WTIS and Ontario's Client Registry/EMPI.

Leverage Existing Knowledge and Expertise

The team had to decide what systems model to adopt for the WTIS. In defining project scope and, ultimately, product definition, two possible options were considered:

1. Customize an existing "Wait Times" application to meet the requirements of the WTIS, or
2. Deliver a custom-built solution.

At the time, very few province-wide systems were in operation to leverage or learn from. In Ontario, the team learned from the work of the Cardiac Care Network (CCN). However, the CCN model was specific to cardiac care, whereas the objective for the WTIS was to eventually capture all surgical procedures performed in Ontario.

Nationally, the team looked to other jurisdictions to examine other provincial wait time efforts from the perspective of standardization, implementation, governance and management. One leading example was found in Saskatchewan with the Saskatchewan Surgical Care Network (SSCN). While the project team discovered insights through examining this initiative, key distinguishing requirements such as clinician connectivity, user registration, reporting and integration indicated that Ontario's WTIS required a custom-built solution to deliver a sustainable system and meet the unique requirements of Ontario's healthcare stakeholders.

Clearly Define Scope and Business Requirements

Ongoing stakeholder consultations defined the scope and requirements for core WTIS functionality. Initially, a key metric, the "Wait 2" period - the time between the decision to treat and actual provision of the treatment - would be tracked, monitored and reported. Based on this experience, the intent was to deliver a solution with the capacity to eventually capture the full spectrum of a wait.

It was important to engineer a system that allowed for straightforward adaptability by hospital platforms and workflows. The main development priorities were to build a system that (1) focused only on collecting the data needed to capture the "Wait 2" metric, (2) would work with hospitals' existing technical infrastructure, (3) stakeholders would find relatively easy to use, and (4) appropriately addressed privacy and security issues. The system had to remain adaptable, align and use HL7 (international standards used to exchange, integrate, share and retrieve electronic health information known as Health Level Seven) messaging standards and be built in a manner that would provide integration capability and scalability with future needs and investment.

Scalability was also important to ensure the system could evolve to handle a larger load as additional clinicians used the system and to meet changing business requirements, such as adding new procedures and eventually encompassing all aspects of the wait.

Once the requirements were developed, they were rigorously tested against defined evaluation criteria. These included hospitals being able to successfully send HL7 messages to the WTIS and ensure that hospital procedures mapped to WTIS procedures. In addition, hospitals had to review and/or adjust their business processes and workflows to ensure the application design met stated objectives and allowed end users to successfully open and close wait list entries.

Seek Opportunities for Agility

In building such a complex and large-scale system, the WTIS project team "expected the unexpected" and proactively looked for opportunities to increase their agility. A few factors helped ensure the project team built capacity and could rapidly and effectively respond to IT changes and decisions throughout the WTIS project:

1. 1. User registration - In addition to the registration that occurred within CCO, external new users were required to register with the WTIS separately. In response, a user registration and management process was developed, introducing local registration authorities. The new process easily registered hundreds of users at a time, a much more efficient undertaking for both hospitals and CCO.
   
   
2. Hosting - Each instance of the WTIS required a larger number of servers. A virtual server strategy was undertaken to allow rapid buildup and deployment; it also enabled extensive quality assurance (QA, focused on data, user acceptance and development).
   
   
3. An iterative approach - The WTIS project employed multiple builds of applications, as opposed to single large builds. The decision to use a phased implementation approach provided the project team with clearly stated objectives for each phase. This demanded focused and concise project management and an iterative software development approach. Each phase, through a better understanding of enhancements requested by users, delivered enhanced functionality that refined the system (Figure 1).

Figure 1. Timeline of the evolution in WTIS functionality
Timeline	Number of Hospitals	Functionality
Phase I/Beta - March 2006	5	Core WTIS functionality available to collect data for cancer surgery, cardiac procedures, cataract surgery, hip and knee replacements surgery and CT and MRI scans One-way integration with Ontario's Client Registry/EMPI
Phase II - November 2006	53	WTIS functionality available to additional hospitals Bi-directional integration between WTIS Client Registry/EMPI
Phase III - June 2007	81	WTIS functionality available to all wait time-funded hospitals
WTIS Expansion 2007/08 - March 2008	82	System expanded to collect data for all general, ophthalmic and orthopaedic surgical procedures Pilot functionality to capture Wait 1 data Pilot functionality to capture paediatric procedures
WTIS Expansion 2008/09 - March 2009	86	System expanded to collect data for all adult and paediatric procedures

The Need for a Client Registry

Introducing the WTIS system hastened the development and deployment of another province-wide system, Ontario's Client Registry/Enterprise Master Patient Index (Client Registry). A cornerstone for e-Health in the province, the Client Registry identifies individual patients within and across hospitals. It was introduced concurrently with the WTIS because it provides the engine that allows patients on multiple waiting lists for the same procedure to be correctly identified, preventing duplication. Accurately tracking these individuals is important for maintaining data integrity and measuring true wait times. Consequently, the Client Registry became part of the WTIS rollout.

The WTIS is a custom-built .NET application. Custom-built integration code, designed to help integrate data from disparate sources and enable HL7 message transfer from multiple hospital interfaces, was provided by a third-party vendor. Data can either be entered online, via a standard web browser, or submitted electronically, through a hospital, using HL7 web-browser messaging. At a local level, hospitals determine how they open and close their wait list entries. The scenarios are identified in Figure 2. A central server at CCO acts as the database and stores all required wait time data.

 

[Figure 2]

The Client Registry's architecture permits integration with the WTIS, hospital systems and other provincial e-Health systems. Hospitals can submit patient information electronically to the Client Registry through a standard interface over a secure network connection. Patient information is then stored in databases. The WTIS and the Client Registry are connected over a secure sockets layer virtual private network (SSL VPN) connection that provides message security in environments where trust is an issue, across a managed private network. The two applications communicate over specific application program interface (API) calls that pass patient information between the two systems. In other words, access is simple, security is high and the systems can easily talk to each other. CCO hosts the WTIS, and the Client Registry, initially the responsibility of CCO, is now hosted by the Smart Systems for Health Agency (SSHA, now a part of e-Health Ontario).

Leverage Local Resources for Local Solutions

Rather than customize a solution for every hospital, the project team delivered a flexible platform that allowed for a local integration solution led by hospital IT professionals who could determine how best to adapt their systems to enable connection with the WTIS.

Clinicians would connect to the system directly from their offices using standard Internet protocols to ensure timely, accurate and near real-time capture of wait time data. Data capture at source minimized concerns about the impact of time delays and inaccuracies due to human error on data quality (e.g., interpretation of handwritten notes), and clinicians would have direct access to their wait list information for managing their wait lists.

Integration - Building on New and Existing Systems

Over time, the project team gained a solid and growing understanding of the various hospital systems, workflows and integration capabilities. This learning was critical in ensuring that, wherever possible, data available in other information systems would not be re-entered in the WTIS. An integrated WTIS/Client Registry solution ensured that the data-entry process was streamlined and enabled system consolidation and reconciliation of duplicate wait time entries.

Hospitals defined their integration requirements based on one of three levels: basic, standard or complex operating room (OR) integration. A custom-built integration engine enabled WTIS integration with hospital systems through a secure exchange of HL7 messages between hospital systems and the WTIS and HL7 adapters. Figure 3 shows the entry and flow of information between the WTIS and integrated hospital systems, involving the case of a sample patient.

 

[Figure 3]

Supporting Hospital Transition to the WTIS

Numerous supports and tools helped hospitals, their staff and clinicians in meeting the technical requirements of the WTIS. The project team highlighted key technical milestones, testing requirements, due dates and completion status for each participating hospital, tracked using the Indicator Report.

Key technical milestones included procedure mapping and conformance testing for go-live. Procedure-mapping exercises would match WTIS procedure codes to codes used by the hospitals' systems. This exercise was essential to the success of HL7 interface messages, which would be used to submit, open, modify or close wait list entry messages between the hospital system and the WTIS. Extensive conformance-testing activities were required to ensure hospital wait list data submissions complied with specified WTIS standards. This was key in validating that interface messages met current WTIS HL7 specifications and business rules. Issues with procedure maps or hospital interface customization were identified and resolved early, prior to deployment. Hospital operating room and decision support resources worked with hospital IT and project team resources to refine interfaces, ensuring their readiness for WTIS deployment.

Given that hospitals had different levels of technical ability and IT support, the project team's technical resources, working with the Site Leads, provided ongoing support to hospital technical teams to ensure an appropriate understanding of WTIS deployment requirements. Teleconferences and checkpoints were scheduled for information exchange and to provide opportunities for hospitals with similar systems to problem-solve among each other. Pre- and post go-live support was also available.

Lessons Learned Leverage existing knowledge and expertise to help define the technology. Take a cross-functional view (not just technical view) of decision-making to enable quick resolutions that consider all impacts to stakeholders. Consult with clinical leads early in the process. Ensuring end users become partners in the initiative will result in a greater likelihood of IM/IT success and buy-in. Execute repeated dry runs, supported with input from all project work streams, to establish roles and responsibilities, identify risks early and streamline the go-live process. Work as "one team" (no matter how many vendors are involved) to ensure a cohesive approach to product development and deployment. Use an iterative development approach to foster the team commitment and focus required to deliver a product within aggressive timelines. Establish clear and definitive feedback mechanisms to evolve the system in line with user and changing business requirements. Establish a strong technical support model during and post deployment to effectively escalate and resolve issues in a timely and positive manner. Conduct a rigorous testing program to identify risks early and resolve defects. This, in turn, delivers a product that minimizes business impact on go-live and builds credibility and trust in the overall process. Establish a privacy governance structure to mitigate and manage privacy concerns. Provide opportunities for integration, as adoption and sustainability are enabled if existing systems can be integrated with new ones.

Assuring Quality of Delivery

Well-defined processes were put in place to ensure each WTIS rollout was delivered to meet stakeholder needs and expectations, as well as ensure the appropriate level of support to hospitals.

Provincial Testing Program

The WTIS and the Client Registry had different and unique success criteria for testing. WTIS testing involved extensive hospital conformance testing activities to ensure that HL7 messages would successfully pass between the WTIS and hospital interfaces.

Building upon the initial testing program, the project team used specialized automated testing to augment the rigour and speed of the initial manual testing programs. This enabled greater breadth of functionality and accelerated release development and deployment cycles. Automated testing provided flexibility, while reducing overall risk. It was valuable in managing user expectations by identifying and addressing performance issues, thereby delivering a more stable product to hospitals. It also allowed the project team to gain the confidence of the hospitals, which understood that a release would not be deployed unless it conformed to their environments, workflows and business processes. Details of the application testing success factors can be found in "Key Success Factors for Testing the WTIS".

Process and Practice: Run Book and Dry Runs

Detailed run books (a written set of technical procedures) and dry runs for the WTIS' promotion to a live environment were an important element in the go-live process. Repeated dry runs were conducted via daily run book meetings leading up to the go-live, identifying issues and risks and providing predictability. The run book highlighted every step and detail of the work to be completed and assigned responsibility for each element.

Where run book exercises usually involve only technology resources, the WTIS project made the decision to include resources from other work streams, including communications, deployment and the CCO operations team, as well as representation from Ontario's Client Registry operations team. This meant all scenarios were included and a comprehensive plan was in place to reduce risk and ensure each participant was clear on role, responsibilities and timing. Any issues or risks found triggered a series of escalations that, within hours, could resolve the situation or result in the project sponsor calling a delay. Supporting communications and back-out plans were also developed in case a release or rollout was rescheduled.

Dashboards

Dashboards, like the Indicator Report for hospitals, provided snapshots of technology activity status. Although dashboards were internal tools, they helped the project team track progress to ensure that once live in hospitals, the WTIS would perform as planned. Dashboards were shared with key stakeholders, and they required multiple approvals before a decision to deploy or delay an application release was made. This ensured that cross-functional risks and impacts were considered, with clear criteria to determine whether or not the provincial team could or should proceed with next steps.

Before a go-live date, dry runs using the application in an environment that mimicked each hospital production environment assured that hospital systems would not be negatively impacted at go-live. Any impact would trigger a "go/no go" decision.

Key Success Factors for Testing the WTIS As information technology continues to be deployed within the healthcare system, healthcare and IT professionals need to be aware of the risk associated with these systems. WTIS's mitigation strategy in this case included the investment in an independent professional testing and verification capability. This capability was configured to manage multiple test phases, meet aggressive project timelines and deliver a high-quality product to hospitals. [Figure 1] ABOUT THE AUTHOR Harbir Singh, MBA, is a Consultant with xwave, a division of Bell Aliant. She specializes in the design and management of independent test and verification programs for information technology applications in healthcare.

 

Success Factor	Description	Benefits
Effective, Independent Testing Model	With in-depth knowledge of WTIS requirements, Cancer Care Ontario (CCO) worked closely with a quality assurance and testing vendor to develop a plan to execute a robust and iterative testing program. This model included remote testers (near-shore testers) who automated and executed tests that were then validated by the local team.	Delivers an independent and comprehensive quality assurance testing program that reduces risk associated with the WTIS application while meeting the requirements of the WTIS project. Segments roles and responsibilities to deliver cost efficiencies and provides the flexibility and expertise required to deliver a stable product within aggressive timelines.
Automated Regression-Testing Tools	A testing tool to automate regression tests was introduced. Regression tests ensure that existing system functionality has not been "broken," based on modifications made during a release. At the time, automated testing tools were considered innovative in healthcare IM/IT system deployment.	Ensures appropriate test coverage, while augmenting the rigour and speed of manual testing programs. Regression testing is typically repetitive, labour intensive and prone to human error. Automation of these tests made the testing of new releases more efficient and accurate. Once up and running, automation greatly streamlines regression testing. More importantly, it allows managers to re-run automated regression tests to validate an unplanned fix mid-cycle - an otherwise significant impact to cost and schedule under traditional manual methods. In Phase III, automated testing reducing the amount of time spent on execution of tests by 87%.
Automated Performance Testing Tools	Performance testing tools imitate production system usage and simulate a large number of people using the system simultaneously.	Enables observation of system behaviour under a similar amount of stress/load to what would be encountered in the production system. Early identification and management of performance issues ensures delivery of a more stable product to hospitals and enhances credibility.
Daily Tracking and Reporting	Defects were reviewed daily with the development team and requirements team, and issues/risks that could impact release timelines were escalated as soon as they were identified.	A daily "dashboard" to display progress at a glance proved to be a very effective tool for communicating progress and status and identifying risks during testing. (Figure 1)

Evolving the WTIS

As with any system, user feedback was critical to evolving the WTIS. A regular release management program allowed the project team to maintain focus on priorities and ensure successful deployment within stated timelines. A quarterly release strategy was implemented, and requirements were prioritized into major and minor releases. Major releases would deliver mission-critical requirements, enhancements and functionality, while minor releases would focus on less critical, "nice to have" functionality.

This release strategy, used throughout each phase, delivered many advantages. It focused the development lifecycle on clearly stated objectives, aligned the project team and hospital stakeholders in an increased level of interaction, minimized schedule slippage and ensured that the WTIS was deployed with minimal impact to business operations. It also managed stakeholder expectations and change requests, since stakeholders understood that the quarterly release cycle provided ample opportunities for system upgrades.

Post go-live support was available to support business users through a formalized help desk model, which ensured business and technical issues were addressed in a timely and useful manner. User feedback has played a critical role in improving help desk support. For example, the service window was expanded from 9 a.m. to 5 p.m. to 8 a.m. to 8 p.m., based on user feedback, to ensure the appropriate level of support was available to WTIS users.

The WTIS has moved from project to product and is now in the hands of the operations team at CCO, which manages and supports its evolution and maintenance through product and release management. As of March 2009, the WTIS is on version 12 of the application, an indication of its ability to evolve and meet the needs of its users.

Clinical Leads remain engaged and continue efforts to expand the WTIS to report on all surgical procedures. User feedback enabled CCO to build a more robust technical infrastructure to maintain a high-performance system and support WTIS expansion. Indeed, much of the success of the WTIS project can be attributed to the fact that the application was built by clinicians for clinicians.

Comments

Be the first to comment on this!