Objectives: (1) To develop an online repository of policy-relevant documents, other than and complementary to those from the peer-reviewed scientific literature, addressing healthcare renewal in Canada; and (2) to describe the distribution of document contents.

Methods: An iterative scoping review approach was undertaken. Documents were identified through website hand-searches and referrals from 19 Canadian health organizations. Descriptive frequencies were calculated, such as for document type.

Findings: In July 2014, 1,034 documents were in the Evidence-Informed Healthcare Renewal Portal. The top three types of documents were situation analyses (n = 390, 38%), health and health system data (n = 191, 18%) and jurisdictional reviews (n = 115, 11%). The top three national priority areas addressed were health human resources (n = 778, 75%), quality as a performance indicator (n = 502, 49%) and information technology (n = 385, 37%).

Conclusion: The process of developing a systematic method for identifying these documents has yielded a new resource to support evidence-informed health policy making and has identified a large volume of policy-relevant documents addressing healthcare renewal priority areas in Canada.

Health policy makers need quick and easy access to many different types of evidence that can help them to make well-informed decisions about health systems. Researchers, policy makers and other stakeholders have undertaken many initiatives to facilitate the use of research evidence in health policy making, such as preparing evidence briefs and convening stakeholder dialogues (Lavis et al. 2006). Online one-stop-shops have also been created that identify, add value to and make readily available different types of evidence (e.g., Health Evidence¹ for systematic reviews of the effects of public health programs and services and Health Systems Evidence [HSE]² for many types of research evidence – including systematic reviews – about governance, financial and delivery arrangements, as well as implementation strategies, in health systems). The majority of these online repositories focus on research evidence. There is a need for an online repository that identifies, adds value to and makes readily available other types of evidence, such as colloquial evidence (Lomas et al. 2005). This type of evidence can support a better understanding of contextual factors related to the health system.

Policy-relevant documents can be considered a type of colloquial evidence, and are typically not published in peer-reviewed journals; Lomas and colleagues (2005) explain that the use of colloquial evidence prevails among decision-makers. This type of evidence is expansive and includes the expertise, views and realities of experts and professional opinion, political judgment, values, habits and traditions, input from lobbyists and pressure groups and the particular pragmatics and contingencies of the situation, such as resources. Although policy-relevant documents are primarily of the colloquial evidence type, these documents may also fit under the context-free or context-sensitive types of scientific evidence, as defined by Lomas et al. (2005). Context-sensitive scientific evidence is adapted to the circumstances of the local context and often addresses attitudes, implementation, organizational capacity, forecasting, economics/finance and ethics (Lomas et al. 2005). The most narrow type is context-free scientific evidence and is defined as methodologically explicit, systematic and replicable and is most closely aligned with the evidence-based medicine approach. Ultimately, a combination of these three types of evidence, considered alongside the many other influences on the policy process (possibly through deliberative processes), can best support evidence-informed health policy making.

Evidence-Informed Healthcare Renewal Initiative

In Canada, the Evidence-Informed Healthcare Renewal (EIHR) initiative was created to support the producers and users of evidence to work collaboratively to "translate evidence for uptake into policy and practice to strengthen Canada's healthcare systems" (CIHR 2012a). In October 2011, the EIHR initiative convened the first EIHR Roundtable with participants from 41 federal, provincial and territorial governments and a broad range of national and provincial stakeholders and analysis organizations. One of the objectives of the EIHR Roundtable is to work together to ensure the collective outputs (i.e., policy-relevant documents) of each of the partner organization's work are available and accessible to the public and health system leaders through a public knowledge platform (CIHR 2012b). To operationalize this objective, the EIHR Roundtable chose to collaborate with the HSE team at McMaster University (HSE 2012) to create the EIHR Portal, which is "a continuously updated repository of policy-relevant documents that address 'healthcare renewal' in Canada" (CIHR 2012c). The development and contents of the EIHR Portal are the focus of this paper. More detailed information about this process is available through the master's thesis on which this paper is based (Kowalewski 2012).

Methods

An iterative approach similar to Arksey and O'Malley's (2005) scoping review framework was used in developing the methods for creating the EIHR Portal (Figure 1). The Arksey and O'Malley (2005) scoping review framework suggests five stages: identifying the research objective; identifying relevant studies, or in this case documents; selecting documents; charting the data; and collating, summarizing and reporting the results. Note that the fifth stage of the scoping review approach was modified to collate, summarize and report on the general focus of the documents identified and not the specific content of the documents themselves. An optional sixth stage involves consultation with stakeholders to ensure the identification of all relevant material.

Identifying the objective

As previously described, one of the objectives of the EIHR Roundtable was to create an online repository of policy-relevant documents addressing healthcare renewal in Canada (but not published in peer-reviewed journals). To help define the scope of this repository, a broad Google advanced search was conducted to gain a sense of the different types of policy-relevant documents related to Canadian health systems. A scan of the results revealed that many different types of documents are available on the Internet from a number of different sources. However, the breadth of the search produced a large number of results that included many documents that were not relevant to the scope of this project. This initial scoping revealed that a more targeted search strategy was necessary to feasibly and efficiently identify and retrieve these types of documents.

The EIHR Roundtable provided input for the scope of the repository, which was then used to develop and refine the objective and eligibility criteria (see "Selecting documents" further). Nineteen EIHR Roundtable member organizations agreed to contribute documents to assist with populating the online repository. Consultations with the EIHR Roundtable helped to ensure that the contents of the EIHR Portal are of broad interest and usefulness to different policy maker and stakeholder groups in Canada. A descriptive analysis of the general contents of the documents helped to identify any gaps in the EIHR Portal content to ensure comprehensiveness as it continues to grow.

Identifying relevant documents

The identification of relevant documents occurred in four iterative and gradually more targeted search phases (Figure 1). The first and second phases of this stage were conducted before consultations with the EIHR Roundtable began and helped to gain a sense of the available pool of policy-relevant documents. The first phase targeted the Canadian Health Research Collection electronic database, which is a collection of publications from Canadian research institutes, government agencies and university centres working in the area of health and medical research. The second phase was a hand-search of key national health organization websites. The third and fourth phases were conducted in collaboration with the EIHR Roundtable and were critical to the development of the online repository. The third phase involved identifying documents produced by EIHR Roundtable organizations. The final phase involved identifying and signing up for listservs, as well as establishing a mechanism to support internal referrals from the research team.

All searches were focused on documents produced since January 2003 (and documents from before that time were included only if they were perceived as landmark by EIHR Roundtable members, such as the Canada Health Act, although we return to exceptions below). The start date of 2003 was chosen for two main reasons. First, this was the year of the First Ministers' Accord on Health Care Renewal, signifying a major political shift in focus and support for healthcare renewal in Canada. Second, the (then) time span of 10 years enabled the comparison of document content across a significant length of time but also ensured that the documents are still relevant in the current political and system context.

Selecting documents

The inclusion/exclusion criteria were devised iteratively throughout the four phases of the search strategy. The EIHR Roundtable was consulted extensively during the development of the criteria and specifically, during the development of the taxonomy of document types (Table 1). There were two categories of criteria: process-related and content-related. The main initial process-related inclusion criterion was that the document was referred by an EIHR Roundtable organization or was referred by the research team and approved by an EIHR Roundtable organization. The first content-related inclusion criterion was that the document had to address healthcare renewal, which was defined as renewing, reforming or strengthening governance, financial and delivery arrangements within health systems. The second content-related criterion was that the document had to be one or more of a list of 24 (later 25) document types (see Table 1).

The taxonomy of document types was created using a sample of 50 documents from a broad range of sources that were all referred by the EIHR Roundtable. Two individuals from the research team worked through several different versions of the taxonomy before deciding on the current version. Consultations with the EIHR Roundtable informed this iterative process. The following types of documents/sources were excluded: e-newsletters, one-stop shops, podcasts and videos, peer-reviewed journal articles, derivative products of relevant healthcare renewal documents, annual reports that only describe basic activities and outputs or that only present audited financial statements and opinion pieces showing only one individual's opinion on issues related to healthcare renewal (although one-stop shops, podcasts and videos are captured through an "additional resources" document on the EIHR Portal).

Two independent reviewers assessed for eligibility all documents identified in Search Phases 3 and 4. The same two reviewers also assessed the eligibility of those documents identified in Search Phase 2 and by the research team in Search Phase 4 that were sent to EIHR Roundtable organizations for approval to include in the EIHR Portal (any documents not identified by an EIHR Roundtable member had to be approved by the EIHR Roundtable). Both reviewers read the full documents and then met to reconcile any discrepancies in assessments of eligibility.

Charting the data

Two independent reviewers extracted information from the documents assessed as eligible for the EIHR Portal using a coding taxonomy for health-system topics developed for HSE (Lavis et al. 2014, Manuscript under review). The documents were also coded for national health system priority areas, as identified in the 2003 First Ministers' Accord on Health Care Renewal (Health Canada 2006). Documents can be coded for more than one topic and priority area. Finally, the two independent reviewers extracted descriptive characteristics, such as document type, jurisdictional focus and general citation information (note that documents that did not specify a jurisdictional focus were coded as Canada).

Collating, summarizing and reporting the results

The data were summarized to present a "descriptive epidemiology" of policy-relevant documents addressing healthcare renewal in Canada that are contained in the EIHR Portal. The focus of the analysis was to calculate descriptive statistics for both the general characteristics of included documents and their coverage of national priority areas.

Maintaining the EIHR Portal

While not formally part of a traditional scoping review approach, the McMaster Health Forum has continued to develop and execute a strategy to keep the EIHR Portal up to date. Every month, EIHR Roundtable member organizations are sent a reminder e-mail to submit their list of referrals for the EIHR Portal (please refer to eihrportal.org for a list of all current participating healthcare organizations). On average, the EIHR Roundtable members refer 50 documents per month; however, the number of referrals varies greatly on a month-to-month basis. EIHR Roundtable organization referrals are augmented by internal referrals from the research team that are then vetted by one or more EIHR Roundtable members. The sources of these internal referrals are listservs from and ongoing website hand-searches of a broad range of governments, government agencies, institutional associations, professional associations and colleges, patient- and disease-based groups and research centres. The websites are hand-searched monthly, quarterly or yearly depending on their past yield (with prolific sites searched more frequently). All documents continue to be assessed and coded by two independent reviewers.

Results

A total of 1,034 documents were included in the EIHR Portal as of July 2014. Three quarters of documents were published between 2009 and 2014 (n = 780, 75%). Sixty-six percent (n = 687) of documents focus on Canada as a whole. More than one in ten province-territory-focused documents address healthcare renewal in Ontario (n = 118, 11%). The top three types of documents are situation analyses (n = 390, 38%), health and health system data (n = 191, 18%) and jurisdictional reviews (n = 115, 11%). There is no representation of government/third-party accords (see Table 2).

Document themes by 2003 First Ministers' Accord on Health Care Renewal national priority areas

Many of the documents address a number of national priority areas as identified in the 2003 First Ministers' Accord on Health Care Renewal (Table 3). The top three national priority areas addressed are health human resources (n = 778, 75%), quality as a performance indicator (n = 502, 49%) and information technology (n = 385, 37%). The least commonly addressed national priority areas are technology assessment (n = 25, 2%), prescription drug coverage (n = 128, 12%) and diagnostic/medical equipment (n = 157, 15%). Although more attention is being paid to these issues, as evidenced by the increasing number of documents addressing these topics across the three time points (Table 3), they are still not addressed as frequently as the other national priority areas.

Discussion

The EIHR Portal provides a one-stop shop for policy-relevant documents that can support health policy makers who make decisions about healthcare renewal in Canada. It is continuously updated based on protocols refined during its development. The EIHR Portal contains 1,034 policy-relevant documents that address healthcare renewal in Canada and that can be categorized into 24 document types; the largest category of which is situation analysis (n = 390, 38%). The documents address national priority areas identified by federal, provincial and territorial governments, such as health human resources (n = 778, 75%) and information technology (n = 385, 37%). The lack of focus on certain national priority areas, such as technology assessment (n = 25, 2%), seen from a system level (as opposed to in terms of particular programs, services and drugs) can spur discussion about current gaps in Canadian healthcare renewal.

Strengths and limitations

This study has four key strengths and two limitations. The first strength is that the results of the initiative present the first effort to systematically identify, gather and describe policy-relevant documents that address healthcare renewal in Canada and that have not been published in the peer-reviewed literature. The novel taxonomy of document types can contribute to future systematic analyses of this unique body of evidence. Second, the study involved key stakeholders, which was pivotal to its success. This approach is commonly called linkage and exchange in the Canadian context (Lomas 2000). Arksey and O'Malley (2005) explain that including the perspectives of those with knowledge of, and a vested interest in, the area of investigation, in this case Canadian healthcare renewal, is invaluable to the research process. Third, the integration of the EIHR Portal into HSE permits users who identify policy-relevant documents to examine related systematic reviews and other types of research evidence (e.g., economic evaluations and health reform descriptions) and the prompting of users who identify research evidence to examine related policy-relevant documents. Fourth, the EIHR Portal was designed to be continuously updated, which will help to ensure its continued relevance to Canadian policy makers and other stakeholders. The first limitation of this study is that the quality of the documents included in the EIHR Portal was not appraised, because no systematic quality-appraising tool exists for most document types. The second limitation of this study is that, given the nature of the process used to identify documents, there is a possibility that this process may have not captured all documents. However, a comprehensive search of all databases containing grey literature (as would be typical of systematic reviews) would be highly challenging for identifying this unique body of evidence.

Implications for health policy making in Canada

There are a number of implications for health policy making that arise out of the development of a one-stop shop for policy-relevant documents that address healthcare renewal in Canada. The one-stop shop expands the breadth of documents available for policy makers to easily access and use to inform their health system decisions. The EIHR Portal, which provides policy makers with colloquial evidence primarily, in combination with the research evidence available in HSE, facilitates the development of well-informed health policies that can be based on a truly comprehensive understanding of the best available global evidence and of the local health system context.

An analysis of the general contents of the documents in the EIHR Portal also has implications for health policy making in Canada. Based on the descriptive epidemiology of policy-relevant documents, it is evident that certain Canadian health system national priority areas are receiving more attention than others. Policy makers should consider whether all national priority areas are receiving the necessary attention.

Implications for research

The EIHR Portal described in this study is the first of its kind. In time, the EIHR Portal could include documents that address healthcare renewal or other macro-level health system topics in other countries; the potential for inter-country collaboration is great. The process of developing a systematic method for identifying policy-relevant documents and retrieving useful information from these documents can be reproduced by anyone interested in using this type of evidence to inform their health policy making. Of course, the methods described here can only serve as a guide because the process depends on the context and resources available to create one-stop shops like the EIHR Portal. Future research should devise a method to appraise the quality of these documents, so that this could be reported in the EIHR Portal and policy makers would not have to rely on their own critical appraisal skills.

Conclusion

There is growing recognition that a range of evidence is required for evidence-informed health policy making that addresses not only questions of effectiveness but also describes contextual factors, such as the ideas, interests and institutions that shape health policies. Ultimately, only through concerted efforts that facilitate wholly informed policies can we effectively strengthen the health system and improve the health of Canadians.

Appuyer l'élaboration de politiques de santé fondées sur les données probantes: développement et contenu d'un répertoire en ligne de documents ayant trait aux politiques du renouvellement des services de santé au Canada

Résumé

Objectifs: (1) Développer un répertoire en ligne de documents ayant trait aux politiques, autre que et complémentaire à ceux issus de la littérature scientifique évaluée par les pairs, dans le contexte du renouvellement des services de santé au Canada; et (2) décrire la distribution du contenu des documents.

Méthode: Une revue itérative de délimitation de l'étendue a été entreprise. Les documents ont été identifiés au moyen de recherches manuelles sur le site Web et en fonction desréférences provenant de 19 organisations de santé au Canada. Les fréquences descriptives ont été calculées, notamment pour le type de document.

Résultats: En juillet 2014, 1 034 documents étaient présents sur le Portail du renouvellement des soins de santé fondé sur des données probantes. Les trois principaux types de documents étaient les analyses de situation (n = 390, 38 %), les données sur la santé et le système de santé (n = 191, 18 %) et les examens juridictionnels (n = 115, 11 %). Les trois principales priorités nationales traitées étaient les ressources humaines en santé (n = 778, 75 %), la qualité comme indicateur de performance (n = 502, 49 %) et les technologies de l'information (n = 385, 37 %).

Conclusion: Le processus pour le développement d'une méthode systématique afin d'identifier ces documents a mené à une nouvelle ressource pour appuyer l'élaboration de politiques fondées sur les données probantes et a permis d'identifier un grand volume de documents qui traitent des secteurs prioritaires pour le renouvellement des services de santé au Canada.

Comments

Be the first to comment on this!