From Big to Small: A Process for Developing Policy-Relevant Research Summaries
Government departments and not-for-profit agencies expend significant resources commissioning and conducting research, only to find their resulting lengthy reports shelved and unused. In this case study, we describe a project that developed short, policy-relevant research summaries of topics prioritized by the Ontario Women's Health Council, an advisory body to the province's Minister of Health and Long-Term Care. These products were created based on an understanding of policy makers' needs and work contexts and were designed to help users find, understand and utilize research evidence to inform their clinical, programmatic and policy decision-making.
Government and not-for-profit agencies routinely commission or conduct research projects to develop new knowledge to guide decision-making in a range of areas specific to their constituencies and mandates. The products of these initiatives include documents produced in-house, as well as research and evaluation reports contracted to consultants and academic researchers. The resulting repositories of static information are often largely inaccessible - both to those within the organization, who may not know of the reports or be able to find them, and especially to those in the broader field, who could benefit from the knowledge languishing on shelves. For those working in healthcare and health policy, the challenges in accessing, synthesizing and using evidence are well known (Lavis et al. 2003): published journal articles, reviews, government and statistical reports and practice guidelines that highlight the latest and best research for specific types of health-related decisions remain largely unusable by policy audiences and contribute little to informing practice and policy.
The goal of getting research into practice requires, first, understanding the needs and work contexts of end users, that is, those who will ultimately use the synthesized research knowledge for decision-making; and then, developing knowledge products and tools to help users find, understand and utilize evidence in their daily decisions. A commitment to applying user-based design as proposed by knowledge translation (KT) scholars and practitioners (e.g., Jacobson et al. 2003) is required to advance this field and tap into the currently inaccessible and unused knowledge resident in many organizational settings.
The Ontario Women's Health Council (OWHC) was an advisory body to the Ontario Ministry of Health and Long-Term Care (MOHLTC) until its mandate was completed in spring 2007. For a number of years, the OWHC Secretariat, staffed by MOHLTC policy personnel, created and solicited new knowledge on topics in women's health directly relevant to clinical and policy decision-makers in Ontario and beyond. As a result, OWHC had over 50 research, project and program reports covering a broad range of topics. One of the final tasks of the OWHC was to identify a process to make the findings of these reports, not all of which were published, more available and accessible to a broader audience of decision-makers.
The objective of our project was to apply emerging knowledge translation strategies (Graham et al. 2006; Dobbins et al. 2007; Grol and Grimshaw 1999; Lavis et al. 2003) to develop policy-oriented research summaries (Lomas 2005) of high-priority content created by or for the OWHC.
The OWHC-KT Project
The overall process for developing and launching the research summaries included the steps and stages described below and presented in Figure 1.
Step 1: Identify the topics and set priorities
Because it was not feasible to summarize all topics1 and the more than 50 related reports, a system of topic identification and priority setting was implemented to determine those that would be included in the knowledge synthesis and product development. A table of topics and related documentation was prepared, along with a process for surveying current and former Council members, including a data collection tool1 that asked members to rate the importance of each topic. The survey was completed by 79% of members, and the results were collated to determine the main topics for knowledge synthesis and product development.
Of the topics selected, two were deemed to be process-related (i.e., they provided valuable "lessons learned" regarding implementation of specific activities of the Council), while seven provided specific research findings of interest to policy audiences.
Step 2: Identify the end users and their needs
The target group was determined by the OWHC to be policy personnel (at the manager and analyst levels) in the MOHLTC and related provincial ministries. Their needs for research syntheses were determined using a triangulated approach, including (1) a review of KT literature specific to evidence-informed decision-making by policy actors (e.g., Lomas 2005; Innvaer et al. 2002; Lavis et al. 2003; Dobbins et al. 2007), (2) consultation with experts in this area and (3) most importantly, informal discussions, formal interviews and focus groups with representatives from the user groups.
Focus groups - content topics
OWHC secretariat staff identified key users in the Ministry of Health and Long-Term Care and the Ministry of Children and Youth Services, and project staff organized focus groups with them. The sessions covered five main areas: research use, type of content required, value-added aspects to incorporate in the summaries, length and format of the summaries, and dissemination strategies.2 In total, seven policy analysts and managers from various branches within these ministries attended one or more of three focus groups - two initial groups held to determine user needs regarding these topics, and one to solicit feedback on draft products.
Interviews - process learnings
For the two identified process-related topics, key informants (two per topic) with particular knowledge about the issues involved were interviewed and their insights incorporated in the summaries (see below). All interview and focus group participants served as our product testing group.
Step 3: Develop tools
Developing the draft summaries was a multi-stage process including data extraction, gathering more recent, "value-added" information on the topics, producing an initial draft, getting end-user feedback in a focus group and finalizing the summaries for production.
Step 4: Extract the key information
Two stages of data extraction were completed. First, we reviewed all reports specific to a content area and created a new document summarizing the main findings and the authors' recommendations (these were usually found in the Executive Summary and/or Conclusion sections; if not, we reviewed all report sections). For the process-related topics, the key informant interview transcripts were reviewed and key points extracted and summarized.
Step 5: Prepare the draft summary
The second stage of data extraction involved distilling the information collected in Step 4 down to key, implementable and policy-oriented messages. Useful tips on how to formulate these messages and the draft summaries are outlined in Figure 2.
|Figure 2. Structuring the research summary content|
|• Use lay rather than technical language; avoid jargon and acronyms.
• Put as much information as possible in bulleted points rather than narrative paragraphs.
• When possible, avoid reporting statistics (e.g., p-values, t-test results, etc.) that may be meaningless to those without a research background; rather, use percentages or simple descriptive statistics such as means, standard deviations and odds ratios (framed in clear language - i.e., "those in the intervention group were almost twice as likely as those in the comparison group to...").
• When preparing more than one summary, keep the look and feel consistent.
• Include a section on "how to cite" the document, and identify the summary authors.
• Include the author of the original report and the author's contact information. (Prior to doing this, the report author should be given the opportunity to review the material and formally grant permission to include his or her name and contact information.)
• Include information on how to access the full report(s) and any other materials that were used or cited to prepare the summary.
Step 6: Search for "value-added" information
Depending on the topic and the time elapsed since project completion, a section highlighting new information was sometimes required. This situation typically arose when more recent research had been done, or when related policy or program decisions on the topic had been made in other jurisdictions. Methods to assess and meet this need included conducting comprehensive literature searches, asking research or policy experts in the field (including the original authors of the research reports) and consulting recent policy decisions.
Six of the nine summaries included a section with additional information (the two "process" summaries did not, nor did the most recently completed research report). Additional information included citations of specific new research studies, recent guidelines from other organizations, experiences with implementing a policy in other jurisdictions and discussion of existing controversies that could influence decision-making.
Step 7: Incorporate user feedback into the product design
Initial feedback was sought by sending an electronic version of the draft summary to the product testing group early in the development cycle. Next, a focus group was held with four participants who had attended one of the two initial groups; the purpose was to request specific feedback on the appearance, structure/format and content of the summaries.
Step 8: Disseminate the final product
The final summaries were produced by a graphic designer using professional graphics software. Print and electronic versions (in portable document format) were prepared with careful attention to such details as length, layout, colour-matching with the OWHC logo and so on.
Results of the OWHC-KT Project
In total, nine research summaries were prepared: seven specific to content areas and two that described "lessons learned" about process issues related to policy implementation. In addition, the OWHC requested that a step-by-step handbook be developed describing this process.3
Several methods were used to disseminate the summaries, which were printed in colour and were two to four pages long. Following the initial release of the summaries at a farewell reception for the OWHC, electronic versions were posted on the Council's website (< www.womenshealthcouncil.on.ca/English /Knowledge-Translation-Tools.html >). They were also sent to an identified end-user group including CEOs and planning directors of Local Health Integration Networks and Community Health Centres.
Given the nature of the project as the OWHC's "swan song," no specific summary maintenance plan was developed (Step 9 in Figure 1). However, in general, decisions regarding maintaining and updating research summaries can have significant resource implications and should be considered as early as possible in the development process. If the project intends to maintain and update summaries, the issues outlined in Figure 3 should be considered.
|Figure 3. Maintaining and updating research summaries|
|• Determine the time frame for reviewing topics for update. This can be yearly, although the intensity of work in the topic area may dictate more or less frequent reviews.
• Decide whether updates will consist of augmentation of the existing summary with new information only (e.g., "add-on"), significant revision to several sections (e.g., "update") or complete overhaul of the topic - essentially, starting from scratch. The decision will depend on the topic, but having an a priori protocol outlining which factors lead to which decision is helpful. Such factors include:
o the amount of new information;
o the type of new information (i.e., does it support or contradict the previous conclusions?);
o new controversies or other contextual factors that may require a different treatment of the issue.
• Determine who will make decisions regarding whether updating is required and who will conduct the topic reviews. Ideally, some of those who were on the original review and summary team will be part of the updating process.
• Develop a protocol for how to search for new information (sources, tools, etc.) and how much time will be spent doing this.
• Decide what will happen in cases where searches for new information yield nothing of note (i.e., nothing that changes the conclusions of the original summary). For example, will the same summary be re-released with a note indicating when it was last checked for new relevant information?
Similarly, no specific evaluation activities were planned to assess the impact of the summaries. However, informal feedback suggests that those who received them found the summaries visually appealing and useful in their content, size and format. Future projects of this type would provide valuable information by building in an evaluation and follow-up component. Figure 4 offers a brief critique of the project process from the perspective of two end users.
|Figure 4. The policy perspective|
|by Nancy Lewis, Manager, and Sarah Caldwell, Analyst, MOHLTC
Overall, the process of creating the documents was positive and informative. It illustrated how KT can be done effectively and products created that can be disseminated and used by others. The process illustrated an effective way of carrying out a prioritization exercise and provided an opportunity to take time to reflect on, and capture, lessons learned. It was also impressive to see how coherent and digestible products could be generated out of a mass of completed research projects, many of which had often overwhelming methodological detail and, within a body of reports on a content area, sometimes conflicting results and/or interpretations.
A few valuable lessons were learned along the way. For example, a project of this magnitude needs to be well planned and requires a commitment of financial and human resources from the project team and the agency funding the project. In this case, resource constraints and lack of sustainability (by virtue of the OWHC's termination) meant that an ambitious interactive Web platform originally proposed by the project team had to be abandoned. It also proved impossible in some cases to reconstruct the (undocumented) path from research into policy with the resources available, with the result that the documents were often silent on policy implications, even though this was originally intended to be one of the key "value-added" results of the undertaking. In addition, more time should have been devoted at the front end to planning for the later parts of the process - analysis, "thinking through", garnering feedback - in order to strengthen these elements of the project. Finally, the unique circumstances of the OWHC's situation meant that products could not be sustained and revised in the long term, lessening their continued utility.
Nevertheless, the process of creating these summaries was valuable, and should be recommended to decision-makers who wish to create policy-relevant summaries for various audiences.
Pineault et al. (2006) have proposed the concept of a "research collective" - a group comprising decision-makers and researchers actively involved in creating syntheses as research is still underway or just completed. While such an approach provides an ideal context in which to create useable summaries, or at least to engage both sides in meaningful and ongoing dialogue (Pineault et al. 2007; Innvaer et al. 2002), it is not always feasible nor possible in cases where the research is already complete. In these cases, the process described in Figure 1 may prove useful.
Several features of this project were unique in the context of existing models of knowledge translation (e.g., Graham et al. 2006). First, the project team was charged with summarizing and synthesizing multiple studies, conducted by multiple research teams, which were related only in that the OWHC funded them. The synthesis work itself was what Graham et al. (2006) termed "third-generation knowledge" or synthesized products and tools. Second, the project team included the same policy personnel who were (a) involved in funding the research and (b) representative of the end-user group (provincial bureaucrats in health and social services portfolios). Third, and related to the two points above, the research being summarized was highly user-interactive (in the KT sense) because, being funded by OWHC, it reflected direct user involvement in deciding the initial research priorities. Finally, the process actively provided opportunities for researchers and end users to work collaboratively to determine the content and style of the final products (Innvaer et al. 2002).
We therefore propose that this case study exemplifies a blended user-interaction approach for research dissemination that fits well within the "knowledge creation" core of the framework proposed by Graham et al. (2006). As such, this type of activity is one part of a larger KT process that includes a number of steps in an "action cycle": problem identification; knowledge selection and adaptation to the local context; assessment of barriers and tailoring of specific interventions to be implemented in that context; monitoring and evaluation; and sustainability.
This project therefore highlights techniques that might be useful to address a very specific aspect of the full spectrum of knowledge-to-action activities that can occur between researchers and decision-makers (Lomas 2005). We suggest, based on our experiences with this process, that the summaries themselves might best be viewed as an adjunctive strategy to be accompanied by an interactive KT plan, including, for example, ongoing consultations, education sessions or transmission to end users by a knowledge broker who could review, translate and discuss key findings and work to identify strategies for local implementation (Lavis et al. 2003). A key result of the consultation phases of our project was that the simple act of bringing the research team together with potential end users increased awareness of the evidence and, perhaps, the probability that the summaries would be consulted if these users were faced with a related decision or situation. The extent to which these same summaries would be viewed as useful by other types of decision-makers - within the policy context (e.g., politicians) and beyond (e.g., healthcare providers, patients) - is not known and requires further research.
De « volumineux » à « concis » : procédé pour produire des résumés de recherche pertinents aux responsables de politiques
Les ministères et les organismes sans but lucratif affectent une quantité appréciable de ressources dans la commande et l'exécution de recherches, mais les longs rapports qui en découlent sont très souvent oubliés sur les tablettes et inutilisés. Nous décrivons un projet visant à produire de courts résumés de recherche pertinents aux politiques et portant sur des sujets jugés prioritaires par le Conseil ontarien des services de santé pour les femmes, un organisme du ministère de la Santé et des Soins de longue durée de l'Ontario. Ces résumés ont été créés en fonction des besoins des responsables de politiques et de leur contexte de travail. Les résumés sont conçus pour aider à trouver, à comprendre et à utiliser les données de recherche afin d'éclairer les décisions cliniques, politiques ou touchant aux programmes.
About the Author(s)
C. Nadine Wathen, PhD
Faculty of Information and Media Studies, University of Western Ontario
Gillian K. Watson, HBSc
Department of Psychology, Simon Fraser University
Susan M. Jack, RN, PhD
School of Nursing, McMaster University
Sarah Caldwell, MA
Health System Planning and Research Branch, Ontario Ministry of Health and Long-Term Care
Nancy Lewis, PhD
Policy Lead, Colorectal Cancer Screening Program Cancer Care Ontario
Correspondence may be directed to: Nadine Wathen, Faculty of Information and Media Studies, The University of Western Ontario, North Campus Building, London, ON N6A 5B7, tel.: 519-661-2111 ext. 88480, e-mail: email@example.com.
AcknowledgmentThis project was funded by the Ontario Women's Health Council, Ontario Ministry of Health and Long-Term Care. We thank Nancy Sicchia for her graphic design work.
Dobbins, M., S. Jack, H. Thomas and A. Kothari. 2007. "Ontario Public Health Decision Makers' Informational Needs and Preferences for Receiving Research Evidence." Worldviews on Evidence-Based Nursing 4(3): 156-63.
Graham, I.D., J. Logan, M.B. Harrison, S.E. Straus, J. Tetroe, W. Caswell and N. Robinson. 2006. "Lost in Knowledge Translation: Time for a Map?" Journal of Continuing Education in the Health Professions 26(1): 13-24.
Grol, R. and J. Grimshaw. 1999. "Evidence-Based Implementation of Evidence-Based Medicine." Joint Commission Journal on Quality Improvement 25(10): 503-13.
Innvaer, S., G. Vist, M. Trommald and A. Oxman. 2002. "Health Policy Makers' Perceptions of their Use of Evidence: A Systematic Review." Journal of Health Services Research and Policy 7: 239-44.
Jacobson, N., D. Butterill and P. Goering. 2003. "Development of a Framework for Knowledge Translation: Understanding User Context." Journal of Health Services Research and Policy 8: 94-99.
Lavis, J.N., D. Robertson, J.M. Woodside, C.B. McLeod, J. Abelson and the Knowledge Transfer Study Group. 2003. "How Can Research Organizations More Effectively Transfer Research Knowledge to Decision Makers?" Milbank Quarterly 81(2): 221-48, 171-72.
Lomas, J. 2005. "Using Research to Inform Healthcare Managers' and Policy Makers' Questions: From Summative to Interpretive Synthesis." Healthcare Policy 1(1): 55-71.
Pineault, R., P. Tousignant, D. Roberge, P. Lamarche, D. Reinharz, D. Larouche, G. Beaulne and D. Lesage. 2006. "The Research Collective: A Tool for Producing Timely, Context-Linked Research Syntheses." Healthcare Policy 1(4): 58-75.
Pineault, R., P. Tousignant, D. Roberge, P. Lamarche, D. Reinharz, D. Larouche, G. Beaulne and D. Lesage. 2007. "Involving Decision-Makers in Producing Syntheses: The Case of the Research Collective on Primary Healthcare in Quebec." Healthcare Policy 2(4): 1-17.
1 A list of topics and the data collection tool are available from the first author.
2 The interview guide and summary of focus group results are available from the first author.
3 The handbook is available from the first author.
Be the first to comment on this!
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed