In this era of accountability and efficiency, it is only natural that health systems managers are willing to jump on board with such broad, inspirational statements as using research to inform practice and driving evidence-informed decision-making. But actually acting on these concepts in practice requires managers to openly examine the resources their organizations have in place and develop the capacity of these resources (be they people, programs or structures) to use research. The self-assessment tool - IsResearch Working for You? ASelf-Assessment Tool and Discussion Guide for Health Services Management and Policy Organizations - is specifically intended to aid in this evaluation. Developed by the Canadian Health Services Research Foundation (CHSRF), this validated tool is designed precisely to help organizations examine, discuss and understand their capacity to gather, interpret and use research evidence in making decisions about how they organize and deliver health services (CHSRF 2005). As seen on both the national and international fronts, the tool is particularly helpful when implemented as an initial intervention in healthcare organizations to advance evidence-informed decision-making.

Origins of the Self-Assessment Tool: A Brief History

The self-assessment tool and discussion guide were originally developed in 1999 by Christina Ugolini and Steven Lewis of Saskatchewan to assess four general areas of research use: Acquire: can an organization find and obtain the research evidence it needs? Assess: can an organization assess research findings to ensure they are reliable, relevant and applicable? Adapt: can the research findings be adapted to the organization's specific needs? Apply: are there skills, structures, processes and a culture in the organization to promote and use research findings in decision-making? Between 2003 and 2004, a research team led by Anita Kothari of the University of Western Ontario and Nancy Edwards of the University of Ottawa undertook a validation process to find out if the tool was useful in distinguishing between organizations that have clear supports for acquiring, assessing, adapting and applying research and those that have less capacity to do so (CHSRF 2008d). Their investigation involved 32 focus groups across four sectors of Canadian health organizations, including selected branches of federal government and long-term care, non-governmental and community-based organizations. The validation process found the tool to be user friendly and useful for distinguishing among organizations with different degrees of capacity to use research to inform decision-making. The tool was especially beneficial in prompting discussions about evidence-informed decision-making, which is arguably more helpful than the actual scores assigned as the discussions can lead to organization-level insights and subsequent changes within an organization (CHSRF 2008d). Further details about the validation process are available on the foundation's website (CHSRF 2008d) and in a forthcoming paper by the researchers.

EXTRA! EXTRA! National Experiences Using the Tool

Much of the interest nationally in the self-assessment tool has come in relation to the Executive Training for Research Application (EXTRA) program - a two-year fellowship for physician leaders, nurse leaders and other health executives to develop their capacity and leadership to optimize the use of research evidence in managing Canadian healthcare organizations (CHSRF 2008a). The fellowship involves each fellow leading a change management project that confronts a health services management, delivery or policy issue in their host organization or region. To date, three fellows have used the foundation's self-assessment tool as the basis of their intervention project. In each case, the self-assessment tool has been a useful first step in improving evidence-informed decision-making.

Nancy Roberts, a recent graduate of the EXTRA program, used the tool to inform a self-assessment of her own organization, particularly its management and leadership team's knowledge and skills to acquire, assess, adapt and apply evidence. The assessment was a necessary first step in developing a health workforce management workbook and, ultimately, establishing a formal, evidence-informed health human resources planning process in Ms. Roberts' region of the New Brunswick health system.

Another recent graduate, Louise Patrick, consulted the Foundation's tool for her investigation of the decision-making context of an Ottawa, Ontario-based health services organization. Dr. Patrick created and implemented her own assessment tool to explore a range of issues in her organization, such as the basis upon which decisions are made, the capacity to engage in evidence-based decision-making, the interest and willingness of decision-makers to change their practices, the barriers to research use and the extent to which access to an evidence consultation service, for decision support, would increase this use among administrators.

Current EXTRA fellow, Ken Baird, is asking similar questions with his intervention project, which aims to achieve a framework for evidence-informed decision-making at a district health authority in Halifax, Nova Scotia. At a time when his health authority is making a cultural transformation from a service delivery organization to a knowledge-based learning organization (e.g., with a deeper focus on research services and a stronger educational mandate), undertaking a self-assessment of research use capacity is especially important.

Conducting the self-assessment is now engrained in the EXTRA experience. The foundation introduced the tool as a component of the EXTRA program this summer, endorsed by the program's faculty and advisory council. The intention is to further position EXTRA as a lever in stimulating a transformation in organizational culture. As such, every fellow in the latest cohort (cohort V) of the program and all cohorts going forward must administer the tool as a take-home exercise with their senior management teams (CHSRF 2008b, 2008c). For each of the organizations, the findings will supply a helpful baseline measure, provided each organization carefully replicates the self-assessment process in the future. By the same token, the data are important for feeding into the ongoing evaluation activities of the EXTRA program at large.

Outside of EXTRA, the tool has also seen success across Canada. It was particularly helpful in assessing the organizational research capacity of Interior Health, a British Columbia health authority that serves 15% of the province's population. In 2005, Interior Health received funding from the Michael Smith Foundation for Health Research to launch a research department to increase the research capacity in the region. Research facilitator Jennifer Miller tailored the self-assessment tool to suit the needs of the health authority. For example,
Dr. Miller added questions to the assessment about individuals' personal circumstances, such as whether they personally had the time and resources to get involved in research. The assessment helped Interior Health realize necessary steps in launching their research department, such as the need to focus attention on building capacity to assess and adapt research, as well as the need to raise awareness of potential linkages with local university researchers. Interior Health's experience using the tool was summarized in an issue of Promising Practices in Research Use, a series that highlights healthcare organizations that have invested their time, energy and resources to improve their ability to use research (CHSRF 2007).

Meanwhile, in Toronto, Ontario, a research team led by Melanie Barwick of the Community Health Systems Resource Group at the Hospital for Sick Children has used the tool in their work on research use among providers of children's mental health (Barwick et al. 2008). The research team distributed an adapted version of the tool to 80 Ontario provider organizations for children's mental health services, of which 51 executive directors and 483 practitioners responded. Based on a four-point Likert scale (not well, somewhat well, well and very well), the majority of executive directors and practitioners rated their organization's capacity to assess the reliability and quality of research as "somewhat well" (Barwick et al. 2008). The researchers argue that these and other findings can provide a helpful benchmark for tracking changes in research use in the Ontario children's mental health sector, especially in relation to evidence-based practice training and implementation initiatives that may arise. They also recommend the launch of a community of practice - as a strategy to enable practitioners, in particular - to build capacities in research adaptation and application (Barwick et al. 2008).

Lessons from Abroad

The self-assessment tool is also being taken up internationally. This past September, the Alliance for Health Policy and Systems Research program brought together four health policy teams, each from one of four middle-income countries - Columbia, Argentina, Mexico and Georgia - to discuss their experiences in enhancing the application of research in health policy making. The alliance is an international collaboration, based in the World Health Organization (WHO), in Geneva, Switzerland, that aims to promote the generation and use of health policy and systems research as a means to improve the health systems of developing countries (WHO 2008a). As such, the alliance granted funding to the teams to develop capacity for the generation, dissemination and use of health policy and systems research knowledge in their native countries. As part of a first step, each team used the foundation's self-assessment tool. Key messages about the tool emerged, particularly that the tool provides the following:

• Valuable insights into an organization's culture, including but not limited to the value it places on research
• A useful, initial intervention that can build support for longer-term investments in helping organizations use evidence to inform management and policy
• A helpful baseline measure, although only time will tell if the tool shows promise in assessing progress over time in enhancing research use capacity

The teams also acknowledged that the tool's utility may be limited when the assessment does not take place in a democratic environment. If the tool is to be effective, participants must feel free to discuss their organization's research use capacity, without fear for their job security or other negative consequences. Participants felt that in some environments, it may be challenging for decision-makers to be forthcoming and honest about organizational shortcomings and needs for improvement, given that the exercise requires senior management participation. It is only with a sincere and frank approach that organizations can properly identify areas for improvement in research use. The foundation's self-assessment tool is likely to be a feature for projects to be funded by the alliance in low-income countries, as part of a recently issued call with the Wellcome Trust (WHO 2008b).

Request a Copy of the Tool

Since the validation and re-launch of the fourth edition of the self-assessment tool in October 2005, the foundation has received more than 400 requests for the tool. The web page that houses the tool has also seen significant activity; in 2008 alone (from January 1 to September 10), the page had over 600 visits. The self-assessment tool is also well-referenced in the grey and peer-reviewed literature. To read more about the self-assessment tool, visit https://www.chsrf.ca/other_documents/ working_e.php. Or to request a copy for your organization, contact CHSRF by e-mail at orglearning@chsrf.ca.

Comments

Be the first to comment on this!