Healthcare Quarterly

Healthcare Quarterly 23(1) April 2020 : 34-38.doi:10.12927/hcq.2020.26141
Patient-Oriented Research

The Creation of a Tool to Measure Engagement in Patient-Oriented Research

Malori Keller, Christopher Plishka, Malcolm King, Janet Gunderson, Christine Andrews Stobart, Kate Dunn and Charlene R.A Haver


In 2017, the Saskatchewan Centre for Patient-Oriented Research (SCPOR) engaged patient and family advisors (PFAs) to review patient-oriented research grant proposals. The PFAs observed that the reviews would be less subjective if they were assessing the projects based on more rigorous criteria. Together the PFAs and SCPOR staff members developed a tool based on the Canadian Institutes of Health Research's Strategy for Patient-Oriented Research definition for patient-oriented research and the International Association for Public Participation's Spectrum of Public Participation. This article discusses the process of initiating a patient-identified project and co-creating the tool to indicate the level of patient-orientedness.


In 2011, the Canadian Institutes of Health Research (CIHR) launched the Strategy for Patient-Oriented Research (SPOR). "The vision of the Strategy for Patient-Oriented Research is to demonstrably improve health outcomes and enhance patients' health care experience through integration of evidence at all levels in the health care system" (CIHR 2011). Within the strategy was the creation of Support for People and Patient-Oriented Research and Trials (SUPPORT) units:

SUPPORT units are multidisciplinary centres of methodologists including epidemiologists, biostatisticians, social scientists, health economists and clinical trialists as well as research support staff to assist investigators design relevant research studies, conduct biostatistical analyses, manage data and biobanks, learn project management skills and meet all relevant regulatory standards. (CIHR 2011)

In 2017, the Saskatchewan Centre for Patient-Oriented Research (SCPOR) was launched to serve as Saskatchewan's provincial SUPPORT unit. The unit is a partnership of eight organizations, including the Government of Saskatchewan, eHealth Saskatchewan, the University of Saskatchewan, the University of Regina, Saskatchewan Polytechnic, the Saskatchewan Health Quality Council, the Saskatchewan Health Authority and the Saskatchewan Health Research Foundation (SHRF).


In Saskatchewan, patients are often referred to as patient and family advisors (PFAs); however, we encourage each individual to identify a preference. Understanding the individual's preference demonstrates respect and encourages engagement with a broad spectrum of individuals, bringing forward their lived experience as a patient, caregiver, family member or community representative. Furthermore, it aligns with the CIHR's Patient Engagement Framework (CIHR 2014).

SCPOR actively engages with PFAs in governance, priority setting, conducting research and knowledge translation. One key area for engagement has been trialling different approaches to engage PFAs in the review of grants and selection of projects. This has occurred within SCPOR as we sought to identify which patient-oriented projects to support with in-kind support. Similarly, our partner organization, SHRF, which is Saskatchewan's provincial funding agency that supports health research and provides funding opportunities for patient-oriented research (POR), has engaged PFAs in grant review panels for POR proposals.

PFAs on the review panel of POR grants expressed difficulty being consistent in scoring projects as there were no concrete measures or criteria for assessing the "patient-orientedness" of the research. Four PFAs who were part of the grant review panel suggested that we create a tool to grade the projects based on how closely those aligned with the CIHR definition of POR. The purpose of this project was to develop a tool to measure the degree of patient engagement in POR projects. This article outlines the process of developing this tool with PFAs and the plans for evaluating it.

The idea of creating a tool to measure the degree to which a project is patient oriented was advanced by a working group of PFAs and SCPOR staff. The working group was initiated with two PFAs and six SCPOR staff. The first draft of the tool, known as the Patient-Oriented Research Level of Engagement Tool, was created. This tool was designed based on a combination of resources available from the CIHR (2019) and the International Association for Public Participation (IAP2) ( The tool is presented in Appendix 1 (available here). It includes five criteria that are presented on the vertical axis. These criteria are patients are partners, patient-identified priority, outcomes important to patients, the team is multidisciplinary and aims to integrate knowledge into practice. The five criteria are based on wording adapted from two CIHR SPOR frameworks (CIHR 2014, 2015).

The tool scores each criterion on a continuum from 1 to 5. A score of 1 for each criterion suggests that limited patient-orientedness within the project has been achieved. A score of 5 suggests that all aspects of the project meet ideal POR criteria; for instance, patients are actively involved in all aspects of the research project; the team has representation from research, the health system and the patient community; and knowledge translation activities have been planned and approved by all members of the team. Each category should be assessed individually and summed to provide a total score. The highest possible total score for a project is 25.

Cues for the scores are adapted from the wording used in the IAP2 Spectrum of Public Participation (CIHR 2015) and describe the degree to which a patient is engaged. In doing so, the scoring uses IAP2 Spectrum of Public Participation (IAP2 2018) terms, including inform, consult, involve, collaborate and empower. On this spectrum, inform denotes the lowest level of patient engagement and empower denotes the highest level of patient engagement.

An overview of the adapted definitions for each of these terms is presented in Table 1.

Table 1. Definitions for patient engagement adapted from the IAP2 Spectrum for Public Participation
  Inform Consult Involve Collaborate Empower
Patient engagement To provide patients with balanced and objective information to assist them in understanding the problem and potential solutions To obtain patients' feedback on the research project and decision To work directly with patients throughout the process to ensure that their viewpoints are understood and considered To partner with patients in each aspect of the research project To place final decision making in the hands of the patients
IAP2 = International Association for Public Participation.


Integrating the Tool in Practice

Once the tool was created, the working group developed two case studies to test its application. The case studies were based on the previous requests that SCPOR staff had received for support. Each working group member reviewed the case studies and assigned scores. Feedback from these trial case studies was integrated into the tool by the collaborative team. One key piece of feedback obtained was that to use the tool to score patient-oriented projects, the application template needs to ask questions for each of the five categories that a project will be assessed on. Early application of the tool suggested that if questions were not asked of each of the five criteria, then information is missing, and scores are therefore lower and may not accurately depict the level of patient engagement. An overview of SCPOR's recommended questions for application templates can be found in Table 2.

Table 2. Recommended questions for application templates
Criteria Recommended question
Patients are partners Explain how patients have or will be engaged as partners in the research project
Patient-identified priority Explain how patients have identified or will help identify the research priority/priorities
Outcomes important to patients Explain how your project will identify and address outcomes that are important to patients
Team is multidisciplinary Explain the roles of team members and stakeholders in the project
Aims to integrate knowledge into practice Explain how the project aims to integrate knowledge into practice. Describe intended end-of-grant knowledge translation plans and integrated knowledge translation plans. Describe how you will co-develop knowledge translation plans with patients, communities and stakeholders.


Another area of feedback was the need for orientation to the tool. Orientation was deemed necessary because reviewers had varying definitions of POR, which resulted in a variance of scores for the same project. In response, SCPOR developed instructions and an orientation PowerPoint presentation and workshop. PFAs were an integral part of the team who reviewed and co-built the orientation content.

The orientation is recommended for anyone using the tool, including grant applicants (e.g., principal investigators) and those who will be on peer review or grant review committees. The orientation covers a number of topics, including definitions put forward by CIHR SPOR, such as patient, SPOR, POR, CIHR and knowledge translation (KT), as well as definitions put forward by IAP2, such as inform, consult, involve, collaborate and empower. After reviewing definitions, participants are guided through three mock examples developed by SCPOR staff. Working through these examples is first done as a team, then in small groups and, finally, individually. Each example is reviewed and discussed by the full group once it is completed.

Internal Experience and Application

Once the tool and orientation materials were completed, the tool was posted on SCPOR's website and integrated into the SCPOR project intake process (for in-kind support). It was also adopted by partner organizations, including SHRF, the College of Medicine at the University of Saskatchewan and Saskatchewan Polytechnic, in their POR grants and traineeships.

Saskatchewan Health Research Foundation

In this application, SHRF used a modified version of the tool for PFA peer reviewers to assess and score each POR grant proposal. The foundation required that proposals score 2 or more on each criterion to be considered for funding and used the tool score as a weighted contributor to the proposal's overall score. Proposals were also scored for scientific rigour by an academic peer review panel.

College of Medicine

In this application of the tool, graduate students applying for funding to undertake POR submitted proposals to the College of Medicine at the University of Saskatchewan. Students were required to use the tool as a writing guide to describe their proposed project within their application. A group of three PFAs and three SCPOR staff who had participated in the tool orientation used the tool to review the traineeship applications and provided an ordinal ranking of POR proposals. Projects were scored out of a possible score of 25. This ranking was used by the College of Medicine and SCPOR to allocate four POR traineeships – two at the master's level and two at the PhD level.

Saskatchewan Polytechnic

In this application of the tool, instructors applying for release time to do POR submitted a proposal to a Saskatchewan Polytechnic review committee. Prospective researchers were oriented to the tool and were encouraged to use it in their proposal writing to better define patient engagement strategies and write more comprehensive descriptions of their POR projects.

SCPOR in-kind support

At SCPOR, the tool is used to evaluate projects in cases where the research team is making a request for in-kind support. SCPOR has developed an internal policy in which each project submitted for support is evaluated by a SCPOR staff member and two PFAs who are members of the SCPOR detailed support request committee (the committee that evaluates requests for in-kind support). The scoring is used to guide the level of coaching and mentorship that the research team is provided and to help teams attain an appropriate level of patient engagement.

Other applications of the tool

The tool is used for training, coaching and mentorship to those engaged in POR in Saskatchewan. SCPOR requires supported research teams and SCPOR-affiliated researchers to attend the nationally created SPOR Foundations in Patient-Oriented Research Modules 1 and 2. SCPOR has integrated local content into these modules about the Patient-Oriented Research Level of Engagement Tool. The tool is accessible on the SCPOR website (

In addition, SCPOR staff uses the tool in meetings and consultations with researchers and research teams. In this context, the tool provides an introduction to patient engagement. Depending on the researcher or research team's understanding of POR, the tool can guide conversations on all elements of the POR process. However, it is noted that other POR resources exist that may provide more detailed guidance for engaging patients in research. Researchers and research teams are guided to these tools when appropriate. These tools are discussed in more detail in the Limitations section.


It should be noted that the Patient-Oriented Research Level of Engagement Tool serves a specific purpose: to measure the degree to which a project meets the definition of POR, as defined by the CIHR. The tool does not describe the quality of a research project. Additional tools and guidelines have been developed to aid and evaluate other aspects of POR. These include reporting patient engagement in research (Staniszewska et al. 2017), fostering patient engagement in research (Hamilton et al. 2018a), engaging patients in data analysis of qualitative mental health research (Jennings et al. 2018), assessing patient experience on research teams (Hamilton et al. 2018b) and ideating how patients can be involved in research (Kirwan et al. 2017; PCORI 2014). The Patient-Oriented Research Level of Engagement Tool makes a unique contribution to this body of literature.

A further limitation is that the tool has not been developed to assess the degree to which a project adheres to best practices for engaging with communities, especially Indigenous communities. During the development of the tool, the group collaborated with SCPOR's Indigenous Research & Engagement Platform. The working group intended to capture culturally appropriate best practices for engagement of Indigenous peoples and communities. However, after deliberation, we determined that it would be more appropriate to create a separate tool to guide the assessment of Indigenous engagement.


The next steps include an evaluation of the tool for reliability and validity. This will be done using a mixed-methods approach. To assess reliability, the research team will collect scores each time the tool is used internally to assess a POR project. These data will be collected from each individual who scores a project using the tool. Data will include scores for each of the five categories and the total score. Because all current applications require that multiple raters assess each project, this information will be used for statistical analysis of inter-rater reliability.

Validity will be assessed in two ways. First, users will receive a follow-up survey focused on how easy criteria are to understand, how useful they found the tool and the degree to which the tool is a good measure of whether a research project is patient oriented. In addition, once reliability has been established, the tool will be sent to experts in the field of POR for feedback.

This data collection plan was submitted to the University of Saskatchewan Research Ethics Board and was considered exempt as this work falls within the realm of quality improvement.


The Patient-Oriented Research Level of Engagement Tool has had a positive impact on patient engagement and has provided objective criteria to assess patient engagement in research projects. The idea for the project was generated by patient reviewers, who then collaboratively participated in all steps, including creating the tool, trialling the tool, developing the orientation, presenting the tool at national conferences, revising the tool and now evaluating and publishing the tool. The PFA voice has been instrumental in this project and will continue to be impactful as the tool is evaluated and used across multiple avenues for POR.

About the Author(s)

Malori Keller, MPA, is the Patient Engagement & Empowerment Platform lead at the Saskatchewan Centre for Patient-Oriented Research. Her work focuses on supporting the advancement of patient engagement and patient- and family-centred care in health research and health system improvement. She can be reached by e-mail at

Christopher Plishka, BA, MPH, is a methods platform facilitator at the Saskatchewan Centre for Patient-Oriented Research. His work focuses on identifying and addressing the needs of researchers engaging in patient-oriented and community-engaged research.

Malcolm King, PhD, is the scientific director at the Saskatchewan Centre for Patient-Oriented Research. Dr. King is also a former scientific director of the CIHR Institute of Aboriginal Peoples' Health, where he spearheaded the development of a national health research agenda aimed at improving wellness and achieving health equity for people from First Nations Métis and Inuit in Canada.

Janet Gunderson, BEd, has been a patient and family advisor with the Saskatchewan Centre for Patient-Oriented Research and the Chronic Pain Network for two years.

Christine Andrews Stobart, EdD, is a Knowledge Translation and Capacity Development Platform lead at the Saskatchewan Centre for Patient-Oriented Research. Dr. Stobart is an educator with a background in instructional leadership and mentorship. Currently, she develops and facilitates training opportunities for researchers and trainees pursuing patient-oriented research.

Kate Dunn, MPH, is a knowledge translation specialist at the Saskatchewan Centre for Patient-Oriented Research. Kate received the Knowledge Translation Professional Certificate from the University of Toronto and provides knowledge translation support to research teams.

Charlene R.A. Haver, PhD, is a Methods Platform lead at the Saskatchewan Centre for Patient-Oriented Research. Dr. Haver is also an exercise physiologist, serves on the Research Ethics Board for the local health authority and has experience in many research methodologies, including pragmatic trials, surveys, realist methodology and program evaluation.


Canadian Institutes of Health Research (CIHR). 2011, August. Canada's Strategy for Patient-Oriented Research: Improving Health Outcomes through Evidence-Informed Care. Retrieved May 6, 2011. <>.

Canadian Institutes of Health Research (CIHR). 2014. Strategy for Patient-Oriented Research: Patient Engagement Framework. Retrieved March 18, 2019. <>.

Canadian Institutes of Health Research (CIHR). 2015. Strategy for Patient-Oriented Research: Capacity Development Framework. Retrieved March 18, 2019. <>.

Canadian Institutes of Health Research (CIHR). 2019, June 27. Strategy for Patient-Oriented Research. Retrieved March 18, 2019. <>.

Hamilton, C.B., A.M. Hoens, C.L. Backman, A.M. McKinnon, S. McQuitty, K. English et al. 2018a. An Empirically Based Conceptual Framework for Fostering Meaningful Patient Engagement in Research. Health Expectations 21(1): 396–406. doi:10.1111/hex.12635.

Hamilton, C.B., A.M. Hoens, S. McQuitty, A.M. McKinnon, K. English, C.L. Backman et al. 2018b. Development and Pre-Testing of the Patient Engagement in Research Scale (PEIRS) to Assess the Quality of Engagement from a Patient Perspective. PLoS One 13(11): e0206588. doi:10.1371/journal.pone.0206588.

International Association for Public Participation (IAP2). 2018. IAP2 Spectrum of Public Participation. Retrieved March 18, 2019. <>.

Jennings, H., M. Slade, P. Bates, E. Munday and R. Toney. 2018. Best Practice Framework for Patient and Public Involvement (PPI) in Collaborative Data Analysis of Qualitative Mental Health Research: Methodology Development and Refinement. BMC Psychiatry 18(1): 213. doi:10.1186/s12888-018-1794-8.

Kirwan, J.R., M. de Wit, L. Frank, K.L. Haywood, S. Salek, S. Brace-McDonnell et al. 2017. Emerging Guidelines for Patient Engagement in Research. Value in Health 20(3): 481–86. doi:10.1016/j.jval.2016.10.003.

Patient-Centered Outcomes Research Institute (PCORI). 2014, February 4. Engagement Rubric for Applicants. Retrieved March 18, 2019. <>.

Staniszewska, S., J. Brett, I. Simera, K. Seers, C. Mockford, S. Goodlad et al. 2017. GRIPP2 Reporting Checklists: Tools to Improve Reporting of Patient and Public Involvement in Research. BMJ 358: j3453. doi:10.1136/bmj.j3453.


Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed