As part of the Saskatchewan Centre for Patient-Oriented Research (SCPOR)'s commitment to improving the experience of patient and family advisors (PFAs) who engage with local patient-oriented research teams, one of the supports offered is an evaluation that assesses the experiences and effectiveness of PFAs as research team members. In a recent evaluation project, SCPOR used an academically validated questionnaire and a survey co-designed by local PFAs. The results showed that PFAs wanted more frequent updates throughout the research project and greater role clarity. They also reported that the best part of their engagement was feeling like a valued team member.
Patient-oriented research (POR) is research that is conducted in partnership with patients, families, caregivers and communities that investigates questions that matter to them, with an overall aim to improve healthcare (SCPOR 2019b). As a Canadian Institutes of Health Research (CIHR)-funded SUPPORT (Support for People and Patient-Oriented Research and Trials) unit (CIHR 2018), the Saskatchewan Centre for Patient-Oriented Research (SCPOR) works in partnership with eight provincial stakeholders to build capacity and provide support to Saskatchewan research teams working in this field. One of these partners is the Saskatchewan Health Research Foundation (SHRF), which has collaborated with SCPOR on several funding opportunities that were intended to support research teams doing POR in the province. In 2017/2018, eight Saskatchewan POR teams received a SHRF/SCPOR Sprout Grant, designed to support their responsive, equitable and innovative POR projects (SHRF 2019).
One of the supports SCPOR offered to funding recipients was an invitation for their research teams to take part in a patient and family advisor (PFA) evaluation project. A PFA is a patient, or their family/caregiver or community member, who has lived experience relevant to the topic being researched. These individuals share their experience and expertise with the research team and help co-design the project from beginning to end. As integral team members, SCPOR wanted to capture PFA feedback on their experiences as well as gather information from the researchers about the impact PFAs have had on the projects. This initiative was developed in response to a directive from SCPOR's Patient Family Advisory Council that there be a forum for PFAs to provide comments about their experiences being involved in POR. The overall intention behind this evaluation project was to improve the experiences of the PFAs. Research ethics exemption was received from the University of Saskatchewan's Behavioural Research Ethics Board. The project was conducted with recipients of the 2017/2018 SHRF/SCPOR Sprout award (a total of eight teams). Each research team was invited to take part in the evaluation project and connect SCPOR to the PFAs who were working on their projects.
The PFA evaluation project was conducted by a working group consisting of two PFAs from SCPOR's Patient Family Advisory Council and one SCPOR staff member. The evaluation working group met regularly (via WebEx) and co-designed the project. The working group decided to use three different questionnaires to collect feedback from the PFAs from the research teams:
- Quality Involvement Questionnaire (Morrow et al. 2010),
- Saskatchewan Patient Family Advisor (PFA) Experience Survey and
- Saskatchewan Patient Family Advisor (PFA) Effectiveness Survey.
The Quality Involvement Questionnaire is an academically validated tool sourced from the Centre of Excellence on Partnership with Patients and the Public evaluation tool kit (CEPPP 2019). This questionnaire measures PFA experiences in different areas, including perceptions regarding the PFAs: ability, potential, sense of being valued, etc. The tool uses a sliding scale with values ranging from 0 (not at all) to 4 (high). The PFAs on the evaluation working group gave feedback regarding the wording of the survey tool, and it was amended before being used. One edit included the addition of an open-ended question to capture final comments from the PFA respondents.
The other questionnaires that were used were a pair of survey tools co-designed by local Saskatchewan PFAs. The Saskatchewan PFA Experience Tool is a survey completed by PFAs on the research team that uses sliding scale and open-ended questions to assess the PFAs' perception of things such as their understanding of their role, whether they felt heard and their willingness to participate again. The sliding scale ranges from 1 (strongly disagree) to 5 (strongly agree), and each question allows additional comments to be captured.
The Saskatchewan PFA Effectiveness Tool is a survey completed by the lead investigators from the research team. It includes yes/no questions asking whether the PFAs effectively represented the patient perspective and if they positively impacted the project, as well as an open-ended question for additional comments.
After choosing the survey tools that would be used, the evaluation working group, along with feedback from the Patient Family Advisory Council, decided that the surveys would be offered in two ways: via an anonymous online option and as a phone interview with a PFA from the evaluation working group. The phone interview option was for those who felt more comfortable having a conversation with a fellow PFA, and it provided an option for those who did not have access to the Internet.
When the final survey results were received, the evaluation working group met to look at the data. The results that stood out and comments from open-ended questions were collected, and themes were identified. The evaluation working group presented these findings to SCPOR's Patient Family Advisory Council, where discussions were held based on the implications of the results. A report was also shared with the research teams themselves.
Results and Key Findings
Four out of eight lead researchers (50%) and 10 out of a possible 23 PFAs (44%) completed the surveys. One team advised that they were not at a stage of the research to collect meaningful patient engagement data, as these activities had not begun yet, and agreed to be contacted later in the project.
Key results from the Quality Involvement Questionnaire (Morrow et al. 2010) show that 100% of PFA respondents felt there was potential for them to bring their own ideas and values to the research (high = 80%; medium = 20%). All the respondents believed that researchers had the right reasons for wanting to work with them (high = 70%; medium = 30%) and felt they had contributed to the research project (high = 70%; medium = 30%). When considering questions that received neutral responses, 11% of the PFAs were undecided when asked if there was a clear role in the research for them, and 10% indicated a neutral response when commenting on the extent to which they felt valued as a partner on the research team (see Table 1).
|Table 1. Results from the Quality Involvement Questionnaire|
|Survey question||Response (degree to which PFAs agreed with given question) *|
|High (score = 4)||Medium (score = 3)||Neutral (score = 2)|
|To what extent do you feel you are able to …?|
|access research resources (e.g., $, facilities, information)||40%||30%||30%|
|achieve your own goals through the research||50%||40%||10%|
|make a contribution to the research||70%||30%|
|make decisions about how to do the research||20%||60%||20%|
|express your views about research topics||70%||30%|
|discuss research issues||70%||30%|
|take on new research challenges||30%||50%||20%|
|To what extent do you feel there is potential for you to …?|
|choose the type of role you play in the research||50%||40%||10%|
|bring your own ideas and values to the research||80%||20%|
|work in ways that suit you||50%||50%|
|identify and organize your research ideas and priorities||20%||70%||10%|
|To what extent do you feel …?|
|valued as a partner (not controlled)||60%||30%||10%|
|enabled (rather than constrained)||30%||60%||10%|
|empowered (rather than exploited)||50%||50%|
|consenting (happy to be involved)/not coerced (unhappy about it)||80%||20%|
|it is acceptable that different people have different responsibilities and decisions to make about the research||80%||20%|
|Thinking about your research relationships, to what extent do you think …?|
|the researchers have the right reasons for wanting to work with you||70%||30%|
|there is sufficient funding to make engagement work||40%||50%||10%|
|you have enough information about engagement||40%||50%||10%|
|the way the researchers work with you is supportive||50%||50%|
|the way the researchers communicate with you is supportive||50%||50%|
|the types of goals that the researchers want are what you want||50%||50%|
|Thinking about the research itself, to what extent do you think …?|
|there is a clear role in the research for you||67%||22%||11%|
|the skills/experience needed for the role are clear to you||44%||56%|
|the responsibilities for the role are clear to you||67%||33%|
|you are aware of the legal and ethical "rules" for doing research (e.g., confidentiality)||67%||33%|
|Thinking about the research organization, to what extent do you think your involvement is …?|
|not just a part of a project; it is valued as part of the work of the organization||56%||44%|
|supported by research ethics and governance systems||56%||44%|
|helped because of the research structures (networks, links with other studies, etc.)||44%||44%||12%|
|noticed and recorded as part of the work of the research organization||56%||44%|
|PFAs = patient and family advisors. *Total percentage of responses for each given value.|
The Saskatchewan PFA Experience Tool found that all the PFA respondents would be willing to participate in another research project in the future (89% strongly agree; 11% agree), and all the participants found the engagement to be a productive and valuable use of their time (78% strongly agree; 22% agree). Questions that received neutral responses included reflections on adequate orientation to the role of a PFA (11% neutral) and understanding the PFA role on the research team (neutral 13%). The only question in the survey to receive a "disagree" response was regarding the following statement: "I know how/when I will be updated following this work" (11% disagree) (see Table 2).
|Table 2. Results from the Saskatchewan PFA Experience and Effectiveness Tool's scaled questions|
|Saskatchewan PFA Experience Tool|
|Strongly agree||Agree||Neither agree nor disagree||Disagree|
|Staff/researchers actively listened to the experiences and suggestions that I shared.||56%||44%|
|I was adequately oriented to the role of PFA.||44%||44%||12%|
|I understood my role and what was expected of me.||25%||63%||12%|
|I had the information I needed to feel comfortable in my engagement.||56%||44%|
|I know how/when I will be updated following this work.||11%||56%||22%||11%|
|I would be willing to participate again in the future.||89%||11%|
|This engagement was productive and a valuable use of my time.||78%||22%|
|Saskatchewan PFA Effectiveness Tool|
|Did the PFA on this team effectively represent the patient perspective?||75%||25%|
|Did engaging the PFA positively impact, and provide value to, the outcome of this event/activity?||75%||25%|
|PFA = patient and family advisor. *Total percentage of responses for each given value.|
Regarding the Saskatchewan PFA Effectiveness Tool, 75% of the researchers indicated that the PFAs on the team represented the patient perspective, and 75% stated that engaging PFAs had a positive impact and provided value to the outcome of the research.
When considering comments received from PFAs on both questionnaires, three main themes emerged:
- PFAs would like more updates regarding the status of the research project.
- PFAs would like better role clarity before they begin working with a research team so that they know what to expect.
- PFAs believed that the best part of the engagement activity was:
- being part of a team and connecting with other people and
- feeling valued.
Two themes appeared in the questionnaire responses collected from the lead investigators:
- The PFAs provided a unique perspective that had a positive impact on the research project.
- Increased role clarity for the PFAs would have improved their effectiveness on the team (see Table 3).
|Table 3. Key results from the Saskatchewan PFA Experience and Effectiveness Tool's open-ended questions|
|Saskatchewan PFA Experience Tool|
|Key themes||Supporting quotations from PFAs|
|Regular updates regarding the status of the research project||"As a PFA, I felt involved at the beginning but less involved as the time goes on. I would like to be more involved and informed about what is going on. For example: progress reports 1 or 2 times a month. That would keep us in the loop as a PFA."
"Be more informed when work is going on. Keep us in the loop."
"I am wondering when we will go out and do presentations? I wish we stayed in touch more."
"Should inform us more often than they are."
|Role clarity||"My understanding started out slowly."
"I didn't know what to expect on data collection day …"
"I wasn't sure from the beginning what would be expected, but we figured it out as we went."
|The best part of the engagement activity was:|
|Being part of a team||"To meet other people and know that other people know that there is a recognizable problem. Everybody on the team is looking for solutions to address the problem earlier. The communication with the team was good, and you felt like a part of the team."
"Networking with all the different organizations has been a highlight for me."
"Hearing other's stories and experiences and being in a comfortable environment."
"Participation and conversation with other participants."
|Feeling valued||"The project is very close to my heart, and I feel like my participation is valued."
"Contributing to an outcome … that will in turn help the targeted population gain better understanding and find solutions for the issue at hand."
|Saskatchewan PFA Effectiveness Tool|
|Key themes||Supporting quotations from lead investigators|
|Unique perspective that impacted the research project||"During the team meetings, the PFAs enrich the conversations by sharing their personal experiences. These experiences have enabled us to appreciate the nuances of the [research area] landscape."
"They were able to point out issues [with the study intervention] that I would not have been aware of."
|PFA role clarity||"Some patient family advisors don't understand their roles."|
|PFA = patient and family advisor.|
The overall reaction from the evaluation working group to the findings was positive. The reflection "that participation in the various research projects was a valuable use of the PFA's time and something they would do again" was important to both the working group and SCPOR's Patient Family Advisory Council. This response demonstrated that the PFAs were meaningfully engaging with their research teams – a particular emphasis of all POR activities.
When asked if "engaging the PFA positively impacted, and provided value to, the outcome of the research project," the majority of the lead investigators indicated that PFAs had a positive influence. The evaluation working group was disappointed, however, with the one negative response to this question. Because responses were anonymous, a follow-up conversation with the researcher was not possible; thus, it was difficult to surmise what specific issues led to this negative reply. A suggestion from the evaluation working group was for SCPOR to highlight their ongoing and continuous support to POR research teams throughout their research projects so that potential concerns and challenges could be addressed in a timely manner.
When considering the theme that PFAs desire regular updates throughout the research project, the PFAs from the evaluation working group reflected on their own experiences collaborating with research teams and postulated that perhaps new PFAs were unaware of the length of time a typical research project takes. In response, SCPOR has included this information when interacting with POR teams, specifically advising them to address this concern and to build regular PFA updates into their project plan. Examples of possible activities could include drafting/sending regular team newsletters/e-mails, deciding calendar time for holding specific update phone calls, drafting timelines during initial research team meetings and circulating meeting minutes with expected future activity dates to all team members.
Another theme that stood out in the evaluation project was the PFAs' desire for greater role clarity. SCPOR currently provides training sessions for new POR team members, including Strategy for Patient-Oriented Research training modules (SCPOR 2019a), PFA orientation sessions and instructive videos on the SCPOR website that aim to support teams in their patient engagement activities. In response to this finding, SCPOR has increased the emphasis on these resources and encouraged new PFAs to attend relevant workshops. Coaching conversations with individual teams also include reference to this finding, and researchers are asked to consider how they can support their PFA team members in this area.
When reflecting on the overall evaluation project, the PFAs from the evaluation working group agreed that it was a worthwhile endeavour and provided useful information, but the low response rate was disappointing. Suggestions for an improved response rate for future evaluation projects included SCPOR having access to a full list of all research team PFA members to communicate directly with and build supportive relationships throughout their research experience.
The evaluation working group also considered the survey tools that were used and found that the Saskatchewan PFA Experience and Effectiveness tools provided responses that had a direct and immediate impact on SCPOR's work. Although supported by findings from the Quality Involvement Questionnaire, it was felt that the local survey tools were more succinct, and the questions were easier to understand than those found in the academically validated tool. The PFAs from the evaluation working group provided this feedback based on their experiences carrying out the telephonic interviews with fellow PFAs. They were asked to clarify more questions from the academic tool, and reflections from survey participants suggested that these questions were repetitive, hard to understand and not always relevant to their experiences.
The PFAs from the evaluation working group also suggested that future evaluation projects be conducted as phone interviews with all research team PFAs and that the evaluation be framed as an informal peer check-in rather than using the anonymous online survey option.
Evaluation of both the experiences and the effectiveness of PFAs engaging with POR teams is a valuable activity that provides insight into the support needs of those PFAs. Collaboration with PFAs on the evaluation project itself allows for reflective contemplation on the process and findings, resulting in actionable suggestions that will have an impact on the experiences of all members of POR teams, especially the PFAs. Using evaluation tools co-designed by PFAs demonstrates the collaborative process in the initial phase of an evaluation project. The questions created were more relevant to those taking part in the questionnaires and were easier to understand. Having PFAs conduct evaluation interviews over the phone provides an opportunity to answer questions, develop relationships, collect richer data and offer individual support/mentoring on specific issues that may be occurring. The impact that PFAs had on the entire evaluation project, from design to implementation, data analysis and knowledge translation, was important and provided immense value.
About the Author(s)
Michelle Flowers, BAppSc, MSc, is a patient engagement specialist with SCPOR working out of the Saskatchewan Health Quality Council. Her work focuses on supporting patients and research teams in patient-oriented research projects. She can be contacted at firstname.lastname@example.org
Janet Gunderson is a patient family advisor with the SCPOR Patient Family Advisory Council and the Strategy for Patient-Oriented Research's (SPOR) Chronic Pain Network. She also volunteers with the Canadian Arthritis Patient Alliance and the Cochrane Collaboration.
Dale Hall is a patient family advisor who sits on the SCPOR Patient Family Advisory Council. He has experienced the healthcare system as both a patient and a caregiver. Dale was a member of a SCPOR team that organized a speed networking event where patients and researchers connected to discuss their ideas for research questions.
Canadian Institutes of Health Research (CIHR). 2018. SPOR SUPPORT Units. Retrieved September 20, 2019. <http://www.cihr-irsc.gc.ca/e/45859.html>.
Centre of Excellence on Partnership with Patients and the Public (CEPPP). 2019. Patient and Public Engagement Evaluation Toolkit. Retrieved July 7, 2019. <https://ceppp.ca/en/our-projects/evaluation-toolkit/>.
Morrow, E., F. Ross, P. Grocott and J. Bennett. 2010. A Model and Measure for Quality Service User Involvement in Health Research. International Journal of Consumer Studies 34(5): 532–39. doi:10.1111/j.1470-6431.2010.00901.x.
Saskatchewan Centre for Patient-Oriented Research (SCPOR). 2019a. Training Opportunities. Retrieved September 20, 2019. <https://www.scpor.ca/training-1>.
Saskatchewan Centre for Patient-Oriented Research (SCPOR). 2019b. What Is Patient-Oriented Research? Retrieved September 20, 2019. <https://www.scpor.ca/about-us-index/#what-is-por>.
Saskatchewan Health Research Foundation (SHRF). 2019. Sprout Grant. Retrieved July 7, 2019. <https://www.shrf.ca/Opportunities/Sprout-Grant>.
Be the first to comment on this!
This article is for subscribers only. To view the entire article
Note: Please enter a display name. Your email address will not be publically displayed