Healthcare Quarterly

Healthcare Quarterly 21(Special Issue) December 2018 : 31-37.doi:10.12927/hcq.2018.25636
Integrating Research And Education

Evaluating Patient, Family and Public Engagement in Health Services Improvement and System Redesign

Julia Abelson, Anya Humphrey, Ania Syrowatka, Julia Bidonde and Maria Judd

Abstract

As efforts to actively involve patients, family members and the broader public in health service improvement and system redesign have grown, increasing attention has also been paid to evaluation of their engagement in the health system. We discuss key concepts and approaches related to evaluation, drawing particular attention to different and potentially competing goals, stakeholders and epistemological entry points. Evaluation itself can be supported by an increasing number of frameworks and tools, matched to the relevant purpose and approach. The patient engagement evaluation field faces several challenges, including the need for greater specification of both the form and the context of engagement, the need to balance the measurement imperative with the relational aspects of care and the need for supportive organizations with the capacity and commitment to undertake high-quality engagement and its evaluation. 

 


Évaluation de l’engagement du patient, de sa famille et du citoyen à l’amélioration des services de santé et au réaménagement des système

Julia Abelson, Anya Humphrey, Ania Syrowatka, Julia Bidonde and Maria Judd

Résumé 

Tandis que les efforts visant à activement faire participer le patient, les membres de sa famille et le grand public à l’amélioration et au réaménagement des systèmes de santé se sont intensifiés, une attention croissante a également été accordée à l’évaluation de leur engagement au système de santé. Nous abordons d’importants concepts et approches liés à l’évaluation, en attirant une attention particulière aux divers objectifs, parties prenantes et points d’entrée épistémologiques éventuellement en concurrence. L’évaluation elle-même peut s’appuyer sur un nombre croissant de cadres et d’outils adaptés à l’objectif et à l’approche en question. Le domaine de l’évaluation de l’engagement du patient doit relever de nombreux défis, à savoir le besoin de préciser les modalités et le contexte de l’engagement, le besoin d’équilibrer l’impératif de la mesure avec les aspects relationnels des soins et l’aspiration des organismes disposant de la capacité et de la volonté nécessaires à concrétiser et à évaluer ce projet en engagement de grande qualité.  

Key Messages

  1. An increasing focus on evaluation of patient engagement activities advances the science of the field.
  2. Robust frameworks for evaluation should guide organizations to precisely specify the purpose, form and context of their engagement activities so they can understand what is working and what is not.
  3. High-quality engagement and its evaluation aligns with the core features of rapid learning health 

Introduction

Many – perhaps most – of us who choose to volunteer to improve healthcare have had a bad experience. I am referring to something that has made a person feel dismissed, or ignored, or invisible, or mistreated, or otherwise hurt in a way that could have been avoided. We all expect and deserve to be treated considerately when we or our family members are ill. Because that is largely what does happen, when it doesn’t happen that takes us by surprise, when we are at our most vulnerable, and exacerbates an already traumatic situation. So that kind of wound is often the back story to a Patient and Family Advisor’s interest in getting involved. That means that not only do we have a powerful motivation to make things better for the next person, we also need to repair our own relationship with the medical world. And the only way to do that is to restore a respectful, kindly, mutually trusting connection with other people. When that is established, not only do all the participants benefit personally, but that tone radiates out from their committee or project like a fractal. So that’s what I really want to see measured because I think it is vitally important.
– Anya Humphrey, patient and family advisor

Health system organizations in Canada and around the world continue to expand their efforts to actively involve patients, family members and the broader public in health service improvement and system redesign initiatives (Barello et al. 2012; Carman et al. 2013; Richards et al. 2013). Evidence suggests that these efforts have the potential to translate into organizational and system improvements in quality, safety and patient experience, but the mechanisms through which these occur are not well understood (Bombard et al. 2018). This is due, in part, to the many and varied approaches used to engage patients, family members and the public in health services improvement and system redesign initiatives – approaches that typically range from more traditional consultation methods to more inclusive partnership, shared leadership and co-design models (Health Canada 2000). In the young field of patient and family engagement, the task of evaluation and to moving beyond a focus on practice (i.e., the everyday work of engagement) to what some refer to as the science of engagement (Anderson and McCleary 2016) has only recently begun. Questions such as how current approaches are working, which methods are most effective for which types of problems and whether any of this is making a difference have become more commonplace as governments, organizations, researchers and patient organizations seek to demonstrate the benefits, value or return on investment of patient and family engagement.

As both the practice and the science of engagement grow and mature, we offer guidance to health system leaders that will support their patient, family and public engagement efforts. Our specific aims in this paper are threefold: (1) to seek clarity about the different goals and meanings attached to evaluation in relation to patient, family and public engagement; (2) to review current frameworks, tools and approaches for supporting evaluation in this area; and (3) to highlight key challenges faced with suggested strategies for addressing these. Throughout the article, we primarily use the term patient and family engagement, defined as the “involvement of patients and/or family members in decision-making and active participation in a range of activities (e.g., planning, evaluation, care, research, training, and recruitment) … which involves collaboration and partnership with professionals” (Baker et al. 2016). We also use the terms public, citizens and service users in some places to reflect the broader construct of “the public” that may be relevant to health service improvement initiatives and decision-making at the system and policy levels.

An Evaluation Primer for the Engagement Field

Meanings, motivations and measures

Approaching the task of evaluation in the context of patient, family and public engagement requires awareness and receptivity to the different epistemologies of evaluation that are situated in contributing disciplines. Familiar labels such as program, realist and impact evaluation have different underlying theories. Equally, terms such as effectiveness and impact convey different meanings to engagement professionals, scientists and funders. As evaluation efforts expand and proliferate, sensitivity to these different entry points to the field will be needed, as well as toward those who may challenge the evaluation effort altogether given its intrinsic value.

Seeking clarity and agreement about the purpose for evaluation can be a helpful way forward. Just as engagement requires clarity about why (we are engaging), with whom and how (we will do this), the same principles apply to its evaluation. In Table 1, we identify four overarching goals for public and patient engagement evaluation. The research community tends to be preoccupied with summative evaluation and the goal of contributing to the evidence base about what works and under what conditions for the purposes of refining engagement methods. In contrast, organizational staff and patient partners may be more focused on formative evaluation and the goal of improving the quality of their engagement practices. Funders and health system leaders, in turn, may be interested in evaluation from an accountability and value-for-money perspective. Within organizations, this can often take the form of assessing whether organizational requirements (or expectations) for engagement have been met. They and other key stakeholders will be interested in these questions as well as the evaluation of longer-term outcomes, such as what and how the engagement activity, process or strategy contributes to health system planning, system redesign and quality of care.

TABLE 1. Overarching goals for patient, family and public engagement evaluation (linked to relevant stakeholders and examples)

Evaluation goal Key stakeholders Evaluation example
Contribution to engagement science
Researchers
Patient partners
Health system organizations
Summative evaluations to improve the evidence base around engagement methods 
Improvement of engagement practices
Health system organizations
Patient partners
Formative evaluations to track and refine organizational approaches to engagement
Accountability
Funders
Health system leadership
Formative and summative evaluations demonstrating that organizational requirements for engagement have been met
Linking level and quality of engagement to outcomes All of the above Formative and summative evaluations linking meaningful engagement to changes in delivery of healthcare or health outcomes

 

 

The engagement literature is often criticized for offering little in the way of tangible evidence about what works or what added value engagement provides (Conklin et al. 2015; Mockford et al. 2012). This is due in part to the emphasis typically given to evaluating the procedural aspects of engagement rather than its outcomes (Abelson and Gauvin 2006; Rowe and Frewer 2005). Process measures typically focus on the execution or implementation of the engagement activity (e.g., whether participants were supported adequately to participate, whether they felt that they were able to contribute meaningfully or whether the objectives of the activity were clearly communicated). In contrast, outcome measures focus on changes that have taken place as a result of the engagement activity or process. These may include increased knowledge and capacity of patient and family members or a more comprehensive and accountable approach to service planning. In the longer term, they may include services that are more responsive to patients’ and family members’ needs and priorities or an improved patient experience resulting from patients being directly involved in the redesign of a service. An example of an outcomes-oriented approach to patient engagement is a study comparing patients’ and providers’ priorities for healthcare improvement for chronic care in Quebec (Boivin et al. 2014). Using a patient engagement intervention involving interaction between patients and providers (compared to a control that had no patient involvement), the healthcare priorities of patients and providers (the outcome being measured) were found to converge with each other over the course of the engagement and to differ significantly from those of professionals alone (in the control group).

Supporting the Evaluation of Patient, Family and Public Engagement

Current frameworks and tools

Health system organizations are increasingly supported by a broad array of engagement frameworks and tools that have been developed by researchers, engagement practitioners and patient partners (Abelson et al. 2016; Carman et al. 2013; Frampton et al. 2017; Hamilton et al. 2018; Health Quality Ontario 2017). These resources provide the foundation for rigorous evaluation in their specification of the key dimensions of engagement: (1) the principles that should guide these efforts (e.g., partnership, learning, responsiveness, respect); (2) the levels and domains at which engagement is embedded (e.g., governance, program and service design and policy); and (3) the range of approaches or methods used (e.g., consultation, deliberation or full collaboration).

Despite a rich array of frameworks, the development of structured evaluation tools has progressed more slowly, mostly through unpublished, project-specific instruments, limiting opportunities for comparison and mutual learning across engagement initiatives (Boivin et al. 2018). 

Recently, more concentrated efforts have been made to synthesize the evaluation literature, map the dimensions of high-quality engagement on to measurable outcomes and develop generic evaluation tools and instruments (Dudley et al. 2015; Esmail et al. 2015; Forsythe et al. 2018; Gibson et al. 2017). Much of this work has focused on the evaluation of patient engagement in the health research arena, but there are several promising developments in the health services improvement and health system decision-making field, which are discussed in the following sections.

Selected examples

As the number of evaluation frameworks and tools expands, those seeking to evaluate their engagement efforts will need to choose carefully among these resources. These decisions should be guided by clarity about evaluation goals, methods and perspectives. In the sections that follow, we provide examples from our own evaluation work in this area, highlighting the different goals, methods and perspectives featured.

In 2014, the Canadian Foundation for Healthcare Improvement (CFHI) approached the evaluation of patient and family engagement in quality improvement (QI) with the goal of understanding how well patient and family advisors integrated into QI teams as part of a Partnering with Patients and Families Collaborative (CFHI 2016). They aimed to build capacity and enhance organizational culture for partnering with patients and families to improve quality across the healthcare continuum. A mixed-methods approach was used to evaluate the social dynamics, experience and effectiveness of QI teams. Social dynamics were evaluated using the International Association for Public Participation (IAP2) Spectrum and Social Network Analysis (SNA) (IAP2 2015; Valente 2010). Team members provided self-reported assessments of the levels of engagement from the IAP2 Spectrum that best described their interactions with key people involved in the QI project (e.g., inform, consult, involve, collaborate or empower). This information was also used to build a network map showing the positions of the patient and family advisors within the teams – in particular, the number of connections with other team members. Team experience and effectiveness were measured using questions adapted from existing survey instruments: the Team Effectiveness Instrument, the Primary Health Care Team Climate Survey and the Assessment of Interprofessional Team Collaboration Scale (Orchard et al. 2012; Poulton and West 1999; Shortell et al. 2004). The responses provided insight into how well teams were integrating and working together. Focus group interviews and online surveys using open-ended questions with patient and family advisors and other team members were used to better understand the experience of partnering as a part of a QI project.

The overarching goals of this evaluation were: (1) to contribute to engagement science with the specific aim of improving future patient and family engagement practices within CFHI collaboratives and programs and (2) to link the level and quality of patient and family engagement with outcomes – in particular, to understand the impact of high-quality patient and family engagement on the teams’ progress through their QI project. Twenty-two teams were supported to engage patients and family members in the design, implementation and evaluation of a QI project. Based on this work, CFHI has developed tip sheets on how to engage patients and families in building high-quality improvement initiatives from both the healthcare provider and patient perspectives (CFHI 2018a; CFHI 2018b).

Supporting Patient and Family Engagement Using the Public and Patient Engagement Evaluation Tool

In 2015, the Public and Patient Engagement Evaluation Tool (PPEET) (McMaster University 2018) was launched as a simple-to-administer tool intended for use by a wide range of health system organizations to assess the quality and impacts of engagement, with the goal of contributing to both the practice and the science of public and patient engagement. Developed through an iterative, collaborative process involving researchers and practitioners across the country, the tool is structured around core principles of quality engagement informed by a synthesis of published and grey literature (Abelson et al. 2016). The tool consists of three questionnaires aimed at evaluation from the following perspectives: (1) those participating or partnering in engagement activities and processes (patient contributors and partners); (2) those responsible for the planning, execution or sponsoring of engagement activities within organizations (engagement practitioners and users); and (3) those providing the leadership and capacity for public and patient engagement within their organizations (organizational leadership). The tool was recently subjected to additional feasibility testing in seven health system organizations in Ontario in collaboration with staff and patient partner representatives from each organization. The results of this feasibility testing have informed tool modifications through a revised PPEET (launched in August 2018) that focus on increased tailoring for different respondent groups, the creation of separate evaluation modules for different types and stages of engagement (e.g., one time versus ongoing and planning versus implementation) and achieving greater balance between open-ended and scaled questions with opportunities for more in-depth follow-up.

A searchable online resource of engagement evaluation tools

Another recent contribution to the evaluation field is an open-access online evaluation toolkit resource (CEPPP 2018) that features 27 evaluation tools and instruments searchable by user type (whether you are a researcher, a patient partner or a healthcare organization). Each has been assessed for scientific rigour, comprehensibility, usability and the involvement of patients in its development. The toolkit itself was developed through a Canadian collaboration of evaluation tool users from across the country, including patient partners, engagement practitioners and researchers.

A systematic review carried out in conjunction with the development of the toolkit yielded two key areas for improvement: (1) the scientific rigour with which these evaluation tools were developed and (2) the level of patient and/or public engagement in their design and reporting (Boivin et al. 2018). These findings signal some important challenges that lie ahead as the science and practice of patient engagement continue to evolve.

Evaluation Challenges and Strategies for Addressing Them

Specifying the purpose, dimensions and context of engagement

A core principle of evaluation is the need to be clear about the dimensions or attributes of the concept, phenomenon or activity being evaluated. The evaluation of patient, family and public engagement faces considerable challenges on this front. The engagement often takes many forms (different types of contributions on an ad hoc or ongoing basis); it involves different people in different roles (advisors, partners, committee members) and occurs at different levels (e.g., governance, project based) and over different time periods (e.g., weeks, months or years). As a result, patient, family and public engagement needs to be understood as a complex and dynamic set of context- and process-dependent activities that require unpacking. As Staley (2015) pointed out:

… we need to precisely define the form it takes, paying close attention to the context and the detailed mechanism, rather than using a loose definition of “public involvement” that in fact describes many different types of activity.

Through more precise specification of the purpose, form and context of engagement, organizations will be in a stronger position to know what is working, what is not working and what contributed to things going well or not.

Balancing the measurement imperative with the relational aspects of care

Engagement professionals and evaluation researchers often prefer questionnaires and survey instruments to collect information about engagement activities. Although these tools allow for the efficient collection of data, they can be limited in what they are able to capture about what really matters to people and may be viewed negatively when used on their own, without opportunities for personal interactions (Abelson et al. 2018). A complementary approach to collecting both quantitative and qualitative evaluation data provides a richer portrait of the public and patient engagement experience and mirrors the relational aspects of care that are so central to patient- and family-centred health systems.

There are some situations that cannot be addressed by any survey, no matter how carefully designed it is. So in order to hear about them, we have to talk to people.

– Anya Humphrey, patient and family advisor

Evaluation requires organizational readiness, capacity and commitment

Just as high-quality engagement requires a major commitment from organizations, its evaluation requires the same level of commitment. Organizations that have not articulated clear goals for their engagement efforts or a coherent engagement program or strategy will struggle to meaningfully evaluate their efforts. In fact, the lack of a coherent engagement program and supporting organizational capacity is often revealed in the early stages of evaluation. Commitment to engagement and its evaluation is more than simply putting a patient and family advisory council or a patient advisors program in place and administering the occasional questionnaire or set of interviews to see how things are going. Organizations need to go beyond this “virtue signalling (Johannesen 2017) to carefully attend to the tasks of what is being sought through the evaluation, what are the most appropriate methods for carrying it out, and with whom and how they will share the results – all key considerations, just as they are for any engagement process or activity. This requires considerable organizational readiness and capacity. As more and more organizations are mandated to engage with patients, family members and the public, the degree to which this capacity currently exists and can be sustained over time will need to be addressed. Creative and meaningful collaboration with university-based researchers and knowledge organizations may provide fruitful avenues for addressing these capacity gaps.

The concept of engagement-capable environments (addressed in another paper in this issue) should consider the inclusion of evaluation as one of its core attributes (Baker and Denis 2011; Baker et al. 2016). Receptivity to the involvement of relevant stakeholders in the evaluation process is another important consideration for organizational readiness. Recent advances in the evaluation field support collaborative efforts that emphasize partnerships between evaluation experts and key stakeholders such as patients, family members, health professionals and decision-makers (Gilbert and Cousins 2017). These principles also align with the core features of rapid-learning health systems and their emphasis on patient needs, perspectives and aspirations (focused on improving care experiences); timely data and evidence, enabled through a culture of, and competencies for, learning and improvement (Lavis et al. 2018).

Even in the absence of the more systematic approaches to conducting and supporting evaluation described above, health system leaders should also recognize the intangibles associated with evaluation. The introduction of an evaluation tool within an organization can play an influential, consciousness-raising role even without collecting any data and take some steps toward modelling a caring approach within organizations.

Remember why we are doing this

Patient- and family-centred care is about putting human needs ahead of those of the system; it is about taking care of people. This relational piece can be the most difficult to evaluate but in many ways is the most important. If the quality of the engagement in health service improvement and system redesign initiatives is a good indicator of how patients and families are treated across an organization, then evaluating this experience is a great opportunity to gain insights into what is going on across the larger organization. The problems that patients and families encounter are less about the people and more about the fact that the health system does not ask how things are going or does not ask this in appropriate ways. Attending to this is the heart of patient and family engagement. If we want to know how well it is working, we have to ask.

Conclusion

We have articulated key concepts and summarized the current state of evaluation in the context of patient, family and public engagement in health services improvement and system redesign. The focus to date in this nascent field has primarily been on understanding how to engage each of these perspectives in various roles and levels within organizations and health systems and how to determine which approaches work best in different contexts. This is a necessary first step to evaluating the links between quality engagement and care delivery and health outcomes. As the engagement landscape shifts, future efforts will also need to consider the interactions between these different perspectives (e.g., patient/user and the broader public). If the recent growth in this area is any indication, we expect considerable progress to be made in addressing these important evaluation questions over the coming years. Careful attention will be needed to ensure that all interested parties are meaningfully involved in this work and that high-quality engagement and evaluation efforts are adequately supported at the organization, system and policy levels. 

 

About the Author

Julia Abelson, PhD, is a professor in the Department of Health Research Methods, Evidence and Impact at McMaster University. Her research program focuses on supporting policy makers and health system organizations to establish and maintain high-quality public and patient engagement structures and processes.

Anya Humphrey is a retired psychotherapist. Motivated by the deaths of her husband and son, she is a patient/family advisor in the healthcare system, with the goal of improving care for critically ill and dying patients and their families.

Ania Syrowatka, PhD, is a health system impact postdoctoral fellow at the Canadian Foundation for Healthcare Improvement (CFHI) and the University of Toronto. Ania is a health services researcher with interests in knowledge translation, quality improvement and patient engagement.

Julia Bidonde, PhD, is a health research scientist whose work focuses on evidence synthesis and methods, chronic illness and vulnerable populations. Julia is currently a CIHR health system research fellow documenting and evaluating patient involvement’s impact on health technology management processes.

Maria Judd, BScPT, MSc, is vice president, CFHI. She previously led the Patient and Citizen Engagement portfolio at CFHI. Maria’s patient-first perspective has evolved from her diverse roles and experiences within the health system and guides her work.

References

Abelson, J. and F.-P. Gauvin. 2006. Assessing the Impacts of Public Participation: Concepts, Evidence and Policy Implications. Retrieved July 26, 2018. <http://www.ipea.gov.br/participacao/images/pdfs/abelson%20and%20gauvin_assessing%20pp%20impacts_2006.pdf>.

Abelson, J., S. Kandasamy, K. Burrows and L. Tripp. 2018. “Supporting High-Quality Public and Patient Engagement: Assessing the Feasibility of the Public and Patient Engagement Evaluation Tool (PPEET).” Presentation at the Canadian Association of Health Services and Policy Research Conference, May 30, 2018, Montreal, QC.

Abelson, J., G. Wilson, K. Li, K. Shields, C. Schneider and S. Boesveld. 2016. “Supporting Quality Public and Patient Engagement in Health System Organizations: Development and Usability Testing of the Public and Patient Engagement Evaluation Tool.” Health Expectations 19(4): 817–27. doi:10.1111/hex.12378.

Anderson, M. and K. McCleary. 2016. “On the Path to a Science of Patient Input.” Science Translational Medicine 8(336): 11. doi:10.1126/scitranslmed.aaf6730.

Baker, G.R. and J.-L. Denis. 2011. “Patient Engagement and System Transformation.” Presentation to Patient Engagement Workshop, Edmonton, AB, November 2011.

Baker, G.R., M. Judd, C. Fancott and C. Maika. 2016. “Creating ‘Engagement-Capable Environments’ in Healthcare.” In G.R. Baker, M. Judd and C. Maika, eds. Patient Engagement: Catalyzing Improvement and Innovation in Healthcare (pp. 11–34). Toronto, ON: Longwoods Publishing Corporation.

Barello, S., G. Graffigna and E. Vegni. 2012. “Patient Engagement as an Emerging Challenge for Healthcare Services: Mapping the Literature.” Nursing Research and Practice 2012: 905–34.

Boivin, A., P. Lehoux, R. Lacombe, J. Burgers and R. Grol. 2014. “Involving Patients in setting Priorities for Healthcare Improvement : A Cluster Randomized Trial.” Implementation Science 9: 24.

Boivin, A., A. L’Espérance, F.-P. Gauvin, A. Macaulay, P. Lehoux and J. Abelson. 2018. “Patient and Public Engagement in Research and Health System Decision-Making: Systematic Review of Evaluation Tools.” Health Expectations 18(5): 1114–26. doi:10.1111/hex.12084.

Bombard, Y., G.R. Baker, E. Orlando, C. Fancott, P. Bhatia, C. Casalino et al. 2018. “Engaging Patients to Improve Quality of Care: A Systematic Review.” Implementation Science 13(1): 98. doi:10.1186/s13012-018-0784-z.

Canadian Foundation for Healthcare Improvement (CFHI). 2016. “Partnering with Patients and Families Collaborative.” Retrieved April 4, 2018. <https://www.cfhi-fcass.ca/WhatWeDo/recent-programs/partnering-with-patients-and-families-collaborative>.

Canadian Foundation for Healthcare Improvement (CFHI). 2018a. How to Successfully Engage Patients and Families in Building Quality-Improvement Initiatives: 10 Insights from Healthcare Providers and Leaders. Retrieved April 4, 2018. <https://www.cfhi-fcass.ca/sf-docs/default-source/patient-engagement/pe-patient-tip-sheet-from-providers-e.pdf?sfvrsn=1456a944_2>.

Canadian Foundation for Healthcare Improvement (CFHI). 2018b. How to Successfully Engage Patients and Families in Building Quality-Improvement Initiatives: 10 Lessons Learned from Patient and Family Advisors. Retrieved April 4, 2018. <https://www.cfhi-fcass.ca/sf-docs/default-source/patient-engagement/pe-patient-tip-sheet-from-advisors-e.pdf?sfvrsn=1956a944_2>.

Carman, K.L., P. Dardess, M. Maurer, S. Sofaer, K. Adams, C. Bechtel et al. 2013. “Patient and Family Engagement: A Framework for Understanding the Elements and Developing Interventions and Policies.” Health Affairs 32(2): 223–31. doi:10.1377/hlthaff.2012.1133.

Centre of Excellence on Partnership with Patients and the Public (CEPPP). 2018. Patient and Public Engagement Evaluation Toolkit. Retrieved July 26, 2018. <https://ceppp.ca/en/our-projects/evaluation-toolkit>.

Conklin, A., Z. Morris and E. Nolte. 2015. “What Is the Evidence Base for Public Involvement in Health-Care Policy? Results of a Systematic Scoping Review.” Health Expectations 18(2): 153–65. doi:10.1111/hex.12038.

Dudley, L., C. Gamble, J. Preston, D. Buck, The EPIC Patient Advisory Group, B. Hanley et al. 2015. “What Difference Does Patient and Public Involvement Make and What Are Its Pathways to Impact? Qualitative Study of Patients and Researchers from a Cohort of Randomised Clinical Trials.” PLoS ONE 10(6): e0128817. doi:10.1371/journal.pone.0128817.

Esmail, L., E. Moore and A. Rein. 2015. “Evaluating Patient and Stakeholder Engagement in Research: Moving from Theory to Practice.” Journal of Comparative Effectiveness Research 4(2): 133–45. doi:10.2217/cer.14.79.

Forsythe, L., A. Heckert, M.K. Margolis, S. Schrandt and L. Frank. 2018. “Methods and Impact of Engagement in Research, from Theory to Practice and Back Again: Early Findings from the Patient-Centered Outcomes Research Institute.” Quality of Life Research 27(1): 17–31. doi:10.1007/s11136-017-1581-x.

Frampton, S.B., S. Guastello, L. Hoy, M. Naylor, S. Sheridan and M. Johnston-Fleece. 2017. “Harnessing Evidence and Experience to Change Culture: A Guiding Framework for Patient and Family Engaged Care.” NAM Perspectives. Washington, DC: National Academy of Medicine. doi:10.31478/201701f.

Gibson, A., J. Welsman and N. Britten N. 2017. “Evaluating Patient and Public Involvement in Health Research: From Theoretical Model to Practical Workshop.” Health Expectations 20: 826–35. doi:10.1111/hex.12486.

Gilbert, N. and J.B. Cousins. 2017. “Advancing Patient Engagement in Health Service Improvement: What Can the Evaluation Community Offer?” Canadian Journal of Program Evaluation 32(2): 202–21. doi:10.3138/cjpe.31120.

Hamilton, C.B., A.M. Hoens, C.L. Backman, A.M. McKinnon, S. McQuitty, K. English et al. 2018. “An Empirically Based Conceptual Framework for Fostering Meaningful Patient Engagement in Research.” Health Expectations 21(1): 396–406. doi:10.1111/hex.12635.

Health Canada. 2000. The Health Canada Policy Toolkit for Public Involvement in Decision Making. Retrieved July 26, 2018. <https://www.canada.ca/content/dam/hc-sc/migration/hc-sc/ahc-asc/alt_formats/pacrb-dgapcr/pdf/public-consult/2000decision-eng.pdf>.

Health Quality Ontario. 2017. Ontario’s Patient Engagement Framework: Creating a Strong Culture of Patient Engagement to Support High Quality Health Care. Retrieved April 4, 2018. <http://www.hqontario.ca/Portals/0/documents/pe/ontario-patient-engagement-framework-en.pdf>.

International Association for Public Participation (IAP2). 2015. “Public Participation Spectrum.” Retrieved April 4, 2018. <http://iap2canada.ca/page-1020549>.

Johannesen, J. 2017. “Two-Way Street? A Critical Look at Patient and Family Engagement.” Presentation to the Canadian Family Advisory Network session at the CAPHC conference, Montreal, October 29, 2017. Retrieved July 26, 2018. <https://johannesen.ca/2017/10/two-way-street-critical-look-patient-family-engagement>.

Lavis, J.N., F.-P. Gauvin, R. Reid, H. Bullock, W. Wodchis and A. Hayes. 2018. Rapid Synthesis: Creating a Rapid-Learning Health System in Ontario. Hamilton, ON: McMaster Health Forum.

McMaster University. 2018. “Overview of the Public and Patient Engagement Evaluation Tool.” Retrieved July 26, 2017. <https://fhs.mcmaster.ca/publicandpatientengagement/ppeet.html>.

Mockford, C., S. Staniszewska, F. Griffiths and S. Herron-Marx. 2012. “The Impact of Patient and Public Involvement on UK NHS Health Care: A Systematic Review.” International Journal for Quality in Health Care 24(1): 28–38. doi:10.1093/intqhc/mzr066.

Orchard, C.A., G.A. King, H. Khalili and M.B. Bezzina. 2012. “Assessment of Interprofessional Team Collaboration Scale (AITCS): Development and Testing of the Instrument.” Journal of Continuing Education in the Health Professions 32(1): 58–67. doi:10.1002/chp.21123.

Poulton, B.C. and M.A. West. 1999. “The Determinants of Effectiveness in Primary Health Care Teams.” Journal of Interprofessional Care 13(1): 7–18.

Richards, T., V.M. Montori, F. Godlee, P. Lapsley and D. Paul. 2013. “Let the Patient Revolution Begin.” BMJ 346: f2614. doi:10.1136/bmj.f2614.

Rowe, G. and L. Frewer. 2005. “A Typology of Public Engagement Mechanisms.” Science, Technology, & Human Values 30(2): 251–90.

Shortell, S., J.A. Marsteller, M. Lin, M.L. Pearson, S.Y. Wu, P. Mendel et al. 2004. “The Role of Perceived Team Effectiveness in Improving Chronic Illness Care.” Medical Care 42(11): 1040–48.

Staley, K. 2015. “‘Is It Worth Doing?’ Measuring the Impact of Patient and Public Involvement in Research.” Research Involvement & Engagement 1: 6.

Valente, T.W. 2010. Social Networks and Health. Oxford, UK: Oxford University Press.

Comments

Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed