Healthcare Quarterly

Healthcare Quarterly 24(Special Issue) April 2022 : 81-85.doi:10.12927/hcq.2022.26769
Patient- Or Community-Driven Projects

Patient Engagement in a Multi-Stakeholder Workshop to Plan the Collection of Patient-Oriented Outcomes for Children with Inherited Metabolic Diseases

Kylie Tingley, Maureen SmithP, Nicole PalloneP, Pranesh Chakraborty and Beth K. Potter


Building on a study to develop core outcome sets for children with rare inherited metabolic diseases, the purpose of this workshop was to inform the design of longitudinal pediatric registries that support registry-based clinical trials. This workshop was co-designed by two patient/family partner investigators and attended by two family advisors who received preparatory training. Patient partners and advisors recommended integrating the collection of registry data into everyday life and highlighted the importance of transparent communication and attention to the issue of integration of patient-reported data into clinical care. We propose a need to explore strategies for engaging patients in post-project knowledge translation.

Key Points
  • Building on previous work, we successfully co-designed a workshop with patient/family partner investigators to inform the design
    of longitudinal pediatric registries that support registry-based clinical trials.
  • Patient partners and advisors recommended integrating the collection of registry data into everyday life and highlighted the importance of transparent communication and attention to the integration of patient-reported data into clinical care.
  • We attribute much of our success and sustainability of our partnership to co-developing a comprehensive patient engagement strategy that included regular feedback to patient partners/advisors about the positive impact of their contributions.


Core outcome sets (COSs) are an agreed list of the minimum standardized outcomes that should be measured and described for all clinical trials in a specific disease area (Williamson et al. 2017). Ideally, COSs should be developed by a multi-stakeholder team that includes patients and family members, healthcare providers, health policy decision makers and methodologists (Williamson et al. 2017). Incorporating patients and family members as partners in COS development is key so that outcomes included in a COS are reflective of what is important to those affected by the disease and those who stand to benefit most from clinical research (Young and Bagley 2016). Our team established the first COSs for two inherited metabolic diseases (IMDs) in children: phenylketonuria (PKU) and medium chain acyl-CoA dehydrogenase (MCAD) deficiency (Pugliese et al. 2021).

Overview of the COS study

Methods and findings from our COS development study are described elsewhere (Potter et al. 2017; Pugliese et al. 2020, 2021). Briefly, following guidance from the Core Outcomes Measures in Effectiveness Trials (COMET) Initiative (Williamson et al. 2017), we systematically reviewed published studies for each condition to identify candidate outcomes (Pugliese et al. 2020). Subsequently, parents of children with PKU or MCAD deficiency, clinicians and policy advisors participated in a multi-round Delphi consensus survey (Pugliese et al. 2021). Final COSs were selected by discussion and voting at an in-person multi-stakeholder meeting, where more than 30% of the attendees were patients and caregivers (Pugliese et al. 2021).

Our patient engagement strategy for the COS study has also been described in another study (Vanderhout et al. 2021). Briefly, two patient or family member partners were engaged as co-investigators throughout the study and a Family Advisory Forum (FAF), which included seven parents of children with IMDs, were engaged at key stages. Patient partner investigators contributed to protocol development, co-designed and co-led all patient engagement activities, identified challenges and solutions to incorporating patient perspectives and communicated with FAF members. FAF members advised and provided feedback at several points during the study, including reviewing patient/family member–facing materials and contributing to outcome selection. Our team adapted existing resources from the COMET Initiative on patient and public engagement to support this work (COMET Initiative 2021). In addition, the principal investigator (BKP) and one patient partner investigator (MS) attended training on patient-oriented research from the Ontario Strategy for Patient-Oriented Research (SPOR) SUPPORT Unit (OSSU).

Planning the implementation of COSs to support registry-based clinical trials

The next phase of our work involved identifying barriers and facilitators to the implementation of our COSs. We were specifically interested in the potential for the outcomes to be collected across Canadian centres in new disease registries designed to support the development and implementation of registry-based randomized trials (Li et al. 2016). Registry-based randomized trials use patient registries as the platform for recruiting clinical trial participants and to optimize trial data collection (Li et al. 2016; Mathes et al. 2018). To integrate our COSs in registries to support trials, we needed to better understand preferences of stakeholders, including patients and families, regarding their collection and use. One of our patient partners suggested that we co-develop an OSSU Engaging Multi-stakeholders for Patient Oriented-research Wider Effects and Reach (EMPOWER) Award application, which was successful and helped fund a knowledge translation workshop to gain this multi-stakeholder perspective. Here, we report the methods and results from the workshop, emphasizing the contributions of patient partners, who were integral in its design and conduct, and of the family member participants. We have used the Guidance for Reporting Involvement of Patients and the Public, Version 2 (GRIPP2)–Short Form reporting checklist to guide our reporting (Staniszewska et al. 2017).


This project is built on our previously funded research to develop COSs for children with rare IMDs. The workshop's purpose was to solicit ideas and preferences from multiple stakeholders to inform the design of a registry focused on longitudinal collection of the COSs for PKU and other IMDs and to outline what would be required to successfully implement such a registry across Canada.


Two patient/family partner investigators (MS and NP) from the original study were involved as patient partner co-lead investigators. These patient partners co-designed the knowledge translation workshop and led our patient engagement strategy. They were involved in all the stages from the writing of the funding application to the final workshop report. Specifically, the patient partner investigators provided feedback and refined the scope of the grant proposal, co-designed preparatory training materials for parent workshop participants, contributed their own perspectives at the workshop, led a breakout session with parent participants about the selection of outcome measurement instruments and conducted an informal evaluation among parent workshop participants about their experience. In addition, one patient partner (MS) co-led the opening workshop session and presented the patient engagement strategy from the original COS study. Finally, the patient partner investigators reviewed and revised the final workshop report and continued their collaboration with the research team based on a grant for subsequent successful research, which has made use of the findings from the workshop for designing the disease registries.

Alongside these patient/family partner co-lead investigators, all seven FAF members from the original COS study were invited to attend the workshop as participants; two FAF members attended. These advisors were provided with preparatory training co-designed by the patient partner investigators, including a list of possible questions to consider for each workshop discussion and a document explaining the process of selecting outcome measurement instruments. Workshop topics that emphasized patient partner and advisor contributions included the following: selection of specific outcome measurement instruments for patient- or family-reported outcomes in the disease registries and methods for the regular collection of such outcomes, and sharing and integration of patient-/family-reported research registry data for use in clinical care.

Honoraria and travel expenses for patient partners and advisors were included in the grant proposal, in line with the Canadian Institutes of Health Research (CIHR) SPOR guidelines for compensation (CIHR 2019).


Collecting patient- and family-reported outcomes

Workshop participants recognized the importance of engaging with patients and family members throughout the registry design process to encourage participation and ensure that the registries contribute meaningful data. Regarding patient- or family-reported outcomes, family advisors commented on the relevance and acceptability of specific outcome measurement instruments for measuring the outcomes from the COS study. We also discussed strategies for collecting patient- and family-reported data. Participants identified a need to carefully consider the frequency of data collection to minimize respondent burden – for example, requesting data quarterly or less frequently. They also noted that incentives may facilitate registry participation. Several patient or family member participants recommended using approaches that integrate the collection of registry data into "everyday life" – for example, using mobile apps with functions such as appointment reminders. Access to technology was raised as a potential concern, which could be mitigated with an option to answer questionnaires when visiting a care clinic.

With respect to the degree of integration of patient- or family-reported outcomes data from a disease registry into clinical care, both patient or family member participants and clinician participants strongly emphasized a need for clear and transparent communication. This is particularly important if patients or parents are answering questionnaires for research purposes at the time of a clinic visit (e.g., in the waiting room). This could lead to misunderstandings about whether data are incorporated into the medical chart or otherwise shared with and considered by clinicians. Patient and family member partners and advisors also discussed that while completing a survey may help to organize one's thoughts before engaging with clinicians, parents may worry about compromising their child's care if the research data were shared, depending on the sensitivity of the information. They also expressed concern that research data, particularly data focused on parental (vs. a child's) experiences, may distract from priority clinical discussions during children's appointments. Parent participants also felt that data specific to parental well-being should not be invariably integrated into the child's medical chart.

Communication and consent

Workshop participants agreed that regular and effective communication with registry participants would be critical for the success of the registry. Concerning consent and privacy, they emphasized a need for transparent communication to inform decision making and to build trust. Consent from children or their family members should be viewed as a process rather than a one-time event. For example, ongoing opportunities for consent should be incorporated at the stage when a child reaches the designated age to give assent or their own informed consent, or if new information becomes available that may change the decision to consent.

Discussion: Reflection on Strengths and Limitations

This workshop followed the completion of a COS study that incorporated a comprehensive patient engagement strategy (Vanderhout et al. 2021). This facilitated the co-designing of the workshop with the patient partners who had been investigators on that study and contributions from some of the same family advisors. Our established continuity of patient and family investigators and advisors also enabled all the members of our team to benefit from mutual learning, strong team cohesiveness and collaboration toward a shared goal.

From the patient-partner perspective, we attribute much of our success to the co-development of a comprehensive patient engagement strategy that included regular feedback to patient partners and advisors about how their contributions positively impacted the project, and we feel this contributed to the sustainability of our partnership. In addition, patient-partner co-investigators felt empowered in their participation as research team members and appreciated the study team's openness to new ideas and responsiveness to their suggestions. Taking into account the perspective of other research team members, we recognize the importance of the insights brought forward by patients and family members, which have been critical to our team's ongoing design of patient registries. Partners and advisors also raised questions that the team has taken on in further research. For example, the discussion about whether and how to integrate patient-/family-reported outcomes data collected for research purposes into the clinical chart requires further investigation from multiple perspectives.

The patient-partner co-investigators have continued to co-design the patient engagement strategy for our program of work, including our ongoing design of registries to support registry-based trials. Recruiting a group of family advisors who received training and support to contribute to the COS study and the knowledge translation workshop has also allowed us to build capacity in the field, including engaging new patient and family advisors.

Our patient engagement strategy for this workshop was not without limitations. Inviting only the patient-partner co-investigators and members of the FAF from the COS study rather than recruiting additional advisors meant that we had a relatively small number of patients and family members contributing to the knowledge translation workshop (participation in this workshop was not part of the original commitment of the partners and advisors to the COS study). In addition, there was limited diversity among advisors who contributed to our study. The need to increase diversity among patient and public partners in health research is an established priority (Reynolds et al. 2021). To address this, we have increased our reach to include a more diverse group of patient partners in our current work to implement registries. Furthermore, we did not engage children or youth themselves in the original COS development study or in the knowledge translation workshop. Although the COSs we developed were targeted toward children aged 12 years and younger, our ongoing work has specifically engaged a youth advisory group so that the registries we develop can meaningfully include older children. Finally, we did not formally evaluate the patient engagement strategy for the workshop. The patient-partner co-investigators and advisors informally reported that they were supported and able to fully participate, but, in the future, a more formal approach would be beneficial.


This co-designed workshop confirmed the importance of understanding and incorporating the preferences of patients and families as key stakeholders in the collection and use of patient- and family-reported data and in the design of disease registries. Their perspectives on issues such as consent, frequency of data collection, tools that are patient friendly and expectations about how data translate to clinical visits are all fundamental to developing registries that meet the needs of all stakeholders. We propose a need to further explore knowledge translation strategies for patient engagement in post-COS activities, including those that inform registry design.


This knowledge translation workshop was funded by an OSSU EMPOWER Award (fall 2019) and the University of Ottawa Conference or Workshop on Campus Opportunity (fall 2019). The preceding COS development study was funded by the CIHR (grant #151614).

About the Author(s)

Kylie Tingley, PhD, is an epidemiologist from the University of Ottawa. Kylie's research has focused on evidence generation and synthesis for rare diseases and their associated treatments.

Maureen Smith, MEd, is a patient partner whose interest in partnerships with researchers stems from her diagnosis with a rare disease in childhood.

Nicole Pallone, ArtDp, is a mother of a teenager diagnosed with PKU and served as the vice president of Canadian PKU and Allied Disorders Inc. for six years and as a board member for 10 years.

Pranesh Chakraborty, MD, FRCPC, FCCMG, is a medical biochemist and pediatrician with clinical and research activities in the treatment of children with inherited metabolic diseases. Pranesh is also the chief medical officer of Newborn Screening Ontario.

Beth K. Potter, PhD, leads a research program that aims to generate evidence to improve health outcomes and experiences with care for children with rare diseases and their families. Beth can be contacted by e-mail at


The authors would like to thank the members of the FAF, who contributed their expertise through lived experience during the knowledge translation workshop. The authors would also like to thank Alison Howie for her assistance with preparing this manuscript and for providing editorial comments. Finally, the authors would like to extend their thanks to all the investigators who belong to the Canadian Inherited Metabolic Diseases Research Network for their participation in the foundational work that led to this study.


Canadian Institutes of Health Research (CIHR). 2019. Considerations When Paying Patient Partners in Research. Strategy for Patient-Oriented Research. Retrieved September 9, 2021. <>.

COMET Initiative. 2021. Patients and the Public. Retrieved October 28, 2021. <>.

Li, G., T.T Sajobi, B.K. Menon, L. Korngut, M. Lowerison, M. James et al. 2016. Registry-Based Randomized Controlled Trials – What Are the Advantages, Challenges, and Areas for Future Research? Journal of Clinical Epidemiology 80: 16–24. doi:10.1016/j.jclinepi.2016.08.003.

Mathes, T., S. Buehn, P. Prengel and D. Pieper. 2018. Registry-Based Randomized Controlled Trials Merged the Strength of Randomized Controlled Trails and Observational Studies and Give Rise to More Pragmatic Trials. Journal of Clinical Epidemiology 93: 120–27. doi:10.1016/j.jclinepi.2017.09.017.

Potter, B.K., B. Hutton, T.J. Clifford, N. Pallone, M. Smith, S. Stockler et al. 2017. Establishing Core Outcome Sets for Phenylketonuria (PKU) and Medium-Chain Acyl-CoA Dehydrogenase (MCAD) Deficiency in Children: Study Protocol for Systematic Reviews and Delphi Surveys. Trials 18: 603. doi:10.1186/s13063-017-2327-3.

Pugliese, M., K. Tingley, A. Chow, N. Pallone, M. Smith, P. Chakraborty et al. 2021. Core Outcome Sets for Medium Chain Acyl-CoA Dehydrogenase Deficiency and Phenylketonuria. Pediatrics 148(2): 1–13. doi:10.1542/peds.2020-037747.

Pugliese, M., K. Tingley, A. Chow, N. Pallone, M. Smith, A. Rahman et al. 2020. Outcomes in Pediatric Studies of Medium-Chain Acyl-CoA Dehydrogenase (MCAD) Deficiency and Phenylketonuria (PKU): A Review. Orphanet Journal of Rare Diseases 15(1): 12. doi:10.1186/s13023-019-1276-1.

Reynolds, J., M. Ogden and R. Beresford. 2021. Conceptualising and Constructing ‘Diversity’ through Experiences of Public and Patient Involvement in Health Research. Research Involvement and Engagement 7(1): 1–11. doi:10.1186/s40900-021-00296-9.

Staniszewska, S., J. Brett, I. Simera, K. Seers, C. Mockford, S. Goodlad et al. 2017. GRIPP2 Reporting Checklists: Tools to Improve Reporting of Patient and Public Involvement in Research. The BMJ 358. doi:10.1136/bmj.j3453.

Vanderhout, S.M., M. Smith, N. Pallone, K. Tingley, M. Pugliese, P. Chakraborty et al. 2021. Patient and Family Engagement in the Development of Core Outcome Sets for Two Rare Chronic Diseases in Children. Research Involvement and Engagement 7(1): 66. doi:10.1186/s40900-021-00304-y.

Williamson, P.R., D.G. Altman, H. Bagley, K.L. Barnes, J.M. Blazeby, S.T. Brookes et al. 2017. The COMET Handbook: Version 1.0. Trials 18: 280. doi:10.1186/s13063-017-1978-4.

Young, B. and H. Bagley. 2016. Including Patients in Core Outcome Set Development: Issues to Consider Based on Three Workshops with Around 100 International Delegates. Research Involvement and Engagement 2(1): 1–13. doi:10.1186/s40900-016-0039-6.


P = Patient partner.


Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed