Abstract

This commentary, in response to Tamblyn et al. (2016), provides a case study of the uptake of metered-dose inhalers with spacers to deliver respiratory medication in a pediatric emergency department in Nova Scotia. Our objective was to demonstrate the opportunities and challenges in engaging researchers and their trainees in planning and evaluating a clinical practice change to improve drug therapy. We document the use of community engaged scholarship (including experiential learning) to increase the capacity and capability of researchers in academia and healthcare organizations, healthcare providers, and managers. We note lessons learned from Dalhousie University’s Drug Use Management and Policy Residency and four individual research projects conducted between 2006–2016. 

Introduction 

The optimal use of pharmaceuticals is a key component of a safe, effective and sustainable healthcare system, yet prescribing, medication management and pharmaceutical policy often lag in implementing recommendations from scientific evidence. Tamblyn et al. (2016) note the high level of healthcare spending in Canada compared with other Organisation for Economic Co-Operation and Development (OECD) countries. Canada spends US$761 per capita on pharmaceuticals. This is higher than the OECD reported average (2015) of $527 USD per capita, and ranks 29th of 30 countries. The Canadian Institute for Health Information (CIHI) reports (2015) that 15.7% of healthcare spending is directed toward pharmaceuticals ($959 CAD per capita). Canada, without a comprehensive national pharmaceutical strategy with quality indicators, has identified and unidentified performance gaps in prescribing and pharmaceutical use in areas such as access, safety, quality and value for money. 

Tamblyn et al. (2016) note the need for conducting and learning from natural experiments in health service delivery in Canada. Various approaches are being implemented by Canadian public and private sector organizations to improve prescribing and medication use focusing on the patient, provider, organization and/or system levels; however, these are often implemented for a specific patient population or jurisdiction with limited spread across the country (Sketris et al. 2009). 

This paper presents a case example, a series of studies around the uptake of metered-dose inhalers with spacers (MDIs) for respiratory medication delivery in a pediatric emergency department (PED) in Nova Scotia. Throughout this case, we sought to increase capability and capacity of health services researchers, and innovation in the healthcare system using community-engaged scholarship including experiential learning. Our approach paired scientists in academia with policy, clinical, managerial and research leaders in the health system to co-produce research. 

The case highlights two strategic directions identified by the Canadian Health Services and Policy Research Alliance: (1) accelerating the creation of a cadre of scientists working in a learning health system and (2) measuring research impact (CIHR Institute of Health Services and Policy Research 2014). The case also highlights two themes that have received limited health services research funding, identified by the Alliance’s 2007–2012 scan of 27 federal and provincial health research funding agencies and health charities: managing for quality and safety (receiving 11.9% of total health services research funding) and change management (receiving only 0.3%) (Tamblyn et al. 2016). 

DUMPR and DEANS 

In 1999, the Canadian Health Services Research Foundation (CHSRF) in partnership with the Canadian Institutes of Health Research (CIHR) launched the Capacity for Applied and Developmental Research and Evaluation in Health Services and Nursing (CADRE program) with $6.5 million in annual funding (Grudniewicz et al. 2014; Potvin and Armstrong 2013; Tamblyn et al. 2016). Under this program, Ingrid Sketris received a Chair, which allowed her to form partnerships to conduct pharmaceutical policy research in Nova Scotia. This Chair used community-engaged scholarship and operated in Mode 2 research (co-production of research with relevant provincial government departments and healthcare organizations). One component was the Dalhousie Drug Use Management and Policy Residency (DUMPR), an experiential learning approach that embedded trainees (38 in total) with the support of their academic advisors in organizations to conduct research of relevance to their decision-making preceptors (Conrad et al. 2005, 2013). 

Some of the key components of community-engaged scholarship and the experiential learning program used in the case are noted below. While they are presented linearly, academics and healthcare decision-makers interacted frequently with multiple planning iterations and processes that may or may not have occurred concurrently. 

Establishing Key Partnerships 

DUMPR leveraged the Drug Evaluation Alliance of Nova Scotia (DEANS), which was established in 1998 by the Nova Scotia government to provide a secretariat for and funding of pharmaceutical prescribing and use improvement interventions. DEANS identifies and prioritizes critical drug therapy issues, analyzes scientific evidence and connects with local clinical expertise to understand and improve drug therapy. Along with partners, DEANS develops, implements and evaluates interventions to increase the uptake of evidence-informed practices and policies (Sketris et al. 2006). 

Lessons learned and future considerations 

Many trainees’ research topics originated from issues identified by DEANS. As trainees presented research results to DEANS, they received feedback from a broad group of practitioners and policy makers, advice on research dissemination channels and suggestions for future research. More formal and long-term approaches to developing research priorities could be developed, taking into account the priorities of both the academic and decision-maker organization. 

Utilizing the Academic Health Sciences Network 

Our case example was set at the IWK Health Centre, which has a memorandum of understanding with Dalhousie University for student placements and a mission of patient care, education and research (Brimacombe et al. 2010). Tamblyn et al. (2016) discuss the need for research in geriatrics. However, our case addresses children who are similarly vulnerable because randomized controlled trial (RCT) evidence can be limited (Council of Canadian Academies 2014). 

Lessons learned and future considerations 

The IWK Health Centre and Dalhousie University partnership facilitated experiential education to allow trainees and their academic advisors, especially those without clinical backgrounds, to conduct or participate in research in healthcare settings. Canada’s Strategy for Patient-Oriented Research (SPOR) may involve rural, primary and continuing care organizations, and these partnerships could be expanded to improve drug use outside academic settings. Further ties (both formal and informal), and mechanisms to integrate research strategies of the health system and academia could be developed, such as those in the UK’s academic health sciences centres’ networks, and Collaborations for Leadership in Applied Health Research and Care (CLAHRC) and other partnerships (Graham and Tetroe 2009; Rycroft-Malone et al. 2011; Sibbald et al. 2014; Soper et al. 2015; Spyridonidis et al. 2015). 

Identifying the Drug Policy Issue for Intervention 

With over 15,000 marketed drug products and many gaps in optimal prescribing, the research teams identified priorities (Sketris et al. 2009). The initiative to increase the uptake of MDIs in the delivery of respiratory medications for children who wheeze was noted as a priority by the clinical leadership at the IWK PED. One of the members of our clinical leadership, Dr. Douglas Sinclair (then Chief of Emergency Medicine), had been part of the earlier DEANS initiative to increase the uptake of MDIs in adults and was able to build on this experience. For adults, DEANS developed and evaluated educational, drug reimbursement policy and financial incentives to hasten the uptake of MDIs (Bowles et al. 2007; Kephart et al. 2005; Lowe et al. 2008; Murphy et al. 2005). The evaluations noted that the interventions resulted in a three-fold decrease in respiratory medication delivered by nebulization and approximately $1 million in annual drug cost savings to the Nova Scotia Seniors’ Pharmacare Program (Kephart et al. 2005; Sketris et al. 2006). This approach had not yet been taken up in pediatric patients.

Lessons learned and future considerations

Experience in adults related to increasing the uptake of MDIs could be adapted for children. In the future, gaps in optimal prescribing identified by other organizations (e.g., National Prescription Drug Utilization Information System) and pan-Canadian approaches to improve drug therapy (e.g., Canadian Agency for Drugs and Technologies in Health, Choosing Wisely Canada) could be better leveraged to improve drug use in Nova Scotia.

Assembling a Research Team

Conducting research often requires multi- and transdisciplinary teams and collaboration across academic and healthcare organization boundaries (Denis et al. 2003; National Research Council (NRC) 2015). We had access to and employed the skills of 14 researchers with diverse backgrounds (Table 1). The team had multilevel learners: one Masters student in health informatics and one in library and information sciences, one PhD student in economics and one medical student. While we worked with our DUMPR trainees, the IWK Emergency Department hosts about 150 medical students per year and many medical residents and students of other health professions who may have also been exposed to the intervention.

Table 1. Skillsets for multi- and transdisciplinary team involved in research on intervention to increase uptake of salbutamol metered-dose inhalers (MDIs) and spacers for a pediatric emergency department  

Four pediatric respiratory medication research projects were conducted from 2006 to 2016 (Table 2). Healthcare practitioners were interested in improving quality quickly to enhance safe and effective care. Researchers were also interested in applying theories and rigorous methods to produce generalizable knowledge (Bauer et al. 2015). The team established common objectives, invested time to build trust, and negotiated approaches, timelines, priorities and financial and human resources to improve healthcare quality and advance science.

Table 2. Timeline for experiential learning related to intervention to increase uptake of salbutamol metered-dose inhalers (MDI) and holding chambers for a pediatric emergency department 

Lessons learned and future considerations 

The teams’ methods of operation were facilitated by the DUMRP, which defined roles and responsibilities for trainees, preceptors and faculty advisors, and provided trainees with frequent in-person and electronic communication and an ongoing support network; however, team science has developed and provides further research and training guidance (Conrad et al. 2005, 2013; Forrest et al. 2009; Morgan et al. 2010; National Research Council 2015). Patients, families and voluntary health organizations had limited involvement in the research process and this could be strengthened. 

Identifying Evidence Sources and the Gap between Evidence and Practice 

Multiple sources of evidence were gathered and critiqued: scientific literature, an environmental scan of PEDs (Osmond et al. 2007; Scott et al. 2009) and a site visit to the Saint John Regional Hospital, New Brunswick, where the switch to MDIs had already been adopted, and where Dr. Katrina Hurley, who was a DUMPR trainee, had trained as a resident. As early as 1984, there was evidence that MDIs were at least equivalent to nebulization in children with acute asthma (Freelander and Van Asperen 1984). By 1997, there were recommendations that MDIs be considered the preferred mode of treatment in children with acute exacerbations of asthma and over the next years evidence was strengthened as multiple trials, syntheses and guideline recommendations were published. (Appendix 1) contains a record of publications related to the use of MDIs versus nebulization in children with acute wheezy illnesses. This literature search was conducted by another DUMPR trainee, Barbara Hill Taylor.) Following this literature scan, investigators appraised and synthesized relevant articles for specific presentations. By 2006, when the intervention was being planned, evidence showed that the two treatment methods could no longer be considered equivalent for most pediatric asthma emergency cases. There was established evidence that MDIs offered clinical and societal benefits, with potential for a better comfort and safety profile (Hill-Taylor et al. 2013). 

Lessons learned and future considerations 

Multiple evidence sources needed to be acquired, synthesized and exploited to plan the intervention and research (Zahra and George 2002), and teams were needed as the task would be daunting for a single clinician (Greenhalgh et al. 2014). This speaks to the ongoing need for communities of practice comprising healthcare practitioners and researchers to collaboratively identify and address gaps between evidence and practice. 

Understanding the Context 

Health services researchers need to understand the context in which to apply evidence and conduct research and then determine which aspects will facilitate or hinder their quality improvement and research efforts (Bowen and Graham 2015; Greenhalgh and Fahy 2015; Jackson and Greenhalgh 2015; Sargeant et al. 2008; Squires et al. 2015). There are various definitions of context, for example, from Tuen van Dijk, the “situational, historical, geographical, social or cultural environment of a phenomenon being studied” (Bate 2014: 4). 

Our teams included embedded clinician researchers, who practiced in and understood the changing context, and could develop tailored interventions and adapt these over time. 

One team examined perceptions surrounding the use of MDIs for delivering respiratory medications in the PED, using focus groups and interviews. Four main themes that needed to be addressed prior to implementing the intervention were identified: workload, misconceptions related to cost, need for education, and clarity related to professional roles (Hurley et al. 2008). 

The team also needed to understand the decision-making structures, operational processes (e.g., role of the Quality and Operations Committee), clinical and policy levers and resources available to support the intervention. They needed to determine sources of local support and/or resistance for the intervention as well (Bate 2014). 

Lessons learned and future considerations 

Our teams learned the difficulties of working in the real world context. We echo Donald Berwick and Paul Bate who note “the punishing contextual terrain,” which “so clearly labels the facts on the ground for ambitious even courageous clinicians, managers, executives and others in healthcare who seek to make care far better” (Bate 2014: 12). 

Future work needs to use emerging frameworks and theories in understanding context and provide greater community supports for embedded researchers (Bate 2014; Damshroder et al. 2009). 

Development of the Intervention 

The intervention was developed by identifying the scientific evidence and leading practices as well as understanding the local context through qualitative interviews and informal consultations with clinicians and managers (Hurley et al. 2008). The embedded researchers were able to develop a practical intervention, which recognized the clinicians’ strong internal motivations to improve care, and provided them with their own quantitative data, myths uncovered during qualitative research and opportunities to share experiences. It used clinical managers to leverage authority and provided numerous opportunities to refine the intervention (Pannick et al. 2015; Wieringa and Greenhalgh 2015). 

The intervention included the development of an asthma care map (with MDIs as one component) and educational sessions with healthcare providers and patients. The implementation of the care map took over two years – discussions began in 2006. The map was introduced as a pilot in January 2008 and became standard practice in July 2009. Piloting the intervention was key and recommended by the clinical leader, Douglas Sinclair. The intervention also included public service announcements to request that parents bring in their child’s holding chamber when seeking emergency care. 

The intervention adhered to the characteristics of simplicity, trialability, observability, reinvention and risk minimization (Ganz et al. 2009), and the team understood that changing clinician mindlines required the incorporation of both scientific knowledge and an understanding of factors that influence practice and decisions to change practice such as leadership, relationships, and personal and group beliefs. 

Lessons learned and future considerations 

The strong preparation related to understanding the context and assembling multiple sources of evidence as well as involvement of clinicians and managers helped make the intervention successful. We used grounded theory, knowledge translation principles and some aspects of learning theories (Graham et al. 2006; Hurley et al. 2008). Newer theories, models and approaches have much to offer in developing and implementing interventions (Bate 2014; Damschroder et al. 2009; Helfrich et al. 2010; Jagosh et al. 2012, 2015; Riley et al. 2015; Rycroft-Malone et al. 2013; Soper et al. 2015; Swanson et al. 2012; Waltz et al. 2015). Mechanisms need to be put in place for continued learning and adaptation of care maps as evidence and guidelines change over time (Fleiszer et al. 2016; Kastner et al. 2015). 

For our intervention, the information technology infrastructure could not be harnessed to provide electronic clinical decision support tools and continues to remain a challenge, making it an avenue for further work. 

These interventions were developed within a single institution and knowledge translation opportunities across the province have not been fully exploited; however, care maps are faxed on request to other institutions and Translating Emergency Knowledge for Kids (TREKK) is used as a national knowledge sharing network. Other jurisdictions employ national networks (e.g., Quality Enhancement Research Initiative (QUERI and NPS MedicineWise) to develop and share interventions for large systems with the potential for increased efficiency and effectiveness (Graham and Tetroe 2009). 

Developing the Evaluation 

There were three evaluation components. Hill- Taylor et al. (2013) examined PED inventory data for salbutamol inhalation formulations and received patient data from decision support services. They found a 1,215% (95% CI 1032 to 1396, p less than 0.001) increase in the proportion of salbutamol delivered as MDIs following the intervention. 

To understand medication dispensing data quality, Wing et al. (2012) explored the agreement between salbutamol administration records abstracted from the paper-based emergency department charts and electronic records generated by an automated dispensing device. While they noted substantial agreement (kappa 0.71), there were still many discrepancies to address. 

Spin et al. created a model for evaluating the cost-effectiveness of both salbutamol inhalation methods (MDIs vs nebulization) using local IWK Health Centre patient chart review and inventory data, Nova Scotia wages and salaries, and Canadian data from the National Ambulatory Care Reporting system, the Canadian Institutes of Health Information Discharge Abstract Database and Patient Cost estimator, and Statistics Canada (consumer price index) for patient costs, length-of-stay in the PED and hospital (if admitted) and the probability of admission combined with a time and motion study from the UK. (Spin et al. in press) They confirmed that use of MDIs was associated with a lower average patient cost. 

Lessons learned and future considerations 

Each study contributed not only to the overall evaluation of this intervention but also to the building of datasets and methods for future intervention evaluations. Other aspects that could have been evaluated include documentary evidence related to the implementation process and fidelity, resources and costs used in the development of the intervention, qualitative approaches to determine clinician satisfaction with the intervention and remaining barriers and patient perspectives. 

In future projects, we could better use theory-based evaluation, evaluate contribution of the research process to the outcome and determine endurance and adaptations of the innovation (Bate 2014; Fleiszer et al. 2016; Soper et al. 2015). 

Incentives and Disincentives for Community-Engaged Scholarship and Experiential Learning 

Tamblyn et al. (2016) note the need for incentive systems for health system innovation and the need to align incentives to further the engagement of practitioners in the innovation process. Our case provided value to the healthcare system by facilitating, documenting and evaluating the switch to a more cost-effective approach to deliver drug therapy in pediatric patients who wheeze. 

It was facilitated by a studentship award to Katrina Hurley and others and a chair award to Ingrid Sketris. Physician compensation models, which include not only clinical care but also quality improvement and research, may assist in their involvement in community-engaged scholarship to improve healthcare system performance, but accountability frameworks and performance measurement systems may be difficult to establish as these new payment models emerge (Damschroder et al. 2014; Hanney and Gonzalez-Block 2014; Soper et al. 2015; Wilsdon et al. 2015). Clinician scientists need time for practice reorganization, quality improvement activities and/or research and incentives to move up the clinical scientist or managerial ladder and to continue to engage in community-engaged scholarship (Belkhodja 2014; Harvey et al. 2015). 

Our teams produced five papers, received best oral presentation awards at local departmental and faculty-wide research days, and a poster award at the 2015 CAPHC (Canadian Association of Paediatric Health Centres) national conference. We also presented our research over 20 times to various decision-making audiences, locally and nationally. These awards, presentations and publications can be used by academics in their submissions for tenure and promotion files, and salary and operational grant competitions. Our teams conducted evaluations consistent with the human and financial resources available, but did not comprehensively examine research impact (Bauer et al. 2015; Panel on Return on Investment in Health Research 2009; Soper et al. 2015). 

Conclusion 

This case, which examined an intervention and its evaluation between 2006 and 2016, has been used to illustrate how community-engaged scholarship including experiential learning can catalyze the improvement of healthcare, specifically in the improved delivery of medication for children with acute exacerbations of asthma in a PED, and can decrease overall healthcare costs. 

Creating and maintaining a collaborative learning system and trusting partnerships across academic and health services delivery organizations, and engaging embedded health service researchers, were key to the success of our intervention. A vision, strategy and process could be developed to expand relationships to other patient populations and across other healthcare and social care organizations. The result of these productive relationships between academia and the health system can lead to increased health services researcher capacity and ongoing rigorous and relevant research and knowledge translation so needed in meeting the challenges of healthcare transformation, as outlined by Tamblyn et al. in their paper. 

Acknowledgements 

We would like to acknowledge Ethel M. Langille Ingram for her careful review of the manuscript and Sara Rehan for assistance with literature retrieval. 

The following sources of funding were used for the projects referred to in this commentary: Category A Grant from the IWK Health Centre, Osman Summer Research Studentship from the Faculty of Medicine, Dalhousie University and Ingrid Sketris’ Chair in Health Service Research, funded by the Canadian Health Services Research Foundation (CHSRF)/Canadian Institute of Health Research (CIHR), co-sponsored by the Nova Scotia Health Research Foundation (NSHRF). 

About the Author

Ingrid Sketris, PharmD, MPA(HSA), Professor, College of Pharmacy, Dalhousie University, Halifax, NS 

Barbara Hill Taylor, BSP, MLIS, Research Associate, IMPART, College of Pharmacy, Dalhousie University, Halifax, NS 

Joan Sargeant, PhD, Professor and Head, Division of Medical Education, Faculty of Medicine, Dalhousie University, Halifax, NS 

Katrina Hurley, MD, FRCPC, MHI, Research Director, Division of Pediatric Emergency Medicine, Faculty of Medicine, Dalhousie University and IWK Health Centre, Halifax, NS  

References

Bate, P. 2014. “Context is Everything.” In Perspectives on Context: a Selection of Essays Considering the Role of Context in Successful Quality Improvement. London, UK: Health Foundation. pp. 1–30. Retrieved January 22, 2016. <www.health.org.uk/sites/default/files/ PerspectivesOnContext_fullversion.pdf>. 

Bauer, M.S., L. Damschroder, H. Hagedorn, J. Smith, A.M. Kilbourne. 2015. “An Introduction to Implementation Science for the Non-Specialist.” BMC Psychology 3: 32. doi:10.1186/s40359-015-0089-9. 

Belkhodja, O. 2014. “Knowledge Utilization in Canadian Health Service Organizations: a Path Analysis.” International Journal of Public Administration 37(6): 339–52. doi:10.1080/01900692. 2013.837070. 

Bowen, S., and I.D. Graham. 2015. “Backwards Design or Looking Sideways? Knowledge Translation in the Real World Comment on “A Call for a Backward Design to Knowledge Translation”.” International Journal of Health Policy and Management 4(8): 545–47. doi:10.15171/ijhpm.2015.71. 

Bowles, S.K., I. Sketris, G. Kephart and Drug Evaluation Alliance of Nova Scotia. 2007. “Use of Wet Nebulized Inhaled Respiratory Medications Under Criteria-Based Reimbursement Guidelines in a Publicly Funded Seniors’ Pharmacare Program in Nova Scotia, Canada.” American Journal of Geriatric Pharmacotherapy 5(2): 120–28. doi:S1543-5946(07)00022-0. 

Brimacombe, G.G. and Association of Canadian Academic Healthcare Organizations, Academic Health Sciences Centres National Task Force (Canada). 2010. Three Missions, One Future: Optimizing the Performance of Canada’s Academic Health Sciences Centres. Ottawa, ON: Association of Canadian Academic Healthcare Organizations. 

Canadian Institute for Health Information. 2015. National Health Expenditure Trends, 1975 to 2015. Ottawa, ON: CIHI. 

CIHR Institute of Health Services and Policy Research in collaboration with Terrence Sullivan & Associates. 2014. Pan-Canadian Vision and Strategy for Health Services and Policy Research: 2014–2019. Montreal, QC: CIHR Institute of Health Services and Policy Research. Retrieved January 22, 2016. <www.cihr-irsc.gc.ca/e/13733.html>. 

Conrad, P., J. Murphy, I. Sketris. 2005. “Drug Use Management and Policy Residency: a Service- Learning Application.” American Journal of Pharmaceutical Education 69(5): 1–10. 

Conrad, P., I. Sketris and E. Langille-Ingram. 2013. “Participants’ Perceptions of a Multidisciplinary Training Program for Graduate and Postgraduate Students in Drug Use Management and Policy.” American Journal of Pharmaceutical Education 77(5): 102. doi:10.5688/ajpe775102. 

Council of Canadian Academies. 2014. Improving Medicines for Children in Canada. Ottawa, ON: The Expert Panel on Therapeutic Products for Infants, Children, and Youth, Council of Canadian Academies. Retrieved January 21, 2016. <www. scienceadvice.ca/uploads/eng/assessments%20 and%20publications%20and%20news%20releases/ therapeutics/therapeutics_fullreporten.pdf>. 

Damschroder, L.J., D.C. Aron, R.E. Keith, S.R. Kirsh, J.A. Alexander, J.C. Lowery. 2009. “Fostering Implementation of Health Services Research Findings into Practice: a Consolidated Framework for Advancing Implementation Science.” Implementation Science 4: 50. doi:10.1186/1748-5908-4-50. 

Damschroder, L.J., C.H. Robinson, J. Francis, D.R. Bentley, S.L. Krein, AM Rosland et al. 2014. “Effects of Performance Measure Implementation on Clinical Manager and Provider Motivation.” Journal of General Internal Medicine 29(S4): 877–84. doi:10.1007/s11606-014-3020-9. 

Denis, J.L., P. Lehoux, M. Hivon and F. Champagne. 2003. “Creating a New Articulation between Research and Practice through Policy? The Views and Experiences of Researchers and Practitioners.” Journal of Health Services Research & Policy 8(S2): 44–50. doi:10.1258/135581903322405162. 

Fleiszer, A.R., S.E. Semenic, J.A. Ritchie, M.C. Richer, J.L. Denis. 2016. “A Unit-level Perspective on the Long-term Sustainability of a Nursing Best Practice Guidelines Program: an Embedded Multiple Case Study.” International Journal of Nursing Studies 53: 204–18. doi:10.1016/j.ijnurstu.2015.09.004. 

Forrest, C.B., D.P. Martin, E. Holve and A. Millman. 2009. “Health Services Research Doctoral Core Competencies.” BMC Health Services Research 9: 107 doi:10.1186/1472-6963-9-107. 

Freelander, M. and P.P. Van Asperen. 1984. “Nebuhaler Versus Nebuliser in Children with Acute Asthma.” British Medical Journal (Clinical Research Edition) 288(6434): 1873–74. 

Ganz, D.A., E.M. Yano, D. Saliba and P.G. Shekelle. 2009. “Design of a Continuous Quality Improvement Program to Prevent Falls among Community- Dwelling Older Adults in an Integrated Healthcare System.” BMC Health Services Research 9: 206. doi:10.1186/1472-6963-9-206. 

Graham, I.D., J. Logan, M.B. Harrison, S.E. Straus, J. Tetroe, W. Caswell et al. 2006. “Lost in Knowledge Translation: Time for a Map?” Journal of Continuing Education in the Health Professions 26(1): 13–24. doi:10.1002/chp.47. 

Graham, I.D. and J. Tetroe. 2009. “Learning from the US Department of Veterans Affairs Quality Enhancement Research Initiative: QUERI Series.” Implementation Science 4(13). doi:10.1186/1748-5908-4-13. 

Greenhalgh, T., J. Howick, N. Maskrey and the Evidence Based Medicine Renaissance Group. 2014. “Evidence Based Medicine: a Movement in Crisis?” BMJ 348. doi:10.1136/bmj.g3725. 

Greenhalgh, T. and N. Fahy. 2015. “Research Impact in the Community-Based Health Sciences: an Analysis of 162 Case Studies from the 2014 UK Research Excellence Framework.” BMC Medicine 13: 232. doi:10.1186/s12916-015-0467-4. 

Grudniewicz, A., L. Hedden, S. Kromm, R. Lavergne, M. Menear, S. Sivanantha et al. 2014. “Capacity Development in Health Systems and Policy Research: a Survey of the Canadian Context.” Health Research Policy and Systems 12: 9. doi:10.1186/1478-4505-12-9. 

Hanney, S.R. and M.A. Gonzalez-Block. 2014. “Four Centuries on from Bacon: Progress in Building Health Research Systems to Improve Health Systems?” Health Research Policy and Systems / BioMed Central 12: 56. doi:10.1186/1478-4505-12-56. 

Harvey, G., R.J. Marshall, Z. Jordan and A.L. Kitson. 2015. “Exploring the Hidden Barriers in Knowledge Translation: a Case Study within an Academic Community.” Qualitative Health Research 25(11): 1506–517. doi:10.1177/1049732315580300. 

Helfrich, C.D., L.J. Damschroder, H.J. Hagedorn, G.S. Daggett, A. Sahay, M. Ritchie et al. 2010. “A Critical Synthesis of Literature on the Promoting Action on Research Implementation in Health Services (PARIHS) Framework.” Implementation Science 5: 82. doi:10.1186/1748-5908-5-82. 

Hill-Taylor, B.J., K.F. Hurley, I. Sketris, C. O’Connell, D. Sinclair and A. Wing. 2013. “Evaluating A Clinical Practice Intervention to Promote Delivery of Salbutamol by Metered-Dose Inhalers with Holding Chambers in a Pediatric Emergency Department.” Canadian Journal of Emergency Medicine 15(2): 101–08. 

Hurley, K.F., J. Sargeant, J. Duffy, I. Sketris, D. Sinclair and J. Ducharme. 2008. “Perceptual Reasons for Resistance to Change in the Emergency Department Use of Holding Chambers for Children with Asthma.” Annals of Emergency Medicine 51(1): 70–77. doi:S0196-0644(07)00485-4. 

Jackson, C.L. and T. Greenhalgh. 2015. “Co-creation: a New Approach to Optimising Research Impact?” Medical Journal of Australia 203(7): 283–84. doi:10.5694/mja15.00219. 

Jagosh, J., A.C. Macaulay, P. Pluye, J. Salsberg, P.L. Bush, J. Henerson et al. 2012. “Uncovering the Benefits of Participatory Research: Implications of a Realist Review for Health Research and Practice.” Milbank Quarterly 90(2): 311–46. doi:10.1111/j.1468-0009.2012.00665.x/. 

Jagosh, J., P.L. Bush, J. Salsberg, A.C. Maccaulay, T. Greenhalgh, G. Wong et al. 2015. “A Realist Evaluation of Community-Based Participatory Research: Partnership Synergy, Trust Building and Related Ripple effects.” BMC Public Health 15: 725. doi:10.1186/s12889-015-1949-1. 

Kastner, M., O. Bhattacharyya, L. Hayden, J. Makarski, E. Estey, L. Durocher et al. 2015. “Guideline Uptake is Influenced by Six Implementability Domains for Creating and Communicating Guidelines: a Realist Review.” Journal of Clinical Epidemiology 68(5): 498–509. doi:10.1016/j.jclinepi.2014.12.013/. 

Kephart, G., I.S. Sketris, S.K. Bowles, M.E. Richard and C.A. Cooke. 2005. “Impact of a Criteria-Based Reimbursement Policy on the Use of Respiratory Drugs Delivered by Nebulizer and Health Care Services Utilization in Nova Scotia, Canada.” Pharmacotherapy 25(9): 1248–57. doi:10.1592/ phco.2005.25.9.1248. 

Lowe, D.O., H. Lummis, Y. Zhang and I.S. Sketris. 2008. “Effect of Educational and Policy Interventions on Institutional Utilization of Wet Nebulization Respiratory Drugs and Portable Inhalers.” Canadian Journal of Clinical Pharmacology 15(2): e334–43. 

Morgan, S., K. Orr and C. Mah. 2010. “Graduate Attributes for Master’s Programs in Health Services and Policy Research: Results of a National Consultation.” Healthcare Policy 6(1): 64–86. 

Murphy, A.L., N.J. MacKinnon, P.S. Flanagan, S.K. Bowles and I.S. Sketris. 2005. “Pharmacists’ Participation in an Inhaled Respiratory Medication Program: Reimbursement of Professional Fees.” Annals of Pharmacotherapy 39(4): 655–61. doi:aph.1E286. 

National Research Council. 2015. Enhancing the Effectiveness of Team Science. Committee on the Science of Team Science, N.J. Cooke and M.L. Hilton (Eds.), Board on Behavioral, Cognitive, and Sensory Sciences, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academics Press.

OECD. 2015. Health at a Glance 2015: OECD Indicators. Paris, France: OECD Publishing. doi:10.1787/health_glance-2015-en. 

Osmond, M.H., M. Gazarian, R.L. Henry, T.J. Clifford, J. Tetzlaff and PERC Spacer Study Group. 2007. “Barriers to Metered-Dose Inhaler/ Spacer use in Canadian Pediatric Emergency Departments: a National Survey.” Academic Emergency Medicine 14(11): 1106–13. doi:j.aem.2007.05.009. 

Panel on Return on Investment in Health Research. 2009. Making an Impact: a Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Ottawa, ON: Canadian Academy of Health Sciences. Retrieved January 22, 2016. <www.cahs-acss. ca/e/assessments/completedprojects.php>. 

Pannick, S., N. Sevdalis and T. Athanasiou. 2015. “Beyond Clinical Engagement: a Pragmatic Model for Quality Improvement Interventions, Aligning Clinical and Managerial Priorities.” BMJ Quality & Safety. doi:10.1136/bmjqs-2015-004453. 

Potvin, L. and P. Armstrong. 2013. Shaping Academia for the Public Good: Critical Reflections on the CHSRF/CIHR Chair Program. Toronto, ON: University of Toronto Press. 

Riley, B.L., K.L. Robinson, J. Gamble, D.T. Finegood, D. Sheppard, T.L. Penney et al. 2015. “Knowledge to Action for Solving Complex Problems: Insights from A Review of Nine International Cases.” Health Promotion and Chronic Disease Prevention in Canada: Research, Policy and Practice 35(3): 47–53. Retrieved January 22, 2016. <www.phac-aspc.gc.ca/publicat/ hpcdp-pspmc/35-3/ar-01-eng.php>. 

Rycroft-Malone, J., J.E. Wilkinson, C.R. Burton, G. Andrews, S. Ariss, R. Baker et al. 2011. “Implementing Health Research Through Academic and Clinical Partnerships: A Realistic Evaluation of the Collaborations for Leadership in Applied Health Research and Care (CLAHRC).” Implementation Science 6(74). doi:10.1186/1748-5908-6-74. 

Rycroft-Malone, J., K. Seers, J. Chandler, C.A. Hawkes, N. Crichton, C. Allen et al. 2013. “The Role of Evidence, Context, and Facilitation in an Implementation Trial: Implications for the Development of the PARIHS Framework.” Implementation Science 8: 28. doi:10.1186/1748-5908-8-28. 

Sargeant, J., K.F. Hurley, J. Duffy, I. Sketris, D. Sinclair, J. Ducharme, J. 2008. “Lost in Translation or Just Lost?” Annals of Emergency Medicine 52(5): 575–76. 

Scott, S.D., M.H. Osmond, K.A. O’Leary, I.D. Graham, J. Grimshaw, T. Klassen et al. 2009. “Barriers and Supports to Implementation of MDI/ spacer use in Nine Canadian Pediatric Emergency Departments: A Qualitative Study.” Implementation Science 4: 65. doi:10.1186/1748-5908-4-65. 

Sibbald, S.L., J. Tetroe and I.D. Graham. 2014. “Research Funder Required Research Partnerships: A Qualitative Inquiry.” Implementation Science 9(176). doi:10.1186/s13012-014-0176-y. 

Sketris, I.S., D.M. Frail, P. MacLean-Veysey, G. Kephart, C. Cooke, M. Allen et al. 2006. “A Multidisciplinary, Multi-Sectoral Alliance to Improve Drug use in Nova Scotia.” In Canadian Institutes of Health Research – Institute of Health Services and Policy Research (Ed.). Evidence in Action, Acting on Evidence: A Casebook of Health Services and Policy Research Knowledge Translation Stories. Ottawa, ON: CIHR: 107–11. 

Sketris, I.S., E.M. Langille Ingram and H.L. Lummis. 2009. “Strategic Opportunities for Effective Optimal Prescribing and Medication Management.” Canadian Journal of Clinical Pharmacology 16(1): e103–25. 

Soper, B., S. Hinrichs, S. Drabble. O. Yaqub, S. Marjanovic, S. Hanney et al. 2015. “Delivering the Aims of the Collaborations for Leadership in Applied Health Research and Care: Understanding their Strategies and Contributions.” Southampton, UK: NIHR Journals Library; Health Services and Delivery Research, No. 3.25. Retrieved January 21, 2016. <www.ncbi.nlm.nih.gov/books/NBK294356/>. 

Spin, P., I. Sketris, B. Hill-Taylor, C. Ward and K. Hurley. (in press). “A Cost Analysis of Salbutamol Administration by Metered-Dose Inhalers with Spacers versus Nebulization for Patients with Wheeze in the Pediatric Emergency Department: Evidence from Observational Data in Nova Scotia.” Canadian Journal of Emergency Medicine. 

Spyridonidis, D., J. Hendy and J. Barlow. 2015. “Leadership for Knowledge Translation: The Case of CLAHRCs.” Qualitative Health Research 25(11): 1492–505. doi:10.1177/1049732315583268. 

Squires, J.E., I.D. Graham, A.M. Hutchinson, S. Michie, J.J. Francis, A. Sales et al. 2015. “Identifying the Domains of Context Important to Implementation Science: a Study Protocol.” Implementation Science 10: 135. doi:10.1186/ s13012-015-0325-y. 

Swanson, R.C., A. Cattaneo, E. Bradley, S. Chunharas, R. Atun, K. Abbas et al. 2012. “Rethinking Health Systems Strengthening: Key Systems Thinking Tools and Strategies for Transformational Change.” Health Policy and Planning 27(S4): iv54–61. doi:10.1093/heapol/ czs090. 

Tamblyn. R., M. McMahon, J. Nadigel, B. Dunning and E. Drake. 2016. “Health System Transformation through Research Innovation.” Healthcare Papers 16(Special Issue, September): 8–19. doi:10.12927/hcpap.2016.24719. 

Waltz, T.J., B.J. Powell, M.M Matthieu, L.J. Damschroder, M.J. Chinman, J.L. Smith et al. 2015. “Use of Concept Mapping to Characterize Relationships Among Implementation Strategies and Assess their Feasibility and Importance: Results from the Expert Recommendations for Implementing Change (ERIC) Study.” Implementation Science 10: 109. doi:10.1186/s13012-015-0295-0. 

Wieringa, S. and T. Greenhalgh. 2015. “10 Years of Mindlines: a Systematic Review and Commentary.” Implementation Science 10: 45. doi:10.1186/ s13012-015-0229-x. 

Wilsdon, J., L. Allen, E. Belfiore, P. Campbel, S. Curry and E. Hill. 2015. The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. doi:10.13140/ RG.2.1.4929.1363. Retrieved January 22, 2016. 

Wing, A., B. Hill-Taylor, I. Sketris, J. Smith, S. Stewart and K.F. Hurley. 2012. “Medication Records in the Emergency Department: Agreement Between Paper-Based Charts and Automated Dispensing Device.” Canadian Journal of Hospital Pharmacy 65(4): 265–71. 

Zahra, S. and G. George. 2002. “Absorptive Capacity: a Review, Reconceptualization and Extension.” Academy of Management Review 27: 185–203.