Early psychosis intervention (EPI) is a complex model of care designed to be delivered by a large multidisciplinary team. However, in practice, it is often delivered by very small teams, particularly in rural areas. This study analyzed fidelity data from over half of Ontario EPI programs (n = 24) to compare model fidelity in programs with smaller (≤2.1 staff) and larger (≥4.3 staff) teams. Few differences were identified, suggesting that small teams may be a viable option to deliver the EPI model, although both large and small teams were challenged to deliver almost a third of the elements of care.

Background

Early psychosis intervention (EPI) is internationally recognized as a best practice to treat young people experiencing their first episode of psychosis (Bertolote and McGorry 2005). EPI is a comprehensive model of care that integrates multiple evidence-based practices and is designed to be delivered by a multidisciplinary team (Bennett and RAISE Connection Program Investigators 2018; Mueser et al. 2015; NHS England 2016). EPI programs have been implemented across Canada, and several provinces, including Ontario, Quebec, Nova Scotia and British Colombia, have prioritized EPI as a core component of their mental health services (Bertulies-Esposito et al. 2022; Durbin et al. 2016; Iyer et al. 2015).

Canada, similar to many jurisdictions, has large sparsely populated regions (Statistics Canada 2022). Supporting equitable healthcare delivery in these rural and remote regions is an ongoing challenge (Weinhold and Gurtner 2014; Whaley 2020). For a specialized program like EPI, the lower population density in rural and remote regions may mean insufficient cases to justify a large multidisciplinary team and difficulty recruiting professionals to fill all team roles (Cheng et al. 2014; Pipkin 2021). Similar challenges have been raised for other team-based mental health treatment models (Luciano et al. 2014; Meyer and Morrissey 2007). As a result, smaller teams than the model specifies may be tasked with treatment delivery, with less certain results. In Ontario, there are currently 45 EPI programs and, according to a 2014 survey, almost half had two or fewer full-time equivalent (FTE) staff (Standards Implementation Steering Committee 2015).

Limited prior research in Canada or elsewhere has examined whether small teams are able to deliver the full EPI model. One US study of 36 EPI teams reported a positive but non-significant correlation between team FTEs and fidelity to the model (Addington et al. 2020). A 2004 Australian study found that fidelity was variable for three EPI teams that consisted of two EPI providers embedded in general mental health teams (O'Kearney et al. 2004). Finally, a 2012 survey of Ontario programs found that programs serving catchment areas with lower population density, all of which had two or fewer FTEs, were more challenged to deliver some elements of care but more likely to deliver others (Durbin et al. 2016). This research, however, is now over a decade old and was based on a key informant survey rather than more rigorous fidelity assessments.

Fidelity assessments are a strategy to measure whether the delivery of an intervention adheres to the intended practice model. In Ontario, the Early Psychosis Intervention Ontario Network (EPION) and the Centre for Addiction and Mental Health partnered to assess EPI program fidelity to the model to support quality improvement. In the present study, we utilized these data to compare fidelity for large and small EPI teams.

Methods

Fidelity assessment scale and process

Fidelity assessments were conducted using the First Episode Psychosis Services Fidelity Scale (Addington et al. 2016, 2020). The scale includes 31 items, with each item rated between 1 (not implemented) and 5 (fully implemented). A rating of 4 is considered satisfactory. Fidelity assessments were conducted by teams of two to three trained independent assessors. Fidelity ratings were assigned based on interviews with staff, clients and family members, an audit of 10 randomly selected client health records and program administrative data. Prior to the COVID-19 pandemic, data were collected through a two-day site visit. During the pandemic, interviews were conducted remotely and chart audits were conducted by local staff, trained and supervised remotely. More detail on the fidelity model has been published previously (Selick et al. 2021). Data on program characteristics were obtained through the fidelity assessment or existing administrative data collected by EPION.

Sample

Between 2017 and 2022, all Ontario EPI programs were invited to receive an assessment. Participation was voluntary and programs were included annually on a first come, first served basis until capacity was reached based on the available budget to support assessments. Twenty-four of Ontario's 45 EPI programs received at least one fidelity assessment. If programs received more than one assessment, only the most recent was included in this study.

Analysis

Given that a mix of small and large teams were assessed each year, we did not expect time-based differences to impact results. Therefore, data for the 24 programs were combined and a mean fidelity rating was calculated per item. Over time, a small number of items were modified by the scale developer to increase clarity, rating reliability and alignment with the most recent evidence (Addington et al. 2020), with additional minor modifications made by the study team for the Ontario context. Of the 31 items, rating criteria for 19 items were unchanged and could be calculated for all 24 programs; criteria for 12 items that had changed could be calculated for 16 programs.

While specific guidance on the minimum FTEs necessary for EPI delivery is lacking, it seemed likely that delivering the model with two or fewer FTEs would pose a challenge. In our study sample, nine programs reported 2.1 or fewer clinical FTEs and 15 reported 4.3 or more clinical FTEs, excluding psychiatry (Table 1). Programs were therefore grouped as larger (≥4.3 clinical FTEs) and smaller (≤2.1 clinical FTEs). Mann-Whitney U tests were used to compare the mean item ratings between groups, and the percentage of item mean scores that met adherence (≥4) was calculated per group. Analyses were conducted using IBM SPSS Statistics 27 (2020). Ratings are presented in a heat map to visually show patterns in the findings. Ethics approval was obtained from the Centre for Addiction and Mental Health research ethics board.

Results

Sample

Of the 24 participating programs, nine were categorized as small teams and 15 as large teams (Table 1). Programs were located across Ontario. Although large teams had much larger overall caseloads (mean = 146 vs. 20), large and small teams had similar caseloads per clinical FTE. Compared with large teams, small teams were more likely to operate in rural areas and serve smaller catchment area populations.

Fidelity scores

Large teams had a mean fidelity score of 4 or greater for 19 items (63%); small teams had a mean fidelity score of 4 or greater for 17 items (55%) (Table 2). There were three items where large teams met the target rating of 4 but small teams did not: psychiatrist role on team, multidisciplinary team and practicing team lead. There was one item where small teams met the target rating of 4 but large teams did not: timely contact with referred individual. Mean item fidelity ratings were significantly different (p < 0.05) for only one item: practicing team lead(small = 3.4 vs. large = 4.5). For both large and small teams, there were 11 items with mean scores below 4.

Discussion

The present study compared model fidelity in large and small Ontario EPI programs. Our data showed that small teams delivered similar care to large teams in most areas of practice. The main differences identified pertained to team structure and how team members worked together, elements intended to support high-quality service delivery. Small teams were significantly less likely to have a practicing team lead. Although not significant, small teams were also less likely to achieve high fidelity for psychiatrist integration into the team and weekly team meetings. From the study data, it is unclear whether lower fidelity in team practices affected care delivery as few additional areas were identified where large teams performed better.

In addition, there were a number of items for which both large and small teams did not achieve satisfactory fidelity scores, especially related to psychosocial treatment delivery. Manualized models of EPI such as NAVIGATE and OnTrack may help strengthen consistency and quality of care (George et al. 2022). These models, however, expect that EPI is delivered by larger multidisciplinary teams with specialized skills. In Ontario, the feasibility of implementing these manualized models in smaller EPI teams and possible need for adaptations is currently being investigated (Kozloff et al. 2020). It is also possible that some low-performing items (e.g., active engagement and retention, which focuses on community visits) were impacted by the COVID-19 pandemic and will improve without system intervention.

Implications for research

An important next step is to examine the strategies used by small teams to support fidelity in order to identify effective strategies that could be implemented more widely. Small programs in Ontario historically operated as part of networks (Standards Implementation Steering Committee 2015), although current data are lacking. Network structures varied, but, similar to other jurisdictions (Behan et al. 2017; Pipkin 2021), they typically included a larger hub team supporting multiple smaller spokes or multiple small teams that partnered to support each other. Some networks were formalized and some were informal. Levels and types of support received from network partners ranged widely, including staff training, standardized tools, supervision and specialist consultation. Some small teams were also embedded within a general mental health team and leveraged supports from the broader team. For small teams, it is likely that an ability to leverage supports from partners, both EPI and non-EPI, is key to supporting fidelity.

It is also important to compare small teams to other strategies for rural delivery. Specialist outreach or multi-site programs are another potential approach to serving rural areas. In these models, a large central team delivers treatment to surrounding rural areas through telemedicine, travelling clinics, operating multiple sites and/or consultation to local providers. Currently, there is limited evidence on the relative effectiveness of these different models, and it has been suggested that different models may be necessary in different contexts (Behan et al. 2017; Cheng et al. 2014; Pipkin 2021).

Implications for policy and practice

The present study suggests that small teams may be a viable model for delivering EPI in regions that cannot support a full team. Formalizing and systematically implementing networks, particularly with the expansion in telemedicine use during the COVID-19 pandemic, may help support small team fidelity. Ongoing quality monitoring, including fidelity and outcome measurement, can help identify successful practices for serving rural and remote areas. There were also a number of items for which both large and small teams did not achieve satisfactory fidelity scores, suggesting that all programs require additional support to deliver the full model. Improving practice in these areas may require centralized intervention to provide clear guidance on expected practice, training, monitoring and coaching. In many jurisdictions, this role is performed by intermediary organizations, which work with programs and funders to support system implementation of evidence-based practices (Durbin et al. 2021; Proctor et al. 2019).

Limitations

A limitation of applied system evaluation is that measures must evolve in response to system needs and emerging evidence on best practices. In the present study, changes to the fidelity scale over time reduced the sample for some items. Participation in fidelity assessments was voluntary and it is possible that the study sample included higher-performing programs. That said, extensive efforts were used to encourage participation and the study sample included diverse programs from across the province. It is also possible that there are elements of practice where large and small programs differed that were not captured in the fidelity scale. The scale can only capture quality of delivery in a limited way and it does not include some elements of care (e.g., peer support), which are receiving increasing recognition as important components of EPI. The study does not allow us to reach conclusions on the minimum number of staff necessary to delivery EPI with fidelity; however, it provides some preliminary evidence on the feasibility of delivering the model with very small teams.

Conclusion

This study found that EPI programs with very small teams had similar fidelity scores to programs with larger teams for most elements of practice, suggesting that small teams may be a viable way to support equitable access to EPI services across the province. However, both large and small programs struggled to deliver almost a third of the elements of care. All programs may need centralized support to deliver the full model.

Correspondence may be directed to Avra Selick by e-mail at avra.selick@camh.ca.

Les petites équipes constituent-elles une stratégie viable pour offrir des services d'intervention précoce contre la psychose dans les régions rurales? Une étude ontarienne sur la fidélité

Résumé

L'intervention précoce en cas de psychose (IPE) est un modèle complexe de soins conçus pour être prodigués par une grande équipe multidisciplinaire. Cependant, dans la pratique, ces soins sont souvent fournis par de très petites équipes, en particulier dans les zones rurales. Cette étude analyse les données sur la fidélité de plus de la moitié des programmes d'IPE en Ontario (n = 24) afin de comparer les modèles de fidélité entre les équipes plus petites (≤2,1 employés) et plus grandes (≥4,3 employés). Peu de différences ont été relevées, ce qui laisse entendre que les petites équipes peuvent constituer une bonne solution de rechange pour offrir le modèle d'IPE, bien que les grandes et les petites équipes aient rencontré des défis pour offrir près du tiers des éléments de soins dudit modèle.

Comments

Be the first to comment on this!