Developing Indicators for the Child and Youth Mental Health System in Ontario
When the Government of Ontario launched a comprehensive mental health and addictions strategy, the Institute for Clinical Evaluative Sciences (ICES) was tasked with developing a scorecard for ongoing monitoring of the child and youth mental health system. Using existing administrative and survey-based healthcare and education data, researchers at ICES developed a scorecard consisting of 25 indicators that described at-risk populations, child and youth mental healthcare and relevant outcomes. This scorecard is the first in Canada to report on performance indicators for the child and youth mental health system and provides a model for monitoring child and youth mental health using routinely collected administrative data.
At any time, 14% of children and youth in Canada experience mental disorders (Waddell et al. 2012). Yet, only one in five receive the specialized services they need (Offord et al. 1987). In 2011, the Government of Ontario released a comprehensive mental health and addictions strategy for the province (Government of Ontario 2011). The first three years of this strategy focused on child and youth mental health (CYMH). As part of the strategy, the Institute for Clinical Evaluative Sciences (ICES) was tasked with developing a scorecard for the CYMH system in Ontario. Monitoring and reporting are important for policy and program planning; yet historically, there has been very little monitoring and reporting within the CYMH sector in Canada (Junek 2012). Therefore, the first CYMH scorecard produced by ICES would provide an important baseline against which to measure future performance.
The Mental Health and Addictions Scorecard and Evaluation Framework (MHASEF) Research Team at ICES set about this work with a vision to develop a scorecard that reflected the complex, multisectoral nature of CYMH care in Ontario using available data sources. The scorecard would be an empirical snapshot that described the contexts of the population at risk, delivery processes of CYMH care and relevant outcomes. In order to develop a robust scorecard, information from all sectors providing CYMH care and from all of Ontario would need to be captured.
MHASEF undertook a systematic approach to selecting indicators for the scorecard (Figure 1). In order to populate an inventory of existing indicators, a jurisdictional scan was conducted, which included grey and peer-reviewed literature and identified CYMH indicators used in English-speaking jurisdictions. To align the scorecard with Ontario's strategy, MHASEF team members gathered information from each of the four provincial ministries involved in order to develop logic models that described each strategy-funded initiative in terms of inputs, activities, outputs and outcomes (Goeschel et al. 2012). Strategy outcomes were compared to outcomes from the indicator inventory to identify relevant indicators. A feasibility review significantly reduced the list of potential indicators to those where provincial-level data were available. Throughout this process, MHASEF consulted with a scientific advisory committee of Canadian CYMH experts. In particular, this group provided critical input on the importance and face validity of the selected indicators. Reviews by the committee resulted in a final set of 25 indicators selected for reporting.
The Mental Health of Children and Youth in Ontario: A Baseline Scorecard was released in March 2015 (MHASEF Research Team 2015) and presented two types of indicators. Contextual indicators described prevalence rates of mental illnesses, service usage rates and relevant outcomes. System performance indicators analyzed how well the system met the needs of the population and could be used to track performance over time. The indicators were arranged under six domains (Table 1). Where feasible, indicators were reported over 10 years, from 2002/2003 to 2011/2012, to describe temporal trends. Indicators were also stratified by Local Health Integration Networks (LHINs) and by Child and Youth Mental Health Service Areas, the latter reflecting the sub-regional oversight of CYMH services.
|Table 1. Indicators included in the baseline scorecard|
|Domain: Known prevalence
|Domain: System use
|Mental health and addictions system performance indicators|
|Domain: Early identification
Whenever possible, indicators were stratified by age group (from birth to 24 years), sex, immigrant category (refugees, non-refugee immigrants and non-immigrants) and neighbourhood income quintile. In recognition of the years of young adulthood as a significant time of transition and onset of mental illnesses (Kessler et al. 2005; Pottick et al. 2008), the population we measured included individuals up to 24 years of age. The scorecard and indicator results are available online at www.ices.on.ca.
The CYMH system has been described as fragmented (Kutcher and McLuckie 2010; Office of the Auditor General of Ontario 2008); as such, it is critical to measure its performance to be able to address gaps in care. The scorecard produced by MHASEF is a first step in understanding the complex CYMH system in Ontario and provides a baseline against which to compare the system's future performance. Although the indicators reported were limited in scope, quantifying CYMH system performance was lauded as an important development by policy makers and stakeholders.
Available data sources allowed MHASEF, for the first time, to report systematically on the CYMH system in Ontario. Since the ICES administrative databases contain information on the entire population of Ontario, indicators were reported on all children and youth in the province, thus avoiding issues of representativeness and selection bias. The scorecard also commented on the service user population in a number of settings, including schools (using non-ICES, aggregate-level education data), provincial correctional centres (very roughly and proximally), physician offices and hospitals. Finally, linking data across multiple databases allowed MHASEF to develop new indicators such as a measure of "first contact" and to apply an equity lens to each indicator by stratifying results by a number of determinants of health. Compared to other scorecards where equity may be separated into an independent category of indicators, stratifying each indicator by the determinants of health allowed for a more nuanced interpretation of findings.
We were unable to measure the performance of the entire CYMH system. Community-based children's mental healthcare was a major gap in the scorecard. The ICES data holdings are largely comprised of health administrative data. While a scan conducted by ICES revealed multiple Ontario CYMH data sources that were theoretically available, they were not centralized, were governed by different legislative authorities than healthcare data, and were, therefore, not available for use by ICES for system-level CYMH performance measurement. Limited data availability meant that the indicators were restricted mainly to the healthcare sector. This limitation has resulted in an interministerial data strategy to move toward more comprehensive mental health system performance reporting, including contributions of programs and services within other sectors, such as community-based care, education and justice.
There were other notable limitations to the scorecard development process. A structured consensus process, such as a Delphi panel, was not used for indicator selection. However, the majority of indicators were previously developed, often through rigorous processes, and CYMH experts and policy makers were regularly consulted to ensure that the scorecard was relevant. Data could not be linked across sectors, thus restricting the ability to describe a client's journey through the entire CYMH system, and data on certain populations of interest, such as Indigenous groups, Francophones and other ethnocultural groups, were not available.
Indicators are descriptive in nature. However, they point to areas where further targeted investigation would be valuable. For example, based on indicator results, MHASEF researchers have pursued inquiry into an observed increase in mental health–related emergency department visits and hospitalizations (Gandhi et al. 2016) and mental health service use by youth in provincial correctional centres (Khan et al. 2016). Results concerning rates of neonatal abstinence syndrome have also been used to benchmark against other jurisdictions (Davies et al. 2015). Other studies stemming from the scorecard are forthcoming.
Until recently, no monitoring reports for CYMH had been produced by any federal, provincial or territorial government in Canada (Junek 2012). This scorecard is the first in Canada to report on performance indicators for CYMH services. Internationally, Scotland (Tod et al. 2013) and Australia (National Mental Health Performance Subcommittee 2013) are the only other English-speaking jurisdictions that measure CYMH system performance. While other organizations in Canada have produced CYMH indicators in recent years (Canadian Institute for Health Information 2015; Mental Health Commission of Canada 2015), these indicators were descriptive in nature and focused on describing prevalence and service utilization.
This scorecard is a first step in understanding the complex CYMH system and provides a baseline against which to compare future trends in system use and outcomes within Ontario. MHASEF has committed to repeating the scorecard every two years to allow for continuous monitoring. This will also provide an opportunity to improve upon the reported indicators and to add new ones through a process of continuous quality improvement. An opportunity exists to develop standardized reporting across the country. As most of the indicators included in this scorecard are calculated from routinely collected sources of administrative data, the indicators in the scorecard can be replicated in other Canadian jurisdictions as well as internationally. Enhancing performance monitoring will lead to better policy and program planning and to improved outcomes for children and youth.
About the Author
Julie Yang, MA, is a senior research project manager at ICES.
Paul Kurdyak, MD, PhD, FRCPC, is a senior scientist and program lead for the Mental Health and Addictions Research Program at ICES, director of the Health Outcomes and Performance Evaluation Research Unit at the Centre for Addiction and Mental Health and associate professor in the Department of Psychiatry and the Institute of Health Policy, Management and Evaluation at the University of Toronto.
Astrid Guttmann, MDCM, MSc, FRCPC, is a senior scientist and chief science officer at ICES, staff paediatrician in the Department of Paediatrics, The Hospital for Sick Children, and associate professor at the Institute of Health Policy, Management and Evaluation at the University of Toronto.
The authors thank MHASEF Research Team members, John Cairney, Sima Gandhi, Karey Iron, Saba Khan and Kelvin Lam.
MHASEF is funded by the Ontario Ministry of Health and Long-Term Care.
Canadian Institute for Health Information. 2015. Care for Children and Youth with Mental Disorders. Ottawa, ON: Author. Retrieved July 13, 2016. <https://secure.cihi.ca/free_products/CIHI%20CYMH%20Final%20for%20pubs_EN_web.pdf>.
Davies, H., R. Gilbert, K. Johnson, I. Petersen, I. Nazareth, M. O'Donnell et al. 2015. "Neonatal Drug Withdrawal Syndrome: Cross-Country Comparison Using Hospital Administrative Data in England, the USA, Western Australia and Ontario, Canada." Archives of Disease in Childhood Fetal and Neonatal Edition 101(1): F26–30.
Gandhi, S., M. Chiu, K. Lam, J.C. Cairney, A. Guttmann and P. Kurdyak. 2016. "Mental Health Service Use Among Children and Youth in Ontario: Population-Based Trends Over Time." Canadian Journal of Psychiatry 61(2): 119–24.
Goeschel, C.A., W.M. Weiss and P.J. Pronovost. 2012. "Using a Logic Model to Design and Evaluate Quality and Patient Safety Improvement Programs." International Journal for Quality in Health Care 24(4): 330–37.
Government of Ontario. 2011. Open Minds, Healthy Minds: Ontario's Comprehensive Mental Health and Addictions Strategy. Toronto, ON: Queen's Printer for Ontario. Retrieved July 13, 2016. <http://www.health.gov.on.ca/en/common/ministry/publications/reports/mental_health2011/mentalhealth_rep2011.pdf>.
Junek, W. 2012. "Government Monitoring of the Mental Health of Children in Canada: Five Surveys (Part I)." Journal of the Canadian Academy of Child and Adolescent Psychiatry 21(1): 30–36.
Kessler, R.C., P. Berglund, O. Demler, R. Jin, K.R. Merikangas and E.E. Walters. 2005. "Lifetime Prevalence and Age-of-Onset Distributions of DSM-IV Disorders in the National Comorbidity Survey Replication." Archives of General Psychiatry 62(6): 593–602.
Khan, S., M. Chiu, A.I. Simpson, A. Guttmann, N. Jembere and P. Kurdyak. 2016. "Use of Mental Health Services by Youths and Young Adults Before and During Correctional Custody: A Population-Based Study." Psychiatric Services 67(7): 790–93.
Kutcher, S. and A. McLuckie for the Child and Youth Advisory Committee, Mental Health Commission of Canada. 2010. Evergreen: A Child and Youth Mental Health Framework for Canada. Calgary, AB: Mental Health Commission of Canada. Retrieved July 13, 2016. <http://www.mentalhealthcommission.ca/English/media/3154>.
Mental Health Commission of Canada. 2015. Informing the Future: Mental Health Indicators for Canada. Ottawa, ON: Author. Retrieved July 13, 2016. <http://www.mentalhealthcommission.ca/English/focus-areas/mental-health-indicators-canada>.
MHASEF Research Team. 2015. The Mental Health of Children and Youth in Ontario: A Baseline Scorecard. Toronto, ON: Institute for Clinical Evaluative Sciences. Retrieved July 13, 2016. <http://www.ices.on.ca/~/media/Files/Atlases-Reports/2015/Mental-Health-of-Children-and-Youth/Full-report.ashx>.
National Mental Health Performance Subcommittee. 2013. Key Performance Indicators for Australian Public Mental Health Services (3rd ed.). Canberra, AU: Commonwealth of Australia.
Office of the Auditor General of Ontario. 2008. "Child and Youth Mental Health Agencies." In 2008 Annual Report of the Office of the Auditor General of Ontario. Toronto, ON: Author. Retrieved July 13, 2016. <http://www.auditor.on.ca/en/content/annualreports/arreports/en08/304en08.pdf>.
Offord, D.R., M.H. Boyle, P. Szatmari, N.I. Rae-Grant, P.S. Links, D.T. Cadman et al. 1987. "Ontario Child Health Study: II. Six-Month Prevalence of Disorder and Rates of Service Utilization." Archives of General Psychiatry 44(9): 832–36.
Pottick, K.J., S. Bilder, A. Vander Stoep, L.A. Warner and M.F. Alvarez. 2008. "US Patterns of Mental Health Service Utilization for Transition-Age Youth and Young Adults." Journal of Behavioral Health Services and Research 35(4): 373–89.
Tod, E., J. Parkinson and G. McCartney. 2013. Scotland's Mental Health: Children and Young People 2013. Edinburgh, UK: NHS Health Scotland. Retrieved July 13, 2016. <http://www.scotpho.org.uk/downloads/scotphoreports/scotpho131219-mhcyp2013-fullreportv2.pdf>.
Waddell, C., D.R. Offord, C.A. Shepherd, J.M. Hua and K. McEwan. 2002. "Child Psychiatric Epidemiology and Canadian Public Policy-Making: The State of the Science and the Art of the Possible." Canadian Journal of Psychiatry 47(9): 825–32.
Be the first to comment on this!
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed