Healthcare Policy

Healthcare Policy 5(2) November 2009 : e177-e186.doi:10.12927/hcpol.2013.21180
Research Paper

Indicators for Measuring Mental Health: Towards Better Surveillance

Cara Tannenbaum, Joel Lexchin, Robyn Tamblyn and Sarah Romans

Abstract

Accurate measurement and improvement of population mental health requires the recording of indicators that capture the full spectrum of disease severity. This paper describes four different strategies for measuring the prevalence of depression and anxiety in Canada based on data from the 2002 Canadian Community Health Survey – Mental Health and Well-being (Cycle 1.2) and the 2003 Quebec medical services claims database. The use of multiple indicators provides a more comprehensive picture of mental health needs than a single indicator alone. However, the validity of these indicators raises certain challenges and highlights the complexity of obtaining valid and sustainable measurements of mental health problems over time. We include a discussion of problems related to information availability and management.

The use of health indicators for mental health surveillance is critically lacking. Mental health indicators currently in use in Canada and elsewhere include suicide rates, hospitalization rates (e.g., in-hospital stay or discharge data), utilization rates of health resources (e.g., number of psychiatrists or psychiatric beds per capita) and self-reported use of mental health services or disorders (e.g., derived from national surveys). Unfortunately, these mental health indicators do not capture the broad spectrum of severity that characterizes this field. Suicide represents an extreme and often acute manifestation of distress. Monitoring hospital service use captures only the very small proportion of individuals who are very sick and need to receive in-hospital treatment, compared to the much larger contingent who receive out-patient or community services. Self-reported disorders in surveys often fail to capture isolated, subthreshold or short-lived cases, and mild cases that could be detected and treated early are also missed. Canada has recently created a Mental Health Commission to develop a national mental health strategy (Mental Health Commission 2008; Standing Senate Committee 2006). However, without an effective surveillance system that provides ongoing comprehensive and timely information on the entire spectrum of population mental health, it will be challenging to plan and allocate mental health resources, monitor the effectiveness of new policies and programs, and assess the success of mental health reform.

As part of a larger project on health indicators funded by the Health Policy Research Program of Health Canada, we sought to evaluate available measures of mental illness that span a wider spectrum of disease severity than those currently in use. Key criteria for indicator selection included ensuring that the indicators are measurable, verifiable, meaningful, policy relevant and capable both of assessing trends over time and of providing key regional, provincial, demographic and socio-economic breakdowns. We felt that there were distinct advantages to using existing databases and therefore considered potential data sources, data availability and the challenges of reconciling different databases. We believed that a variety of databases could be used to strengthen the accumulated weight of evidence derived from disparate sources, and at the same time not add any significant administrative burdens that would be necessitated by the creation of entirely new databases.

Canada has the privilege of having population-based administrative databases that document all services provided to the population by physicians under public insurance (approximately 99% of physicians). In addition, Statistics Canada has surveyed Canadians about the state of their mental health through the 2002 Canadian Community Health Survey – Mental Health and Well-being (CCHS Cycle 1.2). Our main objective was to use data from these existing data sources to explore the feasibility of selecting alternative health indicators for depression and anxiety, to be used as part of a national surveillance system on mental health. We chose depression and anxiety as prototype mental health symptoms/disorders as these are two of the most common neuro-psychiatric illnesses leading to days off work, unemployment and years of life lived with disability (WHO 2001; Health Canada 2002; Lim et al. 2000).

Methods

We evaluated four potential indicators of depression and anxiety in the adult (aged 18 and over) Canadian population: self-reported subthreshold mental health symptoms, self-reported full diagnostic disorders, physicians' billings for outpatient mental health visits and use of psychotropic medications.

For self-reported mental health symptoms, we analyzed data from the participants in the 2002 CCHS. The CCHS is a key component of the Population Health Surveys Program of Statistics Canada that aids in the development of public policy; provides data for analytic studies that will assist in understanding the determinants of health; collects data on the economic, social, demographic, occupational and environmental correlates of health; and aims to increase the understanding of the relationship between health status and healthcare utilization. Statistics Canada conducted Cycle 1.2 over the period May 2002 to December 2002, and focused on collecting mental health data from a nationally representative sample of people aged 15 and older (Statistics Canada 2003). The CCHS content was based on a selection of mental disorders from the World Mental Health Survey; input from an expert group of mental health professionals guided the content development and strategic direction of the study (Statistics Canada 2003; World Mental Health Survey 2005).

We included two different measures for self-reported diagnostic disorders: those respondents reporting symptoms that meet the full, gold-standard Diagnostic and Statistical Manual IV (DSM-IV) criteria for depressive and anxiety disorders, and those reporting any two or more criteria for a mental health disorder (a subthreshold condition) (Rowe and Rappaport 2006). For assessment of mental disorders, the CCHS 1.2 employed a Canadian adaptation of the Composite International Diagnostic Interview (CIDI) version used in the WHO Mental Health 2000 initiative (Kessler and Ustun 2004). Only individuals aged 18 years and older living in the 10 provinces across the country were included in the analysis, yielding a final sample of data from 15,889 men and 19,347 women.

The physician billings mental health indicator was derived from the medical services claims database from the province of Quebec. Because each province maintains separate healthcare systems, it was not possible to combine data nationwide. The province of Quebec was selected owing to immediate availability of the database to the study investigators; it covers the costs of essential medical care for 8.5 million provincial residents. The Quebec health insurance agency (RAMQ) is responsible for beneficiary enrolment and reimbursement of all physicians. The RAMQ maintains a database of all Quebec beneficiaries (name, age, sex, residence) and all medical services received by beneficiaries (date, diagnosis, type and location of service and provider). For the physician billings indicator, we calculated the proportion of mental health ICD-9 diagnostic billing codes covering depression and anxiety disorders entered by 28,426 Quebec physicians on 3,204,637 unique patients (55% female, 45% male) aged 18 years and older in the 2003 RAMQ medical services claims database. We report unadjusted results as well as adjusted results, by 0.87 for women and by 0.75 for men, to obtain population prevalence. Only 87% of Canadian adult women and 75% of Canadian adult men consult a physician each year for their health (Statistics Canada 2003). Without the adjustment quotient, outpatient visits for mental health symptoms could be overestimated, assuming that those who do not consult do not have health problems.

We validated billings by general practitioners in the Quebec RAMQ database as an indicator of the proportion of the population with a mental health disorder by using data from the Medical Office of the 21st Century Study (MOXXI). MOXXI is a Quebec-based research project testing the potential benefits of implementing an electronic prescription, drug and disease management system for primary care physicians and other healthcare practitioners (Tamblyn et al. 2006). As part of this project, 100 physicians were asked to verify the accuracy of 340 diagnostic codes for 330 unique patients against their electronic patient chart records. Cases billed for these 330 patients by generalists or specialists for anxiety or depressive disorders were abstracted from these 340 records and presented back to primary care physicians with the question, "Does your patient have this problem, yes or no?" Out of 125 cases of depression that were billed and reviewed, 119 (95%) were coded correctly. Out of 112 cases of anxiety that were billed and reviewed, 110 (98%) were coded correctly.

To derive an indicator based on medication use, we used the Canadian Community Health Survey – Mental Health and Well-being (CCHS Cycle 1.2 for 2002) database. Psychotropic drug use was self-reported, with respondents being asked whether in the past 12 months they had taken any medication to reduce anxiety or nervousness or for depression (Statistics Canada 2003). If respondents reported use of at least one drug from these classes, the CCHS 1.2 collected more comprehensive data by asking respondents to produce medication containers for drugs used in the two days preceding the interview. Self-reported medication names were recorded and coded with a modification of the WHO's anatomical therapeutic chemical codes (WHO Collaborating Centre 2005). We operationally defined psychotropic drugs as those used for the treatment of any type of depression or anxiety. Data from 15,889 men and 19,347 women aged 18 years and older were included in the analysis.

Results

Table 1 shows the annual prevalence derived from these four potential indicators of depression and anxiety. The lowest estimate, 8% of women and 4% of men, comes from self-reported use of medications for these two disorders. Intermediate estimates are based on the proportion of the population who reported symptoms that fulfilled the DSM-IV diagnostic criteria for depression or anxiety. The highest estimates are based on the percentage of the population with a subthreshold diagnosis and the proportion of diagnostic billing codes for either depression or anxiety. It is noteworthy that despite being derived from two different population sets (CCHS respondents from across Canada and Quebec residents' visits to doctors), these latter two estimates are similar – 20% of women and 14% of men, based on CCHS data and the adjusted physician billings data.


 

TABLE 1. Prevalence of depression and anxiety in Canadian men and women aged 18 years and older using different indicators
Annual prevalence (% of population)
  Self-reported symptoms (subthreshold DSM-IV depression and anxiety disorders) Self-reported criteria fulfilling DSM-IV diagnosis of either depression or anxiety Proportion of physician billings for anxiety or depression (unadjusted estimate) Proportion of physician billings for anxiety or depression (adjusted estimate*) Self-reported use of medications for depression and anxiety
Women 20 11 24 20 8
Men 14 10 18 14 4
* Adjustment: Prevalence from RAMQ data multiplied by 0.87 for women and by 0.75 for men according to the proportion of women and men consulting any MD per year according to CCHS Cycle 1.1 data

Discussion

Estimates of the proportion of the Canadian population with depression and anxiety vary widely according to the different indicators used, ranging from 8% to 24% in women, and from 4% to 18% in men. Because each indicator taps a different aspect of mental illness, it is to be expected that the prevalence would get smaller at each step in the therapeutic continuum. Not all subthreshold symptom clusters become a threshold syndrome – which in turn may not warrant a decision to seek treatment – and not all treatment involves medication. Rather than relying on a single set of numbers, the approach we have taken, of comparing and contrasting the results from multiple databases, paints a more informative profile of the different aspects of Canada's overall mental health state, according to different levels of severity. If showcased alongside national suicide rates and psychiatric hospital admission rates, for example, these indicators would enable decision-makers to better predict the need for mental health services across the spectrum of care.

Although our study did not permit repeated measurement of these indicators over time, this would be easy to accomplish. For the physician billings indicator, measurement could be carried out at any time, and the indicators derived from the CCHS could be reassessed with every subsequent cycle of the national survey (every two to three years). As long as the possibility exists for repeated measurements, and the methods for measuring each indicator do not change, these indicators would be useful for surveillance purposes. However, should the questions in the national surveys change and not measure exactly the same symptoms, or if new billing codes were introduced for physicians, or more physicians opted out of public coverage, then the indicators would not be comparable over time.

In order for surveillance to be effective, the importance of linkage between data sets and across time needs urgently to be addressed. Privacy regulations in Canada do not permit an assessment of the extent to which people in the estimates overlap. Nor is it possible to determine remissions, incident cases, or progression from mild symptoms to full-fledged DSM-IV diagnoses. Statistics Canada and other government departments in possession of health data need to allow greater access to the data and facilitate longitudinal follow-up by qualified health researchers, as is the case in the United States and the United Kingdom. National data collection initiatives such as the CCHS must be sustained, and provinces must work together to integrate and validate administrative data. Furthermore, for indicators to be both meaningful and policy relevant, and to provide useful information to policy makers seeking to improve mental health conditions, they must also address underlying root causes of distress and not focus solely on symptomatic outcomes. Individually linked longitudinal data that allow assessments of depth and length of time in low-income circumstances, labour force data and data on unpaid work, time stress and rates of chronic stress all need to be considered during mental health surveillance.

Strengths and limitations

The strengths, limitations and biases inherent in the indicators described in this paper must be acknowledged (Table 2). In the absence of a gold standard or biologic indicator of a mental disorder, there is no way of determining the accuracy of these estimates. The stigma associated with mental health disorders may lead to underreporting of symptoms by participants in the CCHS survey data, as well as by physicians in their billing practices, and thus create problems in predicting the need for mental health services (Patten 2008). Future surveys should consider adding questions about people's perceived need for mental health services as there may be a gap between meeting diagnostic criteria and wanting help. Since medication use and physicians' diagnoses contained in administrative databases do not differentiate the severity of disease, new surveys should also incorporate questions that capture this variable within a DSM-IV diagnosis. The psychotropic drug indicator offers important information about the use of pharmaceutical management, but may also say more about patient preferences for treatment and the prescribing practices of physicians than about the severity of symptoms per se. A more reliable measure of medication consumption for mental illness could be derived from pharmacy databases across the country once all provinces collect data on all prescriptions filled. In countries where centralization of electronic records for dispensation of pharmaceuticals has already occurred, this indicator is easier to extract.


 

TABLE 2. Strengths and weaknesses of the various indicators for deriving estimates of the proportion of the Canadian population aged 18 and older with depression and anxiety
Criteria Method of estimating proportion of the population with anxiety or depression
Self-reported symptoms (subthreshold symptoms and diagnostic disorders) Proportion of physicians’ billings for mental health complaints Self-reported use of medications for depression and anxiety
Feasibility of use Currently can be derived only from national surveys on mental health, which are costly, and are usually conducted only every 5 to 10 years. Could be derived from existing fee-for-service databases that are available in most provinces. Alternatively, electronic medical records (EMRs) will soon be universally available, and this indicator could be derived from the EMR database. Could be easily derived from existing pharmaceutical databases that are available in most provinces in addition to reporting in national health surveys on mental health.
Gender sensitivity Women are more likely to express distress through symptoms of depression and anxiety. Men are more likely to develop problems with alcohol and illicit substance use. Need to adjust for differential health-seeking behaviours by men and women, especially for mental health symptoms. Takes into account higher rates of prescribing and drug use among women, as well as more frequent health service utilization by women.
Capable of providing key regional, provincial and demographic breakdowns Yes Yes Yes
Advantages Use of DSM-IV diagnostic criteria (and subthreshold criteria) make international comparisons possible if mental health surveys using the same instruments are conducted in other countries. 1) Captures both outpatient and inpatient healthcare system utilization for mental distress and disorders.
2) ICD-9 (now ICD-10) codes are used internationally for physician billing.
3) Captures people with emotional distress who seek alternative forms of treatment such as counselling or psychotherapy (within the part of the healthcare system paid for publicly).
International comparisons are possible given accepted international standards for classifying prescription medications, and a growing body of literature on how to measure drug use accurately across populations and time.
Disadvantages 1) Recall bias or reluctance to admit to stigmatizing mental health symptoms might lead to underestimation of prevalence rates.
2) Because the surveys are conducted only every 5 to 10 years, the data may not be up to date. Expense and effort required to conduct surveys mean these data are not sensitive to short-term changing trends.
1) Does not capture people with distress who do not seek care from physicians or who seek care from salaried physicians or other providers outside the public healthcare system.
2) Underestimation of diagnostic claims may occur because mental health providers tend to list the least stigmatizing diagnosis on the billing claims form for reasons of patient confidentiality.
3) Overestimation of procedure codes for counselling may occur if physicians bill discussion of other health conditions under the counselling code.
1) Psychotropic drugs are a mixed group of medications, used for indications other than mental health complaints.
2) Issues with interpretation include erroneous judgments on under- or overtreatment for men and women.
3) Danger of reflecting marketing practices and physician prescribing patterns more than patient disease levels.

Conclusion

Our study highlights clear advantages of using multiple data sources to monitor and track the full spectrum of our population's mental health, and also illustrates that a single measure is not adequate to uncover the severity of disease within a single population. Mental health surveillance will require a conglomerate of indicators, and would best be served by including upstream determinants of mental health in addition to downstream symptomatic outcomes. The potential for linking data across data sources and time, as well as the privacy issues involved in such an endeavour, urgently require regulation in order to accelerate effective mental health surveillance in Canada. The use of a standardized set of indicators that takes into account health determinants, the severity of symptoms and the use of healthcare services would permit more useful international comparisons.


Indicateurs de mesure pour la santé mentale : vers une meilleure surveillance

Résumé

L'établissement d'indicateurs pour l'ensemble de l'indice de gravité des maladies est essentiel pour obtenir des mesures précises et pour améliorer la santé mentale de la population. Cet article décrit quatre stratégies pour mesurer la prévalence, au Canada, de la dépression et de l'anxiété, selon les données de l'Enquête sur la santé dans les collectivités canadiennes – Santé mentale et bien-être 2002 (cycle 1.2) ainsi que celles de la base de données 2003 de la RAMQ (Régie de l'assurance maladie du Québec). L'utilisation d'indicateurs multiples permet une meilleure compréhension des besoins en santé mentale que l'utilisation d'un seul indicateur. Cependant, la validité de ces indicateurs pose certains défis et fait voir la complexité quant à l'obtention de mesures valables et durables en matière de problèmes de santé mentale. Nous discutons ensuite des problèmes liés à la disponibilité et à la gestion de l'information.

About the Author

Cara Tannenbaum, MD, Associate Professor of Medicine, Centre de recherche, Institut universitaire de gériatrie de Montréal, Faculty of Medicine, University of Montreal, Montreal, QC

Joel Lexchin, MD, School of Health Policy and Management, York University, Emergency Department, University Health Network, Department of Family and Community Medicine, University of Toronto, Toronto, ON

Robyn Tamblyn, PHD, Department of Epidemiology and Biostatistics, Faculty of Medicine, McGill University, Montreal, QC

Sarah Romans, MD, Women's College Research Institute, Department of Psychiatry, University of Toronto, Toronto, ON

Correspondence may be directed to: Cara Tannenbaum, MD, Associate Professor of Medicine, University of Montreal, Centre de recherche, Institut universitaire de gériatrie de Montréal, 4565 Queen Mary Rd., Suite 7824, Montreal, QC H3W 1W5; tel.: 514-340-3540, ext. 2526; fax: 514-340-3530; e-mail: cara.tannenbaum@umontreal.ca.

Acknowledgment

The authors gratefully acknowledge the assistance of Hana Partlova and Wei Zhou in gathering and analyzing the data throughout the course of the project.

The authors also extend appreciation to the advisory board members for their invaluable contribution, including Madeline Boscoe, RN, Clarence Clottey, MD, MPH, Ronald Colman, PhD, Janet Currie, MSW, Sharon Davis-Murdoch, MA, Mireille Kantiebo, MSc, PhDc, Nancy Mayo, PhD, Robert Perreault, MD, Nancy Poole, MA, Michel Préville, PhD, Alex E. Schwartzman, PhD, Lisa A. Serbin, PhD, Donna E. Stewart, MD, FRCPC and Bilkis Vissandjée, PhD. All are dedicated individuals committed to improving the mental health of adult men and women in Canada.

References

Health Canada. 2002. A Report on Mental Illnesses in Canada. Retrieved October 17, 2009. <http://www.phac-aspc.gc.ca/publicat/miic-mmac/index-eng.php>.

Kessler, R.C. and T.B. Ustun. 2004. "The World Mental Health (WHM) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI)." International Journal of Methods in Psychiatric Research 13(2): 93–121.

Lim, D., K. Sanderson and G. Andrews. 2000. "Lost Productivity among Full-Time Workers with Mental Disorders." Journal of Mental Health Policy and Economics 3(3): 139-46.

Mental Health Commission of Canada. 2008. Retrieved October 17, 2009. <http://www.mentalhealthcommission.ca/Pages/index.html>.

Patten, S. 2008. "Major Depression Prevalence Is Very High, But the Syndrome Is a Poor Proxy for Community Populations' Clinical Treatment Needs." Canadian Journal of Psychiatry 53: 411–19.

Rowe, S.K. and M.H. Rappaport. 2006. "Classification and Treatment of Sub-Threshold Depression." Current Opinion in Psychiatry 19: 9–13.

Standing Senate Committee on Social Affairs, Science and Technology. 2006. Out of the Shadows at Last: Transforming Mental Health, Mental Illness and Addiction Services in Canada. Retrieved October 17, 2009. <http://www.parl.gc.ca/39/1/parlbus/commbus/senate/Com-e/SOCI-E/rep-e/rep02may06-e.htm>.

Statistics Canada. 2003. Canadian Community Health Survey, Cycle 1.2. Mental Health and Well-being. Questionnaire. Retrieved October 17, 2009. <http://www.statcan.gc.ca/imdb-bmdi/instrument/5015_Q1_V1-eng.pdf>.

Tamblyn, R., A. Huang, Y. Kawasumi, G. Bartlett, A. Jacques, M. Dawes, M. Abrahamowicz, R. Perreault, L. Taylor, N. Winslade, L. Poissant and A. Pinsonneault. 2006. "The Development and Evaluation of an Integrated Electronic Prescribing and Drug Management System for Primary Care." Journal of the American Medical Informatics Association 13(2): 148–59.

World Health Organization (WHO). 2001. The World Health Report 2001. Mental Health: New Understanding, New Hope. Geneva: Author.

WHO Collaborating Centre for Drug Statistics Methodology. 2005. About the ATC/DDD System. Oslo: Norwegian Institute of Public Health.

World Mental Health Survey Initiative. 2005. Retrieved October 9, 2009. <http://www.hcp.med.harvard.edu/wmh>.

Footnotes

The project received a financial contribution from the Health Policy Research Program, Health Canada Project #6795-15-2003/6380006. The views expressed in this report are those of the authors and members of the advisory board committee and do not necessarily represent the official policy of Health Canada. All analyses, interpretations and viewpoints expressed, as well as any errors or misinterpretations, are the sole responsibility of the authors.

Comments

Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed