How does a society know if the health and well-being of children and youth are improving, staying the same or getting worse? If one was to choose a manageable set of indicators to follow over a 20-year period, what would it look like? Since we know that what gets measured focuses attention, programming and funding, the question is this: what measures, covering which aspects of the lives of children and youth, should be selected from a population perspective? Given the variety of world views that exist among the many child and youth service providers, as well as the multiple definitions of health and well-being, this question is particularly difficult to answer. This is what the provincial health officer (PHO) of British Columbia is facing with an upcoming report that will look at the health and well-being of children and youth in the province. To answer these questions the Office of the PHO has partnered with the Canadian Institute for Health Information (CIHI) to identify a set of indicators to define and track child health and well-being in British Columbia. The process of identifying this suite of indicators is committed to transparency, evidence and collaboration.

Over a decade ago, the PHO (1998) published a comprehensive report on the health of children in British Columbia, with more focused reports since then. At the time, the report was groundbreaking in that it looked at child health beyond physical health and considered how the social determinants of health impacted the lives of children. Building on this tradition, the PHO's vision for the updated report is that it will identify the factors and modifiable conditions and actions that truly make the most difference to both positive and negative child and youth health and well-being outcomes, and will inform health system decision-making in terms of policy, programs and services aimed at improving the lives of children in British Columbia. The PHO's goal is to have a sustainable, solid measurement system that will support consistent and ongoing reporting over many years.

Project Governance

The project engages multiple government ministries and stakeholders involved in the delivery of services and programs. The structure of the project consists of three supporting bodies:

• 1. The Project Advisory Committee consists of senior government representatives from the health and social ministries and regional authorities, child health academics and other content experts. The purpose of the committee is to provide overall guidance and direction to the project.
• 2. The Technical Advisory Committee consists of operational staff and methodologists from government, academia and the health information realm to provide guidance on methodological issues such as survey design and interpretation.
• 3. The Project Working Group, consisting of the deputy PHO and representatives from the Ministry of Health, CIHI and the representative for children and youth, is responsible for the day-to-day coordination and administration of the project and for making recommendations to the Project Advisory Committee.

Project Overview

The project can be roughly broken down into two phases with five major tasks: (1) the conceptual framework – the development of a discussion paper and framework, and an expert workshop; and (2) indicator selection – relevance review, evidence review and the final selection of indicators.

Starting Point: A Conceptual Framework

Indicator selection initiatives, whether at the local, provincial, national or international level, typically involve obtaining expert feedback and conducting some form of consensus building on reportable indicators. However, selecting an agreed-upon set of indicators from diverse stakeholder groups is challenging: healthcare experts, families, communities and policy-makers all have differing points of view and bear different responsibilities for protecting and promoting children's health and well-being. In addition to relying on expert advice, indicators are frequently selected for reporting because they have been reported on in the past or because the data are conveniently available. Over 2,500 individual indicators have been identified that could potentially be used to describe children's lives (Hanafin and Brooks 2009). With so many indicators available, a legitimate question this project had to contend with was, why not adopt an existing set of indicators developed by another group?

A unique contribution this project makes to the field of child indicators is the methodology; it began by looking at what creates health and well-being in children prior to identifying indicators. As a starting point, this project undertook an extensive review of the literature on healthy child development. The review produced a discussion paper, synthesizing the literature on what creates child health and well-being into a framework, and highlighting the importance of establishing criteria for selecting indicators prior to beginning the exercise

In September 2009, Dr. Ben-Arieh, an international expert on child health indicators, was invited to consult with members of the project team and comment on the discussion paper. His feedback was integral to the completion of the framework, which formed the basis for a validation workshop held in November 2009. Dr. Ben-Arieh stressed that no set of child health indicators would apply across all countries for all time. The important principle was to identify indicators applicable to British Columbia's children and youth that are transparent and defensible based on existing evidence. This theme was echoed by workshop participants.

The 56 workshop participants – selected experts in child health and well-being, policy development and indicator development – provided feedback on the indicator framework and indicator selection criteria. The purpose of the workshop was to promote a common understanding of the environments, opportunities, intrinsic factors, determinants and supports in a child's development that are most likely to attribute to either positive or negative outcomes; and the interventions which are most likely to result in changes to the desired outcomes in health and well-being. Five overarching dimensions of child and youth health and well-being were identified: social relationships, cognitive development, economic and material well-being, physical health and well-being and mental and emotional health and well-being (Figure 1).

Indicator Selection

The second stage of the project was to identify and select the most appropriate indicators. The goal of the indicator selection was twofold:

• To identify reportable indicators, those for which evidence exists (e.g., linked to positive or negative outcomes in the literature, strong data exist to measure the indicator) to support their inclusion in the PHO's report
• To identify information gaps, specifically aspects of the framework for which there is not sufficient evidence to include an indicator or for which indicators may not yet exist; these gaps represent opportunities for future measurement and reporting, addressing issues for which significant gains could be made in health and well-being as suggested by early-stage research

The discussion paper identified over 240 potential indicators and "concepts" corresponding to the dimensions and ecologies of the framework. In many cases, specific and definable indicators were not available, so the concept was defined as an area for measurement that may have multiple and possibly overlapping indicators. For example, one aspect of physical health is the concept of mortality, which can be measured by indicators such as infant mortality rates and perinatal mortality rates. It is also possible to "drill down" and measure mortality from a variety of causes (e.g., poisoning, motor vehicle accidents etc.). Therefore, it was decided to apply the filtering process to the concepts rather than the indicators, to reduce both the potential confusion between indicators and concepts and the potential burden on experts evaluating as many as 240 indicators. Identifying indicators aligned with specific concepts would happen at a later stage in the process. The process for selecting indicators involved a series of "filters" (Figure 2).

The team identified 125 concepts as likely areas for measurement that would provide the most comprehensive view of the health and well-being of children and youth. To maximize efficiency, the Project Working Group decided to apply the relevance filter before the evidence filter. Identifying relevance first ensured that no potential concepts or indicators were excluded due to a lack of current evidence. One of the objectives of the project was to identify information gaps; identifying relevant indicators even where data sources did not exist helped to achieve this goal.

Relevance Assessment

A modified Delphi approach, engaging validation workshop participants and other child health experts, was used to assess the relevance of the candidate concepts through an online survey. Participants were asked to assess the concepts against four criteria:

1. Relevance to policy (concepts should be amenable to effective action through policy, programs and services)
2. Significance to the health and well-being of children (refers to the combination of the magnitude of the issue being measured, i.e., its prevalence, and the impact it has on the health of children, i.e., its severity)
3. Ability to be easily understood by multiple stakeholders (concepts should be clear and easily understood by a wide variety of stakeholders, including the general public)
4. Ability to drive action (concepts are deemed important enough to engage policy or program change)

Survey respondents were also asked to rank order the 10 most important concepts within each dimension. This provided an additional prioritization of concepts. Participants were encouraged to write in further indicators if they believed the list of 125 was missing something. Few participants added additional items, and no item was added by more than one participant.

In a separate initiative, based on joint work between the PHO and the representative for children and youth, a series of workshops and focus groups were conducted with young British Columbians asking them what they believed to be the most important indicators of youth health and well-being. This information was incorporated into the relevance filter for this project.

Survey results were evaluated in two stages. First, the rank ordering was used to prioritize concepts as there was insufficient variation in relevance rating scores to differentiate between most concepts. Second, the following two decision-making criteria were applied to including concepts for further consideration: (1) concepts should be ordered across the age span of the child and youth population and should reflect issues identified as relevant to those age groups; and (2) concepts should provide information relevant to First Nations and Aboriginal populations. The number of concepts was reduced from 125 to 93 based on this combination of evaluations, rankings and filtering.

Evidence Assessment

Once the number of concepts had been narrowed down, the final phase of the indicator selection process was an evidence assessment. Determining the methodology to be used for assessing evidence required a rigorous approach that addressed both the quality of the evidence for the concepts and indicators and the quality of the relevant data.

As a first step, a modified systematic review was conducted for each concept. Borrowing from systematic reviews, the approach was designed to be targeted, transparent, comprehensive, relevant, synthetic, evaluative, summative and conclusive. Content experts were engaged to conduct the reviews for each of the five dimensions of the framework. Each expert followed the same protocol, designed to minimize the variation inherent in world views that exist across different topic areas. The reviews addressed two specific questions for each concept:

1. What is the recent scientific evidence that supports the adoption of this concept in British Columbia as a compelling expression or experience of at least one of the five dimensions of child health and well-being?
2. What are the pros and cons of the identified means of measuring this concept, that is, the measures that could serve as indicators of child health and well-being in British Columbia?

The evidence reviews have been completed, and the working group is currently in the process of evaluating the information. This will include further consultation with the Technical Advisory Committee to solicit feedback on differentiating between concepts/indicators that may require further evidence assessment and those that are sufficient for recommending for inclusion in the PHO report. These activities will be taking place over the summer and early fall of 2011.

Lessons Learned

The key message from the evidence review phase of this initiative was that evidence is a difficult and complex issue to address. Other learnings were also derived from this process:

• Finding evidence took longer than anticipated. For this initiative, the modified reviews were assigned based on the five dimensions of the framework. Indicator development and theory/construct knowledge require different skills. An understanding of both content and measurement was critical to the success of the evidence assessment process. Additionally, strong theoretical and construct knowledge does not necessarily imply expertise in the context of societal or system intervention through policy, program and service delivery.
• Finding a common language was time consuming. The modified review protocol included very specific and agreed-upon guidelines; however, differences in interpretation of terminologies and definitions still hampered the consistent application of these guidelines across the reviews.
• Determining how evidence is defined and what counts as evidence is difficult. Methodologies for collecting evidence can range from anecdotal to randomized control trials. Individual perceptions of evidence and the quality of evidence are variable as well. Some bodies of evidence (e.g., in the physical health domain) may be more "established" and easier to locate and assess. Recent theoretical developments, due to their relative "youth," may not have sufficient evidence to support their relevance.
• The availability of data is not consistent across the dimensions of the framework. Data on physical health are readily available, in multiple forms. However, within the mental health and social relationship domains, data are not typically as readily available, making assessments of evidence more difficult.

Concluding Remarks

This project has presented multiple opportunities and challenges. Keys to maintaining momentum have been committed leadership, ongoing participation across a broad range of stakeholders, adequate resources and a stable team. The BC Ministry of Health and CIHI have maintained the same working group members throughout the life of the project.

Upon completion of this project, there are significant next steps to be undertaken for both the PHO and CIHI separately. The PHO will use the final suite of indicators to form the basis of the upcoming report on child and youth health and well-being within British Columbia, undertaking the effort to compile the data for each selected indicator. This work will be supported by the Project Advisory Committee, which will provide advice to the PHO. CIHI will have an opportunity to leverage these considerable learnings, applying this knowledge to indicator development at the pan-Canadian level. These discussions are likely to continue to present additional challenges and provide significant opportunities to improve the information and reporting on the health and well-being of children and youth across Canada.

Comments

Be the first to comment on this!