Holding local boards of health accountable presents challenges related to governance and funding arrangements. These challenges result in (a) multiple accountability pressures, (b) population health outcomes whose change is measureable only over long time periods and (c) board of health activity that is often not the key immediate direct contributor to achieving desired outcomes. We examined how well these challenges are addressed in Ontario, Canada at early stages of implementation of a new accountability policy. Findings reveal that senior and middle management are open to being held accountable to the Ministry of Health and Long-Term Care (MOHLTC), but are more oriented to local boards of health and local/regional councils. These managers perceive the MOHLTC system as compliance oriented, and find internal accountability systems most helpful for performance improvement. Like healthcare system accountability metrics, performance indicators are largely focused on structures and processes owing to the challenges of attributing population health outcomes to public health unit (PHU) activities. MOHLTC is in the process of responding to these challenges.

This paper presents findings from a study on the structures, processes and perceived implications of accountability in Ontario's public health sector, using the conceptual framework for accountability outlined in the Introduction to this Special Issue (Deber 2014). Several national and provincial crises have inspired the development of new forms and systems for managing and holding local bodies responsible for public health accountable for their performance (Capacity Review Committee 2005). Yet little is known about the functioning and effects of accountability mechanisms in public health. This study examines opportunities and challenges in existing and emerging accountability systems for Ontario's 36 local boards of health, which provincial legislation has made formally responsible for superintending and ensuring the provision of health programs and services in their geographical region (Government of Ontario 1990).

Accountability in public health presents challenges that set it apart from most other health sectors. Public or population health seeks to ensure and improve the overall health outcomes of populations (Capacity Review Committee 2006). Indicators of population health outcomes can be fairly easily devised and are widely measured. Holding public health organizations to account for maintaining or improving population health outcomes is another matter altogether. A fundamental principle of accountability is that accountability "holdees" be able to effect, through their actions, the results for which they are held accountable (Behn 2001). Yet, "it is often difficult to assess the extent to which variations in health outcome can be attributed to the health system" (Nolte et al. 2009).

Deber and Schwartz (2011) note that there are a number of different ways of describing what public health is, which often intermix services, processes, practices and desired outcomes (Bettcher et al. 1998; Last 1988; Shah 2003). Many, but not all, of public health activities deal with the health of populations. Much, but not all, of the focus is on prevention. Some, but not all, of the activities are carried out by organizations designated as "public health." Such activities include, but are not restricted to, the "health protection and promotion" functions, defined by Kirby and LeBreton (Standing Senate Committee on Social Affairs, Science and Technology 2003) as "disease surveillance, disease and injury prevention, health protection, health emergency preparedness and response, health promotion, and relevant research undertakings."

Yet public health goes beyond even these functions. It can be seen to encompass activities not usually falling within the mandate of public health agencies (e.g., community economic development, education). It may also be heavily involved in provision of services to target populations (sometimes referred to as "indigent" or "vulnerable" persons), usually for populations or services not covered by health insurance systems (a role particularly important in the United States). In Canada, public health is often involved in providing some services to children and new parents, and preventive and clinical dental health services to vulnerable populations. In turn, some public health activities, including immunization, may be offered not only by public health, but also by primary healthcare and occupational health providers, depending on the jurisdiction and the service.

Ontario has a highly decentralized model for funding and delivering public health services. All of its 36 public health units (PHUs) are locally based, and each is governed by a local board of health. These boards are autonomous corporations under the Health Protection and Promotion Act, and their membership is largely made up of elected representatives from the local municipal governments. The costs of these services are shared between the local municipalities and the provincial government. The chief administrative officer, called the medical officer of health, reports to the local board of health, but is also responsible to the provincial Chief Medical Officer of Health.

The province of Ontario has used several approaches in attempts to hold the public health system accountable. Between 1997 and 2009, the Mandatory Health Programs and Services Guidelines was the main accountability platform; it specified expectations of boards of health and accompanying reporting mechanisms. The implementation of this policy intervention was decidedly top-down and widely criticized for: using indicators that did not reflect contributions of public health activities to public health outcomes; inadequate attention to diversity in health needs and capacities of local PHUs; irregularity of reporting; and ineffective use of data to manage public health performance at the provincial level (PHRED 2002).

In 2009, these guidelines were replaced by the Ontario Public Health Standards (OPHS). The OPHS is an evolutionary document with many of the same or similar requirements of the old Mandatory Health Programs and Services Guidelines, made contemporary with the inclusion of foundational standards regarding population health assessment, surveillance, research and knowledge exchange, and program evaluation (MOHLTC 2008). Ontario Public Health Standards are published by the MOHLTC under the authority of section 7 of the Health Protection and Promotion Act. There are also 26 program- and topic-specific protocols that are incorporated into these standards. Program standards cover areas such as chronic disease, injury prevention and emergency preparedness, and include general goals, societal outcomes, board of health outcomes and program requirements.

A system of performance management and accountability built around the OPHS constitutes a new accountability system. Components of this system include accountability agreements outlining specific service and performance expectations between MOHLTC and boards of health (MOHLTC 2013). Embedded within the system are performance targets corresponding with 14 indicators that relate to OPHS program requirements (Table 1). Ten of these indicators relate clearly to processes and outputs of PHU activities (e.g., percentage of high-risk food premises inspected), while only four indicators endeavour to measure outcomes (e.g., percentage of youth who have never smoked a whole cigarette). Organizational standards for boards of health, protocols for the delivery of various programs and services, and reporting requirements are main system components. At that time, MOHLTC was responsible for administering the following standards: Foundational; Infectious Diseases; Environmental Health; and Emergency Preparedness, whereas the former Ministry of Health Promotion had responsibility for Chronic Diseases and Injuries and Family Health, and the Ministry of Children and Youth Services for the administration of the Healthy Babies, Healthy Children components of the Family Health standards (MOHLTC 2008). Full implementation of this system, including the first cycle of data collection and reporting, began in 2011. Subsequently, the Ministry of Health Promotion was reabsorbed by MOHLTC.

Methods

Data were collected through four methods:

1. Key informant interviews were conducted with seven individuals from the following organizations: MOHLTC, Ministry of Health Promotion and Sport (MHPS), Ontario Agency for Health Protection and Promotion (OAHPP), PHU medical officers of health and chief executives.
2. A web-based survey of medical officers of health from 27 of the 36 PHUs.
3. Twelve interviews in three PHUs chosen to reflect size, location and level of complexity.
4. A web-based accountability survey of 53 public health unit managers from 12 PHUs.

We corresponded with the medical officers of health in all 36 PHUs and asked permission to recruit public health managers for the web-based survey; 12 agreed to participate. Once consent was granted by medical officers of health, e-mail recruitment letters were sent out (97 valid surveys distributed), and 53 were completed and returned. While the sample is not representative of all public health managers across the 36 PHUs in the province, it is a strong cross-section of the population. The sample reflects affiliation with one-third of the PHUs in the province. Despite one-third of respondents being from just three PHUs, a varied distribution of rural, mixed rural and urban, and mainly urban environments characterized these sites. Indeed, sample distribution according to these peer groupings is nearly equivalent, with respondents from the urban public health units being slightly underrepresented.

Keysurvey.com was used to develop the online survey and manage recruitment and basic descriptive analysis. Once the survey collection was completed, a data set was exported to SPSS for further analysis. Descriptive analyses as well as some measures of association have been used to generate survey findings. This survey and its findings may not be representative of public health managers in the province and, in most cases, the participating PHUs, owing to the small sample and the clustering of data. The findings herein do, however, provide an initial step in exploring public health accountability in Ontario, which may aid in the development of future research, evaluation and performance management and accountability planning.

Findings

1. To whom, for what: Accountability holders

Ontario PHUs belong to the category of hybrid public sector organizations. As noted above, they are funded by both local and provincial governments and governed by local boards of health; but they also function under the mandate of provincial legislation (Health Promotion and Protection Act) and are responsible for promoting and protecting public health in accordance with the provincial Ontario Public Health Standards and associated guidance documents.

Several major national and provincial NGOs, as well as multiple local and grassroots organizations, engage with PHUs for collaboration, advocacy and contracted service provision. Additionally, most PHU employees are members of health professional associations that might be expected to elicit professional accountability relations (Romzek and Dubnick 1987).

This is a seemingly typical situation of multiple accountabilities, often with competing accountability pressures (Hood 1991; Schwartz and Sulitzeanu-Kenan 2004). Interestingly, results from our surveys and interviews consistently indicated that PHU management perceives only three formal accountability holders to be of significant importance – local boards of health, municipal or regional councils and MOHLTC. Survey respondents and interviewees attributed very little importance to accountability relations with NGOs or with professional associations.

Most respondents indicated that both local boards of health and MOHLTC were important accountability holders, partially because both fund PHU activities. Generally, respondents did not feel conflicting accountability pressures from these accountability holders. There was a fairly strong element of accountability ultimately being to the public: "I think that demonstrating accountability has more to do with building public trust than it does in terms of sort of ministry reporting" (senior PHU respondent). Slightly more managers who responded to the survey noted accountability to MOHLTC as being highly important (89%) than accountability to boards of health (77%). However, interviewees suggested some primacy of accountability to local/regional councils and boards of health that were considered to represent local publics: "I think ultimately I would probably put it down to our council – they represent our public" (medical officer of health respondent). "So we are accountable back to council primarily right now through our budgeting process and our planning processes" (medical officer of health respondent).

In contrast, for some respondents, accountability to MOHLTC was rooted primarily in the funding relationship. In the words of one respondent, "because a significant chunk of our funding comes from the MOHLTC, they are a key player in this…" (senior PHU respondent). Another respondent was somewhat reluctant in agreeing to the legitimacy of accountability to MOHLTC: "So I'm not suggesting that the ministry should just get out of the business of holding people accountable. Clearly they need to also demonstrate that. There are conditions tied to the funding" (senior PHU respondent).

Some respondents noted the importance of internal accountability mechanisms. Several PHUs have internal performance measurement and management systems rooted in strategic plans, planning and budgeting relations with boards of health and accreditation:

We've actually put some mechanisms in place that we feel make us increasingly accountable. So we did a 10-year strategic plan. We identified nine priorities and we are actually following them. The accountability on the strategic directs is that people leading them have to report to the senior team twice a year on the progress. (medical officer of health respondent)

While there was some frustration with effort needed to report differently to different accountability holders, overall respondents did not indicate this as being particularly problematic. There was quite a strong sense of the local public being the ultimate accountability holder and being represented by local/regional councils. There was no parallel sense that the PHU was accountable to the general public of the province through MOHLTC.

2. Using which tools: Accountability mechanisms

There are a reasonably large number of potential mechanisms through which PHUs might be held accountable. These range from financial incentives through legal channels, performance audit, performance reporting and performance management. In the managers' survey we asked a series of questions about perceived issues of abuse of funds, preferential treatment, inefficiency and ineffectiveness. We then asked about the use of various accountability mechanisms to address perceived issues in these areas.

For the most part, respondents did not perceive serious issues in these areas. Only 2% indicated that abuse of funds was common or very common, and 8% perceived preferential treatment of friends, family or close associates to be common or very common. Sixteen per cent perceived inefficiency and 15% ineffectiveness in achieving desired results to be common or very common. Generally, when asked about the likelihood that issues and threats to accountability are or would be detected, most respondents agreed that they would be. However, a quarter of the sample perceived detection as being unlikely in the areas of preferential treatment, inefficiency and ineffectiveness.

Managers were also asked about their perceptions concerning inspectors, auditors or evaluators checking for various issues and threats to accountability in their PHU. Although a great proportion of the sample perceived these three professional groups as actively checking for accountability issues, this question item elicited much more uncertainty than the previous questions in this section. What seemed most telling was the large number of respondents, ranging from a quarter to nearly half of the sample, reporting that they did not know whether inspectors, auditors or evaluators were purposefully looking for the various accountability issues.

Performance measurement and reporting is the most prominent accountability mechanism mentioned by respondents (22 mentions each). Financial incentives were rarely mentioned, and legal accountability was not mentioned at all. Accountability agreements were noted by 12 respondents and accreditation by 11. Some respondents viewed accreditation as continuous quality improvement (CQI) rather than accountability:

I'm just kind of looking at what I sort of do here for the accountability piece of accreditation. Because this is a voluntary piece and because we look at this as CQI, … the only consequences that would be there is if we are not identifying the gaps and working towards continuously improving the program delivery; then there could be gaps in getting that information out to our community. I know with some of the programs there are things you definitely have to do, you have to meet this.

A common theme across several PHUs is the importance of internal accountability mechanisms. These internal mechanisms – generally performance measurement and management systems – include long-term strategic planning, the evidence review process, performance management, stand-alone evaluations and priority setting. For example, in one PHU internal reporting mechanisms occur among senior management on a bi-weekly basis and also culminate in a year-end report:

There is also reporting to our executive committee, and what happens there is [that] every two weeks they gather the senior management team and they meet. They review what's happening there and it gets reported back to the division heads that way. That's all documented through meeting minutes, things like that. Then the other reporting piece is when we do our mid-year, year-end review as a management team across the department and report back on what we are doing at the program level.

Internal accountability was seen almost exclusively as oriented towards learning and performance improvement:

Our goal is to make better mistakes. We don't expect "perfect" from us. I don't expect "perfect" from our staff. I'd like our mistakes to get – I don't mean more spectacular, I do mean that they are more refined, that the mistakes we make are mistakes that come from processes that got better because of the mistakes we made before those. That everything just improves over time. That we are learning. That we are taking those lessons learned and implementing them and then just getting better. That is what I think we are really trying to do. Do things better for the public.

3. How well is it working? Accountability to MOHLTC

Our data collection took place during the months in which the new accountability system was in initial stages of implementation. Accountability agreements between MOHLTC and PHUs were being signed and the first round of performance data was being collected. Findings therefore reflect early impressions of the new accountability system. Many respondents appreciated the efforts made to improve on the previous system and expressed some optimism that the new system might indeed be used to help improve performance: "I fully support the accountability agreements – what gets measured gets done – we need the data to be able to tell our story and explain how the funds are being spent and our impact."

At the same time, there was considerable skepticism and some frustration with the new accountability system. Some of this stemmed from the objective difficulty of identifying meaningful indicators when, as one respondent noted, "this is particularly difficult in public health, where outcomes are truly occurring far into the future." The indicators that were chosen were criticized as "not indicative of the effectiveness of our services," "lacking relevancy to us," "number counting" and being "beyond our control." Importantly, MOHLTC has responded to dissatisfaction with the indicator set and convened working groups to identify an improved set of indicators.

Beyond concerns with specific indicators, there is concern with the accountability system more broadly. Overall, many respondents viewed the accountability system as aiming more to ensure bureaucratic compliance with the Ontario Public Health Standards than to generate learning to improve performance and the health of the local population. Ten of the 27 (37%) medical officers of health who completed our web survey anticipated that the accountability agreements would not contribute at all or would contribute only to a small extent to the performance of their PHUs. Thirteen respondents (48%) anticipated a moderate contribution, and only four (15%) anticipated a large contribution to performance. Eighty-four per cent of PHU manager respondents felt that the system had the intent of capturing compliance with public health performance expectations. While most agreed that there was also a focus on learning, one-third (33.3%) disagreed that the system would primarily focus on learning about and improving performance.

Some respondents felt that the system did not include adjudication of performance information provided or imposing consequences in light of the results of such adjudication. Several respondents noted that there is no tie between expected performance and funding, indicating that poor results may be caused by inadequate funding but that increased funding was not being offered to improve those results. While almost all PHU managers were moderately or greatly familiar with accountability agreements, performance targets and performance indicators, half of them (46%) indicated little or no familiarity with potential sanctions or remedial actions resulting from poor performance.

Discussion

Findings confirm the expectation that holding PHUs to account presents considerable challenges. Expected challenges emerge from the fact that PHUs draw authority and funding from both provincial and regional/local mandates and provide a diverse range of services, many in collaboration with governmental, third-sector and sometimes for-profit organizations. Four accountability holders emerge as playing key roles in the eyes of medical officers of health and PHU managers: boards of health/regional–local councils; MOHLTC; the local/regional public; and senior management through internal accountability channels. Overall, PHUs are more attuned to local accountability holders than to MOHLTC. They see more learning and performance improvement coming from internal systems and from relationships with boards of health/regional–local councils than from the new (and certainly the previous) accountability system with MOHLTC. While agreeing with the premise of accountability to the ministry and open to its being used for performance improvement, respondents see the new accountability system, at this time, as more oriented towards compliance than performance improvement.

An additional major challenge stems from the nature of much of the work conducted in PHUs. Attributing population outcomes to activities of PHUs is fraught with issues. Change in population-level outcomes can be slow and hard to detect, especially over short periods of time. These outcomes are often influenced by environmental and ecological factors more powerful (by far) than the potential contribution of anything that the PHU itself can do through its activities. Performance measurement systems that recognize this issue tend to end up with structure and process measures that resemble bean counting and compliance auditing. Ontario's new public health accountability system is struggling with this issue and looking to improve on its first round of performance indicators at this time.

Espoirs et réalités des politiques de responsabilisation en santé publique

Résumé

Tenir responsable les conseils de santé locaux présente des défis quant aux arrangements de gouvernance et de financement. Ces défis donnent lieu à (a) de multiples pressions liées à l'obligation de rendre compte, (b) des changements dans les résultats sur la santé de la population qui ne sont mesurables qu'après de longues périodes de temps et (c) une activité des conseils qui, souvent, ne représente plus une contribution directe et immédiate pour l'atteinte des résultats souhaités. Nous avons examiné comment sont traités ces défis en Ontario, Canada, aux toutes premières étapes de mise en œuvre d'une politique de responsabilisation. Les résultats révèlent que les cadres supérieurs et intermédiaires sont prêts à assumer l'obligation de rendre compte auprès du ministère de la Santé et des Soins de longue durée (MSSLD), mais se sentent plus près des conseils de santé locaux et des conseils régionaux. Ces cadres estiment que le système du MSSLD est axé sur la conformité et trouvent que les mécanismes d'obligation internes sont plus utiles pour une amélioration du rendement. Tout comme les paramètres liés à l'obligation de rendre compte dans le système de santé, les indicateurs du rendement sont très axés sur les structures et les processus en raison des défis liés à l'attribution des résultats sur la santé de la population aux activités des bureaux de santé publique. Le MSSLD a mis en marche un processus pour adresser ces défis.

Comments

Be the first to comment on this!