Equity – a blog
Ontario’s Ministry of Health and Long-Term Care (MOHLTC) has identified equity as a key component of quality care. The Ministry has developed a Health Equity Impact Assessment (HEIA) to support improved health equity, including the reduction of avoidable health disparities between population groups. HEIA also supports improved targeting of healthcare investments—the right care, at the right place, at the right time. And that is what this blog covers. Join us in the conversation.

A Pocket Guide to Evaluating Health Equity Interventions – Some Questions for Reflection

Interventions, whether of the programmatic or policy variety, are offered as solutions for addressing health inequities. It is often unclear before implementation whether the goals of health equity interventions are largely aspirational—or if the intervention planners have a well thought out plan that is likely to impact health equities.

In my experience, health equity interventions are often implemented without clarity about the underlying mechanism. Understanding intervention mechanisms helps answer: what is it about the program that is likely to impact equities?   Without understanding how a program is likely to impact health equities, a program is unlikely to make that difference.  Impacting health equities is a long term journey – usually you need to know how to get to a destination before you get there.  Thinking evaluatively can help make this journey.

Despite our enthusiasm for evaluations, the role of evaluations in helping to clarify policy and practice choices to address problems of health inequities continues to remain ambiguous.   One of the critical gaps in the literature on health inequities is highlighted by Starfield (2006, 14):   “Despite the very large research literature on social determinants of health, relatively little is written that would inform the choice of policy alternatives to address inequities.” What is also not clear, given the complexity of the interventions needed to address the deep-seated problems of inequities, is whether and how evaluations need to be better designed and structured to create more useful and pertinent information for policy makers and practitioners.

This blog argues that thinking evaluatively about interventions is important. Questions are offered as ways to think evaluatively about interventions.

Problem space and solution space

These questions recognize and incorporate the difficulties in applying the hierarchy of evidence, developed from the evidence-based medicine paradigm, to population health interventions and the need for alternative typologies. It is useful to differentiate between evidence of the ‘problem space’ and evidence for the ‘solution space.’  The ‘problem space’ provides knowledge of what variables or systems of relationships are associated with health inequities (e.g. information on gradients of health inequities) while the ‘solution space’ offers knowledge of what kinds of interventions are likely to ‘work,’  ‘for whom’ and under what contexts.

Much of the research on health inequities to date has focused on the problem space of health inequities.  The ‘solution space’ has not received the attention in the literature it most certainly deserves.  Existing knowledge of the ‘solution space’ is often incomplete for the successful implementation of interventions in specific settings. For example, ‘off the shelf’ literature on best practices does not often provide information on the contexts necessary for the program to work.   Similarly most ‘what works’ solutions rarely describe how long is it necessary for a program to be implemented to have their intended impacts. In my view, evaluations play a critical role in building knowledge of the solution space.

Useful questions for evaluating Health Equity Initiatives

The questions we have found useful in evaluating health equity initiatives from the start include:
What is the key evidence that has informed the implementation of an initiative?  What are the key uncertainties in the evidence base?

The evaluation needs to be explicit about the evidence that informs the initiative, as well as the areas of uncertainty that are in the evidence base. It needs to clarify if the evidence is of the ‘problem space’ or ‘solution space’ variety.  In our experience, many proponents of evidence-based programs tend not to discuss the uncertainties that exist in the evidence base of specific interventions. Such a lack of focus on the uncertainties in the knowledge base has negative consequences for translating programs in specific settings.

Why bother with the evaluation? What are its multiple purposes?

Evaluations often invoke concern. Smart learning organizations need to constantly communicate the multiple purpose of evaluation to their stakeholders, which can include:

• clarifying the nature of the activities and how they are expected to achieve outcomes, bringing to the surface the assumptions by which the activities impact short, intermediate and long-term equity outcomes;
• helping to define what success means to different stakeholders;
• examining whether some of these assumptions are being met ;
• testing to determine if  an initiative is “working”.

What is a theory of change for a health equity initiative?

A theory of change describes the relationship between the intervention activities, intervention contexts and expected heath equity outcomes. Surprisingly, a large number of health equity interventions are implemented without clarity on a theory of change.  This, again has serious negative implications for knowledge translation of the interventions.

A good theory of change needs to:

• be explicit about the mechanisms needed to ‘fire’ to affect outcomes;
• define what key terms in the theory mean for different contexts before there is movement to operationalize and measure the concepts. The measurement cart should not be driving the conceptual horse, so conceptualization should respect heterogeneous understandings across communities and stakeholders;
• be explicit about the key assumptions in the theory and which of these assumptions will be tested by the evaluation;
• describe the theory in a way that can aid implementation in local settings.

What is the timeline of impact of an intervention?

A surprisingly large number of programs are implemented without clarity on when such interventions can impact health equity. Without explicit understanding of an anticipated timeline of impact, it is difficult to tell if a program is successful.
One important area in which evaluative thinking can help is to develop an anticipated timeline of impact by involving stakeholders who are best situated to understand how the intervention is likely to work.

How can the monitoring and evaluation design help assess the impacts of an intervention?

A fundamental step in the evaluation framework is developing clarity on the design needed to understand if an intervention is actually impacting health equities.  A good design can help rule out alternative explanations for changes in key outcomes over time.

Elements to consider for the design include:

• reflection on what successful impact means for an intervention;
• clarity on the timeline and also a ‘trajectory’ of impact (Woolcock, 2009);
• clear and reliable measures — the measures need to be informed by the theory of change;
• measures of the dynamic contexts that might be necessary for the intervention to work;
• ideally the design needs to integrate both monitoring and evaluation approaches. Monitoring aims to study progress against selected indicators and measures the system indicators progress against targeted goals. Evaluations, on the other hand, study the “why” or “why not” of performance and attempt to provide remedial action if the performance is not up to expectations.

What will get generalized at the end of the evaluation?  And how will learning be spread?

An evaluation needs to focus on what learning can be generalized and used to inform the development of other future initiatives.  Some of the questions an evaluation will need to address include: Will the evaluation be making recommendations regarding scaling up or replicating the initiative?  What kinds of learning will be spread as a result of the evaluation?
What are the unintended consequences of an evaluation?

Much of the existing literature on heath equity interventions assumes that only ‘good’ or the ‘predictable’ will come of the intervention. However, there is also a distinct possibility that the intervention can result in less than favorable impacts including, in some cases, exacerbating health equities. A good evaluation will need to be sensitive to the possibility of unintended consequences of interventions.

The “so what?” How did the health equity initiative improve lives?

Good evaluations are eventually about performance stories. The most credible performance stories reflect on how and why investments in interventions make a difference in the lives of individuals, especially those in greatest need.  Impacts on individuals might be outside the immediate sphere of influence of some interventions and might also take a long time, but evaluation can serve the important goal of aligning intervention activities with the long term impacts on health outcomes, especially for those who need the intervention the most.

Addressing the above questions will help develop a culture of “integrated evaluation” in which evaluators work closely with policymakers and practitioners at the beginning, middle and end of implementing policies. Evaluation has a role to play to rethink programs (and policies) by comprehensively assessing the impacts on a number of health outcomes; reshape programs (and policies) based on lessons learned across a number implementations; and  build a knowledge base of how best to reform programs, policies and systems based on what has worked and what hasn’t worked with other reform efforts.

References
Starfield, B. (2006). State of the art in research on equity in health. Journal of Health Politics, Policy and Law, 31(1), 11–32.
Woolcock, M. (2009). Toward a plurality of methods in project evaluation: a contextualised approach to understanding impact trajectories and efficacy. Journal of Development Effectiveness, 1, 1-14.

About the Author:
Sanjeev Sridharan is Director of the Evaluation Centre for Complex Health Interventions at Li Ka Shing Knowledge Institute at St. Michaels Hospital and Associate Professor at the Department of Health Policy, Management and Evaluation at the University of Toronto.  Prior to his position at Toronto, he was the Head of the Evaluation Program and Senior Research Fellow at the Research Unit in Health, Behaviour and Change at the University of Edinburgh.    He is a former Associate Editor of the American Journal of Evaluation and is presently on the boards of the Canadian Journal of Program Evaluation and the Journal of Evaluation and Evaluation and Program Planning.

This entry was posted on Wednesday, May 2nd, 2012 at 2:20 pm and is filed under Longwoods Online.