This article presents a framework for thinking about the key questions that need to be answered to develop new policy and program-relevant knowledge that can be used to make more informed decisions. It is a primer for administrators, policy makers and others about how to identify the knowledge they need to make decisions regarding new or existing programs. The article covers three related dimensions in evaluation: types of evaluations, key domains of inquiry and generic research questions. While the questions are generic, they can be readily adapted to any new and/or existing healthcare program evaluation. Examples of how the generic questions can be adapted to primary healthcare clinics and home care are presented.

Program evaluation is an extensive topic and it is beyond the scope of this article to outline all relevant aspects. Rather, this article presents a framework for thinking about the key questions which need to be answered to develop new policy and program-relevant knowledge that can be used to make more informed decisions. Thus, this article is essentially a primer for administrators, policy makers and others about how to identify the knowledge they need to make decisions about new or existing programs. It covers three related dimensions in evaluation: types of evaluations, key domains of inquiry and generic research questions. The questions are fairly generic but, as will be shown later, can be readily adapted to the evaluation of any new or existing healthcare program.

Getting Started

Evaluation can only take place in fertile ground. This means that key actors should have an appreciation of the analysis enterprise and a desire to use analysis as a tool for decision-making. In addition, there needs to be active discussion between the evaluator and decision-makers about what new knowledge is to be developed and what research questions and operational definitions of key concepts will be used in the evaluation. (An operational definition is one which identifies objective and measurable variables as being representative of an abstract concept. For example, one may use the variable of family income as a proxy for the abstract concept of social class). In order for an operational definition to be seen as relevant and/or appropriate, key actors have to agree that it is a valid representation of the concept one is trying to measure. This can sometimes be quite contentious. Disconnects can occur if there is insufficient discussion at the outset about what new knowledge is a high priority for the administrator. There must be adequate discussion before the study is carried out, otherwise the results may be discredited because they do not answer the "real" question(s) of concern, or because there is no consensus on the validity of the operational definitions used in the evaluation. Analysis which misses the mark is disappointing for all parties.

A Brief Word on Logic Models and Indicators

It is conventional wisdom that one needs to start with a logic model and indicators when conducting an evaluation. While logic models can be helpful they are not absolutely necessary as long as there is a clear description of the program and its objectives. More troubling is the trend to simplify knowledge development into creating a list of indicators as a first step in an evaluation. This approach has a high probability of producing sub-optimal results. To be useful, indicators need to be understood in a context. For example, is low cost a marker for program efficiency or a reflection of cost shifting from the organization to the client? To be relevant, indicators need to have a shared meaning, and relevance, to key actors. Unfortunately, indicators may be more a reflection of form rather than content; that is, people come up with indicators because they think they need to do so. In addition, while the indicators developed may be interesting they may not capture the essence of what administrators and policy makers need to know. A preferred approach is to develop a set of key questions which clearly reflect the new knowledge that administrators and policy makers want and need to know in order to make informed decisions. Indicators should be a product of a process of analysis. When this is the case they are much more likely to have a shared meaning and relevance to decision-makers.

Types of Evaluation

Formal evaluations of new initiatives are conducted to ensure that the initiatives are working as planned and are achieving intended results. Process (or formative) evaluations are conducted to determine if services are being delivered in a manner that is consistent with the model of care adopted and with the policies of the program. Process evaluations can be used to improve how services are delivered. Outcome (or summative) evaluations are conducted to determine if a program is meeting its stated objective(s) and/or to determine if it is better than one or more other models of care (including the model which was in place before the new program). Outcome evaluations can be used to determine the relative "worth" of a program and can be used to make decisions about whether a program will be maintained, modified or ended. Two other approaches, a proof of concept evaluation and an implementation evaluation, can also be conducted in the early stages of a new program. (The application of the proof of concept approach to healthcare evaluation was developed by Hollander Analytical Services Ltd. A description of this approach is presented in Appendix 1.) These evaluations should, ideally, precede both process and outcome evaluations. The first looks at the consistency of the proposed care model with best practices for similar initiatives (i.e., assesses the face validity of the model). The second evaluates the implementation phase of an initiative. It is beneficial to independently evaluate these two approaches as problems in design and/or implementation can lead to sub-optimal results.

Thus, one can think of a progression of four types of evaluation: evaluation of the model that is developed (proof of concept evaluation); evaluation of the implementation of the model (implementation evaluation); evaluation of how the model is operating (process evaluation); and evaluation of whether the model should be continued in its existing form (outcome evaluation) (see Figure 1).

The Evaluation Framework, Key Domains of Inquiry and Key Evaluation Questions

While the above provides an overview of the main types of evaluations, other approaches provide more specific domains of inquiry to be considered in conducting an evaluation. Two typologies which have been used on projects for process and outcome evaluations are those developed by the Canadian Institute for Health Information (CIHI) and the original Health Transition Fund (HTF). The CIHI performance domains are acceptability, accessibility, appropriateness, competence, continuity, effectiveness, efficiency and safety. The original HTF was a federal program to fund and evaluate new and innovative models of care delivery. As such, the HTF evaluation domains are particularly relevant for new models of care and/or pilot projects. The HTF domains are quality of service, accessibility, care coordination/integration, health impacts/effects, cost-effectiveness and transferability/generalizability. Sustainability, which refers to the extent to which a program appears to be well funded and supported and is likely to continue to exist over time, can be added to the above criteria.

We developed 11 domains of inquiry using the CIHI and HTF domains, the types of evaluation and our own experience. Table 1 presents the proposed evaluation framework and evaluation domains. Table 2 presents an initial set of generic evaluation questions for developing instruments for each domain of inquiry. The questions in Table 2 can be used to develop a series of indicators of relevance to a particular evaluation, for new or existing programs. It is recognized that not all questions may be included in each evaluation. Rather, the generic questions can serve as a guide for thinking about how any specific evaluation could be conducted. For larger programs of research, where multiple programs are being evaluated, the generic questions could be used to develop a core set of questions which should be covered in all evaluations.

Table 3 shows examples of how the framework noted above can be adapted to different types of health services.

Conclusion

This article has presented a framework for thinking about the key questions that need to be answered to develop new policy and program-relevant knowledge that can be used to make more informed decisions. It is essentially an evaluation primer for administrators, policy makers and others and was prepared because evaluations can be done well or they can be done poorly. They can be well thought out and executed in a methodologically rigorous way or they can be reduced to an a priori list of indicators. They can add critical, new knowledge for decision-making or they can simply reflect things that are mostly already known. Ultimately, high-quality evaluations have the power to bring about significant improvements in service delivery resulting in an improved and sustainable healthcare system for all Canadians.

Appendix 1. Proof of Concept Evaluation

A proof of concept evaluation is a method for externally and independently validating a new care delivery model using a panel of experts in the topic area.

Steps in Conducting a Proof of Concept Evaluation

Conducting a proof of concept evaluation involves the following steps:

1. Evaluators obtain a consensus that the parties involved in the model are prepared to participate in a proof of concept evaluation.
2. Program staff prepare a description of the model of care.
3. Using the above description, and discussions with program staff, the evaluators prepare a comprehensive yet concise description of the model so that an external expert panel can understand what the model is, why it was developed, what its component parts are and how a client would be treated in the model.
4. Evaluators conduct a literature and key informant scan to determine what best practices may exist for similar models, or what the key characteristics of good, similar models are.
5. Evaluators develop a document which outlines an "ideal," or best practices, model and its component parts based on information collected in point four above.
6. Evaluators and program staff identify key experts to be considered for the independent expert panel. Evaluators invite panel members and finalize panel membership.
7. Program staff identify a key person who does not have a vested interest in the model, but who understands both the model and the context in which it operates, to participate in the panel as a resource to the experts (e.g., a knowledgeable retired or former member of the program team with the ability to objectively answer questions about the program). This is done to ensure that the experts can clearly understand the model and its context. The resource person will be required to agree to maintain confidentiality about panel discussions.
8. Convene the panel of experts for a two-day review of the model.
9. In advance of the meeting, provide an overview of the model to be assessed (see step three above) and an overview of best practices or key characteristics from other models (see step five above) to the expert panel. On the first day, facilitate discussions so that the experts develop a consensus on what characteristics or best practices would define an ideal model, that is, what would constitute a "gold standard."
10. During the second day, the actual program model is compared to the ideal template or "gold standard" and a step-by-step determination is made as to the strengths and weaknesses of each main component, or characteristic, of the model under review.
11. Based on the two-day meeting, a draft report of the findings is prepared by the evaluators and circulated back to the members of the expert panel for their review and comment.
12. Based on this input, a confidential final proof of concept evaluation report is prepared and submitted to the funder, or other appropriate group or body.

Comments

Be the first to comment on this!