Home and Community Care Digest

Home and Community Care Digest January 2006 : 0-0

A 'how to' guide to detect, and minimize, common flaws in economic evaluations

Abstract

Economic evaluations are increasingly utilized to assist decision-makers make resource allocation and funding decisions. The growing prominence of cost-effectiveness analyses and other types of economic evaluations has led to increased scrutiny of economic studies and their methods. Numerous deficiencies have been cited, some of which potentially impact final reimbursement decisions. This article was written to assist health care decision-makers and other users of economic evaluations detect key flaws in economic evaluations, focusing on those flaws that are likely to be most important to decision-makers when making final funding decisions.
Background: Economic evaluation is a tool increasingly utilized by health care decision makers that assists them make decisions about the funding and/or reimbursement of health technologies. However, criticism has been raised that an increasing number of studies contain methodological deficiencies that influence the final decision. This article outlines for health care decision-makers the most common and fundamental methodological flaws in economic evaluations. It also offers guidance on how to detect and deal with those flaws that are likely to be of high importance to decision-makers when making final reimbursement decisions.

Methods: Methodological flaws are identified as a flaw in study design, analysis, or reporting of results.

Findings: The common flaws in study design include (1) omitting important costs or benefits, and (2) selecting inappropriate alternatives for comparison. It is therefore important that the decision-maker ask whether, given the perspective of the analysis (e.g., health care system, societal, government), it is likely that omitted costs and/or benefits will have a significant impact on study results. It is also important to consider whether the selected comparator is the best option, or whether better alternatives are available.

Common flaws in data collection and analysis include (1) inadequate representation of the effectiveness data, (2) inappropriate extrapolation beyond the period observed in clinical studies, (3) excessive use of assumptions rather than data, and (4) inadequate characterization of uncertainty. Health care decisionmakers should question whether all available and relevant clinical studies are used in the costeffectiveness study, and, whether there is sufficient justification for the excluded studies. Decisionmakers should further question whether the costs and effects were costed over the appropriate length of time (i.e., is the time horizon of the study justified?). Lastly, with respect to data collection and analysis, decision-makers should note whether key assumptions used in the analysis were tested and supported by other studies.

Decision-makers should also be aware of flaws in the interpretation and reporting of results. Common flaws include (1) inappropriate aggregation of results, (2) reporting average cost-effectiveness ratios, (3) lack of consideration of generalizability issues (i.e., will the findings be useful to other jurisdictions or decision-makers?), and (4) selective reporting of findings. Decision-makers should note whether all component parts of the cost-effectiveness ratio are included, and whether comparisons between alternatives are expressed in incremental form (i.e., is the cost per additional hospital day reported?). Decision makers are also advised to beware of descriptive words like "substantial" and "minor" in the reporting of key results. It is important to question whether potentially negative results are downplayed while positive results are unduly emphasized.

Conclusions: Economic evaluations are increasingly utilized when making funding and reimbursement decisions. While it is important for health care decision-makers that make funding decisions to be aware of methodological flaws that may impact the decision outcome, the authors note the importance of recognizing that most decisions must be made with imperfect information, and that the value of an economic evaluation should therefore be determined by the extent to which it enhances the decisionmaking process. The guidance provided on how to detect and mitigate key study flaws will assist health care decision-makers make informed judgments about whether the economic evaluations presented to them are of sufficient quality to contribute to, and enhance, their decision-making process.

Reference: Drummond M, Sculpher M. "Common methodological flaws in economic evaluations." Medical Care, 2005; 43(7) suppl, II5-II14.

Comments

Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed