Background: Concordance between Common Drug Review (CDR) recommendations and provincial plans has been studied previously. However, no study has, to the best of the authors' knowledge, examined the characteristics of CDR recommendations that may be associated with concordance.

Methods: Recommendation–decision pairs were collected from the CDR and the provincial plans of Ontario, British Columbia and Alberta. Concordance was evaluated by province. Characteristics of each CDR recommendation were collected, and associations with concordance were evaluated by logistic regression.

Results: Recommendation–listing concordance was high. Positive references to cost and clinical outcomes compared to placebo were statistically associated with concordance. Negative references to cost and to the consistency and certainty of economic evidence were statistically associated with discordance. However, these findings were inconsistent across the jurisdictions studied.

Conclusions: Although concordance was high, the ability of recommendation characteristics to explain the relationship between province and CDR listing decisions was limited. This exploratory study highlights the complexity of the reimbursement process and possible reasons for drug listing differences across jurisdictions.

Introduction

The reimbursement process for publicly funded drug plans in Canada is complex, with multiple decision-makers acting across several jurisdictions. Reimbursement evaluations are conducted under the framework of health technology assessment, defined by the World Health Organization (2018) as the "systematic evaluation of properties, effects and/or impacts of health technologies and interventions". This multidisciplinary holistic approach broadens the evidence base to include medical, economic, social, organizational and ethical aspects of current and emerging technologies. Canada, like many European countries, has formally adopted health technology assessment practices as a means of informing decision-makers for drug reimbursements on public drug plans (Angelis et al. 2018; Mitton et al. 2006).

Until recently, the Common Drug Review (CDR) process, housed in the Canadian Agency for Drugs and Technologies in Health (CADTH), typically began once Health Canada approved a drug as safe for use. In June 2018, it was announced that Health Canada, CADTH and Quebec's Institut national d'excellence en santé et en services sociaux will align their respective drug review processes under Health Canada's Regulatory Review of Drugs and Devices initiative. Yet, the purpose of the review process remains to evaluate whether a drug qualifies for public payer reimbursement based on evidence of the drug's clinical and economic properties, alongside input from patient groups. Reimbursement decisions are issued and disseminated to the 18 participating drug plans delivered by federal, provincial and territorial ministries and agencies. However, the recommendation is non-binding and the final listing decision is at the discretion of the plan administrator, with funding allocated in the context of their unique jurisdictional mandates and priorities.

Given that CDR recommendations are non-binding, the final reimbursement decisions by participating drug plans may vary. Several studies have examined Canadian drug reimbursement decision concordance, both before and after the establishment of the CDR (Allen et al. 2016; Anis et al. 2001; Attaran et al. 2011; Gamble et al. 2011; MacDonald and Potvin 2004; Morgan et al. 2006; Morgan et al. 2009). However, the findings and conclusions from this body of evidence vary because of variations in study methodology, the jurisdictions represented and the period being studied.

Reimbursement submissions have also been studied to explore factors that may predict listing recommendation outcomes. For example, a study by Rocchi and colleagues examined factors associated with negative recommendations (Rocchi et al. 2012). Through univariate and multivariate analyses, the authors concluded that clinical uncertainty and price considerations, but not the findings from economic evaluations, were strongly predictive of listing recommendations. Similar research has been conducted in other reimbursement contexts, such as by Linley and Hughes, who examined the influence of policy and clinical and economic factors in Wales (Linley and Hughes 2012). Based on multivariate analysis, it was observed that the use of probabilistic sensitivity analyses to examine uncertainty positively influenced listing decisions. Interestingly, citing high-quality randomized controlled trials as supporting clinical evidence was statistically significantly predictive of negative listing decisions. This finding raised questions about the perception of study quality in the context of decision-making, particularly with respect to the use of surrogate end points.

Although concordance and reasons for listing recommendations or decisions have been explored separately in the literature, no study has, to the best of the authors' knowledge, examined potential predictors of concordance between the CDR recommendations and Canadian provincial listing decisions. Thus, the first objective of our study is to describe the concordance between CDR recommendations and the listing decisions of the provinces of Ontario, British Columbia and Alberta. Beyond describing concordance between the CDR and provincial decisions, there is some value to investigators seeking formulary listings to make explicit the key elements that are associated with a decision outcome. Thus, the second objective is to propose predictors of concordance based on the published recommendation rationales issued by the CDR.

Methods

Study design

A database of CDR recommendations (2009–2017), including the listing recommendation, brand and generic drug name, indication, submission type and the recommendation date, was procured from the CADTH website (Canadian Agency for Drugs and Technologies in Health 2018). For the purpose of this study, only the initial drug submissions were considered (i.e., not re-submissions). The list of CDR recommendations was cross-referenced with the listing decisions by the provincial drug plan administrators of the Ontario Public Drug Programs (PDP) (Ontario Ministry of Health and Long-Term Care 2018), British Columbia PharmaCare (Government of British Columbia 2018) and the Alberta Drug Benefit List (DBL) (Alberta Health 2018) based on the intervention, indication and CDR recommendation date. The CDR recommendation date must have preceded the date of the provincial decision. We selected these provinces because, together, these represent approximately 80% of the Canadian population living in jurisdictions that participate in the CDR process. Recommendations were coded as "positive" irrespective of whether those were conditional (Allen et al. 2016). Similarly, provincial listing decisions were considered positive irrespective of limitations or special authorization requirements. Oncology drugs, which are evaluated through the separate pan-Canadian Oncology Drug Review process, were not considered.

A list of predictors reflecting various aspects of clinical evidence, economic evidence and evidence synthesis methods was developed by reviewing a sample of 25 CDR recommendation decisions purposefully selected to reflect a range of years, interventions and indications. Through a thematic analysis, novel reasons for recommendations were extracted from subsequent recommendation documents until no new themes emerged. Then, for each treatment–indication pair, the "Reasons for recommendation" section of published CDR recommendations was evaluated and predictors categorized as follows: "Present, in favour of a positive recommendation"; "Present, in favour of a negative recommendation"; or "Not referenced".

All data were collected by a single reviewer (MZ) and maintained in Microsoft Excel workbooks. Analyses were conducted in R (v3.4.3, r-project.org).

Statistical analysis

For each province, concordance between CDR recommendations and provincial listing decisions was described first by the percentage of positive recommendations and decisions. The crude percentage of recommendations and decisions in concordance was then estimated and supplemented by Cohen's unweighted kappa coefficient to account for the possibility of agreement occurring by chance (Cohen 1960). Kappa coefficients were interpreted according to the guideline proposed by Landis and Koch (1977). Missing data (i.e., provincial listing decisions) were not imputed.

For each predictor, a binomial logistic regression model was fit to the data for each province–CDR paired set to estimate the odds ratio (OR) of concordance between the recommendation and listing decision when the predictor was present. The positive and negative predictor levels were evaluated separately against the reference category of the predictor being "Not referenced". Outcomes were interpreted as follows: Compared to when the predictor is not referenced, ORs greater than 1 and less than 1 are predictive of concordance and discordance, respectively, between the CDR and the province. For a predictor to be evaluated, we specified a priori that 10 observations, either in favour of a positive or negative recommendation, must be available in a given province–CDR paired set. Statistical significance was evaluated at the α = 0.05 level.

In cases where every observation has the same outcome, it is not possible to evaluate the province–CDR relationship by logistic regression. In these instances, a descriptive analysis was supplemented by Fisher's "exact test" to estimate the probability that this association was observed by chance (Fisher 1922).

Results

Common Drug Review recommendations

A database of 309 CDR recommendations, from January 28, 2009, to November 23, 2017, was available. Of these, 193 were identified as "New" submissions, with 125 (64.8%) positive listing recommendations (Figure 1). Over the time interval studied, the percentage of assessments with a conditional recommendation increased, from a low of 28.6% (2/7) of recommendations in 2009 to all recommendations in 2013 (8/8) through 2014 (17/17). Nearly all recommendations issued in 2017 included some condition (95.5%, 21/22). A complete list of recommendations and corresponding provincial decisions is presented in Appendix 1.

From the set of 25 CDR recommendations selected and used to generate predictors, 13 predictors were initially proposed. Data saturation achieved after 20 recommendations was reviewed. These predictors, with definitions and examples extracted from CDR recommendations, are presented in Appendix 2 (available here). Across the complete set of CDR recommendations, the median (interquartile range) number of predictors, positive or negative, was 2 (2, 3). From the 13 initial predictors, 5 were excluded from the analysis for having fewer than 10 references in either the positive or negative direction: clinical outcomes from non-comparative trials; replacement vs. currently available treatments; patient subgroup-specific evidence; evidence synthesis methods, only citing use of a network meta-analysis; and evidence synthesis methods, other than network meta-analysis. Many predictors were only evaluable at only the positive (clinical outcomes vs. placebo; safety or tolerability) or negative (consistency or certainty in clinical evidence; clinical methods; incremental cost-effectiveness ratio; consistency or certainty in economic evidence) level. The pattern of evaluability was similar across provinces. Appendix 3 (available here) presents the predictors as coded in CDR recommendations. A summary count of each predictor, arranged by province, is presented in Appendix 4 (available here).

A summary of the results of the logistic regressions, by predictor level and province, is presented in Figure 2.

Ontario

A review of the Ontario PDP database returned 105 listing decisions (70.5% positive) that corresponded to CDR recommendations. However, several drug list decisions (n = 34) were unavailable because of errors in the provincial database. Despite numerous attempts to contact provincial administrators, we were unable to obtain these data. Crude agreement between recommendations and listing decisions was estimated at 81.9%. Cohen's unweighted kappa was estimated at 0.625 (p < 0.001), indicating substantial agreement.

Positive references to cost (OR = 17.76) and to clinical outcomes compared to placebo (OR = 6.03) were statistically significantly associated with concordance. No negative references to predictors were statistically associated with concordance or discordance.

British Columbia

From the BC PharmaCare database, we identified 140 listing decisions (55.7% positive) corresponding to CDR recommendations. Crude agreement between recommendations and listing decisions was estimated at 81.4%. Cohen's unweighted kappa was estimated at 0.629 (p < 0.001), indicating substantial agreement.

Negative references to cost (OR = 0.23) were statistically significantly associated with discordance between CDR and BC PharmaCare. No other predictors were statistically significant.

Alberta

A review of the Alberta DBL returned 134 corresponding listing decisions (58.2% positive). Crude agreement between recommendations and listing decisions was estimated at 85.8%. Cohen's unweighted kappa was estimated at 0.71 (p < 0.001), indicating substantial agreement.

In all 45 observations of positive references to cost, we observed concordance between the CDR recommendation and the provincial listing decision. The probability of observing this by chance was evaluated by Fisher's "exact test" (p < 0.05). Negative references to consistency or certainty in the economic evidence (OR = 0.30) were statistically significantly associated with discordance between CDR recommendations and Alberta DBL listing decisions. No other predictors were statistically significant.

Interpretation

The purpose of this exploratory study was to examine concordance between CDR reimbursement recommendations and the subsequent decisions by three of the largest participating provincial plans as well as to explore possible predictors of concordance. Provincial concordance with recommendations was high across provinces. However, the ability of predictors to explain this relationship varied, both with respect to the type and direction of the predictor. Positive references to cost were statistically associated with concordance for Ontario and Alberta. Positive references to clinical outcomes compared to placebo were statistically associated with concordance for Ontario only. Negative references to cost were associated with discordance for British Columbia, whereas negative references to consistency or certainty in economic evidence were associated with discordance for Alberta. Interestingly, based on the evaluations by logistic regression, no positive reference to a predictor was statistically associated with discordance and no negative reference was statistically associated with concordance. This suggests that favourable evidence may be more likely to transcend jurisdictional boundaries, whereas negative evidence may need to be more closely examined in the reimbursement context. However, as many predictors were unevaluable because of the small number of observations and trends were largely inconsistent across jurisdictions, this inference is based on limited evidence.

The independent variables that met our a priori threshold for analysis were those that we anticipated as being particularly relevant to the decision-making process, namely, elements of clinical performance and cost. Generally, these considerations are the primary focus of much of the research leading to a decision to apply for reimbursement and, in some cases, are necessary to greenlight next steps in the market access pathway. For example, issues with safety, tolerability or unfavourable outcomes against a placebo may stall or terminate a drug's development because, under the current Canadian reimbursement framework, approval by Health Canada is required before seeking reimbursement. In addition, given that CDR recommendation rationales are short summaries, it would be unexpected to have reviewers comment positively on the consistency of clinical or economic evidence or on the methods of the supporting clinical research. Typically, we expected that reviewers would reserve comments on these factors for instances where evidence was unfavourable. Yet, given the relatively few statistical associations we observed, the summary rationale presented to support the CDR recommendation statement does not appear to sufficiently capture the decision-making process, and the elements highlighted by the CDR may not be the primary driving factors at the provincial level. Future work may explore this by examining correlations between CDR recommendation justifications and the decisions provided in provincial listings.

Alignment of listing decisions in the Canadian context has been studied extensively, and conclusions have proven to be divisive (Allen et al. 2016; Anis et al. 2001; Attaran et al. 2011; Gamble et al. 2011; MacDonald and Potvin, 2004; Morgan et al. 2006; Morgan et al. 2009). Based on CDR recommendations from 2009 to the end of 2014, Allen and colleagues observed concordance rates with the CDR of 81.1%, 78.9% and 78.8% for British Columbia, Alberta and Ontario, respectively (Allen et al. 2016). With the exception of Alberta, we observed similar rates in the current study. However, whereas the current study was based only on new drug submissions, Allen and colleagues sourced the most recent recommendation listing. Gamble and colleagues examined agreement before and after CDR's implementation, collecting data from May 1999 to May 2009 (Gamble et al. 2011). Based on a set of 53 CDR recommendations, agreement varied from a low of 64.2% with Ontario to a high of 90.6% with New Brunswick and Nova Scotia, with concordance estimated at 69.8% and 83.2% for British Columbia and Alberta, respectively. Writing on the effectiveness of the CDR in Canada's national drug strategy, Attaran and colleagues criticized the considerable variability across the country based on a set of 369 pairwise observations (Attaran et al. 2011). Thus, there is a lack of consensus on the ability of the CDR process to standardize drug reimbursements across Canada's public payer drug plans.

Indeed, the establishment of the CDR in 2003 was heralded as a means of consolidating expertise and funding to support health technology assessments through a rigourous, evidence-based, centralized process. Yet, controversy on the usefulness of the process persists in the literature, given that individual drug plans ultimately have the decision to list or not list a drug irrespective of the CDR recommendations (Morgan et al. 2009). Arguably, the establishment of the CDR process particularly benefitted smaller provinces, such as those in Atlantic Canada, which may lack the resources and funding to fully execute health technology assessments on every new drug submission. Larger provinces, such as Ontario, may be more able to fund independent reviews specific to their jurisdictional context. Moreover, decision-makers representing larger patient populations may be afforded more flexibility in drug price negotiations. Ultimately, limitations on funding and patient needs decide whether recommendations are adhered to (Spitz 2013).

The reorganization of the Canadian reimbursement recommendation processes under Health Canada's Regulatory Review of Drugs and Devices initiative has the potential to further enhance the comprehensiveness and rigour by which submissions are assessed. However, the final reimbursement decision will remain with the individual public drug plans. Future research may examine differences in the rates of positive or negative recommendations and the reasons cited for these recommendations under this revised, integrated framework.

Strengths and limitations

The current study is characterized by several strengths. Observations of concordance were made based on a large, valid and directly sourced sample of CDR recommendations using robust methods, including both crude concordance and Cohen's kappa. Although missing data, specifically provincial listing decisions, were not imputed, the cause of missingness was assessed as either outside the jurisdiction of the provincial funding body being queried or that a decision had not yet been reached at the provincial level. The evaluation of predictors was conducted using robust statistical methods and was based on systematically collected evidence. However, there are several limitations to this work. This was an exploratory study, intended to be hypothesis generating rather than hypothesis testing, and thus, it is not appropriate to draw causal conclusions based on the data or analyses. In addition, given the number of analyses performed, it is reasonable to expect at least some relationships to generate spurious statistically significant results. Although the number of CDR recommendations identified was large, the number of predictor observations, either positive or negative, was comparatively small and fell below the a priori specified threshold in many cases. The predictors were restricted to the "Reasons for Recommendation" section of the CDR rationale, as the primary factors driving recommendations are highlighted here. Future studies may expand the scope to explore factors such as indication, first-in-class, year of submission and whether recommendations were conditional. Given that predictors were rarely referenced in isolation, interactions between predictors should be considered, as this may reveal important combinations of predictors that may better explain (dis)concordance. The jurisdictions studied, although representing the majority of patients in provinces participating in the CDR process, were restricted to three provinces. Evidence suggests differences in the level of concordance between large and small provinces, which may impact the evaluation of predictors of concordance. Nevertheless, the findings of the current study are valid with respect to the provinces studied, and we do not intend to extend these inferences beyond the jurisdictions specified here.

Conclusion

Previous studies have examined reasons for listing decisions, but this is, to our knowledge, the first study to examine factors for concordance across different decision-maker contexts. We observed substantial concordance between CDR recommendations and the provinces of Ontario, British Columbia and Alberta. In the predictor analysis, it was observed that positive references to cost (Ontario, Alberta) and clinical outcomes compared to placebo (Ontario) were statistically associated with concordance. Negative references to cost (British Columbia) and to the consistency and certainty of economic evidence (Alberta) were statistically associated with discordance between CDR recommendations and provincial listings. However, these findings were not consistent across the jurisdictions studied. Moreover, the exploratory nature of this study precludes the ascertainment of causal relationships between predictors and province–CDR concordance.

This study was motivated by the patchwork of drug coverage that persists in Canada, with inconsistent listing decisions across the country despite receiving a common CDR recommendation. Although the current study failed to reveal clear explanations for concordance, future work may describe trends, particularly with respect to the interplay of various predictors, that may help investigators preparing reimbursement submissions anticipate outcomes and prepare a better case when evidence to support important predictors is lacking. In addition, qualitative studies with provincial decision-makers may reveal motivations that are not sufficiently captured in the publicly available documentation.

Drug reimbursement decision-making is a complex and multidimensional process, and decision-makers must balance multiple forms of evidence, stakeholders and competing funding priorities. Thus, this study highlights the difficulty and limitations of presenting a simple explanation of a complex process.

Analyse exploratoire des prédicteurs de la concordance entre les recommandations de remboursement du Programme commun d'évaluation des médicaments et les décisions subséquentes de l'Ontario, de la Colombie-Britannique et de l'Alberta

Résumé

Contexte : La concordance entre les recommandations du Programme commun d'évaluation des médicaments (PCEM) et les régimes provinciaux a déjà fait l'objet d'études. Cependant, à la connaissance des auteurs, aucune étude n'a encore examiné les caractéristiques des recommandations du PCEM qui peuvent être associées à la concordance.

Méthode : Des paires recommandation–décision ont été recueillies auprès du PCEM et des régimes provinciaux de l'Ontario, de la Colombie-Britannique et de l'Alberta. La concordance a été évaluée selon les provinces. Les caractéristiques de chaque recommandation du PCEM ont été recueillies et les liens avec la concordance ont été évalués au moyen de la régression logistique.

Résultats : La concordance des listes de recommandations était élevée. Les références positives aux coûts et aux résultats cliniques par rapport au placebo étaient statistiquement associées à une concordance. Les références négatives aux coûts, à la cohérence et à la certitude des données économiques étaient statistiquement associées à une discordance. Cependant, ces résultats n'étaient pas uniformes parmi les provinces étudiées.

Conclusion : Bien que la concordance soit élevée, les caractéristiques des recommandations expliquent de façon limitée la relation entre une province et les décisions d'inscription au PCEM. Cette étude exploratoire met en évidence la complexité du processus de remboursement et les raisons possibles des différences entre les listes de médicaments d'une province à l'autre.

Comments

Be the first to comment on this!