This paper presents a novel conceptualization of policy making as social drama. The selection and presentation of evidence for policy making, including the choice of which questions to ask, which evidence to compile in a synthesis and which syntheses to bring to the policy making table, should be considered as moves in a rhetorical argumentation game and not as the harvesting of objective facts to be fed into a logical decision-making sequence. Viewing policy making as argument does not mean it is beyond rationality - merely that we must redefine rationality to include not only logical inference and probabilistic reasoning, but also the consideration of plausibility by a reasonable audience. We need better evidence, but we also urgently need better awareness by policy makers of the language games on which their work depends.

[To view the French abstract, please scroll down.]

The Cochrane Collaboration was built on a myth - that the judgments required for evidence synthesis are fundamentally technical ones, achieved through the skilled application of tools of the trade such as protocols, data extraction sheets, methodological checklists and evidence hierarchies. Quality in Cochrane reviews is assured by the robustness of the protocol, the exhaustiveness of the data extraction and the ruthlessness with which "methodologically inferior" studies were rejected.

In the evaluation of simple clinical interventions (such as drug therapies), this myth approximates reality so closely that it is entirely appropriate to operate as if the world were actually thus. But the world of policy making is not one of transferable and enduring scientific truths, nor is it exclusively (or even predominantly) concerned with "what works," and the systematic review movement must adapt accordingly (Lomas 2005; Lavis et al. 2005a). In this paper, we argue that the first step in this process is to change the way we conceptualize the policy making process.

Policy making - which might be defined as the authoritative exposition of values - is about defining and pursuing the right course of action in a particular context, at a particular time, for a particular group of people and with a particular allocation of resources. Policy making is about making and implementing collective ethical judgments. Most of us are painfully aware that "evidence," as the evidence-based medicine movement would define it, fits obliquely and sometimes very marginally into this process. But if evidence is marginal, what is central?

Sociologist Judith Green (2000) undertook a detailed ethnographic study of the work of multi-professional Accident Alliances in the United Kingdom. Her fieldwork demonstrated that in establishing credibility for a proposed course of action in accident prevention policy, advocates drew judiciously (and often very eloquently) upon a variety of sources, including professional expertise, local knowledge, appeals to common sense and personal experience. Research evidence on "what works" was rarely crucial to the case. For example, while randomized trial evidence unequivocally supports the efficacy and cost-effectiveness of hip protectors worn by frail elderly people in the prevention of fall-related injury, the policy making decision turned on the argument that "padded knickers" were seen as unpopular and even comical by patients and staff in nursing homes.

This example - in which a randomized trial reported in the language of risk prevention ("hip protectors") was displaced from its perch atop the evidence hierarchy by a rhetorical trope ("padded knickers") designed to draw the audience's attention away from issues of risk and towards those of individual dignity and self-determination - vividly illustrates that the "evidence" for policy making is not sitting in journals ready to be harvested by assiduous systematic reviewers. Rather, it is dynamically created through the human interaction around the policy making table - and, probably more significantly, the lobbying, campaigning and interpersonal influencing going on in the back rooms and corridors leading up to official policy making meetings.

Before we set any rules about what sort of systematic review policy makers need, we must understand in more detail what policy making is. Policy making is not a series of decision nodes into which evidence, however robust, can be "fed," but the messy unfolding of collective action, achieved mostly through dialogue, argument, influence and conflict and retrospectively made sense of through the telling of stories (formally in the minutes of meetings and informally in personal accounts of who said what and how, and how people reacted) (Birch 1997; Czarniawska 2004; Fischer and Forester 1993; Majone 1989; Stone 1997; Young et al. 2002).

We propose that the selection, compilation, presentation, negotiation, contestation and reframing of evidence as part of the "stuff happening" of policy making can usefully be construed as social drama - that is, as a real, enacted story in which all concerned, whether they want to or not, become actors (Turner 1980). Furthermore, the policy making stage is a slippery one, fraught with ambiguity, unpredictability and multiple interpretations. Playing one's part in it can be a frustrating experience - one that lobbyists and the media understand far better than the humble systematic reviewer.

On this stage, the protocols, checklists and hierarchies that are set so securely in stone in the Cochrane Handbook can crumble to dust or be distorted at will by the skilled or passionate orator. In social drama, personal testimony ("anecdotal evidence") is a uniquely authentic and powerful force. In a recent high-profile litigation in the United States against Dow Chemical, falsely blamed for a link between silicon breast implants and connective tissue disorders, one witness successfully refuted a library of epidemiological evidence by pointing to her own evident rheumatological disorder and proclaiming "I am the evidence" (Angell 1996).

The concept of evidence as rhetorically constructed on the social stage so as to achieve particular ends for particular people raises an important question (to which we have for too long assumed the answer to be "yes"): to what extent should policy making be driven by evidence? (Sanderson 2003). The very expression "evidence-based policy making" suggests that there are technical solutions to what are essentially political problems - an assumption that, some have argued, devalues democratic debate and plays down the ethical, moral and political ambiguities and dilemmas inherent in the lived reality of planning, implementing and evaluating in social-political life (Hammersley 2001; Schwandt 1997, 2000).

The normative goals of evidence-based practice (finding out what works and then implementing it) are closely aligned with those of the new public management (defining explicit performance outputs and promoting efficiency and cost-effectiveness) (Webb 2001; Hammersley, 2001). Critics of this approach argue that what matters is not merely "what works" but what is appropriate in the circumstances and what is agreed to be the overall desirable goal (Sanderson 2003; Dobrow et al. 2004).

Here's a provocative question: is the "methodological fetishism" of which the Cochrane Collaboration has been accused an extreme example of the politicization of science by the new managerialists? MacLure (2005) has argued that systematic review

assumes that evidence can be extracted intact from the texts in which it is embedded, and "synthesised" in a form that is impervious to ambiguities of context, readers' interpretations of writers' arguments (i.e. bias). Most significantly of all, systematic review systematically degrades the central acts of reviewing: namely, reading and writing, and the unreliable intellectual acts that these support, such as interpretation, argument, and analysis. By replacing reading and writing with an alternate lexicon of scanning, screening, mapping, data extraction, and synthesis, systematic review tries to transform reading and writing into accountable acts. It tries to force their clandestine operations - the bits that happen inside people's heads - or in the incorporeal gaps between decoding and comprehension, thought and expression - up into plain view, where they can be observed, quality-controlled and stripped of interpretation or rhetoric.

Perhaps, then, clarity, transparency, explicitness, reproducibility and other virtues held dear by the Cochrane community have more to do with the discourse of accountability than with the essential quality of the judgments they are assumed to underpin. Deborah Stone (1997) believes that much of the policy process involves debates about values masquerading as debates about facts and data: "The essence of policy making in political communities [is] the struggle over ideas. Ideas are at the centre of all political conflict. … Each idea is an argument, or more accurately, a collection of arguments in favour of different ways of seeing the world."

Stone's work, and other critiques of the evidence-into-policy model, shift the challenge of "synthesizing evidence for policy making" from a scientific-rationalist frame (ensuring that "objective" evidence is available in an easily assimilable format and in a timely manner to policy makers) to a rhetorical-interpretive frame (acknowledging that all evidence is, and must remain, value-laden and will be rhetorically and judiciously brought to bear in the contact sport of policy development) (Fischer and Forester 1993; Majone 1989; Stone 1997). In this latter perspective, there is no "view from nowhere," so systematic reviewers might as well give up looking for it:

As politicians know only too well but social scientists too often forget, public policy is made of language. Whether in written or oral form, argument is central in all stages of the policy process. … Argumentation is the key process through which citizens and policymakers arrive at moral judgments and policy choices. … Each participant [in policy debates] is encouraged to adjust his view of reality, and even to change his values, as a result of the process of reciprocal persuasion. (Majone 1989)

Whereas the technical model of policy making ("evidence into practice") sees group decision-making as a sequence of logical moves to weigh evidence and reach a single, "rational" course of action, the argumentation model proposes (a) that someone presenting evidence to others tailors the presentation to what he or she believes the audience will find persuasive and (b) that what we will accept as evidence depends on what we have already agreed (what has been established or accepted among the team so far) and what we consider to be an acceptable link between the two states (Crawshay-Williams 1957; Toulmin 1958; Perelman and Olbrechts-Tyteca 1971; van Eemeren et al. 1996).

The roots of argumentation theory lie in Aristotle's philosophical treatises on analytic (logical argument using premises based on certain knowledge), dialectic (debating moves to argue for and against a standpoint) and rhetoric (influencing by reference to laws, documents, etc. or by appeal to emotions, authority or previously acceded premises). Most modern-day scientists (including those in the evidence-based medicine movement) hold that "rationality" is restricted to analytic argument. But for the ancient Greeks, all three dimensions of argumentation were seen as rational, and a respectable scholar was expected to achieve competence in all of them. As the "padded knickers" example illustrates, it is neither "unscientific" nor "biased" to employ rhetorical techniques to get an audience to frame a problem in a new light.

In analytic logic, "evidence" might be thought of as that which is provably true (as in, "Socrates is a man; all men are mortals; therefore, Socrates is mortal") or probably true (in the sense of Bayesian notions such as odds ratios, effect estimates and confidence intervals). But in rhetorical argument, the bounds of rationality extend to what is plausibly true - that is, "evidence" is whatever will convince a reasonable audience.

In their polemical work, The New Rhetoric, Perelman and Olbrechts-Tyteca (1971: 45) showed that rhetorical argumentation techniques persuade by increasing the "intensity of adherence among those who hear it in such a way as to set in motion the intended action." There are, of course, implicit agreements within particular audiences, expressed by their shared language (e.g., jargon, professional practices) and the initiation required to join such a group. There are also "preferable premises" - that is, values, value hierarchies and loci (preferences of one abstraction over another, which are the basis of value hierarchies). All these form what are known as the audience's points of departure.

Taking account of points of departure, the arguer uses rhetorical schemes, such as association or dissociation. Association brings together through metaphor or analogy elements that were seen as separate ("we value the input of independent experts; X is an independent expert"). Dissociation does the opposite; it separates elements previously assumed to be part of a whole (as in "that 'peer reviewed journal' was actually published by the pharmaceutical industry"). Argumentation can be viewed as a performance of "regulated disputation" held according to agreed rules of engagement. Fallacies (that is, things an audience rejects in an argument) are seen as the non-adherence to these agreed rules (van Eemeren et al. 1996). Any argument can be systematically deconstructed to expose the use of rhetorical devices such as association and dissociation, and to expose the (unwritten) rules that the audience uses to accept (as rational) or reject (as fallacious) the conclusions and recommendations made by different players.

Applying these concepts to policy making, Schon and Rein (1990) have suggested that difficult policy making tasks should be faced by acknowledging that controversy is inherent in such work. The way to deal with this inherent and irreducible messiness is not to produce more rigorous, more relevant, less ambiguous, more timely or more appealingly presented evidence but for policy makers to develop a better awareness of their own behaviour as players in the argumentation game.

Reflection on the underlying differences that lead to frustrations and conflicts - differences of backgrounds, values, norms and on what constitutes evidence (the points of departure) and therefore what follows as acceptable conclusions or actions (rules of engagement) - is a critical step for managers and policy makers in moving towards a new rationality of policy making (that is, one in which a linear link between evidence and policy is explicitly rejected, and in which the skills of argumentation are acknowledged, promoted and reflected upon rather than dismissed as underhand, biased or "anecdotal").

Jeremy Grimshaw, who heads the Cochrane Collaboration's Effective Practice and Organization of Care Group, has recently lamented that despite 30 years' research, we still lack a generalizable evidence base to inform management and policy making (Grimshaw et al. 2004), but his proposed solution - that we should do more of the same research, only bigger and better - is naive. There never will be a "generalizable evidence base" on which managers and policy makers will be able to draw unambiguously and to universal agreement, and however hard we strive for methodological rigour in systematic review, there never can be a policy that is unambiguously "evidence-based."

Where does this leave us? The "new systematic review methodology" - pragmatic, pluralistic, context-sensitive and cutting its cloth according to local resources, needs, contexts and timescales - is an important epistemological breakthrough. Disseminating its principles, and raising awareness of the growing range of tools and techniques available to the methodologically discerning reviewer (Dixon-Woods et al. 2005; Lavis et al. 2005b; Pawson et al. 2005; Greenhalgh et al. 2005; Lomas 2005), is a high priority. But equally important is the task of disabusing the healthcare community of the misconception that policy making is, or ever could be, "evidence-based" in the way this term is conventionally construed.

A more fruitful, and certainly more original, use of research funding would be to promote and evaluate the training of policy making teams in the art of rhetoric, and particularly in what Schon (1990) has called "frame reflective awareness," designed to ensure that the players in the policy making drama acknowledge and take account of their respective points of departure. Making explicit the values and premises on which each side has built its case will not only highlight "evidence gaps" more systematically but will also generate light rather than heat at the policy making table.

Recadrer la synthèse des preuves comme une mesure de pure forme dans le drame de l'élaboration de politiques

Résumé

Cet article présente une nouvelle conceptualisation de l'élaboration de politiques en tant que drame social. La sélection et la présentation des preuves servant de base à l'élaboration de politiques, y compris le choix des questions à poser, des données à compiler dans une synthèse et des synthèses à amener à la table d'élaboration de politiques devraient être considérées comme faisant partie d'un jeu d'arguments rhétoriques et non comme une collecte de faits objectifs qui iront alimenter un processus logique de prise de décisions. Le fait d'envisager l'élaboration de politiques comme un argument ne signifie pas qu'elle est dénuée de rationalité, mais simplement que nous devons redéfinir la rationalité pour y inclure non seulement l'inférence logique et le raisonnement probabiliste, mais également la plausibilité aux yeux d'un auditoire raisonnable. Nous avons besoin de meilleures preuves, mais il existe aussi un besoin urgent de sensibiliser les décideurs aux jeux de langue dont dépend leur travail.

Comments

Be the first to comment on this!