Wednesday, June 14, 2006

Peer review and Trial by jury

There has been a big shouting match debate going on in the physics blogosphere over the last couple of days. The topic under discussion is the role that democracy plays, can play or should play within science.

Now, it is easy to make up various kinds of strawmen and bash them to death, e.g. the idea of determining the values of the Standard Model parameters by public voting (which nobody advocates), or the notion of a scientific dictatorship where one single person decides on what is science and what isn't (which hopefully also nobody advocates).

To actually perform a serious analysis of what we want the scientific community to look like is much more difficult: On the one hand, there is clearly a lot to be said in favour of a scientific aristocracy of experts; on the other hand, do we really want some small self-recruiting in-group to decide about everyone else's funding, especially given that they will still be human and hence their decisions may be guided by personal like or dislike of a person just as well as by scientific analysis of his or her proposal?

These are not easy issues to discuss and decide (and of course any discussion of them in the blogosphere is going to have virtually no effect at all, given that the physics blogosphere is dominated by lowly postdocs, or at most assistant professors, and hence does not exactly represent the views of the major policy-makers within the community).

Here, I would just like to point out that the use of the word "democracy" may be slightly misleading, at least as far as common connotations go. Most people, when hearing "democracy", will think of voting, and possibly the absence of an individual or group with dictatorial powers, leading quickly to the kind of strawman arguments that dominated this debate. However, there is another crucial feature of (at least British and American) democracy: I'm speaking of trial by jury. This is a profoundly democratic institution; nobody can be found guilty of and punished for a crime, unless he either admits it himself by pleading guilty, or the prosecution manages to convince a panel of twelve people chosen at random from among the accused's peers (rather than some group of politicians or experts) of his guilt.

This is in many ways a much better analogy for the kind of democracy that can, should, and in fact does exist in the scientific community. Peer review is not that different from trial by jury, with the reviewers acting as the equivalent of jurors (randomly chosen peers of the author), and the editor as the equivalent of the judge. There are even appeals, and many journals have a kind of voir dire where potential conflicts of interest are examined before selecting referees. Of course the analogy is not perfect, because there are no opposing parties to the proceedings, but this is (at least in my opinion) a much closer analogy. In fact, in many respects the work of the scientist is somewhat similar to that of the judiciary (weighing evidence and coming to a conclusion), just as it is hugely different from that of the legislative and executive branches (which are used as flawed analogies in the strawman arguments mentioned above).

Comments are welcome.

Update: More on the debate in a new post by Sabine (to whom we extend our warmest congratulations on her recent marriage) on Backreaction.