Wednesday, June 14, 2006

Peer review and Trial by jury

There has been a big shouting match debate going on in the physics blogosphere over the last couple of days. The topic under discussion is the role that democracy plays, can play or should play within science.

Now, it is easy to make up various kinds of strawmen and bash them to death, e.g. the idea of determining the values of the Standard Model parameters by public voting (which nobody advocates), or the notion of a scientific dictatorship where one single person decides on what is science and what isn't (which hopefully also nobody advocates).

To actually perform a serious analysis of what we want the scientific community to look like is much more difficult: On the one hand, there is clearly a lot to be said in favour of a scientific aristocracy of experts; on the other hand, do we really want some small self-recruiting in-group to decide about everyone else's funding, especially given that they will still be human and hence their decisions may be guided by personal like or dislike of a person just as well as by scientific analysis of his or her proposal?

These are not easy issues to discuss and decide (and of course any discussion of them in the blogosphere is going to have virtually no effect at all, given that the physics blogosphere is dominated by lowly postdocs, or at most assistant professors, and hence does not exactly represent the views of the major policy-makers within the community).

Here, I would just like to point out that the use of the word "democracy" may be slightly misleading, at least as far as common connotations go. Most people, when hearing "democracy", will think of voting, and possibly the absence of an individual or group with dictatorial powers, leading quickly to the kind of strawman arguments that dominated this debate. However, there is another crucial feature of (at least British and American) democracy: I'm speaking of trial by jury. This is a profoundly democratic institution; nobody can be found guilty of and punished for a crime, unless he either admits it himself by pleading guilty, or the prosecution manages to convince a panel of twelve people chosen at random from among the accused's peers (rather than some group of politicians or experts) of his guilt.

This is in many ways a much better analogy for the kind of democracy that can, should, and in fact does exist in the scientific community. Peer review is not that different from trial by jury, with the reviewers acting as the equivalent of jurors (randomly chosen peers of the author), and the editor as the equivalent of the judge. There are even appeals, and many journals have a kind of voir dire where potential conflicts of interest are examined before selecting referees. Of course the analogy is not perfect, because there are no opposing parties to the proceedings, but this is (at least in my opinion) a much closer analogy. In fact, in many respects the work of the scientist is somewhat similar to that of the judiciary (weighing evidence and coming to a conclusion), just as it is hugely different from that of the legislative and executive branches (which are used as flawed analogies in the strawman arguments mentioned above).

Comments are welcome.

Update: More on the debate in a new post by Sabine (to whom we extend our warmest congratulations on her recent marriage) on Backreaction.

Friday, June 09, 2006

The Satanic Papers

Since -- surprise, surprise -- no apocalyptic cataclysm occurred on 6/6/6 (which was 6/6/2006 really anyways, and is also variously known as 24/5/2006, 10/9/5766, 10/3/5766, 9/5/1427, 16/3/1385, 16/3/1928, 12.19.13.6.10, 2006-23-2, 2006-157 or even 18.9.CCXIV, not to mention that the count of years A.D. isn't even historically accurate), maybe we should look for the hand of Satan where it has a greater chance of manifesting itself: I mean in the arXiv, of course! So here are the papers bearing the cleverly disguised mark of the beast:

astro-ph/0606006: M. Meneghetti et al., Arc sensitivity to cluster ellipticity, asymmetries and substructures. Studies how the distribution of galaxies within a cluster affects the shape of the arcs created by gravitational lensing by that cluster.

cond-mat/0606006: J.-P. Wüstenberg et al., Spin- and time-resolved photoemission studies of thin Co2FeSi Heusler alloy films. Studies the properties of some material of interest in spintronics.

gr-qc/0606006: S. Deser, First-order Formalism and Odd-derivative Actions. Studies how a separate Palatini-type reformulation of standard and Chern-Simons terms affects electromagnetism and gravity, with some surprising results.

hep-ex/0606006: ICARUS Collaboration (A. Ankowski et al.), Measurement of Through-Going Particle Momentum By Means Of Multiple Scattering With The ICARUS T600 TPC. Describes the collaboration's liquid argon detector and the methods to be used to measure particle momenta with it.

hep-lat/0606006: C. Hagen et al., Search for the Theta^+(1540) in lattice QCD. Looks for the Θ+(1540) pentaquark in a quenched simulation and doesn't find it.

hep-ph/0606006: C.A. Salgado, Overview of the heavy ion session: Moriond 2006. Summary of proceedings.

hep-th/0606006: S. Creek et al., Braneworld stars and black holes. Studies sperically symmetric brane solutions in a Randall-Sundrum scenario.

math-ph/0606006: R.G. Smirnov and P. Winternitz, A class of superintegrable systems of Calogero type. Studies the relationship between the three-body Calogero system in one dimension and a class of three- and twodimensional superintegrable systems.

nucl-ex/0606006: S. Salur (for the STAR Collaboration), Statistical Models and STAR's Strange Data. Compares various models to the collaboration's data on strange hadron production in p+p and Au+Au collisions.

nucl-th/0606006: L. Platter, The Three-Nucleon System at Next-To-Next-To-Leading Order. Computes the triton binding energy to NNLO in an effective field theory with only contact interactions.

physics/0606006: W.A. Rolke and A.M. Lopez, A Test for the Presence of a Signal. Studies some methods for statistical hypothesis testing.

quant-ph/0606006: F. Cannata and A. Ventura, Scattering by PT-symmetric non-local potentials. Studies one-dimensional scattering by separable non-local potentials and looks at the contraints imposed on the transmission and reflection coefficients by hermiticity and P- and T-invariance.

Not sure if any of this looks particularly satanic, but it was fun to look (however superficially) at papers from archives I would normally never go near.

Quarkonia and MEM

On the arXiv today is a paper by Peter Petreczky about the spectral functions of heavy quarkonia at finite temperature.

People generally expect that at high temperatures, heavy quarkonia will be suppressed, because the gluons will be screened by thermal effects (Debye screening, and possibly chromomagnetic screening as well), leading to an exponential fall-off of the interquark potential at large distances and hence allowing the heavy quarks to drift apart. This suppression of quarkonia is supposed to be an important signature of the formation of a quark-gluon plasma, and hence confirming it in a model-independent way is important. One way to do this is to look at the spectral functions for the corresponding correlators and to see whether the peaks in the spectral function that correspond to the bound states in that channel will broaden and eventually vanish as the temperature is increased.

The results in this case are that the 1P charmonia (the $$J/\psi$$ and its kin) do dissolve just above the deconfinement transition, whereas other quarkonia appear to persist up to considerably higher temperatures.

Now how do people obtain these kinds of results? The spectral function is the function σ(ω) appearing in the Euclidean periodic-time equivalent of the Källén-Lehmann spectral representation

$$D(t) = \int_0^\infty d\omega \sigma(\omega)\frac{\cosh(\omega(t-\beta/2))}{\sinh(\omega\beta/2)}$$

where the latter expression is the correlator for a free particle of mass ω, with β being the extent in the Euclidean time direction. So if you have measured the correlator D(t), you just invert this to get the spectral function, which contains all the information of the spectrum of the theory.

There is one lie in this last sentence, and that lie is the little word "just". The reason is that you are trying to reconstruct a continuous function σ(ω) from a small number of measured data points D(β i/Nt), making this an ill-posed problem.

The way around that people use lies in a method called Maximum Entropy Method (MEM) image restoration, which is also used to restore noisy images in astronomy. (Unfortunately it is bound by the rules of logic and hence cannot do all the wonderful and impossible things, such as looking through opaque foreground objects or enlarging a section to reveal details much smaller than an original pixel, that the writers of CSI or Numb3rs are so fond of showing to an impressionable public in the interest of deterrence, but it is still pretty amazing -- just google and look at some of the "before and after" pictures.)

The basis for MEM is Bayes' theorem

$$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$$

which relates the conditional probability for A given B to that for B given A. Using Bayes' theorem, the probability to have the spectral function σ given the data D and fundamental assumptions H (such as positivity and high-energy asymptotics) is

$$P(\sigma|D,H) = P(D|\sigma,H) P(\sigma|H)$$

where conventionally P(D|σ,H) is known as the likelihood function (it tells you how likely your data are under the assumptions), and P(σ|H) is known as the prior probability (it tells you how probable a given σ is prior to any observation D). The likelihood function may be taken to be

$$P(D|\sigma,H) = Z \exp\left(-\frac{1}{2}\chi^2\right)$$

where χ2 is the standard χ2 statistic for how well the D(t) given by σ fits your data D(β i/Nt), and Z is a normalisation factor. For the prior probability, on takes the exponential

$$P(\sigma|H) = Z' \exp(\alpha S)$$

of the Shannon-Jaynes entropy

$$S = \int_0^\infty d\omega \left[\sigma(\omega)-m(\omega)-\sigma(\omega)\log\left(\frac{\sigma(\omega)}{m(\omega)}\right)\right]$$

where m is a function called the default model, and α is a positive real parameter.

The most probable "image" σα for given α (and m) is then the solution to the functional differential equation

$$\frac{\delta Q_\alpha}{\delta \sigma_\alpha} = 0$$

where

$$Q_\alpha = \left(\alpha S - \frac{1}{2}\chi^2\right)$$

The parameter α hence parametrises a tradeoff between minimising χ2 and maximising S, which corresponds to making σ close to m. Some MEM methods take α to be an arbitrary tunable parameter, whereas in others, to get the final output σMEM, one still has to average over α with the weight P(α|D,H,m), which can be computed using another round of Bayes' theorem. In practice, people appear to use various kinds of approximations. It should be noted that the final result

$$\sigma_{MEM}(\omega) = \int d\alpha \sigma_\alpha(\omega) P(\alpha|D,H,m)$$


still depends on m, although this dependence should be small if m was a good default model.

This is pretty cool stuff.