Community Peer Review (a great new idea)

I have just come across a great site. It is for commenting on scientific papers that appear on the arXiv preprint server.

http://communitypeerreview.blogspot.de/

This is something I have been discussing in general terms with many colleagues over the past few months. It’s really great (if it takes off) to combine the collective knowledge of all the readers of a paper. It is like collecting all the comments that are made at conferences or over lunch. Of course, if the arXiv were to incorporate this functionality, it would really take off. This is a nonlinear process, and the more comments are generated the more will follow. Since all scientists in the relevant fields have accounts on arXiv with their full names, flame wars and polemics in general can easily be avoided. The discussion would at least be at the level of the typical conference question and answer sessions after a talk. And many little questions (and confusion created by definitions and typos etc.) could be addressed by the authors right away.

Dreaming ahead, in the long run, this might even turn into the prevalent model of publishing (without the need for extra peer-reviewed journals). Comments would take the place of peer reviews. To this end, one could then possibly also implement the possibility of submitting anonymous comments, which would be sent out to other registered users for moderation (to prevent polemics and flame wars). And then there’s the whole issue of also ranking the level of experience of commenters (like in physics.stackexchange.com).

Virtually all of the colleagues I’ve talked to agree that they would much prefer their paper being refereed and ranked by many of their knowledgeable peers from the community — rather than two somewhat randomly picked referees (I know it’s pretty hard for editors to pick good referees, there are so many topics; I am an editor myself). What makes matters worse currently is that the referee decision, with all its random factors, ultimately is a digital black/white decision that makes your paper get much attention (like in Nature or Physical Review Letters) or much less attention if it’s rejected there.

Here are some additional thoughts that came up in discussions with colleagues and upon further reflection:

  • You could choose a subset of your colleagues, and then display the arXiv papers that they have commented on (or rated favourably). In the long-term, this could effectively generate community-run topical journals. These are even dynamically generated and tailored exactly to your needs, since it is you who selects the relevant subset of colleagues. These colleagues in this way become something  like editors for that journal addressed to you (and you, in the same way, are effectively selecting articles for others).
  • A simple “like/dislike” or simple rating system would probably be rather counterproductive and likely subject to manipulation. But if the system is more fine-grained, both in terms of rating various aspects of the paper and in terms of keeping track of who did the rating (and how competent they are rated by others!), it could become useful. Imagine you find that a certain paper is rated very favourably by the ‘general, uneducated public’ (those distant to the topic at hand), but rated very unfavourably by the experts. That would tell you something.
  • Unlike the journals, whose publication decision tries to rank the paper immediately, the comment system could actually push a paper to the forefront even after some years. That would be when at least a few experts realize it is important (or some experiments confirm the predictions) and this is reflected in a few high-profile favourable comments (by the experts), which may trigger further interest and discussion. This is certainly much more reliable than the initial decision by a journal.
  • I have heard the following argument in favour of judging a paper in the traditional peer-review/selective journal style: It gives unknown authors at least a better chance to garner some attention. However, it is fair to say that even the traditional peer-review system has some bias towards the more established authors. Even if you ask me in my role as a referee, I have to confess that if I am skeptical about some part of the paper, I may be more inclined to accept it if it comes from a well-known group with a good track record. So I do not think that this argument really speaks against community peer review. On the contrary, one might hope that if enough interest is generated in the course of a discussion, the more established experts will take a look (what the fuss is all about), and if they then come to a positive conclusion and post that publicly, this will boost the paper very much.

Some further links:

Paul Ginsparg’s discussion from 1997 about the role of the arXiv

Paul Ginsparg’s discussion from 2002

Paul Ginsparg’s most recent discussion of the arXiv, from 2011

A more recent discussion on another blog: http://physicsnapkins.wordpress.com/2012/01/16/occupy_scientific_journals/