A rant on single-blind peer review

Let's say you write something novel and clever and technical, and you submit it to a conference for consideration. If the paper is accepted, you will go to present the paper, and (depending on the conference) it can be published in some digital or paper proceedings. To determine if your paper (or poster, or whatever) is accepted, it undergoes what is called peer review. The program organizers electronically corral experts (or budding experts) in the field to read the submissions and provide critical feedback. These are your peers because they are in the same general field as you are. Some reviewers have more experience than you do, and others have less.

There are usually three or more reviewers per paper, and the reviews are usually in the following format.

• There is a score which correlates the degree to which the reviewer thinks the paper should be accepted. For example:
• -3: Strong reject
• -2: Reject
• -1: Weak reject 
• 0: Neutral
• 1: Weak accept
• 2: Accept
• 3: Strong accept
• There is a narrative describing the paper's strengths and weaknesses, suggestions for improvement, layout and organization, and critique of the bibliography. Reviewers answer the question of whether the paper makes a significant impact or contribution to whatever field the paper is representing.

Sometimes there is a meta-review, provided by another peer, which synthesizes the other reviews into a short blurb.

Selecting papers for publication goes something like this. The reviewers' scores are tallied for each paper, and the papers are ordered from highest to lowest score. The program organizers decide how many papers to accept, and that number of the top-scoring papers is selected. If the reviewers provided any notes to the program organizers, or made specific suggestions for accepting or rejecting a paper, these submissions are taken on a case-by-case basis.

There are two types of peer review that are used most frequently: single-blind and double-blind.

For single-blind peer review, you submit your paper as it would be published, with your name and affiliation at the top. Reviewers can see this. But when you receive your paper reviews, you do not know who wrote which review because the reviewers' names are not provided. It is single-blind because it is blind to you, the author.

In double-blind peer review, you remove your name and affiliation from the paper, and try to anonymize it as much as possible. For example, if you write somewhere, "Our previous work at University of Waterbucket, our home institution," you would take out the reference to your institution's name. Reviewers are discouraged from trying to infer the authors of the paper, and thus should (in theory) not be biased based on your identity and the identities and affiliations of your co-authors. The reviews are also anonymous. It is double-blind because it is blind to the reviewers, and also to you, the author.

There are two reasons that I dislike single-blind peer review.



Because you are remembered


Pretend for a second that a particular reviewer has a chip on his (or her) shoulder about your research area. Or about specific methods which you might have used. For example, the researcher hates video games and thinks that people that play games are worthless in society, and your paper is about a game to examine social interactions, such as Prom Week (which is a fun and lauded new Facebook game; if you have not tried it yet, do). The review will say something like this.

Rating: -2 (Reject) -- The authors present a video game that lets the player make and break friendships in the week before the high school prom. The authors failed to cite relevant literature regarding the misuse of gaming technology and associated aggressiveness in players. The proposed game promotes poor behavior and sensationalizes high school relationships which are fundamentally flawed. Arguing for not accepting this paper.

Now, you can get a review like this even with an anonymous paper. But what can happen next is that the reviewer remembers your name or affiliation -- consciously or otherwise. And when your name comes up again, either in another conference for which he (or she) is peer-reviewing, or dropped in conversation when networking, or presented as a Nobel Peace Prize recipient, the reviewer will remember you for the paper he (or she) rejected. Your name has been associated with That Thing He (or She) Hates (With a Passion). Even if you write on another topic entirely, you are remembered for the paper that was a flop.


Because of the discourse


The second reason is that when you receive this angry or scathing or unfair review following the single-blind process, there is nothing you can do. When you do not know who he (or she) is, you cannot engage the reviewer in conversation; you cannot further the discourse of your disagreement. And when you do not know where he (or she) lives, you certainly cannot mail the reviewer a box of flaming poop.