Skip to main content

Archived Comments for: Classical peer review: an empty gun

Back to article

  1. In defence of peer review

    Dorothy Bishop, University of Oxford

    4 January 2011

    Richard Smith seems not only to underestimate the benefits of peer review, but also to misrepresent its function, at least in the context of journal publishing. Thus he states that the process determines “which papers will be published…” etc. Unfortunately, all too often editors relinquish their responsibilities and treat the peer review process as a vote, but this is a distortion of the real function of peer review, which should be to offer advice to the editor and the author.

    Of course, Dr Smith is right to point out the arbitrary and unreliable features of peer review, and the lack of hard evidence for its benefits. But most of those who’ve tried to evaluate the system seem to focus just on ‘scores’ given to papers. There should be much more to a review than this. In this regard Dr Smith notes “many can tell of anecdotes of how a study they published was much improved by peer review”, but he then dismisses such evidence as balanced by anecdotes of bad experiences with reviewers. But has anyone attempted to quantify this aspect of peer review, i.e. by asking authors whether, on balance, they think the system improves their manuscripts? Dr Smith noted that when he gave a talk on the topic, most scientists were strongly in favour of peer review. Are they all deluded?

    I write this at the end of a week where I have struggled to master a new analytic method, having been convinced by a reviewer that it is relevant to my study. My initial response on receiving the review was the one I usually have: anger at this ignorant fool who hadn’t understood the paper and was wasting my time. But as I struggled to formulate a response, I realised that there was a point to what s/he was saying. I have learned a lot, and the paper has benefited as a result. This is not uncommon, in my experience, especially when I am moving into a new area. I’ve also had reviewers force me to explain my arguments more clearly, to read a literature that I was ignorant of, and to look critically at my assumptions. So this, I think, is the up side of peer review: it stops us from being egocentric and ignoring alternative viewpoints. In the extreme case, reviewers have preserved my reputation by pointing out ignorant errors. Of course, I’ve experienced my fair share of muddled and biased reviewers, but I have to say that in general, I find them outnumbered by reviewers who make an honest attempt to engage with what I’ve written and offer constructive advice. I’d be sorry if this were not available to me. And, yes, I spend time on the other side of the fence, reviewing papers by my ‘peers’, but I don’t regard it as a total waste. Here too, there is benefit in being forced to engage with other people’s ideas.

    I do think the real problem is editors, a topic I have blogged on (see Increasingly, one sees editors who don’t use any judgement at all, but just keep going back to reviewers until there is agreement. I recently had a journal secretary get very tetchy with me when I refused to re-review a paper where the only recommendations I’d made were minor and could easily have been checked by a competent editor. I doubt that the editor read the paper at all.

    Some of Dr Smith’s arguments puzzle me. Why does it cost £100 to review an article, when the reviewers aren’t paid and the process is largely automated? Yes, it can be hard to find reviewers, but this seems an overestimate, once the system is up and running. I started acting as an editor for PLOS One this year, and I usually aim to find 2 reviewers for each paper I handle. It typically takes me about 10-20 minutes per paper. PLOS One also achieves much faster turnaround times than the months cited in Dr Smith’s article, indicating that this can be achieved (though seldom is with medical journals, in my experience).

    Dr Smith and I are clearly in agreement about many aspects of science publishing, as might be expected given our mutual involvement in PLOS journals. However, his uniformly negative analysis of peer review cannot go unchallenged.

    Competing interests