Peer review

Written evidence submitted by Dr Thomas J Webb (PR 62)

1. Peer review diagnosis: fine tuning, not overhaul

1.1 Peer review is the primary means of quality control in science. It is used to assess the merits of a very great variety of work, but the most significant areas of its application are in reviewing manuscripts intended for publication in the primary literature, and in supporting decisions over funding applications. The issues concerning peer review differ substantially between these two areas, so it is sensible to consider them separately.

2. Peer review for publication

2.1 Some hold that science isn’t science until it appears in the peer-reviewed literature; peer review thus acts as an essential filter to distinguish fully competent work from preliminary results or unsupported conclusions. I have considerable experience here as an author, reviewer, and editor, and my overwhelming feeling is that the current system works rather well. Sloppy or poor quality work is rarely published (certainly not in the more rigorous journals); when it is, it is usually identified as such by readers, who are likely to write a (peer-reviewed) response to correct the record. Likewise, it is exceedingly rare either for fraudulent results get through the peer review filter, or for excellent science to be thwarted. At the same time, many of the more alarmist comments that peer review for scientific journals is in crisis or plagued by biases are simply not supported by anything other than the flimsiest of anecdotal evidence.

2.2 Concerns have been raised that with the ever increasing numbers of submissions to journals, the supply of willing and able reviewers is drying up. I disagree. True, it can be hard to find reviewers on occasion, but this can be addressed by journals having a broad editorial board (which most already do) to maximise the number of potential reviewers of whom we are aware. I would also encourage senior academics, rather than simply declining to review a manuscript due to time constraints, to recommend their more junior group members (post-docs and PhD students) as alternative reviewers. This already happens often, but making it common practice would introduce the next generation of reviewers into the system more quickly (developing their skills at the same time).

2.3 It is also worth clarifying exactly how most journals – certainly most in my field, ecology – currently operate, as I’m not sure this is well understood by some critics of the present system. Usually, each submission is seen by a senior editor, who makes an initial decision regarding its overall worth and, especially, its fit to the aims and scope of the specific journal. If it is deemed potentially publishable, the manuscript is then assigned to a handling editor – almost always an (unpaid) academic – who will critically read the work, and take the decision either to send it for full peer review (usually by two independent reviewers) or to recommend rejection on the basis of obvious and serious flaws, or again, a lack of fit to the journal (‘not of sufficient general interest’ is a common comment). Although having one’s work rejected at this stage is galling for an author, it has the benefit of saving everyone (authors, editors, reviewers) a lot of time and effort. By filtering out a large proportion of submissions in this way (frequently 20-40%, significantly more in some higher-profile journals), the burden on the peer review process is greatly reduced.

2.4 Those manuscripts that do pass this filter will be reviewed by two or more independent reviewers, whose job is to assess the quality of the work – importantly, although their report may include a recommendation (e.g. ‘accept with minor revisions’), reviewers do not themselves reject or accept manuscripts. The handling editor will collate the reviewers reports and also make a recommendation, pending the senior editor’s final decision.

2.5 Given this strong editorial role, it’s not that unusual for a manuscript to be rejected despite reasonably positive reports from the referees; this can be very frustrating for authors, but it does mean that if they submit their manuscript to another, more appropriate or specialised journal, it stands a very good chance of being published there. One idea that has some support is to allow authors to include reviews of a previous version of their work from one journal when submitting to another. This has considerable merit – essentially, making best use of the overall pool of reviewers by including some ‘memory’ in the system. But it need not be formalised, as there is nothing to stop authors doing it now: certainly, as a handling editor, I have received submissions in which the cover letter includes a full report of a previous round of peer review (reviewers’ comments, and authors’ responses) from a different journal, which I have found very helpful.

2.6 In almost all journals in my field, the names of the reviewers are not revealed to the authors (unless the reviewers decide to sign their report), but the names of the authors are known to editors, handling editors, and reviewers. The option of reviewer anonymity is important, in my opinion, to enable reviewers to criticise freely and without fear of consequences. (It is the job of the editors to ensure that this criticism is fair.) The practice of ‘double blind’ reviewing, whereby authors’ names are concealed from reviewers (it is not clear how far this ‘blinding’ should proceed up the editorial ladder, given complications with making sure that authors aren’t assigned their own work to review!) has been promoted by some, but there is no evidence that the costs of implementing such a system bring any benefits – nor even that the process of blinding works, as reviewers can often guess the identity of the authors in any case. (There is a good deal of published evidence to back up my assertions here, some of which is summarised in Webb et al. (2008) Trends Ecol Evol 23: 351-353.)

2.7 Among the most compelling arguments against radical change in the way that journals manage the peer review process is the case of PLoS ONE, which started with the aim of revolutionising scientific publishing. PLoS ONE has been a great success, but it is revealing that initial plans to take a new approach to peer review, with most decisions taken rapidly by an editor and manuscripts rarely sent to review, and with an emphasis on post-publication review (i.e., comments posted online by readers), have slowly drifted back to a situation in which >90% of manuscripts are sent to review, most reviews are anonymous, and very few articles attract significant online comments. (The reason for this last failure is that we (academics) get no credit for online comments, which do not form an official part of the peer review process. For instance, I pointed out a basic error in one PLoS ONE paper, through an online comment; but the authors never responded, the paper still exists to be read and cited in its original state, and my comment is essentailly lost to the scientific record.)

2.8 So, most attempts to reform peer review for publication regress reasonably rapidly back to the existing norm, which is imperfect but generally popular, most importantly, effective in its primary purpose of ensuring that good science is published, and that bad science is not.

3. Peer review of funding applications

3.1 Having a paper rejected on the basis of an unfair review is frustrating, but ultimately part of the process of being a professional scientist. Having a funding application rejected, on the other hand, can end the career of a promising young scientist, and can have huge effects on the career progression of all of us. Getting the review of proposals right, therefore, is essential for the efficient functioning of UK science.

3.2 It is hard to argue completely dispassionately about these decisions, as all of us will have felt hard done by at one stage or another by at least one funding panel. The reality is that there are a lot of us chasing a very limited pot of money, and that disappointment inevitably is the most frequent outcome. We should recognise, however, the tremendous pressure put on academics to pursue any possible source of funding (often to the detriment of their research), the amount of time spent preparing what are frequently lengthy and very detailed proposals which have perhaps one chance in twenty of being successful; and of course, the requirements to find suitably qualified reviewers to assess the merits, and weight up the risks and rewards, of all of these good ideas.

3.3 Most research funding in the UK remains project-based, i.e. funding is allocated to specific projects, rather than to individuals or groups. Each of these project proposals must be critically assessed. To achieve this, research councils now tend to operate a peer review college, whose members are paid a small stipend in order to assess a large number of funding applications every year. Perhaps 40% of these applications will be rejected by the research council on the basis of college members’ reports; the remaining applications will proceed to full review, being sent to independent referees who are (hopefully) experts in the field.

3.4 A frequent complaint with this system is the lack of relevant expertise among peer review college members, but this is potentially self correcting: those complaining should apply to join the college. My major problem with this system is rather the colossal waste of time that it involves for some of the brightest and most highly qualified people in the UK, who must spend up to a third of their working year preparing very detailed funding proposals, only for four in ten of them to be rejected without proceeding to full review. This wastes the time of the academics preparing the applications (in that they are unable actually to do research, due to the time spent proposing researc), their support teams who help with calculating budgets and so on, all staff at the research councils who have to process the applications, and the reviewers who have to read through numerous lengthy applications. Much better to have a two stage submission process, as some organisations do at present, whereby outline proposals (a couple of pages, with a brief description of the major research question and methodology, track records of the investigators, and a guideline budget) are rapidly sifted by peer review college members and the authors of those identified as having promise are invited to prepare a full proposal. We all complain already that the pre-review sifting of proposals is unfair; this would be no less fair, and much quicker for all involved.

3.5 Second, the emphasis in a lot of grant reviews is to identify negatives. For instance, frequently as a reviewer one is asked to comment on the strengths and weaknesses of a proposal. Now, there are only so many ways that it is possible to say ‘this is great’, and every proposal will have weaknesses; so inevitably there ends up being substantially more text in the ‘weaknesses’ box than in the ‘strengths’ box. This can be taken as a sign that the review is more negative than it in fact is. A consequence of this is that it tends to favour ‘safe’ proposals: ones which employ well-established methods, using a well-studied system, and often where substantial preliminary results are already available. In other words, it favours incremental advances over truly innovative thinking. There is a trade-off here, between the advantages in terms of consistency of a structured review process, and the obligation this places on reviewers to identify, for example, weaknesses in what they actually consider to be a very strong proposal. My view is that too many reviews get bogged down in the fine details of methodology, and emphasis (certainly at the early stage – or assessment of the preliminary proposal in the system advocated above) should rather be on the key questions: Is this exciting science? Are the methods generally appropriate? Does the applicant have the requisite skills with which to apply them?

3.6 Of course, there will always be more good proposals than there are funds available, and there is no failsafe way to decide between very different, but perhaps equally good proposals. Various ideas have been proposed to reduce the number of proposals submitted. My preferred option would be to account more rigorously for investigator time. Thus, if you are already committed to two or three projects, it is unlikely you will have sufficient hours left in the week to act as principal investigator on a further major project. Limiting the number of hours that any one investigator can have allocated to projects at any one time would be popular among academics (it would relieve the pressure on successful academics continually to chase more money, for example) and would reduce the total number of funding applications, thus easing the burden on reviewers.

4. Declaration of interests

4.1 I am a Royal Society University Research Fellow, and interact with the peer review system for publication as an author, a reviewer of manuscripts for many journals in the fields of ecology, marine biology, and general life sciences, and as a handling editor for Journal of Animal Ecology and Oikos. I have also published (peer reviewed) work on the merits (or lack thereof) of double blind review, and have written on peer review on my blog Mola mola on Nature Network. With regard to funding proposals, I have interacted with the peer review process with some success and (rather more) failure as an applicant to bodies including the Royal Society, the Natural Environent Research Council (NERC), and the Leverhulme Trust; and I have reviewed proposals for various organisations, most recently NERC and the Greek Ministry of Education.

Dr Thomas J Webb

10 March 2011