Peer review in scientific publications - Science and Technology Committee Contents


Written evidence submitted by Philip Campbell (PR 60)

1.  I welcome the enquiry on peer review set up by the House of Commons S&T Committee. Peer review, after all, is central to the allocation of research grants and to the publication of valid reports of research.

2.  I am the Editor-in-Chief of Nature and the Editor-in-Chief of the Nature Publishing Group. In the first of the roles, I am responsible for the content of Nature, including the 800-odd research papers that we publish every year. In the second role, I am responsible for the long-term support and development of the quality and editorial policies shared by Nature and all of the other 33 Nature-branded journals published by the Nature Publishing Group.

3.  In this submission I will not attempt a general definition or overview of peer review. My purpose is to highlight a number of issues surrounding peer review, and explain our responses and policies. In some cases these issues relate directly to those highlighted by the Committee. I hope that my discussion of other issues will assist the Committee in its thinking.

4.  We keep our policies under review, in order to reflect changing needs within science and changing factors that affect research. In particular, the Chief Editors of the journals meet on a monthly basis in order to keep abreast of relevant developments inside and outside the company, to reflect on researchers' needs, and to develop policies when required.

5.  Nature is the most highly cited multi-discipline natural science journal. It has a long and distinguished history (first published in 1869) of covering the most important science and of rigorously applying peer-review to the papers it publishes.

6.  Nature-branded journals for the most part focus on specific scientific disciplines. Based on their Impact Factors (a measure of the typical level of citations of a journal), they are often the most highly cited in their fields or if not, are one of a handful of top journals in their fields.

7.  Nature and the Nature journals are untypical journals in that they do not have editorial boards of active researchers. All selection decisions are the responsibility of the fully independent and Chief Editors of each journal and their teams, informed (but not instructed) by the advice from the peer review process. These editorial staff are employees of the Nature Publishing Group, fully dedicated to the task of selecting papers for publication. Through their frequent visits to conferences, laboratories and through constant scrutiny of the literature, they keep abreast of the state of their disciplines and develop contacts with existing and new reviewers.

8.  Full details of our approach to peer review and details of our journals are available on the Nature Publishing Group website. This site includes articles that we have written about peer review and also community discussions about the topic: http://www.nature.com/authors/peer_review/index.html and, with particular relevance to this submission: http://www.nature.com/authors/policies/peer_review.html.

How does our peer review add value?

9.  We occasionally survey our authors and are encouraged by the fact that the substantial majority believe that peer review has added to the quality of their paper. This reflects the evidence from published surveys: http://www.publishingresearch.net/PeerReview.htm and http://www.senseaboutscience.org.uk/index.php/site/project/395.

10.  Referees are expected to read a submitted paper with close attention in order to provide a reasonable assurance of its validity, checking whether its internal logic is consistent and also that its content is consistent with firmly established knowledge of the techniques involved and of the phenomena being discussed.

11.  As well as checking for validity, referees are expected to suggest improvements where they feel, for example, that the logic is insufficiently clear, that the previous literature is inadequately represented, or where certain experiments might close loopholes or otherwise strengthen the conclusions of a paper.

12.  The task of refereeing a paper takes significant time - anything between several hours and several days.

13.  A reviewer is not expected to check the results of a paper by replication. In exceptional circumstances, referees will undertake considerable work on their own initiative to replicate an aspect of a paper.

14.  Journal peer review adds value not only to authors and to publishers, but also in a resulting sense of confidence amongst stakeholders and publics that the scientific literature's claims, often based on highly specialized evidence or argument, have credibility.

15.  Even after peer review, a scientific paper should be considered provisional, either in its detail or even in its fundamental conclusions. It will ultimately stand or fall on its subsequent replication or corroboration by other scientists in the short term, and on the deepening of scientific understanding in the longer term.

16.  It is part of the editor's and peer-reviewer's responsibilities to ensure that data and materials required for other researchers to replicate or otherwise verify and build on the work are subsequently available to those who need it. Such availability may be problematic eg when expensive materials or organisms are involved, but we have policies to ensure such availability where there are facilities such as public databases that enable it, and in general such availability is a condition of publication.

Anonymity in peer review

17.  Nature commissions confidential reviews of papers on the basis that the identity of the referee will be kept anonymous. This is a firmly held principle in practice - we have on occasion declined requests from university enquiries to reveal referees' identities, for example, though we usually seek referees' agreement to assist in such circumstances.

18.  Our key motivation in peer review is always to act on behalf of our readers in ensuring that we publish only the most scientifically significant papers that are submitted to us. The relationship between an editor and his/her referees, and their mutual confidence, can become a key element in optimising the basis for an editor's judgements to that end.

19.  Why anonymity? Because otherwise an author rejected on the basis of a review may resent the reviewer and subsequently act against him or her either subconsciously or in deliberate retaliation. This is a particular concern for up-and-coming researchers, who often provide excellent reviews of papers. Also, it is not unknown for authors who have identified a referee to attempt to pressurise them during their assessment of a revised version.

20.  Can anonymity be abused? It is an editor's responsibility to apply judgement to protect the interests of authors and readers against such abuse and, if disputes arise, to seek advice from others, and thereby do whatever can be done to ensure both ethical and technical integrity.

21.  Surveys suggest that anonymity in peer review is widely accepted and indeed preferred by most researchers (see
http://www.senseaboutscience.org.uk/index.php/site/project/395).

22.  Where referees ask to be named, we will usually identify them.

Peer review by competitors

23.  Our policy is that submitting authors may request that individuals be avoided as peer reviewers on grounds of competing research interests. Provided the number is not excessive (the usual limit is three) we almost always abide by such requests, though reserve the right to choose whoever we need to provide the best advice.

24.  In our letter of invitation to referees, we ask them to decline our invitation if they have any interests that they consider to be dangerously conflicting, or at least to notify us of such interests.

25.  It is an editor's responsibility to use their knowledge of the field and of individuals to take possible competing interests into account, and to seek additional advice where necessary.

Alleged undue influence by a single referee or a coterie

26.  It is sometimes asserted that a single negative referee can force the rejection of a paper. While that can indeed happen on the technical aspects of a paper, our decision-making is structured in a way that helps prevent undue influence by referees, whether negative or positive.

27.  Editors are expected to apply their own judgements in selecting what Nature should publish. Nature's use of referees is not a process of voting, nor a process where the referees' views are the final arbiter of a decision whether to publish.

28.  Editors will involve their in-house colleagues where controversies arise or where several scientific disciplines need to be involved in an assessment of a paper. Except in the most clear-cut cases, no paper is accepted without also being considered by the Chief Biology Editor or Physical Sciences Editor. These senior editors and, in some cases, I will also become involved where a dispute arises between the authors and editors.

29.  We value referees' reports both for their technical assessments and for the explanations of their judgements about a paper's significance. Most papers are seen by several referees of diverse specialist expertise in order specifically to assess particular aspects of the paper.

30.  Our editors will almost always bow to a referee's expertise on the technical validity of a paper's content. Where a dispute arises about technical matters, we will seek advice from other experts.

31.  However, we rely on our editors to make the final decision as to a paper's appropriateness for Nature, based on its scientific significance as judged by them. A referee's opinion on this aspect of a paper is often invaluable, but is in no way binding on our editors. Indeed referees' opinions about a paper's importance often differ.

32.  Thus, in some cases the editor may choose to accept the paper against a referee's advice about its significance. There have been occasions where a paper has been accepted despite negative judgements of its significance by all of the two, three or (in one case that I am aware of) four reviewers involved. These are inevitably subjective judgements, quite distinct from judgements of the technical merits, where the expert views of the referees take priority.

33.  The difference in judgement may operate in the opposite sense also, resulting in rejection despite a positive endorsement by one or more referees.

34.  Authors may appeal against negative decisions where they specifically disagree with referees' or editors' comments.

35.  Editors frequently try out new reviewers alongside established reviewers. We also encourage senior reviewers to include their younger colleagues (whom they must identify) in the process, in order to grow the reservoir of advice while also ensuring that our reviews are fully informed of the current state of the art in evolving experimental techniques.

Alleged influence of where and who a paper comes from

36.  It is a fundamental principle of our processes that papers are judged on their scientific importance, not on their authorship or geographical origins. Any other approach ultimately undermines our perceived value and also may lose us valuable papers.

37.  A relevant consideration is that many papers are co-authored by international collaborations, with different components of the work being done in different laboratories. We estimate the average number of authors on Nature papers to be between 5 and 6.

38.  Our decisions frequently require us to reject authors of high reputation and influence in prestigious institutions, who may also serve us as referees. Such decisions are a part of everyday decision-making.

Double-blind peer review

39.  It is sometimes suggested that papers being sent out for peer assessment should be anonymized, in order to prevent bias.

40.  Our current policy is not to offer this facility. A full discussion of this and our policy can be found at: http://www.nature.com/nature/journal/v451/n7179/full/451605b.html.

41.  I would acknowledge that double-blind peer review is favoured in some surveys (see http://www.publishingresearch.net/PeerReview.htm). We are therefore keeping this policy under review.

Open peer review

42.  In 2006, Nature ran an experiment in open peer review, in which over a period of four months, submitting authors were invited to post their papers on an open website for open assessment by peers. Their papers were also peer-reviewed in the usual way.

43.  The details of the process and the outcome can be found at:
http://www.nature.com/nature/peerreview/debate/nature05535.html.

44.  In brief, the take-up by authors was low, as was the amount of open commenting. Furthermore it was judged by the editors that the comments added little to the assessment of the paper.

45.  It is my view, consistent with this outcome, that scientists are much better motivated to comment on an interesting paper when directly requested to do so by an editor.

Post-publication review and commenting

46.  Nature provides facilities for open commenting on its research papers. Our experience is that this facility is little used.

47.  Our experience of the open peer review experience and the lack of commenting on papers in whichever journal provides it suggests that such unsolicited commenting has yet to take hold as a part of scientific activity. One can readily speculate why: there is no prestige or credit attached, there is the risk of alienating colleagues by public criticism, and everyone is busy.

48.  Sometimes public online debate about a paper may arise after its publication, but that usually happens in the blogs of scientists and others.

Publication of referees' and editors' reports and correspondence

49.  One journal published by the Nature Publishing Group, the EMBO Journal, has pioneered the online display of anonymized referees and editors/authors' correspondence after publication, alongside the paper. For an account of this policy, see the article by the publisher of the European Molecular Biology Organisation (EMBO), Bernd Pulverer at Nature v468, p29-31, 4 November 2010, doi:10.1038/468029a, http://www.nature.com/nature/journal/v468/n7320/full/468029a.html.

50.  Nature and the Nature journals have so far not gone down this route. This reluctance is partly based on a precautionary fear that it might upset the relationship between editors and referees. Moreover, the documents reflect only a part of the process of discussions within the editorial team, between the editors and the referees, and between the editors and the authors. There is also a belief that few people will want to wade through this copious information.

51.  Nevertheless, transparency has its own virtues, and we are keeping this policy under review.

Pressure on peer review

52.  There is without question a pressure on peer review, in that academics are very busy, the number of papers produced has grown, and so forth. Nature journals are privileged in that the papers submitted and selected for peer review tend to be interesting, thus most have not been affected by this.

53.  Our editors reject 70-80% of submitted papers (the exact proportion varies with discipline) on purely editorial grounds, in order not to burden referees unnecessarily.

54.  As explained elsewhere, we are always seeking new reviewers, and welcome the advice of younger (but fully qualified) reviewers, who often bring the greatest technical expertise to bear on newer techniques or fast-moving areas of science given their very active involvement in research.

Transfer of reports between publishers

55.  Since 2008, Nature Neuroscience (NN) has participated in an experiment involving other publishers in which, at the request of authors rejected from NN, and with the explicit agreement of referees, NN's referees' reports and their identities can be passed to another publishers' journal. (Similar transfers are routinely offered as a service to authors rejected by Nature or one of our Nature research journals and wishing to submit to another Nature journal.)

56.  This experiment was conceived in the belief that it would save referees work - it is quite common for the same referee to be asked to referee a paper several times by a succession of journals as an author seeks its publication. Moreover, an editor might decide to publish or reject a paper immediately, without troubling referees, on the basis of received reports.

57.  The take-up by authors of this facility has been small - typically amongst the participating journals, a few percent of rejected authors request it. For those that do, the processing time at subsequent journals has been reduced, though I do not have ready access to the statistics.

58.  This facility is controversial within NPG. We invest significant sums of money in our professional editors spending time both in the office and in visits cultivating contacts with referees, and fostering insightful refereeing as best we can. To then hand on the reports and names to another publisher is to some extent undermining our competitive edge. Indeed, the principle competitor of NN, Neuron, is not a part of the experiment, and we might well not have joined the experiment if it was.

59.  However, if there was stronger evidence that the community wanted this to be happening, and that the burden on referees would indeed be reduced, we would consider developing this approach.

Can peer review detect misconduct?

60.  Given that editors and peer-reviewers need to take everything that authors submit on trust, and do not seek to replicate the work, it is almost impossible for referees to detect misconduct. There have been occasions where a sharp-eyed referee has detected an inconsistency or other flaw in reported results that can only have arisen through inappropriate manipulation, but these are few and far between.

61.  In some of the most severe cases of misconduct, a problem has arisen because of insufficient critical scrutiny between co-authors. But that problem lies beyond the scope of this discussion.

Can misconduct arise during peer review?

62.  The unfortunate answer is: yes. It is not unheard of, for example, for editors to be visiting a lab and to see lying on a table in an open space a paper that is being refereed, supposedly confidentially, by someone at the lab.

63.  The most egregious type of misconduct is for a referee to sit on a paper and to steal its results in order to publish them under his or her own name. Such incidents are very uncommon, in our experience.

64.  We are alert to delays by referees' (which will usually be for the reason that they have conflicting commitments), and will cut off a process or seek alternative advice rather than have a referee unduly delay a paper.

65.  This brings to a conclusion my description of our practice and some key policies associated with our peer review processes. I hope that the committee find it helpful.

Philip Campbell PhD
Nature

10 March 2011


 
previous page contents next page


© Parliamentary copyright 2011
Prepared 28 July 2011