Peer review in scientific publications - Science and Technology Committee Contents

Written evidence submitted by Dr Andrew Sugden, Deputy Editor & International Managing Editor, Science (PR 91)

1.  Peer-reviewed scientific publications represent the primary useful archive of scientific progress. Scientific publications have served other main functions as well: They are a primary means of evaluating scientists and institutions, and they have become a main pathway for informing the general public about science, through coverage in the media and press releases (new results are news). Peer-reviewed publications are also now relied upon in official and legal affairs: In the U.S., the Supreme Court codified the use of peer-reviewed publications in the courtroom, and several acts of Congress have codified their use in government regulations. Many advisory groups rely on, and/or are mandated to use, the peer-reviewed literature. Thus scientific publishers recognize that one of their major responsibilities is ensuring and enhancing the integrity of journal publications for these diverse uses.

2.  Science magazine's remit is to publish research papers with high conceptual novelty and broad interest, in all disciplines of science, both physical and biological. Science is a weekly publication, and the number of research papers we publish is small—typically about 18 in each issue. We place strict limits on the length of papers, and aim to publish expediently. Science is currently the world's widest-circulation general science journal, with a global subscriber base of 130,000 with many more readers accessing the journal online via institutional site licenses. Hence, the number of submissions of research papers from scientists is high, and we can accept few of them for publication: fewer than 10% of submissions are published. Science's editorial workflow and review process are designed to facilitate high-quality choice.

3.  The responsibility for managing the peer review process and for making decisions on rejection/revision/acceptance of submissions for publication rests with the staff editors. Staff editors at Science are PhD-qualified scientists with postdoctoral research experience (and in some cases subsequent experience as professional editors at other journals). Editors are appointed to Science primarily on the basis of the strength of their research record, and their task is to represent a particular discipline (chemistry, astronomy, immunology, ecology etc) and handle the submissions in that discipline. Editors work in teams of four or five, and Science's working practice is that all such decisions are made in consultation with at least one other staff colleague via a common electronic database containing all manuscript records. There is some overlap in expertise between editors, which ensures that no decision need be made in isolation, but also leads to consistency in decision-making. The editorial staff is managed by the Executive Editor, reporting to the Editor-in-Chief who is a senior research scientist appointed by the AAAS Board.

4.  Science has a 2-stage review process for submitted manuscripts. The first stage is primarily a filter, designed to identify the 25% potentially most innovative and original submissions, which, if correct, would qualify for publication in Science. This initial process takes an average of 7 days, which allows authors of solid papers that we deem inappropriate for Science to remain competitive at other journals. This stage is carried out through consultation with the Science's Board of 150 Reviewing Editors. Submissions may be sent to one or more Board members, depending on the discipline or disciplines represented. A typical evaluation from a Board member consists of a paragraph of explanation, a score on a rating of 1-10, a rating of the Board member's judgement of his/her own confidence in the score, and (if the submission is being recommended for in-depth peer review) suggestions for appropriate referees. The evaluation is designed to assess the potential scientific importance of each submission rather than to assess its technical qualities in any detail. The staff editor then decides, on the basis of this advice received, whether to proceed to in-depth peer review (see below) or to reject the paper. In keeping with the purpose of this first stage of selection, there is no precisely-defined threshold score required for this editorial decision, which occurs after an average of seven days. When authors are notified of the decision to reject or review at the end of this first stage, the identity and views of the Board member(s) involved remain confidential.

5.  Why does Science not send more (or all) submissions for in-depth review? The number of manuscripts submitted exceeds the number published by more than a factor of 10. Hence, reviewing a larger proportion would be a hindrance to all parties: authors would suffer delays in finding an alternative journal in which to publish; referees would be spending time reviewing submissions that have a high likelihood of being rejected; the attention of editorial staff would be diverted from those submissions with the highest promise.

6.  The Board of Reviewing Editors consists of c. 170 individuals from 20 countries, appointed by the staff editors to represent the spread of subdisciplines across the sciences. They are mostly mid-career active research scientists with a strong record in their respective fields. On appointment, which is usually for three to five years, each Board member agrees to evaluate up to six Science submissions per week. The Board's role is purely advisory. Members are not expected to do in depth review or decide the fate of submissions, but may occasionally be consulted by staff editors at later stages of peer review (see below) and on appeals (see below). Science's view is that the involvement of the Board at the first stage of review is an important element in the effort to maintain editorial consistency, and it substantially improves the research community's confidence in the fairness of the initial cut.

7.  The second stage of review, for the 25% of submissions not rejected at the first stage, is in-depth review by peers. Referees are selected by the staff editor based mainly on the editor's own knowledge of researchers in the field(s) of the submission, plus suggestions from the Board. At the time of submission, authors are asked to submit their own nominations for referees, and staff editors will occasionally follow these nominations where they coincide with the Board's and/or their own suggestions. We also encourage authors to tell us if they believe that certain individuals have conflicts of interest and should not be consulted as reviewers. The number of referees varies depending on the scope of the submission. The minimum is two, but three or more are frequently used, especially where a submission is multidisciplinary and/or combines a number of components/techniques requiring input from individuals with special expertise. Science's editors always seek referees' agreement to review a manuscript before it is sent to them; the referees are asked to return their reviews within two weeks.

8.  The role of referees at the in-depth stage of review is typical of that followed at most scientific journals. Referees are asked to assess the technical merits and integrity of the submission, and to recommend improvements and revisions that should be made before the submission can be considered acceptable. In our view, the role of a properly-operating peer review system is to maximise the quality of the published account of any piece of research, within it own limits, regardless of where it is ultimately published. Thus, we expect referees to make detailed recommendations regardless of whether they consider the submission ultimately suitable for our journal. Even if the submission is rejected by Science on the basis of in-depth peer review, the referees' comments will generally be helpful to authors in revising the manuscript for submission to a different journal. As a result of the peer review process, many submissions are improved, and improved substantially. Errors are caught (though not always), uncertainties are clarified; standards are met, and even hypotheses can be changed or strengthened. Not all good ideas get delayed through rejection; many become better and stronger.

9.  Peer review, as a system for maintaining the integrity of scientific research and improving the quality of published research, inevitably relies on trust in the integrity and reliability of the scientists and editors tasked with carrying it out. It is not a 100% safeguard against clever fraud, but in the great majority of cases it can be relied upon to fulfil its goal of minimizing the propagation of errors. Nonetheless, not every error is caught, and Science and other journals will publish corrections and clarifications when necessary: in the past decade, about 8% of papers have been corrected; in most cases, these corrections affect matters of detail but not the major conclusions of the work. Severe cases that require complete retraction of a paper are much rarer: over the past decade 0.4% of Science papers have been retracted.

10.  In addition to their written report on the submission that will be provided to the authors, referees are asked to rate the manuscript as either Excellent & Exciting, Above Average, Too Specialized, or Mediocre/Poor, as well as to recommend whether the submission should be published without delay, published after minor revision, re-reviewed or rejected. The referees are also given the opportunity to provide confidential comments to the editor.

11.  The length of the in-depth review process is generally longer and more variable than that of the initial screening described above. The average time for a round of review for a Science submission is currently about three weeks, but may vary depending on the complexity or urgency of the material under review.

12.  In-depth review does not always lead to a straightforward decision for the staff editor. Referees may differ as to the technical quality or potential significance of a submission. Experience shows that the editor is often best advised to follow the more critical opinion in such cases, whether in deciding to reject the submission or in asking for revision. However, our editors are urged to use their own judgement in this regard. In some cases, editors will send a revised submission back to referees (or sometimes new referees) for further checking, especially if the revision contains new material (data, experiments) that was not present in the first version of the submission. At this stage, referees will usually be shown the reports of the other referee(s) and will be asked to assess how the author has responded to all the recommendations. A third round of re-review is rare: generally, the final decision on rejection or acceptance will be taken no later than the second round of review.

13.  Science's policy is to maintain the anonymity of referees in all communication with authors; their reports are unsigned. Although we recognise that the identity of some can be evident from the text of their reports, Science's view is that anonymity gives us access to the widest possible pool of referees, for example those who may be at an early stage of their careers relative to the author of a submission. However, referees are not blinded to the identity of the authors, in common with practice at the majority of other journals.

14.  We recognise the potential for conflicts of interests in the review process. Hence, we allow authors to request that their manuscript not be sent to particular individuals who might be competitors or where there is other reason to suspect the potential for bias or unfair review. Editors are also expected to be alert to potential abuse of this facility on the part of authors, through their knowledge of the research groups involved. We also ask our referees to return or destroy the manuscript without review if they find a conflict of interest or other reason preventing them from reviewing a manuscript in a timely and fair fashion.

15.  In common with other journals, Science does not offer any payment for peer review (though the Board of Reviewing Editors receive an honorarium of a subscription to the journal). Payment would be inimical to the process, yet it is also the case that scientists do not routinely receive the recognition that might be expected for the work that they put into reviewing journal submissions. For example, institutions could recognise peer reviewing activities when assessing a scientist's job application or promotional prospects.

16.  A small percentage of rejection decisions are appealed by authors. For a submission to be reconsidered after a rejection at the first stage, editors need to be convinced that the author has brought some pertinent new information to the table that the editors were not aware of at the time of the rejection. (Disagreement over the degree of novelty or general interest is not enough). For a submission to be reconsidered after a rejection following in-depth review, the editors need to be convinced that the major mistakes were made in the peer-review process, and that the rejection was based on these mistakes. A small proportion of submissions are reconsidered on appeal, and of these even fewer are eventually accepted for publication following further review.

17.  In our experience, the quality of reports from referees is high in most cases, in that they provide pertinent feedback on the key elements of a manuscript and on the importance of the research reported. Nonetheless, we find some reports to be less than ideal in length, detail and focus. Brief reports consisting of a single short paragraph are very rarely adequate for conveying the basis of a decision (whether negative or positive) to an author, yet such reports are sometimes received, and quite often from senior and established scientists. The opposite—reports of excessive length and detail—can also be an occasional problem. And it is not unknown for referees to use inappropriately emotive or forceful language (generally we only edit this out when it is particularly or egregiously offensive).

18.  A common complaint is that referees ask for unnecessary extra details and further experiment before a submission is accepted for publication. All research is part of a wider work-in-progress, and progress is facilitated by publishing rather than withholding. Publication is part of the ongoing scientific process, not the end of the road. Hence, while such requests for further work are often legitimate, referees and editors need to be able to recognise when a piece of work is complete within its own goals and frame of reference. We want our editors to consult with each other in making the final decision on this important, quite frequently encountered, issue.

19.  Anecdotal evidence suggests that there is wide variation in the training that early-career scientists (graduate students, postdoctoral associates) receive in peer review. Most such training is largely ad hoc and informal, dependent on the input of the supervisor or other senior colleagues. Institution-wide principles for practice and training in peer review are not yet the norm. We would recommend that journal editors and academies work together to produce guiding principles for the peer review process that can be adopted and used for instruction at the institutional level.

20.  The chief challenge in the peer review process is the time available for referees. For an editor, the process of finding referees can be time-consuming—not only identifying the appropriate individuals but also sometimes contacting a large number of individuals before finding enough referees willing and able to assess a submission. For referees, assessing a journal submission in the right amount of detail and at the right level involves more than simply reading and commenting on the manuscript and preparing a report. It can involve recalculating results presented in the work, checking citations (including relevant literature not cited), and otherwise checking experiments and analyses reported in the work in order to verify the conclusions. Effective peer review will generally take anywhere from several hours to several days of full-time work. Hence, an editor's first or second choice may well decline to review if other commitments are too pressing.

21.  Editors may become understandably biased in favour of using tried-and-tested referees who are known to be reliable and efficient and understanding of the particular requirements of the editor's journal. This is not in itself a threat to the integrity of the peer review process, but it can become a limit to the size of the potential pool of referees as well as placing a disproportionate burden on a relatively small number of individuals.

22.  Multiple rounds of review can constitute a further problem for the peer review process. As noted above, Science will generally limit the number of rounds of review to a maximum of two, and this is common practice at many other journals. However, a manuscript may be reviewed several more times at several different journals before it eventually finds a home, sometimes by the same referee more than once. Recognising that this is a further drain on the system, Science and other journals have considered sharing referees' reports when a manuscript is submitted to a second or third journal following rejection from the authors' first choice. However, there are several obstacles to such a system, including for example referee anonymity and different editorial policies at different journals.

23.  Multidisciplinarity is a potential pitfall for peer review, requiring extra vigilance on the part of editors to ensure that referees are chosen to cover all the main areas of research that are represented in a submission. There is an increasing amount of contemporary research at the interface of biological and physical sciences (for example, in computational biology or climate change biology), and editors need to be able to recognise the appropriate contributory elements in such cases. Sometimes this means that more than three referees will be needed to adequately review a paper. There has also been a perennial difficulty in reviewing the statistical components of research, where editors and referees are not always qualified in the statistical techniques that have been used in a research project.

24.  There may be procedural differences in peer review between disciplines (for instance, in physics research is made available to readers through preprint servers) but the principles of peer review as a mechanism for improving and maintaining standards in published research are very similar across all disciplines. Where the peer review process becomes harder is in disciplines that are small, with few experts qualified to comment on submissions, or few without conflicts of interest of some kind.

25.  The mobility of scientists, especially younger scientists, coupled with the growth of international collaboration in science and the ease of access to published research via the WWW, means that any national differences in cultural or scientific traditions have become increasingly irrelevant in the context of peer review. National biases in peer review may have been present in the past, because journals have generally been nationally based and hence scientists' work would tend to be reviewed by peers in their own country. The increasing internationalization of research, coupled with the ease of e-communication, will have contributed to the reliability and rigour of peer review in the past two decades. At Science, we have made efforts to ensure that the overall geographical distribution of referees reflects the global nature of the scientific enterprise.

26.  Clearly the impact of information technology has been all-pervasive in science. For peer review, the impact of IT and online resources has been mainly on the efficiency of the process, and not on the underlying principle of peer review (though it has also enabled the exploration of new models or variations on the theme of peer review). E-communication has improved the speed of communication (especially international communication) between editors, referees and authors. It has enabled editors to research a broader range of potential referees for individual submissions, and perhaps has enabled referees to better research the background to the submissions they are asked to review. Electronic submission systems have reduced authors' concerns about the cost and time-lost when submitting to journals with the end result of authors submitting to top journals even when the chance of acceptance is very slim.

27.  Science began featuring supporting online material in the late 1990s (we went online in 1996). Today, most papers (>95%) in Science include an online supplement that describes methods and additional data, and some of these supplements are huge in terms of pages and data. This is also the case for many other journals. While there are obvious advantages to supplying the background data to the reader, these supplements are posing growing problems for peer review. Review of a supplement that is many times the size of the submitted text is a burden to reviewers and hinders requests for rapid consideration. It also raises concerns about the quality of peer review. These issues probably can't be avoided, but standards for reporting and presenting large data sets that allow common analysis tools could help greatly. An additional challenge is providing confidential access to large or complex datasets during review. Currently no databases allow secure posting for the purposes of peer-review, and some authors are unwilling to release data prior to publication. We are in some cases sending data, including large data files, separately to reviewers, but this poses an increasing administrative burden. Raw data for some papers in several fields are too large to transmit, and in some cases special software may be required.

28.  Notwithstanding the pitfalls of the peer review system outlined above, Science maintains (in common with other scientific journals) that it will remain the primary means of validating research for publication. Recognition of the potential pitfalls is the key to ensuring that the system works well, and that errors and poor scientific practices are minimized.

Dr Andrew Sugden
International Managing Editor and Deputy Editor for Life Science
on behalf of and with contribution from Alan Leshner (Executive Publisher); Bruce Alberts (Editor-in-Chief); Monica Bradford (Executive Editor); Brooks Hanson (Deputy Editor, Physical Science), Barbara Jasny (Deputy Editor, Commentary) (Science, 1200 New York Avenue, Washington DC 20005, USA)

25 March 2011


1.  Peer Review at Science Publications (from guidelines for referees at Science website):

2.  Information for reviewers of Research Articles (from guidleines for referees at Science website):

3.  Reviewing Peer Review: Bruce Alberts, Brooks Hanson, and Katrina L Kelner. Science 4 July 2008: 15. [DOI:10.1126/science.1162115]



—  Peer Review at Science

—  Peer Review at Science Signaling

As a peer reviewer for Science magazine, you are part of a valued community. Scientific progress depends on the communication of information that can be trusted, and the peer review process is a vital part of that system.

Only some of the submitted papers are reviewed in depth. For in-depth review, at least two outside referees are consulted. Reviewers are contacted before being sent a paper and are asked to return comments within one to two weeks for most papers. Reviewers may be selected to evaluate separate components of a manuscript. We greatly appreciate the time spent in preparing a review, and will consult you on a revision of a manuscript only if we believe the paper has been significantly improved but still requires input. The final responsibility for decisions of acceptance or rejection of a submitted manuscript lies with the editor.


1.  Reviews should be objective evaluations of the research. If you cannot judge a paper impartially, you should not accept it for review or you should notify the editor as soon as you appreciate the situation. If you have any professional or financial affiliations that may be perceived as a conflict of interest in reviewing the manuscript, or a history of personal differences with the author(s), you should describe them in your confidential comments.

2.  If, as a reviewer, you believe that you are not qualified to evaluate a component of the research, you should inform the editor in your review.

3.  Reviews should be constructive and courteous and the reviewer should respect the intellectual independence of the author. The reviewer should avoid personal comments; Science reserves the right to edit out comments that will hinder constructive discussion of manuscripts.

4.  Just as you wish prompt evaluations of your own research, please return your reviews within the time period specified when you were asked to review the paper. If events will prevent a timely review, it is your responsibility to inform the editor at the time of the request.

5.  The review process is conducted anonymously; Science never reveals the identity of reviewers to authors. The privacy and anonymity provisions of this process extend to the reviewer, who should not reveal his or her identity to outsiders or members of the press. The review itself will be shared only with the author, and possibly with other reviewers and our Board.

6.  The submitted manuscript is a privileged communication and must be treated as a confidential document. Please destroy all copies of the manuscript after review. Please do not share the manuscript with any colleagues without the explicit permission of the editor. Reviewers should not make personal or professional use of the data or interpretations before publication without the authors' specific permission (unless you are writing an editorial or commentary to accompany the article).

7.  You should be aware of Science's policies for authors regarding conflict of interest, data availability, and materials sharing. See

previous page contents next page

© Parliamentary copyright 2011
Prepared 28 July 2011