Written evidence submitted by Dr Andrew
Sugden, Deputy Editor & International Managing Editor, Science
(PR 91)
1. Peer-reviewed scientific publications represent
the primary useful archive of scientific progress. Scientific
publications have served other main functions as well: They are
a primary means of evaluating scientists and institutions, and
they have become a main pathway for informing the general public
about science, through coverage in the media and press releases
(new results are news). Peer-reviewed publications are also now
relied upon in official and legal affairs: In the U.S., the Supreme
Court codified the use of peer-reviewed publications in the courtroom,
and several acts of Congress have codified their use in government
regulations. Many advisory groups rely on, and/or are mandated
to use, the peer-reviewed literature. Thus scientific publishers
recognize that one of their major responsibilities is ensuring
and enhancing the integrity of journal publications for these
diverse uses.
2. Science magazine's remit is to publish
research papers with high conceptual novelty and broad interest,
in all disciplines of science, both physical and biological. Science
is a weekly publication, and the number of research papers we
publish is small—typically about 18 in each issue. We place
strict limits on the length of papers, and aim to publish expediently.
Science is currently the world's widest-circulation general
science journal, with a global subscriber base of 130,000 with
many more readers accessing the journal online via institutional
site licenses. Hence, the number of submissions of research papers
from scientists is high, and we can accept few of them for publication:
fewer than 10% of submissions are published. Science's editorial
workflow and review process are designed to facilitate high-quality
choice.
3. The responsibility for managing the peer review
process and for making decisions on rejection/revision/acceptance
of submissions for publication rests with the staff editors. Staff
editors at Science are PhD-qualified scientists with postdoctoral
research experience (and in some cases subsequent experience as
professional editors at other journals). Editors are appointed
to Science primarily on the basis of the strength of their research
record, and their task is to represent a particular discipline
(chemistry, astronomy, immunology, ecology etc) and handle the
submissions in that discipline. Editors work in teams of four
or five, and Science's working practice is that all such
decisions are made in consultation with at least one other staff
colleague via a common electronic database containing all manuscript
records. There is some overlap in expertise between editors, which
ensures that no decision need be made in isolation, but also leads
to consistency in decision-making. The editorial staff is managed
by the Executive Editor, reporting to the Editor-in-Chief who
is a senior research scientist appointed by the AAAS Board.
4. Science has a 2-stage review process
for submitted manuscripts. The first stage is primarily a filter,
designed to identify the 25% potentially most innovative and original
submissions, which, if correct, would qualify for publication
in Science. This initial process takes an average of 7
days, which allows authors of solid papers that we deem inappropriate
for Science to remain competitive at other journals. This
stage is carried out through consultation with the Science's Board
of 150 Reviewing Editors. Submissions may be sent to one or more
Board members, depending on the discipline or disciplines represented.
A typical evaluation from a Board member consists of a paragraph
of explanation, a score on a rating of 1-10, a rating of the Board
member's judgement of his/her own confidence in the score, and
(if the submission is being recommended for in-depth peer review)
suggestions for appropriate referees. The evaluation is designed
to assess the potential scientific importance of each submission
rather than to assess its technical qualities in any detail. The
staff editor then decides, on the basis of this advice received,
whether to proceed to in-depth peer review (see below) or to reject
the paper. In keeping with the purpose of this first stage of
selection, there is no precisely-defined threshold score required
for this editorial decision, which occurs after an average of
seven days. When authors are notified of the decision to reject
or review at the end of this first stage, the identity and views
of the Board member(s) involved remain confidential.
5. Why does Science not send more (or
all) submissions for in-depth review? The number of manuscripts
submitted exceeds the number published by more than a factor of
10. Hence, reviewing a larger proportion would be a hindrance
to all parties: authors would suffer delays in finding an alternative
journal in which to publish; referees would be spending time reviewing
submissions that have a high likelihood of being rejected; the
attention of editorial staff would be diverted from those submissions
with the highest promise.
6. The Board of Reviewing Editors consists of
c. 170 individuals from 20 countries, appointed by the staff editors
to represent the spread of subdisciplines across the sciences.
They are mostly mid-career active research scientists with a strong
record in their respective fields. On appointment, which is usually
for three to five years, each Board member agrees to evaluate
up to six Science submissions per week. The Board's role
is purely advisory. Members are not expected to do in depth review
or decide the fate of submissions, but may occasionally be consulted
by staff editors at later stages of peer review (see below) and
on appeals (see below). Science's view is that the involvement
of the Board at the first stage of review is an important element
in the effort to maintain editorial consistency, and it substantially
improves the research community's confidence in the fairness of
the initial cut.
7. The second stage of review, for the 25% of
submissions not rejected at the first stage, is in-depth review
by peers. Referees are selected by the staff editor based mainly
on the editor's own knowledge of researchers in the field(s) of
the submission, plus suggestions from the Board. At the time of
submission, authors are asked to submit their own nominations
for referees, and staff editors will occasionally follow these
nominations where they coincide with the Board's and/or their
own suggestions. We also encourage authors to tell us if they
believe that certain individuals have conflicts of interest and
should not be consulted as reviewers. The number of referees varies
depending on the scope of the submission. The minimum is two,
but three or more are frequently used, especially where a submission
is multidisciplinary and/or combines a number of components/techniques
requiring input from individuals with special expertise. Science's
editors always seek referees' agreement to review a manuscript
before it is sent to them; the referees are asked to return their
reviews within two weeks.
8. The role of referees at the in-depth stage
of review is typical of that followed at most scientific journals.
Referees are asked to assess the technical merits and integrity
of the submission, and to recommend improvements and revisions
that should be made before the submission can be considered acceptable.
In our view, the role of a properly-operating peer review system
is to maximise the quality of the published account of any piece
of research, within it own limits, regardless of where it is ultimately
published. Thus, we expect referees to make detailed recommendations
regardless of whether they consider the submission ultimately
suitable for our journal. Even if the submission is rejected by
Science on the basis of in-depth peer review, the referees'
comments will generally be helpful to authors in revising the
manuscript for submission to a different journal. As a result
of the peer review process, many submissions are improved, and
improved substantially. Errors are caught (though not always),
uncertainties are clarified; standards are met, and even hypotheses
can be changed or strengthened. Not all good ideas get delayed
through rejection; many become better and stronger.
9. Peer review, as a system for maintaining the
integrity of scientific research and improving the quality of
published research, inevitably relies on trust in the integrity
and reliability of the scientists and editors tasked with carrying
it out. It is not a 100% safeguard against clever fraud, but in
the great majority of cases it can be relied upon to fulfil its
goal of minimizing the propagation of errors. Nonetheless, not
every error is caught, and Science and other journals will publish
corrections and clarifications when necessary: in the past decade,
about 8% of papers have been corrected; in most cases, these corrections
affect matters of detail but not the major conclusions of the
work. Severe cases that require complete retraction of a paper
are much rarer: over the past decade 0.4% of Science papers have
been retracted.
10. In addition to their written report on the
submission that will be provided to the authors, referees are
asked to rate the manuscript as either Excellent & Exciting,
Above Average, Too Specialized, or Mediocre/Poor, as well as to
recommend whether the submission should be published without delay,
published after minor revision, re-reviewed or rejected. The referees
are also given the opportunity to provide confidential comments
to the editor.
11. The length of the in-depth review process
is generally longer and more variable than that of the initial
screening described above. The average time for a round of review
for a Science submission is currently about three weeks,
but may vary depending on the complexity or urgency of the material
under review.
12. In-depth review does not always lead to a
straightforward decision for the staff editor. Referees may differ
as to the technical quality or potential significance of a submission.
Experience shows that the editor is often best advised to follow
the more critical opinion in such cases, whether in deciding to
reject the submission or in asking for revision. However, our
editors are urged to use their own judgement in this regard. In
some cases, editors will send a revised submission back to referees
(or sometimes new referees) for further checking, especially if
the revision contains new material (data, experiments) that was
not present in the first version of the submission. At this stage,
referees will usually be shown the reports of the other referee(s)
and will be asked to assess how the author has responded to all
the recommendations. A third round of re-review is rare: generally,
the final decision on rejection or acceptance will be taken no
later than the second round of review.
13. Science's policy is to maintain the
anonymity of referees in all communication with authors; their
reports are unsigned. Although we recognise that the identity
of some can be evident from the text of their reports, Science's
view is that anonymity gives us access to the widest possible
pool of referees, for example those who may be at an early stage
of their careers relative to the author of a submission. However,
referees are not blinded to the identity of the authors, in common
with practice at the majority of other journals.
14. We recognise the potential for conflicts
of interests in the review process. Hence, we allow authors to
request that their manuscript not be sent to particular individuals
who might be competitors or where there is other reason to suspect
the potential for bias or unfair review. Editors are also expected
to be alert to potential abuse of this facility on the part of
authors, through their knowledge of the research groups involved.
We also ask our referees to return or destroy the manuscript without
review if they find a conflict of interest or other reason preventing
them from reviewing a manuscript in a timely and fair fashion.
15. In common with other journals, Science
does not offer any payment for peer review (though the Board of
Reviewing Editors receive an honorarium of a subscription to the
journal). Payment would be inimical to the process, yet it is
also the case that scientists do not routinely receive the recognition
that might be expected for the work that they put into reviewing
journal submissions. For example, institutions could recognise
peer reviewing activities when assessing a scientist's job application
or promotional prospects.
16. A small percentage of rejection decisions
are appealed by authors. For a submission to be reconsidered after
a rejection at the first stage, editors need to be convinced that
the author has brought some pertinent new information to the table
that the editors were not aware of at the time of the rejection.
(Disagreement over the degree of novelty or general interest is
not enough). For a submission to be reconsidered after a rejection
following in-depth review, the editors need to be convinced that
the major mistakes were made in the peer-review process, and that
the rejection was based on these mistakes. A small proportion
of submissions are reconsidered on appeal, and of these even fewer
are eventually accepted for publication following further review.
17. In our experience, the quality of reports
from referees is high in most cases, in that they provide pertinent
feedback on the key elements of a manuscript and on the importance
of the research reported. Nonetheless, we find some reports to
be less than ideal in length, detail and focus. Brief reports
consisting of a single short paragraph are very rarely adequate
for conveying the basis of a decision (whether negative or positive)
to an author, yet such reports are sometimes received, and quite
often from senior and established scientists. The opposite—reports
of excessive length and detail—can also be an occasional
problem. And it is not unknown for referees to use inappropriately
emotive or forceful language (generally we only edit this out
when it is particularly or egregiously offensive).
18. A common complaint is that referees ask for
unnecessary extra details and further experiment before a submission
is accepted for publication. All research is part of a wider work-in-progress,
and progress is facilitated by publishing rather than withholding.
Publication is part of the ongoing scientific process, not the
end of the road. Hence, while such requests for further work are
often legitimate, referees and editors need to be able to recognise
when a piece of work is complete within its own goals and frame
of reference. We want our editors to consult with each other in
making the final decision on this important, quite frequently
encountered, issue.
19. Anecdotal evidence suggests that there is
wide variation in the training that early-career scientists (graduate
students, postdoctoral associates) receive in peer review. Most
such training is largely ad hoc and informal, dependent on the
input of the supervisor or other senior colleagues. Institution-wide
principles for practice and training in peer review are not yet
the norm. We would recommend that journal editors and academies
work together to produce guiding principles for the peer review
process that can be adopted and used for instruction at the institutional
level.
20. The chief challenge in the peer review process
is the time available for referees. For an editor, the process
of finding referees can be time-consuming—not only identifying
the appropriate individuals but also sometimes contacting a large
number of individuals before finding enough referees willing and
able to assess a submission. For referees, assessing a journal
submission in the right amount of detail and at the right level
involves more than simply reading and commenting on the manuscript
and preparing a report. It can involve recalculating results presented
in the work, checking citations (including relevant literature
not cited), and otherwise checking experiments and analyses
reported in the work in order to verify the conclusions. Effective
peer review will generally take anywhere from several hours to
several days of full-time work. Hence, an editor's first or second
choice may well decline to review if other commitments are too
pressing.
21. Editors may become understandably biased
in favour of using tried-and-tested referees who are known to
be reliable and efficient and understanding of the particular
requirements of the editor's journal. This is not in itself a
threat to the integrity of the peer review process, but it can
become a limit to the size of the potential pool of referees as
well as placing a disproportionate burden on a relatively small
number of individuals.
22. Multiple rounds of review can constitute
a further problem for the peer review process. As noted above,
Science will generally limit the number of rounds of review to
a maximum of two, and this is common practice at many other journals.
However, a manuscript may be reviewed several more times at several
different journals before it eventually finds a home, sometimes
by the same referee more than once. Recognising that this is a
further drain on the system, Science and other journals
have considered sharing referees' reports when a manuscript is
submitted to a second or third journal following rejection from
the authors' first choice. However, there are several obstacles
to such a system, including for example referee anonymity and
different editorial policies at different journals.
23. Multidisciplinarity is a potential pitfall
for peer review, requiring extra vigilance on the part of editors
to ensure that referees are chosen to cover all the main areas
of research that are represented in a submission. There is an
increasing amount of contemporary research at the interface of
biological and physical sciences (for example, in computational
biology or climate change biology), and editors need to be able
to recognise the appropriate contributory elements in such cases.
Sometimes this means that more than three referees will be needed
to adequately review a paper. There has also been a perennial
difficulty in reviewing the statistical components of research,
where editors and referees are not always qualified in the statistical
techniques that have been used in a research project.
24. There may be procedural differences in peer
review between disciplines (for instance, in physics research
is made available to readers through preprint servers) but the
principles of peer review as a mechanism for improving and maintaining
standards in published research are very similar across all disciplines.
Where the peer review process becomes harder is in disciplines
that are small, with few experts qualified to comment on submissions,
or few without conflicts of interest of some kind.
25. The mobility of scientists, especially younger
scientists, coupled with the growth of international collaboration
in science and the ease of access to published research via the
WWW, means that any national differences in cultural or scientific
traditions have become increasingly irrelevant in the context
of peer review. National biases in peer review may have been present
in the past, because journals have generally been nationally based
and hence scientists' work would tend to be reviewed by peers
in their own country. The increasing internationalization of research,
coupled with the ease of e-communication, will have contributed
to the reliability and rigour of peer review in the past two decades.
At Science, we have made efforts to ensure that the overall geographical
distribution of referees reflects the global nature of the scientific
enterprise.
26. Clearly the impact of information technology
has been all-pervasive in science. For peer review, the impact
of IT and online resources has been mainly on the efficiency of
the process, and not on the underlying principle of peer review
(though it has also enabled the exploration of new models or variations
on the theme of peer review). E-communication has improved the
speed of communication (especially international communication)
between editors, referees and authors. It has enabled editors
to research a broader range of potential referees for individual
submissions, and perhaps has enabled referees to better research
the background to the submissions they are asked to review. Electronic
submission systems have reduced authors' concerns about the cost
and time-lost when submitting to journals with the end result
of authors submitting to top journals even when the chance of
acceptance is very slim.
27. Science began featuring supporting
online material in the late 1990s (we went online in 1996). Today,
most papers (>95%) in Science include an online supplement
that describes methods and additional data, and some of these
supplements are huge in terms of pages and data. This is also
the case for many other journals. While there are obvious advantages
to supplying the background data to the reader, these supplements
are posing growing problems for peer review. Review of a supplement
that is many times the size of the submitted text is a burden
to reviewers and hinders requests for rapid consideration. It
also raises concerns about the quality of peer review. These issues
probably can't be avoided, but standards for reporting and presenting
large data sets that allow common analysis tools could help greatly.
An additional challenge is providing confidential access to large
or complex datasets during review. Currently no databases allow
secure posting for the purposes of peer-review, and some authors
are unwilling to release data prior to publication. We are in
some cases sending data, including large data files, separately
to reviewers, but this poses an increasing administrative burden.
Raw data for some papers in several fields are too large to transmit,
and in some cases special software may be required.
28. Notwithstanding the pitfalls of the peer
review system outlined above, Science maintains (in common
with other scientific journals) that it will remain the primary
means of validating research for publication. Recognition of the
potential pitfalls is the key to ensuring that the system works
well, and that errors and poor scientific practices are minimized.
Dr Andrew Sugden
International Managing Editor and Deputy
Editor for Life Science
on behalf of and with contribution from Alan Leshner (Executive
Publisher); Bruce Alberts (Editor-in-Chief); Monica Bradford (Executive
Editor); Brooks Hanson (Deputy Editor, Physical Science), Barbara
Jasny (Deputy Editor, Commentary) (Science, 1200 New York Avenue,
Washington DC 20005, USA)
25 March 2011
APPENDICES
1. Peer Review at Science Publications
(from guidelines for referees at Science website): http://www.sciencemag.org/site/feature/contribinfo/review.xhtml
2. Information for reviewers of Research Articles
(from guidleines for referees at Science website): http://www.sciencemag.org/site/feature/contribinfo/review.xhtml
3. Reviewing Peer Review: Bruce Alberts, Brooks
Hanson, and Katrina L Kelner. Science 4 July 2008: 15. [DOI:10.1126/science.1162115]
APPENDIX 1
(FROM GUIDELINES FOR REFEREES AT SCIENCE
WEBSITE)
PEER REVIEW AT SCIENCE PUBLICATIONS
— Peer Review at Science
— Peer Review at Science Signaling
As a peer reviewer for Science magazine, you
are part of a valued community. Scientific progress depends on
the communication of information that can be trusted, and the
peer review process is a vital part of that system.
Only some of the submitted papers are reviewed in
depth. For in-depth review, at least two outside referees are
consulted. Reviewers are contacted before being sent a paper and
are asked to return comments within one to two weeks for most
papers. Reviewers may be selected to evaluate separate components
of a manuscript. We greatly appreciate the time spent in preparing
a review, and will consult you on a revision of a manuscript only
if we believe the paper has been significantly improved but still
requires input. The final responsibility for decisions of acceptance
or rejection of a submitted manuscript lies with the editor.
ETHICAL GUIDELINES
FOR REVIEWERS
1. Reviews should be objective evaluations of
the research. If you cannot judge a paper impartially, you should
not accept it for review or you should notify the editor as soon
as you appreciate the situation. If you have any professional
or financial affiliations that may be perceived as a conflict
of interest in reviewing the manuscript, or a history of personal
differences with the author(s), you should describe them in your
confidential comments.
2. If, as a reviewer, you believe that you are
not qualified to evaluate a component of the research, you should
inform the editor in your review.
3. Reviews should be constructive and courteous
and the reviewer should respect the intellectual independence
of the author. The reviewer should avoid personal comments; Science
reserves the right to edit out comments that will hinder constructive
discussion of manuscripts.
4. Just as you wish prompt evaluations of your
own research, please return your reviews within the time period
specified when you were asked to review the paper. If events will
prevent a timely review, it is your responsibility to inform the
editor at the time of the request.
5. The review process is conducted anonymously;
Science
never reveals the identity of reviewers to authors. The privacy
and anonymity provisions of this process extend to the reviewer,
who should not reveal his or her identity to outsiders or members
of the press. The review itself will be shared only with the author,
and possibly with other reviewers and our Board.
6. The submitted manuscript is a privileged communication
and must be treated as a confidential document. Please destroy
all copies of the manuscript after review. Please do not share
the manuscript with any colleagues without the explicit permission
of the editor. Reviewers should not make personal or professional
use of the data or interpretations before publication without
the authors' specific permission (unless you are writing an editorial
or commentary to accompany the article).
7. You should be aware of Science's policies
for authors regarding conflict of interest, data availability,
and materials sharing. See www.sciencemag.org/about/authors/prep/gen_info.dtl.
| 
