Peer review in scientific publications - Science and Technology Committee Contents


Written evidence submitted by The Royal Society (PR 69)

INTRODUCTION

1.  The Royal Society is a Fellowship of the world's most eminent scientists and is the oldest scientific academy in continuous existence. We aim to expand the frontiers of knowledge by championing the development and use of science, mathematics, engineering and medicine for the benefit of humanity and the world. The Society has three main roles: it is the UK academy of science promoting the natural and applied sciences, a learned society, and a funding agency.

2.  The Royal Society has used peer review to make all its publishing decisions since its foundation in 1660 when the Society's Secretary, Henry Oldenburg, first introduced the concept. As a learned society, the Society publishes eight peer reviewed journals, including Philosophical Transactions of the Royal Society, the world's oldest scientific journal.

3.  It is important to understand that the scientific literature is a very large body of analysis and experimental evidence published in the form of articles over several hundred years. These articles are the individual reports of scientific studies on all aspects of the natural world which, together, form a structured and coherent record of the current state of scientific understanding. They are not based on mere opinion or assertion, but are the result of careful observations and experiments which, over time, have been rigorously tested against each other for consistency in order to develop increasingly robust and reliable theoretical frameworks.

4.  We understand that this inquiry is about the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication. Peer review is the best mechanism currently available for this purpose and has stood the test of time. Its use in the evaluation and validation of scientific research prior to publication is essential.

5.  Peer review is the process of subjecting an author's scholarly work or research to the scrutiny of others who are experts in the same field. For most journals, the process generally starts with some form of initial screening of an article ("pre-assessment") to sift out obviously unsuitable material. Items declined at this early stage would include articles clearly out of the subject scope of the journal, or that are obviously non-scientific. The next stage is to select a small number of reviewers expert in the field or fields covered by the article and to send the article to them for review (usually electronically). The reviewers are asked to comment of the scientific validity of the work, the appropriateness and rigour of the experimental methods, the quality of any statistical analysis of the data, and finally to give their opinion of its originality and likely impact on the field. Their recommendations are used by the journal editor to inform the decision as to whether the article should be published or not. In general, there are three categories of decision:

—  Accept for publication ("as is", or with only minor revisions).

—  Accept pending major revisions and/or further work.

—  Reject.

Some journals choose to treat the second of these options as a "re-submission" while others consider it to be a modification of the original submission, but these are technicalities.

There are three types of peer review in use. In order of decreasing frequency of use they are: "single blind," "double blind" and "open".

By far the commonest system in use is "single blind" peer review in which the author's name and institution is known to the reviewer, but the reviewer's name is not provided to the author.

A number of journals instead choose to operate a "double blind" peer review system which is fully anonymised (ie the author(s) are unaware of the identity of the reviewer(s) and vice versa.

Recently, there have been some experiments with a third type, "open" peer review, in which the authors' and reviewers' names are revealed to each other. One major study in which articles were made publicly available on the web prior to publication for open peer review concluded "there is a marked reluctance among researchers to offer open comments." (Nature, 2006) Open peer review can be reasonably described as an experimental system at this stage and is far from common.

SPECIFIC TERMS OF REFERENCE OF THE INQUIRY

The strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

6.  Strengths of peer review

The single greatest strength of peer review is the close scrutiny of new scientific findings using evidence and expertise, to provide the reader with a mark of value, reliability and authority. In the absence of peer review some other method of validating scientific research would need to be used in order to prevent the proliferation of untested ideas, invalid conclusions, incompatible theories, pseudoscience and polemics. The progress of science crucially depends on the interpretation of new findings in the context of existing understanding and experimental evidence. Without any means of distinguishing the scientifically valid and coherent from the unfounded assertion, scientific progress would be severely (perhaps fatally) compromised.

Journals go to considerable lengths to select the most appropriate reviewers and to eliminate any potential sources of bias or conflicting interest in order to select the best material to publish. The reputation of the best journals is intimately involved with the rigour of their peer review systems and, in turn, provides a "kite mark" of authority for the research they publish. It also makes the journal more attractive for researchers to both read and to submit papers.

The method in most common use ("single blind") allows reviewers to provide honest and accurate feedback on an article without fear of repercussions. This is particularly important where the author may be "more established," more senior or have a higher reputation than the reviewer.

Some argue that "double blind" peer review is fairer in that it removes any potential bias in the mind of the reviewer resulting from the name, gender and country of origin of the author. The argument here is that a reviewer might be influenced into thinking an article is better than it is, if it comes from a high profile author in a top institution (and vice versa). However, it can be argued that complete blinding is impossible as it is often easy to identify the authors or their institution from other elements of the article (such as the reference list which is likely to contain references to the authors' own work, or to unique methodologies.)

It should be noted that the process of peer review can often provide very useful feedback to authors. Peer review does not have to produce a simple accept/reject result, and authors are frequently invited to modify their papers in the light of critical comment. This is an important way in which peer review can substantially improve the quality and value of scientific results.

7.  Weaknesses of peer review

Peer review is fundamentally a human operation which relies on individual expertise and judgement. This means that reviewers may occasionally miss methodological errors or invalid conclusions in a research paper. Fortunately, this is relatively rare and its effect tends to be counteracted by the fact that several reviewers are usually invited to review a given article. In the rare event that such errors are not picked up during the process, they are soon picked up by other experts after publication and it is then possible to issue a correction, or in severe cases, a retraction. Where correction is justified there are detailed and robust publishing processes to ensure the scientific record is amended. In the long run, the process is self-correcting; the ultimate test is experimental evidence from the natural world.

Concerns have also been expressed regarding potential reviewer bias. This may be positive or negative and can arise from existing relationships or rivalries between reviewer and author, or from the perceived reputation of the author or their institution appearing to lend (or remove) credibility from the article. Such concerns are often cited as a justification for "double blind" peer review (see above).The risk of an unfair decision is mitigated by the use of several independent reviewers for each article.

Some critics of peer review claim that it can be used maliciously (for example, to suppress the work of rivals or to damage a competitor's career) and it is this concern that has promoted experiments with "open" peer review (under the assumption that reviewers would be less likely to submit malicious reviews in an open system).

Measures to strengthen peer review

8.  Publishers, academies and the wider scientific community are constantly looking for ways to make peer review more effective and efficient. It is important to maintain rigorous standards whilst also minimising the demands on the reviewers (who, it must be remembered, are themselves busy researchers giving their time usually without payment). Traditionally peer review has been a paper-based system involving several communications between author, reviewer and editorial office and such a process carries significant financial and time costs.

In the past 10-15 years, however, the process has evolved away from paper to a fully integrated online process and all major journals will now only accept online submissions. The result has been to make peer review quicker and with a far lower administrative burden. Publishers receive articles, data, references, tables and illustrations as electronic files, usually via a dedicated web-based submission system. Manuscripts are then allocated to reviewers and reviewed online, following which the decision is communicated to the authors in the same system. Many also accept supplementary information such as large datasets, video files etc., which in the case of "paper" journals will only be available on-line. In addition, online peer review systems integrate with the major third-party scientific databases such as PubMed which allow bibliographic reference validation and assist in selecting appropriate reviewers. Recently many publishers (including the Royal Society) have introduced integrated plagiarism detection software.

Another key improvement has been the development and widespread adoption of appropriate ethical policies, regulation and best practice. Like many academic publishers, the Royal Society has a publishing ethics policy covering such issues as authorship, dual publication, plagiarism, conflict of interest and openness in data sharing. Authors, reviewers and editors are required to read and adhere to this policy which reflects the high standards we expect in peer review. A number of organisations define codes of practice and support the "policing" of the peer review process. These include COPE and the International Committee of Medical Journal Editors.

The value and use of peer reviewed science on advancing and testing scientific knowledge

9.  Science progresses by testing a hypothesis against the available evidence obtained through experiment and observation of the natural world. It is not based on the authority or opinion of individuals or institutions. In fact, the Royal Society motto "Nullius in verba" can be roughly translated as "take nobody's word for it". Scientists are trained to be sceptical and during peer review they assess the methods, results and conclusions of a piece of research against existing evidence. Peer review is particularly good at identifying where a claim has no evidence, or if the evidence presented has been arrived at by flawed methods or inappropriate data handling. The greater the apparent claim or discovery, the greater the amount of scrutiny the scientific community tends to give it. In general, an extraordinary claim requires extraordinary evidence.

10.  Without peer review, there would be a need for some other form of validation of scientific findings as it is crucially important for scientists to be able to use each others' work with confidence when interpreting the meaning of their last experiment or designing their next one. The age of the web has meant that anyone can now "publish" whatever they like in the form of blogs. The vast proliferation of information (of hugely variable quality) now accessible would be likely to prove to be more of an obstacle than a benefit to scientists if there were there no system of validation helping them to sift the worthwhile from the worthless.

The value and use of peer reviewed science in informing public debate

11.  We believe that the use of peer review is valuable in informing the public about science as it acts as a "kite mark" that a piece of research has been properly scrutinised and validated by scientists. Barely a day goes by without a science story being reported in the print and broadcast media and there is a vast amount of scientific material (with extremely variable provenance and validity) freely available to the public via the internet. This often results in apparently conflicting messages about an issue (such as a nutritional or medical question) making it very difficult indeed for those with little or no scientific knowledge to distinguish fact from fiction. It is important for scientists, policy makers, educators and journalists to highlight the value of peer-review in providing evidence-based information.

12.  It is also important to put a single piece of published scientific information into the wider context as it may agree or disagree with previously published findings. It is unrealistic to assume that every question will be answered in one go and the public is often unaware of this level of detail, or indeed that doubt is an inherent part of science.

The extent to which peer review varies between scientific disciplines and between countries across the world

13.  Science is a truly international enterprise. The process of peer review is about the scrutiny of factual measurements, observations and data. Just as the laws of nature are the same throughout the world, so is the process of peer review. It is therefore independent of any cultural or legal differences between nations. Indeed this is vital if the universality of scientific knowledge and understanding is to be sustained.

The application of peer review is broadly similar in all scientific disciplines, but there are subtle differences. In some areas of the physical sciences such as particle physics articles are usually deposited on the arXiv pre-print database prior to formal peer-review. This allows the scientists to publish research quickly and get informal feedback and identify any weaknesses. This is then followed by formal peer review in a journal. In the biological sciences there is an imperative to publish research quickly in a peer reviewed journal. In cross-disciplinary research care must be taken to ensure peer-review balances the consideration of all disciplines involved eg both the physical science and biological science. This often requires the use of many more reviewers for each article which can increase both time and cost. In the field of mathematics, the peer review process is not about looking at experimental evidence and data, but rather the strict and logical development of a mathematical argument.

The processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases

14.  Individuals are selected as reviewers based on a proven track record of scientific achievement in the relevant discipline and a thorough knowledge of the existing literature, and are drawn from across the world. This is vital to effectively validate new work against the existing body of knowledge. Selection is supported by electronic databases both within the Society and from major-third party vendors. Such databases provide great detail on potential reviewers such as their area of expertise, publishing and reviewing record.

Reviewing multi-disciplinary research brings specific challenges. One of our journals is dedicated to science at the interface of the physical and life sciences. Such articles are reviewed by both physical and life scientists and require more referees than for single discipline research. In addition, reviewers are required to state their level of confidence in assessing the physical and biological aspects of an article.

The impact of IT and greater use of online resources on the peer review process

15.  IT and other online resources play a vital role in making peer review as rigorous and as efficient as possible. When peer review relied on the postal system it typically took months. With online systems, peer review takes only weeks or sometimes days. The remaining speed limiting factor is the time required to carefully read a paper and produce a considered, evidence based review.

The transition away from paper-based publication has also facilitated bibliometrics, and an increase in its use as a proxy for assessment of research quality and output. This can potentially influence publication behaviour, and further increases pressure on good reviewers.

Possible alternatives to peer review

16.  The scientific publishing community has been proactive in exploring adjuncts and alternatives to peer review. Further review and comment after a paper is published ("post publication review") is already important in testing and checking the quality of science. As described earlier, new concepts and results in science are repeatedly tested to validate their reliability and usefulness.

The online environment has provided the opportunity for numerous experiments such as commenting, voting and citation analysis. However, such measures can be crude and are often unproven. We believe they are important in supporting formal peer-review, but are not a substitute.

The posting of non peer reviewed content on the web would have a number of consequences. It would be more difficult to distinguish evidence-based findings from mere opinion or the downright false. Public opinion about science might be determined by opinion and assertion based on what is the most popular blog or television programme at the time. Formulation of policy requires a solid, non-transient evidence base. Scientists have a central role in the better communication of science by highlighting both the strengths of peer review and its limitations.

Peer review is neither perfect nor infallible, but we believe that dispensing with it is not an option.

DECLARATION OF INTERESTS

17.  The Royal Society is a publisher of journals, under the imprint Royal Society Publishing.

WORKS CITED

Nature. (2006). Nature's peer review trial. Nature , doi:10.1038/nature05535

The Royal Society

10 March 2011


 
previous page contents next page


© Parliamentary copyright 2011
Prepared 28 July 2011