Peer review

Written evidence submitted by the
Public Library of Science (PLoS) (PR 54)

Peer Review-Optimizing Practices for Online Scholarly Communication

About the Public Library of Science (PLoS)

1. PLoS is a nonprofit open-access publishing, membership, and advocacy organization with offices in San Francisco, California, USA, and Cambridge, UK. The mission of the organization is to lead a transformation in scientific and medical research communication, so that the mechanisms for communication are fully adapted to the online medium and so that the literature is a public resource that is open to read and reuse [1] .

2. PLoS publishes seven leading peer-reviewed journals [2] and is developing further innovative online publication venues such as PLoS Currents and PLoS Hubs that, respectively, speed up publication and aggregate content. Most of the work that PLoS publishes is in the health and life sciences, and the evidence that we submit is therefore of most relevance to these fields.

3. PLoS is one of several large publishers (publishing several thousands of research articles each year) that have demonstrated that high quality peer-reviewed open-access journals can be supported sustainably by a publication fee business model. When publishing costs are recovered up front by publication fees (as opposed to downstream recovery, for example, via subscription fees), all restrictions on access and reuse can therefore be removed.

4. PLoS is an innovator. In our view, peer review is one of several aspects of the scholarly communication process that can and should be optimized for online communications systems. We are already exploring a number of ways in which peer review can be reformed so that scholarly communication becomes more efficient and effective. The ultimate goal is to enhance and accelerate the research process itself, while maintaining the quality of the published research literature.

Peer Review and Formal Communication

5. Research articles published in peer-reviewed journals play a central role in communication of research results and ideas. The formal peer-reviewed literature underpins future research, the practical application of research findings, and the development of science and health policy. It is therefore essential that steps are put in place to assess and enhance the reliability of research literature.

6. Research articles are also the primary currency for assessing the contributions of individual researchers and their sponsors or institutions. Journals are currently used to organize such articles, such that publication of an article in a specific journal is an indication that the editors of the journal judge that the work is relevant to a particular audience, and has a certain level of significance for that audience. Thus, in all fields there tends to be a rough hierarchy of journals [1] .

7. Given its critical role, both as a means to communicate new findings and to organize them in terms of relevance and impact, publication in journals is a formal process that is generally regarded to provide the following key functions: registration (date-stamping of the work, so that it can be cited and so that the authors can achieve the appropriate recognition for their achievements); certification (quality control via the editorial and peer-review process); dissemination (ensuring that the work can be read and used); and preservation (for future generations). Peer review therefore sits firmly within the ‘certification’ part of the publication process, but the goals of certification, and the practices by which it is achieved, can and do vary across journals and fields.

8. The differences in terms of the practice of peer review and the variable editorial goals of academic journals mean that there is no accepted definition of peer review. In general, peer review refers to the editorial process that takes place after submission of an article and before it is published (excluding production processes). Operationally, peer review most often involves the assessment of a submission to a journal by a variable number of (frequently two or three) relevant research experts whose identities are not usually revealed to the authors. The opinions of these experts are then used to inform the editorial decision regarding the submission, which ultimately determines whether or not the work will be published in the journal.

Peer Review as One Part of Certification

9. Broadly speaking, there are two types of questions that journals attempt to address during the certification process:

9.1. whether the work is a rigorous contribution to the corpus of scientific knowledge (referred to below as technical assessment);

9.2. and whether the work represents the kind of advance (in terms of relevance and importance) that is appropriate for a given journal (referred to below as impact assessment).

10. The specific issues that are encompassed in the two questions that peer review is attempting to address are complex and highly variable. The challenge for any journal is to develop processes that balance the twin needs for thoroughly assessing submissions to the journal and for avoiding unnecessary delays in communicating new research findings.

11. Although the questions being addressed by peer review are usually not demarcated in the way that we have outlined, this is a helpful distinction because it serves to identify some of the strengths and weaknesses of current practices. This perspective on peer review also identifies ways in which the process might be improved.

Technical Assessment

12. There are many aspects to the technical assessment of a new submission. The first assessments are largely administrative and include an assessment of the financial disclosure information, competing interests declaration, whether the appropriate ethical approvals have been obtained and documentation is available, information about related work under consideration, whether the work adheres to appropriate reporting standards (such as the CONSORT standard for clinical trials), and so on. At PLoS, all submissions go through one or more quality control steps at various stages of the editorial process that are conducted by suitably trained and qualified administrative staff.

13. Peer review by external and internal subject experts also plays a critical role in the technical assessment of submissions. Most academic journals have as editors one or more experts in the subject who are responsible for the oversight of peer review. Once administrative checks are done, these editors can provide further assessment of the submission and its suitability for the journal. Then, relevant external research experts can be asked to assess whether the appropriate methods and materials have been used to investigate a given research question, and whether the data analysis and presentation provide adequate justification for the claims and conclusions of the work. For example, in some methodologies, such as clinical trials research, specific experts are consulted to validate the statistical analysis that might underpin the conclusions. The goal is always to find sufficient appropriate experts so that all of the key parts of the work can be assessed for scientific rigour. The number of experts required will often depend on the nature of the submission. If the work is multidisciplinary, for example, it might be necessary to seek the opinions of a larger number of experts to assess all key aspects of the work.

14. The result of technical assessment by peer review is that errors and weaknesses are frequently identified in article submissions, and revision of the work is required before it can be published. If the technical assessment reveals fatal flaws in design or methodology, then the submission will, however, be rejected.

15. Technical assessment of research articles has an important role in enhancing the reliability of the published literature, and in many ways can be considered to be a reasonably objective process. The questions addressed focus on whether the work adheres to the standards accepted within a given field. Subjective judgments clearly need to be made around certain issues, but in general, given a suitably robust process involving internal checks in combination with expert consultation, the decision as to whether a research article satisfies the technical requirements for publication is often clear-cut.

16. An important point to make is that there are always more technical checks that can be done. The reliability or quality of a research article can never be ‘assured’, and there are many examples of peer-reviewed work that has had to be formally corrected or retracted as a result of straightforward error or, on occasion, deliberate misconduct.

Impact Assessment

17. In contrast to technical assessment, judgments about the importance and relevance of a research article tend to be more subjective and are more susceptible to bias and competing interests. Here, the peer-review process is attempting to judge whether the work meets the criteria for impact and relevance set by a particular journal.

18. Editors consider, and reviewers may be asked, for example, to comment on the ‘strength of the advance’ represented by a given piece of research. As discussed above, current systems for research assessment place heavy emphasis on the journal in which a research article has been published. The outcome of the impact assessment aspect of peer review can therefore have profound consequences for the author. Publication in a high-impact journal, for example, can greatly improve the prospects for obtaining grant funding, promotion, or tenure.

19. Authors will therefore frequently ‘aim high’ when submitting work to a journal, with the result that technically competent work can be rejected from several journals for subjective reasons before it is eventually published. Submissions are often subject to peer review sequentially at multiple journals, and can be revised and resubmitted multiple times, which increases the labour required from authors, reviewers, and editors. The resulting delay between the acquisition of new and important research findings and their eventual communication in a formal journal often extends into years.

20. Although the use of the peer-review process for impact assessment has its weaknesses, all journals strive to ensure that the process is conducted as effectively as possible, free from bias and prejudice. Furthermore, without any form of assessment of impact for specific audiences, the literature would be a disorganized mass of information that would be difficult to navigate and use. Impact assessment is therefore currently an important function of journals, and there are many journals that fulfil this function effectively. However, given the availability of a new medium for the communication of research, it is reasonable to ask whether alternative approaches to impact assessment might be developed that do not rely just on the opinions of the limited number of reviewers and editors that see the work before it is published.

Alternative Approaches to Impact Assessment

21. Having demarcated the peer-review process into two broad sets of issues, it is possible to consider alternatives whereby technical assessment becomes the goal of the pre-publication phase and impact assessment is dealt with after the work is published. It is possible to disaggregate the processes that are currently wrapped up into a single pre-publication phase into components, some of which can usefully be conducted before publication and others which are best left until after publication.

22. At PLoS, we have been pursuing this approach using a journal that was launched in 2006, called PLoS ONE [1] . The editorial criteria for PLoS ONE are that the work must be rigorously performed with appropriate methodology; properly and intelligibly reported; and ethically conducted. In this way, the peer-review process in PLoS ONE is focusing only on technical assessment. Remarkably, in only four years, PLoS ONE has become the largest peer-reviewed scholarly journal, and last year published over 6,700 research articles. It is also very striking that, in light of this success, several major publishers have recently launched journals in various fields that are modelled very closely on PLoS ONE [2] .