Peer review


Written evidence submitted by the
Public Library of Science (PLoS) (PR 54)

Peer Review-Optimizing Practices for Online Scholarly Communication

About the Public Library of Science (PLoS)

1. PLoS is a nonprofit open-access publishing, membership, and advocacy organization with offices in San Francisco, California, USA, and Cambridge, UK. The mission of the organization is to lead a transformation in scientific and medical research communication, so that the mechanisms for communication are fully adapted to the online medium and so that the literature is a public resource that is open to read and reuse [1] .

2. PLoS publishes seven leading peer-reviewed journals [2] and is developing further innovative online publication venues such as PLoS Currents and PLoS Hubs that, respectively, speed up publication and aggregate content. Most of the work that PLoS publishes is in the health and life sciences, and the evidence that we submit is therefore of most relevance to these fields.

3. PLoS is one of several large publishers (publishing several thousands of research articles each year) that have demonstrated that high quality peer-reviewed open-access journals can be supported sustainably by a publication fee business model. When publishing costs are recovered up front by publication fees (as opposed to downstream recovery, for example, via subscription fees), all restrictions on access and reuse can therefore be removed.

4. PLoS is an innovator. In our view, peer review is one of several aspects of the scholarly communication process that can and should be optimized for online communications systems. We are already exploring a number of ways in which peer review can be reformed so that scholarly communication becomes more efficient and effective. The ultimate goal is to enhance and accelerate the research process itself, while maintaining the quality of the published research literature.

Peer Review and Formal Communication

5. Research articles published in peer-reviewed journals play a central role in communication of research results and ideas. The formal peer-reviewed literature underpins future research, the practical application of research findings, and the development of science and health policy. It is therefore essential that steps are put in place to assess and enhance the reliability of research literature.

6. Research articles are also the primary currency for assessing the contributions of individual researchers and their sponsors or institutions. Journals are currently used to organize such articles, such that publication of an article in a specific journal is an indication that the editors of the journal judge that the work is relevant to a particular audience, and has a certain level of significance for that audience. Thus, in all fields there tends to be a rough hierarchy of journals [1] .

7. Given its critical role, both as a means to communicate new findings and to organize them in terms of relevance and impact, publication in journals is a formal process that is generally regarded to provide the following key functions: registration (date-stamping of the work, so that it can be cited and so that the authors can achieve the appropriate recognition for their achievements); certification (quality control via the editorial and peer-review process); dissemination (ensuring that the work can be read and used); and preservation (for future generations). Peer review therefore sits firmly within the ‘certification’ part of the publication process, but the goals of certification, and the practices by which it is achieved, can and do vary across journals and fields.

8. The differences in terms of the practice of peer review and the variable editorial goals of academic journals mean that there is no accepted definition of peer review. In general, peer review refers to the editorial process that takes place after submission of an article and before it is published (excluding production processes). Operationally, peer review most often involves the assessment of a submission to a journal by a variable number of (frequently two or three) relevant research experts whose identities are not usually revealed to the authors. The opinions of these experts are then used to inform the editorial decision regarding the submission, which ultimately determines whether or not the work will be published in the journal.

Peer Review as One Part of Certification

9. Broadly speaking, there are two types of questions that journals attempt to address during the certification process:

9.1. whether the work is a rigorous contribution to the corpus of scientific knowledge (referred to below as technical assessment);

9.2. and whether the work represents the kind of advance (in terms of relevance and importance) that is appropriate for a given journal (referred to below as impact assessment).

10. The specific issues that are encompassed in the two questions that peer review is attempting to address are complex and highly variable. The challenge for any journal is to develop processes that balance the twin needs for thoroughly assessing submissions to the journal and for avoiding unnecessary delays in communicating new research findings.

11. Although the questions being addressed by peer review are usually not demarcated in the way that we have outlined, this is a helpful distinction because it serves to identify some of the strengths and weaknesses of current practices. This perspective on peer review also identifies ways in which the process might be improved.

Technical Assessment

12. There are many aspects to the technical assessment of a new submission. The first assessments are largely administrative and include an assessment of the financial disclosure information, competing interests declaration, whether the appropriate ethical approvals have been obtained and documentation is available, information about related work under consideration, whether the work adheres to appropriate reporting standards (such as the CONSORT standard for clinical trials), and so on. At PLoS, all submissions go through one or more quality control steps at various stages of the editorial process that are conducted by suitably trained and qualified administrative staff.

13. Peer review by external and internal subject experts also plays a critical role in the technical assessment of submissions. Most academic journals have as editors one or more experts in the subject who are responsible for the oversight of peer review. Once administrative checks are done, these editors can provide further assessment of the submission and its suitability for the journal. Then, relevant external research experts can be asked to assess whether the appropriate methods and materials have been used to investigate a given research question, and whether the data analysis and presentation provide adequate justification for the claims and conclusions of the work. For example, in some methodologies, such as clinical trials research, specific experts are consulted to validate the statistical analysis that might underpin the conclusions. The goal is always to find sufficient appropriate experts so that all of the key parts of the work can be assessed for scientific rigour. The number of experts required will often depend on the nature of the submission. If the work is multidisciplinary, for example, it might be necessary to seek the opinions of a larger number of experts to assess all key aspects of the work.

14. The result of technical assessment by peer review is that errors and weaknesses are frequently identified in article submissions, and revision of the work is required before it can be published. If the technical assessment reveals fatal flaws in design or methodology, then the submission will, however, be rejected.

15. Technical assessment of research articles has an important role in enhancing the reliability of the published literature, and in many ways can be considered to be a reasonably objective process. The questions addressed focus on whether the work adheres to the standards accepted within a given field. Subjective judgments clearly need to be made around certain issues, but in general, given a suitably robust process involving internal checks in combination with expert consultation, the decision as to whether a research article satisfies the technical requirements for publication is often clear-cut.

16. An important point to make is that there are always more technical checks that can be done. The reliability or quality of a research article can never be ‘assured’, and there are many examples of peer-reviewed work that has had to be formally corrected or retracted as a result of straightforward error or, on occasion, deliberate misconduct.

Impact Assessment

17. In contrast to technical assessment, judgments about the importance and relevance of a research article tend to be more subjective and are more susceptible to bias and competing interests. Here, the peer-review process is attempting to judge whether the work meets the criteria for impact and relevance set by a particular journal.

18. Editors consider, and reviewers may be asked, for example, to comment on the ‘strength of the advance’ represented by a given piece of research. As discussed above, current systems for research assessment place heavy emphasis on the journal in which a research article has been published. The outcome of the impact assessment aspect of peer review can therefore have profound consequences for the author. Publication in a high-impact journal, for example, can greatly improve the prospects for obtaining grant funding, promotion, or tenure.

19. Authors will therefore frequently ‘aim high’ when submitting work to a journal, with the result that technically competent work can be rejected from several journals for subjective reasons before it is eventually published. Submissions are often subject to peer review sequentially at multiple journals, and can be revised and resubmitted multiple times, which increases the labour required from authors, reviewers, and editors. The resulting delay between the acquisition of new and important research findings and their eventual communication in a formal journal often extends into years.

20. Although the use of the peer-review process for impact assessment has its weaknesses, all journals strive to ensure that the process is conducted as effectively as possible, free from bias and prejudice. Furthermore, without any form of assessment of impact for specific audiences, the literature would be a disorganized mass of information that would be difficult to navigate and use. Impact assessment is therefore currently an important function of journals, and there are many journals that fulfil this function effectively. However, given the availability of a new medium for the communication of research, it is reasonable to ask whether alternative approaches to impact assessment might be developed that do not rely just on the opinions of the limited number of reviewers and editors that see the work before it is published.

Alternative Approaches to Impact Assessment

21. Having demarcated the peer-review process into two broad sets of issues, it is possible to consider alternatives whereby technical assessment becomes the goal of the pre-publication phase and impact assessment is dealt with after the work is published. It is possible to disaggregate the processes that are currently wrapped up into a single pre-publication phase into components, some of which can usefully be conducted before publication and others which are best left until after publication.

22. At PLoS, we have been pursuing this approach using a journal that was launched in 2006, called PLoS ONE [1] . The editorial criteria for PLoS ONE are that the work must be rigorously performed with appropriate methodology; properly and intelligibly reported; and ethically conducted. In this way, the peer-review process in PLoS ONE is focusing only on technical assessment. Remarkably, in only four years, PLoS ONE has become the largest peer-reviewed scholarly journal, and last year published over 6,700 research articles. It is also very striking that, in light of this success, several major publishers have recently launched journals in various fields that are modelled very closely on PLoS ONE [2] .

23. Unlike the vast majority of journals, the editors and peer reviewers for PLoS ONE make no judgments about the relevance or impact of the work that is submitted. Instead, PLoS is working to provide alternative post-publication tools on individual articles with which these more subjective aspects of research can be examined, based on the actions and opinions of a broader constituency.

24. By focusing attention on the article, as opposed to the journal, it becomes possible to assess impact much more rigorously. For the past two years, we have therefore initiated a program of ‘article-level metrics’ whereby every published article in all PLoS journals is enhanced with metrics about Web usage, citations, social bookmarks, user rating and commentary, and blogosphere coverage [3] . The vision is that a ‘dashboard’ will be available for all research articles that will help users and readers to assess the impact of the work, and also be useful for filtering content and identifying the work that is of most relevance to a particular line of investigation. Given a variety of impact indicators, it will thus be possible to establish more sophisticated and meaningful measures of significance and influence than the journal metrics that dominate research assessment today [4] .

25. As a complement to article-level metrics, many publishers are experimenting with post-publication peer review by providing tools for user-based assessment. At the simplest level, it is possible for users to provide comments and ratings directly on articles. So far, however, the usage of commentary tools is fairly modest, and does not make a major contribution to the assessment of research content. That said, increasing amounts of commentary are taking place away from the journal sites themselves in blogs, tweets, and elsewhere, and one current opportunity is therefore to capture the richness of this commentary on the articles themselves.

26. Another interesting and relevant service is provided by Faculty of 1000, which collects comments of selected experts on research articles published in any journal [5] . The Faculty is invited to post short notes and ratings on articles that they find of interest. Although still limited in its effectiveness, Faculty of 1000 is an example of another way in which impact assessment can be added after formal publication.

27. In sum, there are a variety of approaches that are being used to explore how value can be added to content after publication, to help with the continuous assessment of published research. In addition to activities such as Web usage, citation, bookmarking, and so on, users are constructing their own online bibliographies of content and adding subject tags to content at Web sites such as CiteULike [6] and Mendeley [7] . There is thus a wealth of activity that can be aggregated around any given article, and with effective interpretation of such activity, the impact assessment aspect of ‘peer review’ is expanded to a much broader constituency. These new approaches therefore have the potential to make profound improvements to the organization and assessment of research content, and ultimately to facilitate more rapid communication of new findings

Peer Review of Data

28. We are in a research era where large quantities of data are being generated in a wide variety of disciplines. New research articles frequently report the availability and analysis of new and valuable datasets. In online communication it is frequently possible to append (as supplementary files) large bodies of data that are relevant to a particular piece of work.

29. In some fields-for example, genetics and molecular biology-there are well-established curated databases where data can be deposited and linked to particular research articles. Examples of such databases include those available at the European Bioinformatics Institute in Hinxton, UK [1] . The curators who run the databases perform critical quality control checks analogous to the technical assessment of research articles. Newer resources are also being developed, such as Dryad UK [2] , which provides a more flexible site for data deposition in fields where data sharing is less advanced than in other fields.

30. Peer review of data is another important area where online tools could be used, both for technical and for impact assessment as well. Again, it could be fruitful to consider the separation of technical from impact assessment, with the latter occurring after the data have been made available so that the activity and views of entire communities can be leveraged to enhance the data that are being shared.


31. Scientific communication is undergoing a revolution. Online tools allow universal access to all research findings, and new business models can support publishing in an open-access mode in which access and reuse barriers are both removed. New approaches are also emerging for organization and assessment of research articles after publication.

32. Peer review occupies a central locus within the process of formal scholarly communication, and it is helpful to divide its functions into two broad areas: technical and impact assessment. Whereas technical assessment tends to be objective and provides greater confidence in (although cannot assure) the reliability of published findings, impact assessment is subjective and its role is less clear-cut.

33. Impact assessment, as currently performed by the majority of journals during the pre-publication peer-review process, is the means by which research articles are currently organized in journals. Such organization is essential for the navigation and filtering of content by users, but the current process is not particularly reliable and often results in substantial delays in the communication of new findings. However, a new paradigm is emerging and is being tested in several fields whereby articles are subject only to technical assessment (by peer review) before publication, and impact assessment takes place during the post-publication phase, which can broaden the assessment of the work (by peers) to a much wider constituency than can take place before publication.

34. A substantial opportunity for enhancement of research communication exists in the area of research assessment. Rather than relying on the journal in which an article is published, it is now possible to focus on the merits of the article itself. An array of article-level metrics and indicators can be deployed to filter and assess content. Coupled with tools for post-publication commentary and addition of value, there are tremendous prospects for replacing the current impact assessment function of pre-publication peer review with a post-publication system that has the potential to be more efficient and effective.

35. If greater attention is concentrated on the article rather than the journal, the consequences for research communication are that more results can be communicated more effectively and more rapidly, leading to an acceleration of the research process itself.

Declaration of Interests

PLoS is a tax-exempt, 501(c)3, nonprofit corporation with headquarters in San Francisco, California, USA. PLoS's overall revenues and funding information for 2009 are listed in the 2009 Progress report PLoS is an open-access scientific and medical publishing organization. We are exploring many ways in which scholarly communication can be reinvented and fully adapted online by altering business models, editorial (including peer review) processes, and publishing workflows. Broadly, our mission is to lead a transformation in scholarly communication whereby research is open to all to read and use.

Public Library of Science

10 March 2011

[1] Open access is defined as the removal of all barriers to access and reuse of the literature. The legal tool that is frequently used to indicate that a particular work is open access is the Creative Commons Attribution License ( )

[2] PLoS Biology , PLoS Medicine , PLoS Computational Biology , PLoS Genetics , PLoS Pathogens , PLoS Neglected Tropical Diseases , and PLoS ONE .

[1] Supporting this hierarchy of journals is a proprietary metric—the journal impact factor—produced each year by Thomson Reuters. The impact factor is used in various aspects of research assessment. However, PLoS and many others have pointed out the weaknesses of using the impact factor in research assessment and the detrimental consequences to research itself. (See also .)


[1] /

[2] BMJ Open ( )from the BMJ Publishing Group, SAGE Open ( /) from SAGE; AIP Advances ( ) from the American Institute of Physics; Physical Review X ( ) from the American Physical Society; Scientific Reports ( ) from the Nature Publishing Group; and G3 ( ) from the Genetics Society of America.

[3] . See also the PIRUS2 project, which aims to provide standards around the reporting of article Web usage data: .