Written evidence submitted by the Public
Library of Science (PLoS) (PR 54)
PEER REVIEW—OPTIMIZING PRACTICES FOR
ONLINE SCHOLARLY COMMUNICATION
ABOUT THE
PUBLIC LIBRARY
OF SCIENCE
(PLOS)
1. PLoS is a nonprofit open-access publishing,
membership, and advocacy organization with offices in San Francisco,
California, USA, and Cambridge, UK. The mission of the organization
is to lead a transformation in scientific and medical research
communication, so that the mechanisms for communication are fully
adapted to the online medium and so that the literature is a public
resource that is open to read and reuse.[2]
2. PLoS publishes seven leading peer-reviewed
journals[3]
and is developing further innovative online publication venues
such as PLoS Currents and PLoS Hubs that, respectively, speed
up publication and aggregate content. Most of the work that PLoS
publishes is in the health and life sciences, and the evidence
that we submit is therefore of most relevance to these fields.
3. PLoS is one of several large publishers (publishing
several thousands of research articles each year) that have demonstrated
that high quality peer-reviewed open-access journals can be supported
sustainably by a publication fee business model. When publishing
costs are recovered up front by publication fees (as opposed to
downstream recovery, for example, via subscription fees), all
restrictions on access and reuse can therefore be removed.
4. PLoS is an innovator. In our view, peer review
is one of several aspects of the scholarly communication process
that can and should be optimized for online communications systems.
We are already exploring a number of ways in which peer review
can be reformed so that scholarly communication becomes more efficient
and effective. The ultimate goal is to enhance and accelerate
the research process itself, while maintaining the quality of
the published research literature.
PEER REVIEW
AND FORMAL
COMMUNICATION
5. Research articles published in peer-reviewed
journals play a central role in communication of research results
and ideas. The formal peer-reviewed literature underpins future
research, the practical application of research findings, and
the development of science and health policy. It is therefore
essential that steps are put in place to assess and enhance the
reliability of research literature.
6. Research articles are also the primary currency
for assessing the contributions of individual researchers and
their sponsors or institutions. Journals are currently used to
organize such articles, such that publication of an article in
a specific journal is an indication that the editors of the journal
judge that the work is relevant to a particular audience, and
has a certain level of significance for that audience. Thus,
in all fields there tends to be a rough hierarchy of journals.[4]
7. Given its critical role, both as a means to
communicate new findings and to organize them in terms of relevance
and impact, publication in journals is a formal process that is
generally regarded to provide the following key functions: registration
(date-stamping of the work, so that it can be cited and so that
the authors can achieve the appropriate recognition for their
achievements); certification (quality control via the editorial
and peer-review process); dissemination (ensuring that the work
can be read and used); and preservation (for future generations).
Peer review therefore sits firmly within the "certification"
part of the publication process, but the goals of certification,
and the practices by which it is achieved, can and do vary across
journals and fields.
8. The differences in terms of the practice of
peer review and the variable editorial goals of academic journals
mean that there is no accepted definition of peer review. In
general, peer review refers to the editorial process that takes
place after submission of an article and before it is published
(excluding production processes). Operationally, peer review
most often involves the assessment of a submission to a journal
by a variable number of (frequently two or three) relevant research
experts whose identities are not usually revealed to the authors.
The opinions of these experts are then used to inform the editorial
decision regarding the submission, which ultimately determines
whether or not the work will be published in the journal.
PEER REVIEW
AS ONE
PART OF
CERTIFICATION
9. Broadly speaking, there are two types of questions
that journals attempt to address during the certification process:
9.1 whether the work is a rigorous contribution
to the corpus of scientific knowledge (referred to below as technical
assessment); and
9.2 whether the work represents the kind of advance
(in terms of relevance and importance) that is appropriate for
a given journal (referred to below as impact assessment).
10. The specific issues that are encompassed
in the two questions that peer review is attempting to address
are complex and highly variable. The challenge for any journal
is to develop processes that balance the twin needs for thoroughly
assessing submissions to the journal and for avoiding unnecessary
delays in communicating new research findings.
11. Although the questions being addressed by
peer review are usually not demarcated in the way that we have
outlined, this is a helpful distinction because it serves to identify
some of the strengths and weaknesses of current practices. This
perspective on peer review also identifies ways in which the process
might be improved.
TECHNICAL ASSESSMENT
12. There are many aspects to the technical assessment
of a new submission. The first assessments are largely administrative
and include an assessment of the financial disclosure information,
competing interests declaration, whether the appropriate ethical
approvals have been obtained and documentation is available, information
about related work under consideration, whether the work adheres
to appropriate reporting standards (such as the CONSORT standard
for clinical trials), and so on. At PLoS, all submissions go through
one or more quality control steps at various stages of the editorial
process that are conducted by suitably trained and qualified administrative
staff.
13. Peer review by external and internal subject
experts also plays a critical role in the technical assessment
of submissions. Most academic journals have as editors one or
more experts in the subject who are responsible for the oversight
of peer review. Once administrative checks are done, these editors
can provide further assessment of the submission and its suitability
for the journal. Then, relevant external research experts can
be asked to assess whether the appropriate methods and materials
have been used to investigate a given research question, and whether
the data analysis and presentation provide adequate justification
for the claims and conclusions of the work. For example, in some
methodologies, such as clinical trials research, specific experts
are consulted to validate the statistical analysis that might
underpin the conclusions. The goal is always to find sufficient
appropriate experts so that all of the key parts of the work can
be assessed for scientific rigour. The number of experts required
will often depend on the nature of the submission. If the work
is multidisciplinary, for example, it might be necessary to seek
the opinions of a larger number of experts to assess all key aspects
of the work.
14. The result of technical assessment by peer
review is that errors and weaknesses are frequently identified
in article submissions, and revision of the work is required before
it can be published. If the technical assessment reveals fatal
flaws in design or methodology, then the submission will, however,
be rejected.
15. Technical assessment of research articles
has an important role in enhancing the reliability of the published
literature, and in many ways can be considered to be a reasonably
objective process. The questions addressed focus on whether the
work adheres to the standards accepted within a given field.
Subjective judgments clearly need to be made around certain issues,
but in general, given a suitably robust process involving internal
checks in combination with expert consultation, the decision as
to whether a research article satisfies the technical requirements
for publication is often clear-cut.
16. An important point to make is that there
are always more technical checks that can be done. The reliability
or quality of a research article can never be "assured",
and there are many examples of peer-reviewed work that has had
to be formally corrected or retracted as a result of straightforward
error or, on occasion, deliberate misconduct.
IMPACT ASSESSMENT
17. In contrast to technical assessment, judgments
about the importance and relevance of a research article tend
to be more subjective and are more susceptible to bias and competing
interests. Here, the peer-review process is attempting to judge
whether the work meets the criteria for impact and relevance set
by a particular journal.
18. Editors consider, and reviewers may be asked,
for example, to comment on the "strength of the advance"
represented by a given piece of research. As discussed above,
current systems for research assessment place heavy emphasis on
the journal in which a research article has been published. The
outcome of the impact assessment aspect of peer review can therefore
have profound consequences for the author. Publication in a high-impact
journal, for example, can greatly improve the prospects for obtaining
grant funding, promotion, or tenure.
19. Authors will therefore frequently "aim
high" when submitting work to a journal, with the result
that technically competent work can be rejected from several journals
for subjective reasons before it is eventually published. Submissions
are often subject to peer review sequentially at multiple journals,
and can be revised and resubmitted multiple times, which increases
the labour required from authors, reviewers, and editors. The
resulting delay between the acquisition of new and important research
findings and their eventual communication in a formal journal
often extends into years.
20. Although the use of the peer-review process
for impact assessment has its weaknesses, all journals strive
to ensure that the process is conducted as effectively as possible,
free from bias and prejudice. Furthermore, without any form of
assessment of impact for specific audiences, the literature would
be a disorganized mass of information that would be difficult
to navigate and use. Impact assessment is therefore currently
an important function of journals, and there are many journals
that fulfil this function effectively. However, given the availability
of a new medium for the communication of research, it is reasonable
to ask whether alternative approaches to impact assessment might
be developed that do not rely just on the opinions of the limited
number of reviewers and editors that see the work before it is
published.
ALTERNATIVE APPROACHES
TO IMPACT
ASSESSMENT
21. Having demarcated the peer-review process
into two broad sets of issues, it is possible to consider alternatives
whereby technical assessment becomes the goal of the pre-publication
phase and impact assessment is dealt with after the work is published.
It is possible to disaggregate the processes that are currently
wrapped up into a single pre-publication phase into components,
some of which can usefully be conducted before publication and
others which are best left until after publication.
22. At PLoS, we have been pursuing this approach
using a journal that was launched in 2006, called PLoS ONE.[5]
The editorial criteria for PLoS ONE are that the work
must be rigorously performed with appropriate methodology; properly
and intelligibly reported; and ethically conducted. In this way,
the peer-review process in PLoS ONE is focusing only on
technical assessment. Remarkably, in only four years, PLoS
ONE has become the largest peer-reviewed scholarly journal,
and last year published over 6,700 research articles. It is also
very striking that, in light of this success, several major publishers
have recently launched journals in various fields that are modelled
very closely on PLoS ONE.[6]
23. Unlike the vast majority of journals, the
editors and peer reviewers for PLoS ONE make no judgments
about the relevance or impact of the work that is submitted.
Instead, PLoS is working to provide alternative post-publication
tools on individual articles with which these more subjective
aspects of research can be examined, based on the actions and
opinions of a broader constituency.
24. By focusing attention on the article, as
opposed to the journal, it becomes possible to assess impact much
more rigorously. For the past two years, we have therefore initiated
a program of "article-level metrics" whereby every published
article in all PLoS journals is enhanced with metrics about Web
usage, citations, social bookmarks, user rating and commentary,
and blogosphere coverage.[7]
The vision is that a "dashboard" will be available
for all research articles that will help users and readers to
assess the impact of the work, and also be useful for filtering
content and identifying the work that is of most relevance to
a particular line of investigation. Given a variety of impact
indicators, it will thus be possible to establish more sophisticated
and meaningful measures of significance and influence than the
journal metrics that dominate research assessment today.[8]
25. As a complement to article-level metrics,
many publishers are experimenting with post-publication peer review
by providing tools for user-based assessment. At the simplest
level, it is possible for users to provide comments and ratings
directly on articles. So far, however, the usage of commentary
tools is fairly modest, and does not make a major contribution
to the assessment of research content. That said, increasing
amounts of commentary are taking place away from the journal sites
themselves in blogs, tweets, and elsewhere, and one current opportunity
is therefore to capture the richness of this commentary on the
articles themselves.
26. Another interesting and relevant service
is provided by Faculty of 1000, which collects comments of selected
experts on research articles published in any journal.[9]
The Faculty is invited to post short notes and ratings on articles
that they find of interest. Although still limited in its effectiveness,
Faculty of 1000 is an example of another way in which impact assessment
can be added after formal publication.
27. In sum, there are a variety of approaches
that are being used to explore how value can be added to content
after publication, to help with the continuous assessment of published
research. In addition to activities such as Web usage, citation,
bookmarking, and so on, users are constructing their own online
bibliographies of content and adding subject tags to content at
Web sites such as CiteULike[10]
and Mendeley.[11]
There is thus a wealth of activity that can be aggregated around
any given article, and with effective interpretation of such activity,
the impact assessment aspect of "peer review" is expanded
to a much broader constituency. These new approaches therefore
have the potential to make profound improvements to the organization
and assessment of research content, and ultimately to facilitate
more rapid communication of new findings.
PEER REVIEW
OF DATA
28. We are in a research era where large quantities
of data are being generated in a wide variety of disciplines.
New research articles frequently report the availability and
analysis of new and valuable datasets. In online communication
it is frequently possible to append (as supplementary files) large
bodies of data that are relevant to a particular piece of work.
29. In some fields—for example, genetics
and molecular biology—there are well-established curated
databases where data can be deposited and linked to particular
research articles. Examples of such databases include those available
at the European Bioinformatics Institute in Hinxton, UK.[12]
The curators who run the databases perform critical quality control
checks analogous to the technical assessment of research articles.
Newer resources are also being developed, such as Dryad UK,[13]
which provides a more flexible site for data deposition in fields
where data sharing is less advanced than in other fields.
30. Peer review of data is another important
area where online tools could be used, both for technical and
for impact assessment as well. Again, it could be fruitful to
consider the separation of technical from impact assessment, with
the latter occurring after the data have been made available so
that the activity and views of entire communities can be leveraged
to enhance the data that are being shared.
CONCLUSIONS
31. Scientific communication is undergoing a
revolution. Online tools allow universal access to all research
findings, and new business models can support publishing in an
open-access mode in which access and reuse barriers are both removed.
New approaches are also emerging for organization and assessment
of research articles after publication.
32. Peer review occupies a central locus within
the process of formal scholarly communication, and it is helpful
to divide its functions into two broad areas: technical and impact
assessment. Whereas technical assessment tends to be objective
and provides greater confidence in (although cannot assure) the
reliability of published findings, impact assessment is subjective
and its role is less clear-cut.
33. Impact assessment, as currently performed
by the majority of journals during the pre-publication peer-review
process, is the means by which research articles are currently
organized in journals. Such organization is essential for the
navigation and filtering of content by users, but the current
process is not particularly reliable and often results in substantial
delays in the communication of new findings. However, a new paradigm
is emerging and is being tested in several fields whereby articles
are subject only to technical assessment (by peer review) before
publication, and impact assessment takes place during the post-publication
phase, which can broaden the assessment of the work (by peers)
to a much wider constituency than can take place before publication.
34. A substantial opportunity for enhancement
of research communication exists in the area of research assessment.
Rather than relying on the journal in which an article is published,
it is now possible to focus on the merits of the article itself.
An array of article-level metrics and indicators can be deployed
to filter and assess content. Coupled with tools for post-publication
commentary and addition of value, there are tremendous prospects
for replacing the current impact assessment function of pre-publication
peer review with a post-publication system that has the potential
to be more efficient and effective.
35. If greater attention is concentrated on the
article rather than the journal, the consequences for research
communication are that more results can be communicated more effectively
and more rapidly, leading to an acceleration of the research process
itself.
DECLARATION OF
INTERESTS
PLoS is a tax-exempt, 501(c)3, nonprofit corporation
with headquarters in San Francisco, California, USA. PLoS's overall
revenues and funding information for 2009 are listed in the 2009
Progress report http://www.plos.org/downloads/progress_update_lo.pdf.
PLoS is an open-access scientific and medical publishing organization.
We are exploring many ways in which scholarly communication can
be reinvented and fully adapted online by altering business models,
editorial (including peer review) processes, and publishing workflows.
Broadly, our mission is to lead a transformation in scholarly
communication whereby research is open to all to read and use.
Public Library of Science
10 March 2011
2 Open access is defined as the removal of all barriers
to access and reuse of the literature. The legal tool that is
frequently used to indicate that a particular work is open access
is the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/) Back
3
PLoS Biology, PLoS Medicine, PLoS Computational
Biology, PLoS Genetics, PLoS Pathogens, PLoS
Neglected Tropical Diseases, and PLoS ONE. Back
4
Supporting this hierarchy of journals is a proprietary metric-the
journal impact factor-produced each year by Thomson Reuters.
The impact factor is used in various aspects of research assessment.
However, PLoS and many others have pointed out the weaknesses
of using the impact factor in research assessment and the detrimental
consequences to research itself. (See also http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf.) Back
5
http://www.plosone.org/ Back
6
BMJ Open (http://bmjopen.bmj.com/)from the BMJ Publishing
Group, SAGE Open (http://www.sageopen.com/) from SAGE;
AIP Advances (http://aipadvances.aip.org/) from the American
Institute of Physics; Physical Review X (http://prx.aps.org/)
from the American Physical Society; Scientific Reports
(http://www.nature.com/srep/) from the Nature Publishing Group;
and G3 (http://www.g3journal.org/) from the Genetics Society
of America. Back
7
http://article-level-metrics.plos.org/. See also the PIRUS2 project,
which aims to provide standards around the reporting of article
Web usage data: http://www.cranfieldlibrary.cranfield.ac.uk/pirus2/tiki-index.php.
Back
8
http://altmetrics.org/manifesto/ Back
9
http://f1000.com/ Back
10
http://www.citeulike.org/ Back
11
http://www.mendeley.com/ Back
12
http://www.ebi.ac.uk/Databases/ Back
13
http://www.jisc.ac.uk/whatwedo/programmes/mrd/clip/dryaduk.aspx Back
| 
