Written evidence submitted by the Institute
of Physics (PR 61)
The Institute of Physics is a scientific charity
devoted to increasing the practice, understanding and application
of physics. It has a worldwide membership of over 40,000, and
is a leading communicator of physics-related science to all audiences
- from specialists through to government and the general public.
The Institute welcomes the opportunity to respond
to the House of Commons Science and Technology Committee's inquiry
into peer review, which is particularly relevant to our extensive
learned society publishing activities.
The Institute has published academic journals continuously
since its foundation in 1874. Today the Institute's publishing
is carried out through a wholly owned subsidiary company, Institute
of Physics Publishing Ltd (IOP Publishing). The company employs
more than 270 people in Bristol and has offices in the USA, Germany,
Russia, China and Japan.
The Institute publishes 65 journals, which are international
in terms of content and circulation. Around 96% of submitted papers
and 95% of subscription income come from outside the UK. This
performance has been recognised by the award of the Queen's Award
for Export Achievement in 1990, 1995 and 2000. International sales
of IOP journals generate a surplus that is transferred annually
by Gift Aid to the Institute. Income from publishing forms the
largest element of the Institute's total income for its charitable
activities in support of its mission to advance physics for the
benefit of all.
The attached annex details our response to the questions
listed in the call for evidence. If you need any further information
on the points raised, please do not hesitate to contact me.
Dr Robert Kirby-Harris CPhys
FInstP
Chief Executive
(i) The strengths and weaknesses of peer review
as a quality control mechanism for scientists, publishers and
the public
1. The peer review system provides accountability
in science through systematic expert scrutiny of published work.
Manuscripts are subjected to critical review by (usually two)
scientists in a particular research area who are asked to comment
on key issues within the paper including, but not limited to,
whether the content appears to be sound, correct procedures appear
to have been followed, and there are no obvious mistakes. Peer
review in journals can also be guided by quality and scoped criteria
set by the editorial board for that journal.
2. In two independent reports1,2 surveying
between 5,000-6,000 researchers, effective peer review mechanisms
were felt to be fundamental to scholarly communications with 96%
of respondents stating that peer review was very or quite important
in regulating the quality of what is published.
3. There are different methods of peer review
ranging from double-blind peer review (where authors and referees/reviewers
are kept anonymous) through to open peer review where authors
and referees are known. The most commonly used method is single-blind
peer review, where the authors' names remain on the manuscript
but the referees' identities are kept anonymous. There are currently
different experiments underway to look at alternative methods
of peer review but, whichever method is used, peer review currently
remains the only reliable way to ensure quality control of the
increasing amount of papers submitted to scientific journals.
Research into this and attitudes towards peer review have been
conducted and reported in a number of recent independent reports3,
4, 5 demonstrating varying support for the different methods.
4. Our experience supports what has been reported
in other studies: that there is a need for peer review as a check,
if not a cast iron guarantee, of quality; and that it is a part
of the defence against fraud (eg fabrication of data) or misconduct
(eg plagiarism). No system is 100% perfect and if/when incorrect
results make it into the literature there are systematic mechanisms
in place to correct errors and maintain a record of any corrections.
In publishing this is done by the use of corrigenda, retractions
or comments and replies, all of which can be linked back to the
source article maintaining an updated record of changes. There
are specific publishing guidelines for handling cases like this
that occur in journals.6 There are some documented
instances of good papers being rejected by the authors' first
choice of publication, for varying reasons; the diversity in the
publishing landscape, however, offers authors of manuscripts multiple
options and it is likely that good papers will get published
in a peer reviewed publication. For example, most scientists have
a selection of preferred journals that they wish to publish in,
with preferences dictated by impact and readership.
5. With the introduction of a publishing industry
tool called CrossCheck,7 many journals have introduced
additional layers of checks into the peer review system, looking
for duplicated work either by the author, known as self-plagiarism,
or the copying of others' work.
6. The strengths of peer review are that it is
robust, trusted internationally and the mechanisms are well established,
understood and thoroughly tested over many years on a global level.
A further strength of the process is that in general it is based
on the anonymity of the referees. Peer review forces every author
to check his/her work carefully; knowing that it will be
critiqued by a peer generally means that more thought goes into
constructing a paper so that others can more easily read and interpret
the work. At the very least, peer review produces clearer papers
and also allows feedback to the authors, which can help improve
a paper. Many authors acknowledge the help of the referee in their
final paper. In an independent report, 77% of respondents agreed
that the referees' comments on their last paper had been helpful
and improved the work.1
7. Without peer review, flawed and correct research
would have the same scientific status. As a result, the pace of
scientific progress would be significantly slower.
8. The weaknesses of peer review are that some
referees may want to block or delay the publication of a paper
because of competing interests, animosity against the authors,
or may allow publications of flawed works because the authors
are famous and respected. In some research areas there is also
an issue with one branch of science dominating to the possible
detriment of the other. This can be mitigated to a large extent
by careful control of deadlines, the editor's knowledge of the
research area and an open system whereby the authors can appeal
a decision. Authors can also, and do, highlight any potential
clashes when submitting their manuscripts.
9. Another weakness can be the length of time
to publication, but this varies between article type, journals
and across research areas. A 40 page review article will take
much longer to referee than a four page letter. It is important
to note that technology has been used to accelerate this process
but there is an inherent delay in any system that requires assessment
and checks to allow for careful consideration of the work. A balance
must be maintained between speed of publication and time for the
referee to assess a manuscript.
10. Peer review is one of the costs of publication
but any filtering mechanism will incur costs regardless of the
final means of dissemination. For many years physics has had arXiv.org,
a pre-print server running in parallel to the peer reviewed journals.
There are research areas where pre-prints have become the main
conduit for communicating results. However, the majority of papers
posted onto arXiv.org also get into peer reviewed journals, as
publication in a high impact-factor refereed journal is still
seen as the "gold standard".
(ii) Measures to strengthen peer review
11. The peer review process can be strengthened
by continually broadening the pool of referees, setting enforced
deadlines for responses, sending reminders to referees, and monitoring
the time an article is left with a referee and sending it to another
if that referee is not responding. In addition, a possible improvement
to discourage referees holding up publication could be to have
more transparency of submission dates as well as published dates
on articles. Technology
has helped systemise the peer review process by allowing referees
to volunteer for journals they write for, continually updating
their information online and for reminders to be sent automatically.
However, there is a need to understand that referees are volunteers
and care needs to be taken not to overburden them and provide
reasonable timescales for reports to be received.
12. In addition, editors are responsible for
choosing the referees and monitoring their conduct. In many journals
there are now preliminary assessments made to enable quick responses
to authors whose work is either outside of the scope of the journal
or is below the quality criteria set for the journal. This could
be one way of relieving the pressure on overburdened referees
and adds an additional layer of filter. Careful selection of referees
from different research areas is also critical in multi-disciplinary
work and the editor has a key role in ensuring that all aspects
of the research are considered. For controversial research areas,
encouraging additional comments from other specialists provides
as fair an assessment as possible.
13. In some research areas it could be an improvement
not to reveal the identity of the authors to the referees (ie
double-blind refereeing), in order to avoid any form of prejudice.
However, in some research areas this would not be practical as
there are only a handful of groups engaged in a specific type
of research and the authors' identity would be obvious.
14. Good practice in peer review dictates that
referees must be technically competent in the relevant research
area; referees must be independent, impartial and without vested
interest. In some research areas this has to be stated explicitly,
in addition to ensuring that referees are not from the same institution
as the authors, not in direct competition and not had any significant
difficulties with the authors in the past or have any other conflict
of interest. The referees should not allow, in so far as it possible,
their own work to be directly influenced by the privileged access
to the material under review. One important way to strengthen
peer review would be to improve the training and information available
on best practice, particularly as refereeing is a skill predominantly
acquired on the job; there are a number of guides available.5
15. It is also important that the organisation
requesting the review should make every effort to carefully define
the criteria for acceptance or rejection. Report forms guiding
referees can be useful in this regard. When selecting referees
for multi-disciplinary research it is important to get input from
the main contributing research areas to help balance the view.
(iii) The value and use of peer reviewed science
on advancing and testing scientific knowledge
16. Without the "filter" of peer review,
scientists would spend a large part of their time following false
leads. Peer review provides added value to the advancement of
science and plays a critical role in enforcing the highest standards
of evidence. Even in research areas where the pre-print is used
as the main route for communication there is still some initial
peer review/filtering required either internally (eg in larger
collaborations) or by user comments on the article. In these cases
the pre-print is also often cross checked for the existence of
the peer reviewed version to validate the work.1
17. It should also be recognised that referees
often make a significant contribution to the quality of the research
through the suggestion of different methodologies, interpretations
and connections to other previously published work which may be
unknown to the original authors.
18. Peer reviewed publications are also used
as a comparative measure to assess research output for assessment
exercises such as the RAE/REF; they are used as a measure of output
and indication of a researcher's career and used in promotion
decisions and awards. Whilst this is not a perfect measure it
is one that is used internationally, with work published in recognised
peer reviewed journals given higher credibility than non-peer
reviewed work.
(iv) The value and use of peer reviewed science
in informing public debate
19. In the discussion paper, Peer Review and
the Acceptance of New Scientific Ideas,8
Sense About Science reported that: "…science
has become the subject of many public and political controversies.
Exaggeration and anxieties about scientific developments often
relate to research findings that are regarded by scientific experts
as weak or flawed, or that have not been subjected to independent
expert scrutiny at all. These developments have resulted in a
greater public need for clarity about the status of new research
claims. A wider understanding of peer review's role in assessing
whether work is competent, significant and original, is central
to achieving that clarity about the status of research."
The discussion paper made a number of recommendations about how
this might be achieved. For example, it suggested that scientific
bodies should make systematic attempts to explain peer review
and communicate what it is to a wider audience, especially when
there is controversy about particular claims. This is a laudable
aim as the importance of improving the public's understanding
of the role of peer review in the scientific process has not diminished
since this discussion paper was published in 2004. Whether any
progress has been as a result of the discussion paper and the
actions of the numerous bodies involved with communicating science,
is hard to judge, but would be very useful to ascertain.
20. The public should be encouraged to recognise
that a peer reviewed result is the "gold standard" in
research and will produce the most reliable information in the
long term, and needs to understand that this process has to be
carried out in confidence to protect referees from undue harassment.
However, sometimes the peer review process can be too slow to
inform debate and encounters difficulties on those rare occasions
when results are first presented directly to the media. This is
a particular concern in medical research and there are a number
of different experiments trialling what is called open peer review
on blogs, for example, for situations where time is critical.9
Peer review is still a critical part of this and needs to retain
the confidence of the scientific community and the public at large
and is therefore very carefully monitored and mediated.
21. An area for improvement here is to explain
clearly to the public the difference between peer reviewed and
non-peer reviewed content.
(v) The extent to which peer review varies
between scientific disciplines and between countries across the
world
22. Nearly all journal publications in physics
are international and the standards are enforced fairly uniformly.
In large experimental collaborations articles have to undergo
extensive internal peer review (sometimes for many months) before
reaching the submission stage. For example, in particle physics,
this contributes to the high use of the pre-prints on arXiv.org
as this initial peer review is widely understood and acknowledged,
although, as stated previously, the majority of the pre-prints
also get published in journals following additional external peer
review.
23. There are different expectations across research
areas with regards to time to publication, depth of report required,
analysis of data, etc. This is also changing as the technology
evolves and one challenge for the future is how to make data available
for the referees to assess, if required. In medical research there
is sometimes greater urgency to discuss issues outside of the
traditional peer review structure and, as mentioned, there are
a number of different approaches currently being tried.
(vi) The processes by which referees with
the requisite skills and knowledge are identified, in particular
as the volume of multi-disciplinary research increases
24. In multi-disciplinary research there is a
need to use multiple referees with expertise in the different
aspects within the multi-disciplinary research area; referees
with expertise across the whole of the research area are very
limited in number, due to the nature of multi-disciplinary research
and it is therefore important to have an assessment of all sides
of the research where possible. A referee with the broadest appropriate
oversight should be sought and the editor and editorial board/assessment
panel has a critical role in assessing the overall reports.
25. Currently, the problem is also to identify
referees with the requisite willingness or time. The peer review
process is rather peculiar in that there is no obvious reward
offered to referees; in fact it is felt to be an integrated part
of the role of a researcher and the "rewards" are the
opportunity to see work in your research area before it is published,
and an expectation that by refereeing a peer's work you would
in turn expect your work to be reviewed. There is a case for revisiting
this tradition, as other professions generally do not proceed
on this pro bono basis when offering a service, but the
majority of participants in the present debate support the current
practice of peer review and strongly feel that this is part of
their role as a scientist.3, 4
(vii) The impact of IT and greater use of
online resources on the peer review process
26. IT has already had a big effect on the peer
review process. In journal publishing the ability to access manuscripts
online for review has been common practice for many years and
some journals also offer referees access to the literature whilst
they are reviewing a paper; tools such as reference linking are
available on most published articles making it easier to check
the background literature.
27. With the semantic web and changes to PDF
technology there is an opportunity to make the process more interactive
and easier; for example, using the potential of a PDF for inserting
comments into drafts, accessing background data, linking to additional
material, etc. However, the use of IT also has to make the process
simpler and less time consuming and not add more work to already
overloaded referees. In addition to traditional peer review, there
are also experiments utilising technology to enable public, post
publication peer review which supplements the initial confidential
peer review.
(viii) Possible alternatives to peer review
28. There isn't any real alternative, although
there can be improvements, as per our response to question 2.
Of particular concern is the issue relating to grant proposals
as outlined in our response to question 3. There is no obvious
alternative to the use of peer reviewed content as a comparative
measure of output and quality and it is difficult to see how these
processes would work without it.
29. In some areas involving collaborative science,
a high level of internal peer review is undertaken before journal
submission takes place (see previous comments). The policing role
of external peer review is of course still highly important in
these cases.
30. The publishing industry and some research
communities are trialling different approaches to the traditional
models but it is still early days and the experiments to date
have had mixed responses.
REFERENCES
1 Scholarly communication
in the digital environment. Ian Rowlands and David Nicholas. Aslib
Proceedings, Vol. 57 Iss: 6, pp.481-497; http://www.publishing.ucl.ac.uk/papers/2005Rowlands_Nicholas.pdf
2 Journal Futures:
How will researchers communicate as the internet matures? Michael
Made, presentation at the Council of Science Editors annual conference
2006.
3 Peer review: benefits,
perceptions and alternatives. Mark Ware for Publishing Research
Consortium; http://www.publishingresearch.net/documents/PRCsummary4Warefinal.pdf
4 Quality assurance
and assessment of scholarly research. A guide for researchers,
academic administrators and librarians. Research Information Network,
May 2010; http://www.rin.ac.uk
5 Peer review; a guide
for researchers. Research Information Network, March 2010; http://www.rin.ac.uk
6 Preserving the record
of science. STM guidelines, April 2006; http://www.stm-assoc.org
7 http://crossref.org/crosscheck.html
8 Peer Review and
the Acceptance of New Scientific Ideas, Sense About Science, November
2002-May 2004; http://www.senseaboutscience.org.uk/pdf/PeerReview.pdf
9 http://www.healthaffairs.org/1500_about_journal.php
Institute of Physics
10 March 2011
| 
