Peer review in scientific publications - Science and Technology Committee Contents

Written evidence submitted by the Institute of Physics (PR 61)

The Institute of Physics is a scientific charity devoted to increasing the practice, understanding and application of physics. It has a worldwide membership of over 40,000, and is a leading communicator of physics-related science to all audiences - from specialists through to government and the general public.

The Institute welcomes the opportunity to respond to the House of Commons Science and Technology Committee's inquiry into peer review, which is particularly relevant to our extensive learned society publishing activities.

The Institute has published academic journals continuously since its foundation in 1874. Today the Institute's publishing is carried out through a wholly owned subsidiary company, Institute of Physics Publishing Ltd (IOP Publishing). The company employs more than 270 people in Bristol and has offices in the USA, Germany, Russia, China and Japan.

The Institute publishes 65 journals, which are international in terms of content and circulation. Around 96% of submitted papers and 95% of subscription income come from outside the UK. This performance has been recognised by the award of the Queen's Award for Export Achievement in 1990, 1995 and 2000. International sales of IOP journals generate a surplus that is transferred annually by Gift Aid to the Institute. Income from publishing forms the largest element of the Institute's total income for its charitable activities in support of its mission to advance physics for the benefit of all.

The attached annex details our response to the questions listed in the call for evidence. If you need any further information on the points raised, please do not hesitate to contact me.

Dr Robert Kirby-Harris CPhys FInstP
Chief Executive

(i)  The strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

1.  The peer review system provides accountability in science through systematic expert scrutiny of published work. Manuscripts are subjected to critical review by (usually two) scientists in a particular research area who are asked to comment on key issues within the paper including, but not limited to, whether the content appears to be sound, correct procedures appear to have been followed, and there are no obvious mistakes. Peer review in journals can also be guided by quality and scoped criteria set by the editorial board for that journal.

2.  In two independent reports1,2 surveying between 5,000-6,000 researchers, effective peer review mechanisms were felt to be fundamental to scholarly communications with 96% of respondents stating that peer review was very or quite important in regulating the quality of what is published.

3.  There are different methods of peer review ranging from double-blind peer review (where authors and referees/reviewers are kept anonymous) through to open peer review where authors and referees are known. The most commonly used method is single-blind peer review, where the authors' names remain on the manuscript but the referees' identities are kept anonymous. There are currently different experiments underway to look at alternative methods of peer review but, whichever method is used, peer review currently remains the only reliable way to ensure quality control of the increasing amount of papers submitted to scientific journals. Research into this and attitudes towards peer review have been conducted and reported in a number of recent independent reports3, 4, 5 demonstrating varying support for the different methods.

4.  Our experience supports what has been reported in other studies: that there is a need for peer review as a check, if not a cast iron guarantee, of quality; and that it is a part of the defence against fraud (eg fabrication of data) or misconduct (eg plagiarism). No system is 100% perfect and if/when incorrect results make it into the literature there are systematic mechanisms in place to correct errors and maintain a record of any corrections. In publishing this is done by the use of corrigenda, retractions or comments and replies, all of which can be linked back to the source article maintaining an updated record of changes. There are specific publishing guidelines for handling cases like this that occur in journals.6 There are some documented instances of good papers being rejected by the authors' first choice of publication, for varying reasons; the diversity in the publishing landscape, however, offers authors of manuscripts multiple options and it is likely that good papers will get published in a peer reviewed publication. For example, most scientists have a selection of preferred journals that they wish to publish in, with preferences dictated by impact and readership.

5.  With the introduction of a publishing industry tool called CrossCheck,7 many journals have introduced additional layers of checks into the peer review system, looking for duplicated work either by the author, known as self-plagiarism, or the copying of others' work.

6.  The strengths of peer review are that it is robust, trusted internationally and the mechanisms are well established, understood and thoroughly tested over many years on a global level. A further strength of the process is that in general it is based on the anonymity of the referees. Peer review forces every author to check his/her work carefully; knowing that it will be critiqued by a peer generally means that more thought goes into constructing a paper so that others can more easily read and interpret the work. At the very least, peer review produces clearer papers and also allows feedback to the authors, which can help improve a paper. Many authors acknowledge the help of the referee in their final paper. In an independent report, 77% of respondents agreed that the referees' comments on their last paper had been helpful and improved the work.1

7.  Without peer review, flawed and correct research would have the same scientific status. As a result, the pace of scientific progress would be significantly slower.

8.  The weaknesses of peer review are that some referees may want to block or delay the publication of a paper because of competing interests, animosity against the authors, or may allow publications of flawed works because the authors are famous and respected. In some research areas there is also an issue with one branch of science dominating to the possible detriment of the other. This can be mitigated to a large extent by careful control of deadlines, the editor's knowledge of the research area and an open system whereby the authors can appeal a decision. Authors can also, and do, highlight any potential clashes when submitting their manuscripts.

9.  Another weakness can be the length of time to publication, but this varies between article type, journals and across research areas. A 40 page review article will take much longer to referee than a four page letter. It is important to note that technology has been used to accelerate this process but there is an inherent delay in any system that requires assessment and checks to allow for careful consideration of the work. A balance must be maintained between speed of publication and time for the referee to assess a manuscript.

10.  Peer review is one of the costs of publication but any filtering mechanism will incur costs regardless of the final means of dissemination. For many years physics has had, a pre-print server running in parallel to the peer reviewed journals. There are research areas where pre-prints have become the main conduit for communicating results. However, the majority of papers posted onto also get into peer reviewed journals, as publication in a high impact-factor refereed journal is still seen as the "gold standard".

(ii)  Measures to strengthen peer review

11.  The peer review process can be strengthened by continually broadening the pool of referees, setting enforced deadlines for responses, sending reminders to referees, and monitoring the time an article is left with a referee and sending it to another if that referee is not responding. In addition, a possible improvement to discourage referees holding up publication could be to have more transparency of submission dates as well as published dates on articles. Technology has helped systemise the peer review process by allowing referees to volunteer for journals they write for, continually updating their information online and for reminders to be sent automatically. However, there is a need to understand that referees are volunteers and care needs to be taken not to overburden them and provide reasonable timescales for reports to be received.

12.  In addition, editors are responsible for choosing the referees and monitoring their conduct. In many journals there are now preliminary assessments made to enable quick responses to authors whose work is either outside of the scope of the journal or is below the quality criteria set for the journal. This could be one way of relieving the pressure on overburdened referees and adds an additional layer of filter. Careful selection of referees from different research areas is also critical in multi-disciplinary work and the editor has a key role in ensuring that all aspects of the research are considered. For controversial research areas, encouraging additional comments from other specialists provides as fair an assessment as possible.

13.  In some research areas it could be an improvement not to reveal the identity of the authors to the referees (ie double-blind refereeing), in order to avoid any form of prejudice. However, in some research areas this would not be practical as there are only a handful of groups engaged in a specific type of research and the authors' identity would be obvious.

14.  Good practice in peer review dictates that referees must be technically competent in the relevant research area; referees must be independent, impartial and without vested interest. In some research areas this has to be stated explicitly, in addition to ensuring that referees are not from the same institution as the authors, not in direct competition and not had any significant difficulties with the authors in the past or have any other conflict of interest. The referees should not allow, in so far as it possible, their own work to be directly influenced by the privileged access to the material under review. One important way to strengthen peer review would be to improve the training and information available on best practice, particularly as refereeing is a skill predominantly acquired on the job; there are a number of guides available.5

15.  It is also important that the organisation requesting the review should make every effort to carefully define the criteria for acceptance or rejection. Report forms guiding referees can be useful in this regard. When selecting referees for multi-disciplinary research it is important to get input from the main contributing research areas to help balance the view.

(iii)  The value and use of peer reviewed science on advancing and testing scientific knowledge

16.  Without the "filter" of peer review, scientists would spend a large part of their time following false leads. Peer review provides added value to the advancement of science and plays a critical role in enforcing the highest standards of evidence. Even in research areas where the pre-print is used as the main route for communication there is still some initial peer review/filtering required either internally (eg in larger collaborations) or by user comments on the article. In these cases the pre-print is also often cross checked for the existence of the peer reviewed version to validate the work.1

17.  It should also be recognised that referees often make a significant contribution to the quality of the research through the suggestion of different methodologies, interpretations and connections to other previously published work which may be unknown to the original authors.

18.  Peer reviewed publications are also used as a comparative measure to assess research output for assessment exercises such as the RAE/REF; they are used as a measure of output and indication of a researcher's career and used in promotion decisions and awards. Whilst this is not a perfect measure it is one that is used internationally, with work published in recognised peer reviewed journals given higher credibility than non-peer reviewed work.

(iv)  The value and use of peer reviewed science in informing public debate

19.  In the discussion paper, Peer Review and the Acceptance of New Scientific Ideas,8 Sense About Science reported that: "…science has become the subject of many public and political controversies. Exaggeration and anxieties about scientific developments often relate to research findings that are regarded by scientific experts as weak or flawed, or that have not been subjected to independent expert scrutiny at all. These developments have resulted in a greater public need for clarity about the status of new research claims. A wider understanding of peer review's role in assessing whether work is competent, significant and original, is central to achieving that clarity about the status of research." The discussion paper made a number of recommendations about how this might be achieved. For example, it suggested that scientific bodies should make systematic attempts to explain peer review and communicate what it is to a wider audience, especially when there is controversy about particular claims. This is a laudable aim as the importance of improving the public's understanding of the role of peer review in the scientific process has not diminished since this discussion paper was published in 2004. Whether any progress has been as a result of the discussion paper and the actions of the numerous bodies involved with communicating science, is hard to judge, but would be very useful to ascertain.

20.  The public should be encouraged to recognise that a peer reviewed result is the "gold standard" in research and will produce the most reliable information in the long term, and needs to understand that this process has to be carried out in confidence to protect referees from undue harassment. However, sometimes the peer review process can be too slow to inform debate and encounters difficulties on those rare occasions when results are first presented directly to the media. This is a particular concern in medical research and there are a number of different experiments trialling what is called open peer review on blogs, for example, for situations where time is critical.9 Peer review is still a critical part of this and needs to retain the confidence of the scientific community and the public at large and is therefore very carefully monitored and mediated.

21.  An area for improvement here is to explain clearly to the public the difference between peer reviewed and non-peer reviewed content.

(v)  The extent to which peer review varies between scientific disciplines and between countries across the world

22.  Nearly all journal publications in physics are international and the standards are enforced fairly uniformly. In large experimental collaborations articles have to undergo extensive internal peer review (sometimes for many months) before reaching the submission stage. For example, in particle physics, this contributes to the high use of the pre-prints on as this initial peer review is widely understood and acknowledged, although, as stated previously, the majority of the pre-prints also get published in journals following additional external peer review.

23.  There are different expectations across research areas with regards to time to publication, depth of report required, analysis of data, etc. This is also changing as the technology evolves and one challenge for the future is how to make data available for the referees to assess, if required. In medical research there is sometimes greater urgency to discuss issues outside of the traditional peer review structure and, as mentioned, there are a number of different approaches currently being tried.

(vi)  The processes by which referees with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases

24.  In multi-disciplinary research there is a need to use multiple referees with expertise in the different aspects within the multi-disciplinary research area; referees with expertise across the whole of the research area are very limited in number, due to the nature of multi-disciplinary research and it is therefore important to have an assessment of all sides of the research where possible. A referee with the broadest appropriate oversight should be sought and the editor and editorial board/assessment panel has a critical role in assessing the overall reports.

25.  Currently, the problem is also to identify referees with the requisite willingness or time. The peer review process is rather peculiar in that there is no obvious reward offered to referees; in fact it is felt to be an integrated part of the role of a researcher and the "rewards" are the opportunity to see work in your research area before it is published, and an expectation that by refereeing a peer's work you would in turn expect your work to be reviewed. There is a case for revisiting this tradition, as other professions generally do not proceed on this pro bono basis when offering a service, but the majority of participants in the present debate support the current practice of peer review and strongly feel that this is part of their role as a scientist.3, 4

(vii)  The impact of IT and greater use of online resources on the peer review process

26.  IT has already had a big effect on the peer review process. In journal publishing the ability to access manuscripts online for review has been common practice for many years and some journals also offer referees access to the literature whilst they are reviewing a paper; tools such as reference linking are available on most published articles making it easier to check the background literature.

27.  With the semantic web and changes to PDF technology there is an opportunity to make the process more interactive and easier; for example, using the potential of a PDF for inserting comments into drafts, accessing background data, linking to additional material, etc. However, the use of IT also has to make the process simpler and less time consuming and not add more work to already overloaded referees. In addition to traditional peer review, there are also experiments utilising technology to enable public, post publication peer review which supplements the initial confidential peer review.

(viii)  Possible alternatives to peer review

28.  There isn't any real alternative, although there can be improvements, as per our response to question 2. Of particular concern is the issue relating to grant proposals as outlined in our response to question 3. There is no obvious alternative to the use of peer reviewed content as a comparative measure of output and quality and it is difficult to see how these processes would work without it.

29.  In some areas involving collaborative science, a high level of internal peer review is undertaken before journal submission takes place (see previous comments). The policing role of external peer review is of course still highly important in these cases.

30.  The publishing industry and some research communities are trialling different approaches to the traditional models but it is still early days and the experiments to date have had mixed responses.


1 Scholarly communication in the digital environment. Ian Rowlands and David Nicholas. Aslib Proceedings, Vol. 57 Iss: 6, pp.481-497;

2 Journal Futures: How will researchers communicate as the internet matures? Michael Made, presentation at the Council of Science Editors annual conference 2006.

3 Peer review: benefits, perceptions and alternatives. Mark Ware for Publishing Research Consortium;

4 Quality assurance and assessment of scholarly research. A guide for researchers, academic administrators and librarians. Research Information Network, May 2010;

5 Peer review; a guide for researchers. Research Information Network, March 2010;

6 Preserving the record of science. STM guidelines, April 2006;


8 Peer Review and the Acceptance of New Scientific Ideas, Sense About Science, November 2002-May 2004;


Institute of Physics

10 March 2011

previous page contents next page

© Parliamentary copyright 2011
Prepared 28 July 2011