Peer review

Written evidence submitted by
The Academy of Medical Sciences (PR 89)

Summary

· The Academy of Medical Sciences believes that peer review is required to quality assure scientific publications, to the benefit of all those who use them. It helps to generate trust and consistent standards in science.

· The physical sciences communities have adopted a model of pre-publication community peer review. We believe that some level of peer review before publication is essential in the biomedical sciences because of the more subjective and open ended nature of the research.

· The desire to maintain impact factors, to reduce print production costs and the conservative attitudes of some journals can increase the rejection rates of papers. Researchers may have to approach a number of journals before a paper is accepted.

· High rejection rates and the requirement for unnecessary revisions before publication delay access to the outcomes of publicly and charity funded science and waste the time of researchers. Ultimately they slow the progression of science.

· We welcome models that use peer review simply to assess the validity of the scientific approach taken rather than its potential impact – resulting in a faster rate of publication. To identify the likely high impact papers, post-filtering mechanisms such as the Faculty of 1000 are helpful to researchers and other users of scientific knowledge.

· Digitisation of scientific publication has facilitated greater use of, and access to, peer reviewed science. It has also created easier and quicker automated systems for both the submitting authors and the reviewers. Online tools for reference checking, plagiarism monitoring and figure enhancement provide new ways to monitor unethical behaviour.

Introduction

The Academy of Medical Sciences welcomes the opportunity to respond to the Select Committee’s inquiry. The Academy is the independent body in the UK representing medical science that promotes the advances of medical research and campaigns to ensure these are converted into healthcare benefits for society. Progression of scientific knowledge in all science, including medical science, is dependent on maintaining a firm foundation of information on which future scientists can base their research. Similarly, public trust and comprehension of science are better justified by a system whereby data and scientific ideas have been formally scrutinised and subsequently endorsed. Peer review of primary research is essential to both these goals.

Peer review is used to assess research for quality and potential impact (normally by the journal to which it has been submitted). It is initially viewed by the journal editors who can decide to reject the paper outright or send it on to review, based on the relevance of the manuscript’s content to the journal’s scope and on its potential impact. If sent to peer review then experts in the field will review the contents of the paper and can either: accept it in its current form; reject it; or send it back to the authors for further revision and/or experimentation. In the latter case, the paper is re-reviewed to decide on final publication or rejection.

Other models of publication use post-publication peer review as part of a traditional journal or a repository. Additional variations include the identification of peer reviewers and the publication of reviews alongside papers.

Peer review is a vital tool in the process of scientific publication and is ultimately beneficial to both the scientific community and the public. However, as we outline in this response, which addresses each of the issues raised by the Committee in turn, the current system can, and should, be improved to increase the speed of the dissemination of scientific knowledge.

1. The strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

The key strength of peer review is that it helps to ensure that weak research is not endorsed through publication in recognised journals. In this digital age where new information can be disseminated very easily, peer review acts as an important filter or ‘kite mark’ to differentiate between research that has reached an acceptable scientific standard and that which has not. This is particularly important in the biomedical field where research is of great interest to many non-scientists. The peer review process can be important in improving a paper before publication but, as we outline below, it can also unnecessarily delay the rate at which new knowledge becomes available.

Scholarly journals use peer review to assess three main features of potential papers:

· The quality of the research carried out.

· The potential impact of the study.

· The relevance of the research to the remit of the journal.

Peer review is the traditional method of regulating scientific publication and requires the knowledge of experienced experts to scrutinise scientific work. Journals vary in nature, for example by the breadth of their remit. The status of any particular journal depends on its impact factor, a measure of the impact its published studies have on science. The various scientific disciplines use slightly different forms of peer review (see section 5). High impact journals are characterised by having a high rejection rate, only publishing what they regard to be the highest impact research.

Publication prestige and quality control for researchers, publishers and the public

There is increasing pressure on researchers to publish in high impact journals such as Nature and Science. A strong publication record is a key determinant in the allocation of grant funding both to individual researchers and to their universities via processes such as the Research Excellence Framework (formerly the Research Assessment Exercise). This has focused attention on how peer review operates, particularly in high impact journals. Further, the scrutiny offered by the peer review process helps to ensure only scientifically sound work is published, providing a reliable body of information that can be used by other researchers. This also allows publishers to maintain consistent quality and scope of the articles published, justifying subscription costs.

Effective peer review is vital in ensuring that the information subsequently used in the public and policy-making domains, is accountable and trustworthy. Thus peer review can help guarantee that knowledge and ideas derived via appropriate scientific methodology are made available to wider circles of society. Science has also become a stronger part of public culture. Whereas previously peer review would occur in the context of communication between scientists, journals must now consider how new work will be received in a public setting. Concerns have been raised that this could influence the timing of publication of work that may have a public impact (or whether to publish at all) and prevent potentially important findings being scrutinised, and the experiments replicated.

Current publishing models can create a risk-averse publishing environment that can delay progression of scientific knowledge and lead to wasted research time and money

To justify subscription costs and maintain prestige, there is increasing pressure on journals to preserve or improve their impact factor. This in turn places strain on publishers and peer reviewers to raise the rejection rate of papers, effectively leading to greater exclusivity of the work published. In addition, there is an incentive to limit print production costs of hard copies by reducing the numbers of papers published.

We welcome the fact that some journals have a Fast Track review and publication process but ultimately, increased rejection rates and conservative attitudes can delay publication of valid research, which can hamper progress across disciplines. After rejection from one journal, submission to another journal requires not only reformatting of the manuscript, but instigation of an entirely new round of peer review. Even if the research is eventually published, this wastes research time and money and can delay the availability of the findings to the scientists. The behaviour of peer reviewers who may require revisions that are of tangential importance can also contribute to unnecessary time delays associated with publication of research (see below). Many papers will be published eventually and delays attributed to the peer review process can sometimes be beneficial in clarifying a piece of research before publication. However in some of the biomedical disciplines, the delay between submitting a valid scientific paper to the first choice journal and having it published (perhaps in the second or third choice of journal) is causing major concern.

The role of reviewers

Peer review depends on the experts involved having the experience and knowledge to critically appraise scientific research. However, reviewing manuscripts is time-consuming, which can result in delegation of reviewing duties to less senior colleagues, for example post-doctoral researchers. Younger reviewers, perhaps due to positions of less seniority, spend on average more time reviewing a manuscript which can result in a more thorough appraisal of the work. [1] If supervised, this can also be a useful training opportunity for the junior researcher. However, without moderation by the original reviewer it can result in requests for unnecessary additional data or revisions that may be superfluous to the key theme of the study.

Unnecessary changes delay access to new information for scientists and the public and they waste the time and funds of researchers, which is especially detrimental insofar as a significant amount of research is directly (through charitable organisations) or indirectly (through taxation) funded by the public.

Traditionally, the peer review process is a closed system, where the reviewer’s identity remains anonymous. This raises the possibility of unfair treatment from competitors and potentially the ‘theft’ of ideas, but does allow reviewers to give their honest opinion about a paper without fear of repercussions. Journals, such as the British Medical Journal have adopted an open system, whereby the reviewer’s details are made available to authors and the European Molecular Biology Organisation (EMBO) journal publishes reviewer comments in conjunction with the publication of the paper. [2] , [3] One randomised study has shown that reviewers that were identified provided reviews that were of higher quality but took longer to complete than unsigned reviews. [4] Reviewers who were identified were more likely to recommend publication. [5]

2. Measures to strengthen peer review

We believe that pre-publication peer review is the most appropriate method of quality assuring scientific knowledge in the biomedical sciences, although as outlined earlier, the practices of some journals can create delays to the progress of research. One possible method to improve and strengthen how peer review is utilised is to prioritise the quality of the research as the key determinant for publication rather than its perceived impact. This could facilitate quicker access to new knowledge, while still preventing poor science being published.

Public Library of Science ONE journal approach

An example of a system where peer review places emphasis on research validity over potential impact is that taken by the Public Library of Science ONE (PLoS ONE) journal (also mentioned in section 8). In principle, any manuscript submitted that presents valid research will be published, with less emphasis on the potential impact of the paper during the peer review process. Initially, many researchers used PLoS ONE as a final resort to ensure their work was published somewhere (after previous rejections), but the process undertaken by the journals can result in quicker submission to publication times and more researchers are turning to PLoS ONE as a first or second port of call for publication to avoid delays. [1]

The PLoS family of journals (along with an increasing number of other journals) also use an open access model where authors pay publication costs, including those incurred from the peer review process and the papers are free to access from the date of publication. This also speeds the rate of access to new knowledge.

3. The value and use of peer reviewed science on advancing and testing scientific knowledge

Scientific peer review performs vital roles in both advancing and testing scientific knowledge. If all data were released in a way that did not involve scrutiny, then the onus would fall to the reader to judge the relevance of the work. While some may have the knowledge and capability to do this, many do not, and peer review ensures the services of experts who can carry out these roles. This ultimately results in a solid body of information on which junior scientists can base their own work.

4. The value and use of peer reviewed science in informing public debate

As highlighted in section 1, peer review is vital in informing public debate. Journals having access to reviewers who are deemed experts in the field means they can create a ‘kite mark’ of quality associated with the papers they are publishing. This in turn is helpful in informing public trust and reliability of the particular journal and the work contained therein. Work that is released in to the public domain without some level of quality assurance could potentially lead to situations where imperfect or incorrect science is used by the media and others. Ultimately this could be detrimental to the public’s overall trust in research. As highlighted above, this is perhaps the most important role of peer review.

5. The extent to which peer review varies between scientific disciplines and between countries across the world

Variation of peer review on the global stage

Science is a global endeavour with an increasing number of multi-national collaborations. At one level, the country from which the research originates should have little bearing on the peer review process. However countries vary in terms of the conditions under which research is conducted (e.g. the facilities and funding available) and journals need to decide whether to vary their measurements of quality accordingly if they are to take account of local conditions.

Variation of peer review between disciplines

The physical sciences have been quick to adopt newer systems for dissemination that use post-publication, rather than pre-publication, peer review. For example, the database arXiv acts as an online repository for pre-prints of papers in the physical and mathematical fields. [1] Here authors submit original manuscripts to the repository and anyone can view and comment on the work. Authors can subsequently use this information to further improve and modify the paper before submission to the peer review process. Moderators monitor the discussion of each submission to ensure comments do not go ’off topic’ and all comments and discussions are open and transparent. [2] More formal ‘Hybrid Peer Review’ systems exist whereby the journals themselves provide the open forum for community discussion before sending papers off to traditional anonymous peer review for that specific journal. [3] , [4] This method is utilised by The Journal of Atmospheric Chemistry and Physics.

We feel that approaches that involve the initial publication of papers without peer review in an open forum are less appropriate for biomedical research. In the physical and mathematical sciences studies are more likely to present a finding that will be fundamentally right or wrong. However more subjectivity exists in biomedical research where differing and often competing experimental systems and approaches are used to answer the same questions, creating greater scope for ‘incorrect’ or differing results. The dissemination of non-peer reviewed information, for example about medical research, to the public domain could potentially be unhelpful. Hybrid systems and those that ‘kite mark’ the final, accepted, version of the paper address some of our concerns but the process as a whole may result in a delay in overall publication times.

6. The processes by which reviewers with requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases

Identification of reviewers with requisite skills

Peer reviewers are generally experts in their field and thus have the requisite knowledge to review a paper from that subject area. It has been proposed that perhaps, with the author’s permission, reviewers can use the manuscript as part of a group discussion with other members of their laboratory or department, in a similar manner to the universal journal club, where papers that have already been published are presented at a laboratory or departmental level. [1] This can be helpful providing (as outlined in section 1) that it does not result in authors being asked to make unnecessary changes to their manuscript. We can see this approach as being particularly beneficial in contributing to some of the post-publication filtering methods (see section 8).

Identification of suitable reviewers for multi-disciplinary research

Multi-disciplinary studies can be potentially problematic to the standard peer review system. Several approaches may be required to ensure fair treatment of these types of study, especially as they become increasingly utilised to address research problems. [1] Each domain of the submitted work must be scrutinised to the same degree, which increases the complexity of the peer review process.

It has been suggested that in the future, official review groups could be set up to act as excellence centres that could review emerging multi-disciplinary studies. Authors would have to submit their work under the knowledge that their research may be reviewed by peer group as opposed to a single reviewer. Or, less formally, a single peer reviewer could be selected with the knowledge that they may select someone they feel could help them review the research. [2] Ultimately peer review needs to adapt to changes in the way that scientific research is conducted.

7. The impact of IT and greater use of online resources on the peer review process

As with many aspects of science, digitisation has improved how peer review can be conducted – simplifying and quickening the process. With the majority of reputable journals publishing online as well as in hard copy, new models of peer review can emerge due to reduced pressure on number of pages per paper or issue – the main limiting factor in hard copy publication. Indeed, all the aforementioned models and further models discussed in section 8 utilise and are reliant, sometimes entirely, on online resources. [1]

Impact on the peer review process

Digitisation of submission, tracking and reviewing of research papers has in some ways hastened the speed of peer review with many journals using online pro-forma both for authors to submit their work and for reviewers to access and submit their comments and decisions.14 It is also now easier for reviewers to substantiate the author’s claims and detect breaches of ethical behaviour.

Online anti-plagiarism programmes for the ethical misuse of text, or similar programmes for detecting digitally modified figures, are routinely available. [1] Overall therefore, the increased use of online resources aids in raising confidence levels in the reviewers and in the peer review process, which can also contribute to the ability to carry out ’light-touch’ peer review such as that described for the PLoS ONE journal.

8. Possible alternatives to peer review

The Academy believes that some form of pre-publication peer review is a key requirement for any trustworthy and valid system for scrutinising scientific publication, particularly as interest in results from those outside the scientific community increases. The example described in section 2, PLoS ONE model, represents a variation on the traditional peer review model – with quality and validity of the research carried out taking precedence over expected impact of studies. While one criticism of this approach is the risk of flooding journals with reams of mediocre studies, with no ’pre-filtering’ on the perceived impact of the study, some modern ’post-filtering’ approaches can, and are, fulfilling this requirement. [1]

For example, the Faculty of 1000 (F1000) is a dedicated post-publication peer review format by experts, who are elected by peers to determine the paper’s impact (as opposed to the journal’s impact factor), generating an ‘F1000 Article Factor’ (FFa). [2] , [3] A post-filter mechanism like this can identify the most significant papers more quickly than more accurate impact measures such as citation indices. While post-publication filters are subjective estimates of the likely impact of papers, they are helpful when the speed of publication and the number of papers being published is increasing.

While we believe that ‘peer community’ discussions of papers prior to publication are not appropriate to biomedical research (see section 5), post publication community review could also be a potential post-filter mechanism to estimate the impact of papers published through organisations such as PLoS. A so-called ‘Amazon-like’ process where, like the Amazon website, readers’ reviews can generate a collective opinion on the research presented, although we are not aware of a working example of this type of post-filtering. [4]

The Academy of Medical Sciences

The Academy of Medical Sciences promotes advances in medical science and campaigns to ensure these are converted into healthcare benefits for society. Our Fellows are the

UK’s leading medical scientists from hospitals and general practice, academia, industry and the public service.

The Academy seeks to play a pivotal role in determining the future of medical science in the UK, and the benefits that society will enjoy in years to come. We champion the UK’s strengths in medical science, promote careers and capacity building, encourage the implementation of new ideas and solutions – often through novel partnerships – and help to remove barriers to progress.

The Academy’s Officers are:

Professor Sir John Bell FRS HonFREng PMedSci (President); Professor Patrick Sissons

FMedSci (Vice-President); Professor Ronald Laskey CBE FRS FMedSci (Vice-President);

Professor Robert Souhami CBE FMedSci (Foreign Secretary); Professor Susan Iversen CBE

FMedSci (Treasurer); Professor Patrick Maxwell FMedSci (Registrar).

The Academy of Medical Sciences

17 March 2011