Peer review

Written evidence submitted by the UK Research Integrity Office (PR 84)

General comments

The peer review process is used globally in science and the majority of other academic disciplines to examine the quality of scientific research findings prior to publication and also to evaluate the quality of research grant applications as part of the selection process before an award are made. Peer review is also used to inform academic promotion.

Peer review is not a perfect process and almost certainly can be improved. One is reminded of the quotation by Sir Winston Churchill in 1947, "Many forms of government have been tried, and will be tried in this world of sin and woe. No-one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of government except all those other forms that have been tried from time to time."

Response to specific requests for comment

1 The strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

1.1 Peer review is a form of scientific control or self-regulation which aims to check, criticise and improve research. It is a process where scientists open their research to the scrutiny of other experts in the field. It is usually, but not always, a confidential process involving three key partners (i) the investigator/author, (ii) the journal editor or funding agency and (iii) the peer reviewer(s). In most instances, the author/investigator will be unaware of the identity of the peer reviewer, although this is not always the case.

1.2 Science works best in an environment of unrestrained criticism and thus high-quality peer review will aim to detect methodological faults in the design of research studies, identify flaws in the analysis and reviewers should be expected to provide comment on whether the interpretation of the findings is appropriate and make a judgement on the likely impact of the findings on the future development of the particular scientific area. It is the most widely used quality assurance process for selecting papers for publication and for selecting research grants for funding.

1.3 Peer review works well when it is conducted by expert, thorough reviewers who undertake the task in an unbiased and honest way. The process, however, has many weaknesses but, as yet, there is no viable alternative to quality assure the process for the publication of scientific research or to assess the worthiness of grant applications.

1.4 Peer review has many downsides. Peer review has many critics as it is thought to be non-standardised, idiosyncratic and open to bias. This can lead to unreliability, unfairness and a failure to validate or authenticate research. Most disciplines in science are highly competitive. There is a danger that the peer review process can stifle innovation and perpetuate the status quo. Peer reviewers, for example, are more likely to reject a paper or research grant if it challenges their own belief system.

1.5 The most widely used form of peer review is when the author or investigator is unaware of the identity of the peer reviewer, sometimes called "blind" peer review. This secrecy can lead to irresponsibility and failure to produce a fair, balanced review and, on occasions, may invite malice. Many authors will have experienced long delays in the peer review process have suspected that a peer reviewer might be intentionally holding up publication to allow advancement of their own work.

1.6 Blind peer review may allow reviewers to make broad, overarching, destructive statements, such as "this study is fundamentally flawed", without fully substantiating their claims. Such a statement will immediately alert an editor to the potential profound weakness of a study and may trigger a rapid rejection process.

1.7 It is for this reason that many feel that "true experts" may be too close to the work to produce a balanced opinion. While their technical competence cannot be disputed, their proximity to the research area and their competitiveness may be destructive.

1.8 Although in the last decade there have been a number of research studies addressing some aspects of the peer review process, many would still argue that it has been insufficiently tested by objective measures. There is evidence, for example, that the outcomes of peer review are dependent on age, gender, language skills and geographic location, all of which have been shown, to some extent, to influence the outcome of the peer review process.

1.8 By and large these variables are not taken into account when routinely selecting peer reviewers. Most journal editors and funding agencies will be aware that, within the scientific community, there are "hawks" and "doves". In the recent past, editors of some elite science journals have been heavily criticised for their own bias perhaps in selecting reviewers who would produce the answer that they wanted. In a quote from the Wall Street Journal it was said that "Nature and Science are locked in such fierce competition for prestige and publicity that they may be cutting corners to get ‘hot’ papers". The Nobel Laureate, Robert Laughlin, commenting on a series of retractions from these eminent journals said "in this case the editors are definitely culpable… they chose reviewers they knew would be positive".

1.9 There are still uncertainties as to how many reviewers should be invited to comment on a grant proposal or a research publication. Statisticians might suggest that statistically viable results would require at least six opinions, all agreeing to either reject or accept. It is uncommon for as many as six reviewers being asked to assess a paper for publication. Journal editors will differ in the number of reviewers they engage and at the end of the day, most journals give editors the responsibility for the final decision so he or she is at liberty to accept or reject the advice of reviewers. It is not a democratic process.

1.10 Peer review is an expensive, time-consuming process that is largely unrewarded and unrecognised. Many scientists will review large numbers of papers each year and many will serve on review boards for funding agencies which, again, takes them away from their research and other responsibilities.

1.11 Despite all the shortcomings of the peer review process, there is a real danger in placing research findings in the public domain that have not been quality assured. The possibility for the commercial exploitation of poor science is already evident and there is still a tendency for some questionable findings to be published in the national press before having undergone scrutiny by experts in the field. Peer reviewers do not always get it right but the public deserves scientific findings that can have such a profound effect on society to be quality assured by a gateway process managed by experts.

2 Measures to strengthen peer review

2.1 Clear guidance on what is expected of a peer reviewer:

Many journals and funding agencies provide instructions to peer reviewers but there is a lack of consistency and, in some cases, a paucity of detail. Reinforcement of the necessity to understand that this is a confidential process and, in effect, the peer reviewer makes a contract with both the editor/funder and the applicant. A clearer understanding of the variables that influence the outcome of peer review need to be emphasised as is the necessity of disclosing all conflicts of interest.

2.2 Training:

The knowledge and skills required to conduct high quality peer review are usually passed on in an informal way. Peer reviewers in the future perhaps should be required to undertake a period of training which could be largely online but might involve at least one face-to-face group session (possibly via a ‘chat room’) which could be case based.

2.3 Selection of reviewers:

There is some evidence that younger peer reviewers produce a more thorough review. There is evidence that editors can bias the outcome of the peer review process by selecting particular individuals who they know will produce the answer they require. Most journals have a well-established database of reviewers which can be searched by the area of special expertise. It might be argued that, within disciplines and sub-disciplines, peer reviewers might be selected at random to avoid editorial bias.

2.4 The majority of peer reviewers are not paid and their work is largely unrewarded. Most scientists accept that peer reviewing each other's work is part of the job and many will spend a considerable number of hours every year undertaking the process. As yet there is no satisfactory way of identifying this contribution as a serious part of the research and publication process. One of the advantages of "open" peer review is that the name of the reviewer could be published in the scientific journal along with the authors and it has been suggested that the reviewer’s reports might also be available with the online version paper. Contribution to the peer review process might be formally recognise as part of an academic work and the taken into account as a criterion for promotion.

2.5 ‘Open’ peer review:

This approach is used by an increasing number of journals and effectively means that, not only do reviewers know the identity of the authors, but the authors know the identity of their reviewers and reviewers are invited effectively to sign their peer review assessments. Many feel that this leads to greater openness in the peer review process and leads to greater assurance that reviews will be evidence based. There is certainly evidence that reviews conducted under this system are "more constructive and courteous" but no evidence as yet that it improves the quality of the reviews. Further research is clearly required to determine whether this is indeed the way forward. Some younger reviewers, however, are concerned about open peer review as they feel their careers may be at stake if they produce harsh reviews of the work of established giants in the field.

3 Value of peer reviewed science on advancing and testing scientific knowledge

The volume of research and research publications has increased almost exponentially over the last 50 years. Most scientists feel that some pre-selection of published work is required as it would be impossible to perform this function on an individual basis. The peer review process also improves the final published work often making it shorter, more focussed and reduces the chance of technical errors, both in the science and in the published article.

4 Peer review in informing public debate

The public inevitably relies on experts to pre-select and quality assure scientific data that are placed in the public domain. A Mori survey in 2004 clearly showed that the public have a poor understanding of the peer review process but many want peer scrutiny and want scientific findings to be replicated before findings are widely trailed in the public domain.

5 Differences in peer review between scientific disciplines and between different countries

There are differences in the peer review process across the scientific disciplines. The peer review process in life and biomedical sciences generally follows the standard approach described above. However in mathematics and to some extent in physics peer review often takes place in an open and transparent manner often as a collective activity. A good example is the way in which the solution to Fermat's Last Theorem was arrived at. Similarly the peer review process for the ‘big experiments’ in physics are again often conducted as a collective event in an open and transparent manner. The quality of peer review almost certainly varies globally and from Journal to Journal.

6 Identification of peer reviewers

Most journals will construct a large database of potential reviewers whose special interest and expertise has been identified prior to entry onto the database. These databases are searchable by editors and editorial staff and the final selection made of between two and four reviewers being the norm. It is not uncommon for reviewers to decline to review a paper in which case the editor will return to the database and seek alternatives. Reviewer databases are usually refreshed on a regular basis. Tardy or ineffective reviewers may be removed and new and active researchers added.

Multidisciplinary research papers may require a larger number of reviewers to reflect the multidisciplinary approach of the research.

7 Impact of IT on the peer review process

Modern information technology has revolutionised the publication process. The majority of major scientific journals have moved entirely to an online submission and review process. This paperless process has reduced the time to publication which has been largely due to the speeding up of the peer review process.

8 Possible alternatives to peer review

8.1 The most obvious alternative to peer review is to publish all research studies and allow the scientific community to decide whether a particular study is of high quality and whether it contributes significantly to the body of knowledge. This could work in the same way as the now popular social media sites, including the online versions of daily newspapers and magazines which encourage comments. Some of the weekly medical journals such as the British Medical Journal encourage rapid responses to publish papers and other articles which stimulate an interesting debate.

8.2 With the assent of open access online journals it was anticipated that many of these would dispense with formal peer review prior to publication and allow the scientific community to decide. The majority however have retained a formal peer review process before e publication. It has been suggested however that an open online peer review process could take place during a period of say four – six weeks following which an editorial decision could be made as to whether the paper should be formally published.

8.3 During the past 30 years there has been expansion of the number of scientific journals available to authors and a substantial increase in the number of published papers. If there was a viable alternative to peer review then one might have expected expansion to have driven change. This suggests that as yet there is no viable alternative to the established peer review process and that we have to accept it as a relatively expensive, time-consuming and imperfect process.

Declaration of Interests

This submission draws upon the views of the Board of the UK Research Integrity Office (UKRIO) and its staff. These include persons who have: undertaken research which was then submitted for publication via the peer review process; acted as peer reviewers; acted as editors of academic journals which operate a peer review process; and/or hold senior roles in institutions such as universities which commonly disseminate their research via peer reviewed journals. The Committee on Publication Ethics, a forum for editors and publishers of peer reviewed journals to discuss all aspects of publication ethics, has supported UKRIO and its Chair holds a seat on our Board. UKRIO has received funding from bodies that fund research projects which are commonly disseminated via peer reviewed journals, such as Research Councils UK. None of the bodies which fund or support UKRIO had any input into the content of this submission.

About the UK Research Integrity Office

The UK Research Integrity Office (UKRIO), established in 2006, is an independent body which offers confidential and expert advice and guidance about the conduct of research, covering all subject areas. It helps research organisations, individual researchers and members of the public. UKRIO also publishes guidance on good research practice and the investigation of alleged misconduct, and operates a help-line service where concerns can be reported in complete confidence. UKRIO is not a regulatory body and has no formal legal powers. The advice and guidance it offers is not mandatory but reflects best practice in the conduct of research and addressing misconduct. Further information about UKRIO is available from its website: www.ukrio.org .

UK Research Integrity Office

10 March 2011