Written evidence submitted by the UK Research
Integrity Office (PR 84)
GENERAL COMMENTS
The peer review process is used globally in science
and the majority of other academic disciplines to examine the
quality of scientific research findings prior to publication and
also to evaluate the quality of research grant applications as
part of the selection process before an award are made. Peer review
is also used to inform academic promotion.
Peer review is not a perfect process and almost certainly
can be improved. One is reminded of the quotation by Sir Winston
Churchill in 1947, "Many forms of government have been tried,
and will be tried in this world of sin and woe. No-one pretends
that democracy is perfect or all-wise. Indeed, it has been said
that democracy is the worst form of government except all those
other forms that have been tried from time to time".
RESPONSE TO
SPECIFIC REQUESTS
FOR COMMENT
1. The strengths and weaknesses of peer review
as a quality control mechanism for scientists, publishers and
the public
1.1 Peer review is a form of scientific control
or self-regulation which aims to check, criticise and improve
research. It is a process where scientists open their research
to the scrutiny of other experts in the field. It is usually,
but not always, a confidential process involving three key partners
(i) the investigator/author, (ii) the journal editor or funding
agency and (iii) the peer reviewer(s). In most instances, the
author/investigator will be unaware of the identity of the peer
reviewer, although this is not always the case.
1.2 Science works best in an environment of unrestrained
criticism and thus high-quality peer review will aim to detect
methodological faults in the design of research studies, identify
flaws in the analysis and reviewers should be expected to provide
comment on whether the interpretation of the findings is appropriate
and make a judgement on the likely impact of the findings on the
future development of the particular scientific area. It is the
most widely used quality assurance process for selecting papers
for publication and for selecting research grants for funding.
1.3 Peer review works well when it is conducted
by expert, thorough reviewers who undertake the task in an unbiased
and honest way. The process, however, has many weaknesses but,
as yet, there is no viable alternative to quality assure the process
for the publication of scientific research or to assess the worthiness
of grant applications.
1.4 Peer review has many downsides. Peer review
has many critics as it is thought to be non-standardised, idiosyncratic
and open to bias. This can lead to unreliability, unfairness and
a failure to validate or authenticate research. Most disciplines
in science are highly competitive. There is a danger that the
peer review process can stifle innovation and perpetuate the status
quo. Peer reviewers, for example, are more likely to reject a
paper or research grant if it challenges their own belief system.
1.5 The most widely used form of peer review
is when the author or investigator is unaware of the identity
of the peer reviewer, sometimes called "blind" peer
review. This secrecy can lead to irresponsibility and failure
to produce a fair, balanced review and, on occasions, may invite
malice. Many authors will have experienced long delays in the
peer review process have suspected that a peer reviewer might
be intentionally holding up publication to allow advancement of
their own work.
1.6 Blind peer review may allow reviewers to
make broad, overarching, destructive statements, such as "this
study is fundamentally flawed", without fully substantiating
their claims. Such a statement will immediately alert an editor
to the potential profound weakness of a study and may trigger
a rapid rejection process.
1.7 It is for this reason that many feel that
"true experts" may be too close to the work to produce
a balanced opinion. While their technical competence cannot be
disputed, their proximity to the research area and their competitiveness
may be destructive.
1.8 Although in the last decade there have been
a number of research studies addressing some aspects of the peer
review process, many would still argue that it has been insufficiently
tested by objective measures. There is evidence, for example,
that the outcomes of peer review are dependent on age, gender,
language skills and geographic location, all of which have been
shown, to some extent, to influence the outcome of the peer review
process.
1.9 By and large these variables are not taken
into account when routinely selecting peer reviewers. Most journal
editors and funding agencies will be aware that, within the scientific
community, there are "hawks" and "doves".
In the recent past, editors of some elite science journals have
been heavily criticised for their own bias perhaps in selecting
reviewers who would produce the answer that they wanted. In a
quote from the Wall Street Journal it was said that "Nature
and Science are locked in such fierce competition for prestige
and publicity that they may be cutting corners to get 'hot' papers".
The Nobel Laureate, Robert Laughlin, commenting on a series of
retractions from these eminent journals said "in this case
the editors are definitely culpable… they chose reviewers
they knew would be positive".
1.10 There are still uncertainties as to how
many reviewers should be invited to comment on a grant proposal
or a research publication. Statisticians might suggest that statistically
viable results would require at least six opinions, all agreeing
to either reject or accept. It is uncommon for as many as six
reviewers being asked to assess a paper for publication. Journal
editors will differ in the number of reviewers they engage and
at the end of the day, most journals give editors the responsibility
for the final decision so he or she is at liberty to accept or
reject the advice of reviewers. It is not a democratic process.
1.11 Peer review is an expensive, time-consuming
process that is largely unrewarded and unrecognised. Many scientists
will review large numbers of papers each year and many will serve
on review boards for funding agencies which, again, takes them
away from their research and other responsibilities.
1.12 Despite all the shortcomings of the peer
review process, there is a real danger in placing research findings
in the public domain that have not been quality assured. The possibility
for the commercial exploitation of poor science is already evident
and there is still a tendency for some questionable findings to
be published in the national press before having undergone scrutiny
by experts in the field. Peer reviewers do not always get it right
but the public deserves scientific findings that can have such
a profound effect on society to be quality assured by a gateway
process managed by experts.
2. Measures to strengthen peer review
2.1 Clear guidance on what is expected of a peer
reviewer:
Many journals and funding agencies provide instructions
to peer reviewers but there is a lack of consistency and, in some
cases, a paucity of detail. Reinforcement of the necessity to
understand that this is a confidential process and, in effect,
the peer reviewer makes a contract with both the editor/funder
and the applicant. A clearer understanding of the variables that
influence the outcome of peer review need to be emphasised as
is the necessity of disclosing all conflicts of interest.
2.2 Training:
The knowledge and skills required to conduct high
quality peer review are usually passed on in an informal way.
Peer reviewers in the future perhaps should be required to undertake
a period of training which could be largely online but might involve
at least one face-to-face group session (possibly via a "chat
room") which could be case based.
2.3 Selection of reviewers:
There is some evidence that younger peer reviewers
produce a more thorough review. There is evidence that editors
can bias the outcome of the peer review process by selecting particular
individuals who they know will produce the answer they require.
Most journals have a well-established database of reviewers which
can be searched by the area of special expertise. It might be
argued that, within disciplines and sub-disciplines, peer reviewers
might be selected at random to avoid editorial bias.
2.4 The majority of peer reviewers are not paid
and their work is largely unrewarded. Most scientists accept that
peer reviewing each other's work is part of the job and many will
spend a considerable number of hours every year undertaking the
process. As yet there is no satisfactory way of identifying this
contribution as a serious part of the research and publication
process. One of the advantages of "open" peer review
is that the name of the reviewer could be published in the scientific
journal along with the authors and it has been suggested that
the reviewer's reports might also be available with the online
version paper. Contribution to the peer review process might be
formally recognise as part of an academic work and the taken into
account as a criterion for promotion.
2.5 "Open" peer review:
This approach is used by an increasing number of
journals and effectively means that, not only do reviewers know
the identity of the authors, but the authors know the identity
of their reviewers and reviewers are invited effectively to sign
their peer review assessments. Many feel that this leads to greater
openness in the peer review process and leads to greater assurance
that reviews will be evidence based. There is certainly evidence
that reviews conducted under this system are "more constructive
and courteous" but no evidence as yet that it improves the
quality of the reviews. Further research is clearly required to
determine whether this is indeed the way forward. Some younger
reviewers, however, are concerned about open peer review as they
feel their careers may be at stake if they produce harsh reviews
of the work of established giants in the field.
3. Value of peer reviewed science on advancing
and testing scientific knowledge
The volume of research and research publications
has increased almost exponentially over the last 50 years. Most
scientists feel that some pre-selection of published work is required
as it would be impossible to perform this function on an individual
basis. The peer review process also improves the final published
work often making it shorter, more focussed and reduces the chance
of technical errors, both in the science and in the published
article.
4. Peer review in informing public debate
The public inevitably relies on experts to pre-select
and quality assure scientific data that are placed in the public
domain. A Mori survey in 2004 clearly showed that the public have
a poor understanding of the peer review process but many want
peer scrutiny and want scientific findings to be replicated before
findings are widely trailed in the public domain.
5. Differences in peer review between scientific
disciplines and between different countries
There are differences in the peer review process
across the scientific disciplines. The peer review process in
life and biomedical sciences generally follows the standard approach
described above. However in mathematics and to some extent in
physics peer review often takes place in an open and transparent
manner often as a collective activity. A good example is the way
in which the solution to Fermat's Last Theorem was arrived at.
Similarly the peer review process for the 'big experiments' in
physics are again often conducted as a collective event in an
open and transparent manner. The quality of peer review almost
certainly varies globally and from Journal to Journal.
6. Identification of peer reviewers
Most journals will construct a large database of
potential reviewers whose special interest and expertise has been
identified prior to entry onto the database. These databases are
searchable by editors and editorial staff and the final selection
made of between two and four reviewers being the norm. It is not
uncommon for reviewers to decline to review a paper in which case
the editor will return to the database and seek alternatives.
Reviewer databases are usually refreshed on a regular basis. Tardy
or ineffective reviewers may be removed and new and active researchers
added.
Multidisciplinary research papers may require a larger
number of reviewers to reflect the multidisciplinary approach
of the research.
7. Impact of IT on the peer review process
Modern information technology has revolutionised
the publication process. The majority of major scientific journals
have moved entirely to an online submission and review process.
This paperless process has reduced the time to publication which
has been largely due to the speeding up of the peer review process.
8. Possible alternatives to peer review
8.1 The most obvious alternative to peer review
is to publish all research studies and allow the scientific community
to decide whether a particular study is of high quality and whether
it contributes significantly to the body of knowledge. This could
work in the same way as the now popular social media sites, including
the online versions of daily newspapers and magazines which encourage
comments. Some of the weekly medical journals such as the British
Medical Journal encourage rapid responses to publish papers and
other articles which stimulate an interesting debate.
8.2 With the assent of open access online journals
it was anticipated that many of these would dispense with formal
peer review prior to publication and allow the scientific community
to decide. The majority however have retained a formal peer review
process before e publication. It has been suggested however that
an open online peer review process could take place during a period
of say four - six weeks following which an editorial decision
could be made as to whether the paper should be formally published.
8.3 During the past 30 years there has been expansion
of the number of scientific journals available to authors and
a substantial increase in the number of published papers. If there
was a viable alternative to peer review then one might have expected
expansion to have driven change. This suggests that as yet there
is no viable alternative to the established peer review process
and that we have to accept it as a relatively expensive, time-consuming
and imperfect process.
DECLARATION OF
INTERESTS
This submission draws upon the views of the Board
of the UK Research Integrity Office (UKRIO) and its staff. These
include persons who have: undertaken research which was then submitted
for publication via the peer review process; acted as peer reviewers;
acted as editors of academic journals which operate a peer review
process; and/or hold senior roles in institutions such as universities
which commonly disseminate their research via peer reviewed journals.
The Committee on Publication Ethics, a forum for editors and publishers
of peer reviewed journals to discuss all aspects of publication
ethics, has supported UKRIO and its Chair holds a seat on our
Board. UKRIO has received funding from bodies that fund research
projects which are commonly disseminated via peer reviewed journals,
such as Research Councils UK. None of the bodies which fund or
support UKRIO had any input into the content of this submission.
ABOUT THE
UK RESEARCH INTEGRITY
OFFICE
The UK Research Integrity Office (UKRIO), established
in 2006, is an independent body which offers confidential and
expert advice and guidance about the conduct of research, covering
all subject areas. It helps research organisations, individual
researchers and members of the public. UKRIO also publishes guidance
on good research practice and the investigation of alleged misconduct,
and operates a help-line service where concerns can be reported
in complete confidence. UKRIO is not a regulatory body and has
no formal legal powers. The advice and guidance it offers is not
mandatory but reflects best practice in the conduct of research
and addressing misconduct. Further information about UKRIO is
available from its website: www.ukrio.org .
UK Research Integrity Office
10 March 2011
| 
