Written evidence submitted by Professor
Ian A Walmsley, Pro-Vice-Chancellor
(Research, Academic Services and University Collections), University
of Oxford (PR 73)
MAJOR POINTS
1. Peer review is central to effectively assessing
the quality of research.
2. Peer review has been (and is) fundamental
to the UK research/science system, and its successes.
3. Peer review of funding proposals makes use
of an expert knowledge base to identify the best research. We
recognise that it is imprecise, because of the necessarily uncertain
capacity of predicting outcomes based on untested ideas. But as
part of a research eco-system of sufficient capacity it is effective
and indispensible. The alternatives, based on non-expert review
or "top-down" delineation of activity are vastly inferior
at producing transformational ideas.
4. Peer review is the best, though of course
not perfect, system to provide an assurance that what is published
can be relied upon.
5. Peer review does place workload pressures
on researchers as reviewers. However, the vast majority of researchers
are willing to take on this work to ensure the best selection
method, viz. peer review, can operate fairly and effectively.
6. There is now quite a lot of evidence as to
the practical issues which need to be tackled to make the review
of funding proposals and of work submitted for publication fairer,
and more effective and efficient. Oxford has supported, and will
continue to support, efforts to address these issues.
FURTHER DISCUSSION
Peer review of funding proposals
7. The major funding bodies in the UK, incl.
the research councils and biomedical charities, all use peer review
for advice on which research projects should be funded in the
first place, and often to assess the progress of funded projects/programs.
8. The Association of Medical Research Charities
(AMRC) considers that peer review is the best way for charities
to select which research to fund. One of its membership criteria
is that all AMRC member charities must seek expert advice from
external reviewers to help them make decisions about which research
grant applications they should fund. AMRC produces guidelines
and information on peer review and offers training on this topic.
9. The UK Research Councils state that their
policy is to "fund research on a competitive basis employing
independent expert peer review. This system is regarded as an
international benchmark of excellence in research funding, and
this provides a guarantee of the quality of UK research."[21]
10. Proposals for research funding should be
assessed for scientific quality by a number of senior academics
or "peers", from the UK and overseas, who work within
relevant areas of research. This assessment or "review"
provides the basis of the funding decision. This approach does
not exclude others from participating in decisions, but does emphasise
the importance of assessment by experts in the research area.
11. Interestingly, at the National Institutes
of Health (NIH), one of the world's foremost agencies conducting
and supporting biomedical and behavioural research,
the peer review system is mandated by statute
in accordance with section 492 of the Public Health Service Act
and federal regulations.
12. In late 2007 to 2010 the NIH undertook a
far-reaching review of its peer review system. A series of reports
and enhancements to the system focussed on three "Implementation
Goals":
— To
Engage the Best Reviewers.
— To
Improve the Quality & Transparency of Review.
— To
Ensure Balanced and Fair Reviews.
13. NIH's review has been extremely thorough
and highly consultative. It found no basis for recommending any
dilution of the peer review principle at all, including for multi
and inter disciplinary proposals.
14. The changes arising from the review have
sought to improve the scoring system, the review criteria, the
clustering of proposals, as well as the application forms (esp.
to more clearly align the questions to the review and selection
criteria). A December 2010 report on the most recent survey of
key stakeholder groups—NIH grant applicants, NIH peer reviewers,
Scientific Review Officers, Program Officers and Advisory Council
members—indicated higher ratings of the system in terms
of "satisfaction" and "fairness".[22]
15. What is clear from policy review work such
as by the NIH, the Australian Research Council's consultations
on "Peer Review Processes" (2009-10),[23]
the RCUK Efficiency and Effectiveness of Peer Review Project[24]
and the Parliamentary Office of Science and Technology paper on
Peer Review, (2002),[25]
is that peer review is held to be beneficial to the scientific
community and has become central to the process by which science
is conducted.
16. Yes there are strains and tensions. For example,
from time to time one hears that the peer review system is "close
to breaking point" given the demands being placed on researchers
to act as reviewers and serve on grant review panels and awarding
committees. It is true that workload is an issue. However, the
vast majority of researchers accept that there is a price to pay
to enable the best selection method, viz. peer review, to operate
fairly and effectively, and are willing to pay it.
17. Policy analysis by major research funders
and academic research on peer review (incl. from sociological,
psychological, economic and other perspectives) has helped to
identify some of the difficulties with peer review. Even more
importantly, there is increasing experience of ways in which these
can be addressed. Many of the issues apply both to reviewing work
for publication and assessing grant/funding proposals and include:
— Potential
gender bias.
— Conflicts
of interest.
— Recruiting
and retaining referees.
— Training.
— Clarity
of review criteria.
— Opportunities
to rebut criticisms.
— Assessing
multi- and inter-disciplinary research.
18. The experiences of the NIH community of reviewers[26]
echo those of many Oxford colleagues who express frustrations
with, eg:
— "Clunky"
on-line systems.
— Reviewers
being asked questions that, at least to their thinking, bear little
or no obvious relationship to the selection criteria.
— Reviewers
not being clear why they had been asked to review certain proposals.
— Mismatches
in the allocation of referees, where it appears that the information
about the potential "pool of referees" has not been
sufficiently detailed or accurate to select the right referees.
— Reviewers
not receiving any training (on-line or other).
— Reviewers
not receiving any feedback on their reviews.
19. Practical issues such as these can be addressed
by funding agencies, and improvements can be made.
Peer Review of Work for Publication
20. From their beginnings in the mid-17th century,
scientific journals were subjected to criticism about the quality
of what they put into print. Thus from the outset they began to
develop referee systems for the express purpose of controlling
the quality of the papers accepted. The result was an institutionalized
mechanism for the application of standards to scientific work,
which has changed little in the ensuing centuries.[27]
21. As with peer review and funding decisions,
there is now quite a lot of evidence as to the practical issues
which need to be tackled to make the publication system fairer,
and more effective and efficient.
22. It is important that all reviewers uphold
the highest ethical standards.
23. The World Association of Medical Editors
(WAME) has proposed a comprehensive policy which addresses all
the major areas of publication ethics that contemporary science
journals should consider, including:
— Conflict of Interest.
— Study Design and Ethics.
— Authorship.
— Peer Review.
— Editorial Decisions.
— Originality, Prior Publication, and Media Relations.
— Plagiarism.
— Advertising.
24. As WAME observes, peer reviewers are experts
chosen by editors to provide written assessment of the strengths
and weaknesses of written research, with the aim of:
— improving
the reporting of research, and
— identifying
the most appropriate and highest quality material for the journal.
25. Reviewers should be required to meet minimum
standards (as determined and promulgated by each journal) regarding
their background in original research, publication of articles,
formal training, and previous critical appraisal of manuscripts.
Reviewers should be selected for their objectivity and scientific
knowledge and their reviews professional, honest, courteous, prompt,
and constructive.
26. Whilst peer review is commonly accepted as
an essential part of scientific publication, practices vary across
journals and disciplines. Debates continue as to the best method(s)
of peer review, the value-added, the ethics of the review process
and how can new technology be used to improve traditional models.[28]
And new approaches are emerging. The journal Atmospheric Chemistry
and Physics (ACP) is an example of "interactive open
access peer review" based on a two-stage process of publication
and peer review combined with interactive public discussion. One
sees the original manuscripts, any background papers, comments
by peer reviewers and editors, dialogue with authors and revisions
(if any), and there is a distinct section for open, public reaction
and comment.
27. What remains essential is that the "users"
of research have confidence in the quality and integrity of both
the research and the peer review process.
Professor Ian A Walmsley
Pro-Vice-Chancellor
(Research, Academic Services and University Collections), University
of Oxford
10 March 2011
21 http://www.rcuk.ac.uk/research/peers/Pages/home.aspx Back
22
http://enhancing-peer-review.nih.gov/docs/Enhancing_Peer_Review_Report.pdf Back
23
See eg http://www.arc.gov.au/general/peer_consultation.htm Back
24
http://www.rcuk.ac.uk/research/peers/Pages/vfmpeer.aspx Back
25
http://www.parliament.uk/documents/post/pn182.pdf Back
26
See http://enhancing-peer-review.nih.gov/continuous_review.html Back
27
http://www.the-asci.org/addresses/asci1978.pdf Back
28
See Nature's Peer Review Debate - http://www.nature.com/nature/peerreview/debate/index.html Back
| 
