Peer review in scientific publications - Science and Technology Committee Contents


Written evidence submitted by Professor Ian A Walmsley, Pro-Vice-Chancellor (Research, Academic Services and University Collections), University of Oxford (PR 73)

MAJOR POINTS

1.  Peer review is central to effectively assessing the quality of research.

2.  Peer review has been (and is) fundamental to the UK research/science system, and its successes.

3.  Peer review of funding proposals makes use of an expert knowledge base to identify the best research. We recognise that it is imprecise, because of the necessarily uncertain capacity of predicting outcomes based on untested ideas. But as part of a research eco-system of sufficient capacity it is effective and indispensible. The alternatives, based on non-expert review or "top-down" delineation of activity are vastly inferior at producing transformational ideas.

4.  Peer review is the best, though of course not perfect, system to provide an assurance that what is published can be relied upon.

5.  Peer review does place workload pressures on researchers as reviewers. However, the vast majority of researchers are willing to take on this work to ensure the best selection method, viz. peer review, can operate fairly and effectively.

6.  There is now quite a lot of evidence as to the practical issues which need to be tackled to make the review of funding proposals and of work submitted for publication fairer, and more effective and efficient. Oxford has supported, and will continue to support, efforts to address these issues.

FURTHER DISCUSSION

Peer review of funding proposals

7.  The major funding bodies in the UK, incl. the research councils and biomedical charities, all use peer review for advice on which research projects should be funded in the first place, and often to assess the progress of funded projects/programs.

8.  The Association of Medical Research Charities (AMRC) considers that peer review is the best way for charities to select which research to fund. One of its membership criteria is that all AMRC member charities must seek expert advice from external reviewers to help them make decisions about which research grant applications they should fund. AMRC produces guidelines and information on peer review and offers training on this topic.

9.  The UK Research Councils state that their policy is to "fund research on a competitive basis employing independent expert peer review. This system is regarded as an international benchmark of excellence in research funding, and this provides a guarantee of the quality of UK research."[21]

10.  Proposals for research funding should be assessed for scientific quality by a number of senior academics or "peers", from the UK and overseas, who work within relevant areas of research. This assessment or "review" provides the basis of the funding decision. This approach does not exclude others from participating in decisions, but does emphasise the importance of assessment by experts in the research area.

11.  Interestingly, at the National Institutes of Health (NIH), one of the world's foremost agencies conducting and supporting biomedical and behavioural research, the peer review system is mandated by statute in accordance with section 492 of the Public Health Service Act and federal regulations.

12.  In late 2007 to 2010 the NIH undertook a far-reaching review of its peer review system. A series of reports and enhancements to the system focussed on three "Implementation Goals":

—  To Engage the Best Reviewers.

—  To Improve the Quality & Transparency of Review.

—  To Ensure Balanced and Fair Reviews.

13.  NIH's review has been extremely thorough and highly consultative. It found no basis for recommending any dilution of the peer review principle at all, including for multi and inter disciplinary proposals.

14.  The changes arising from the review have sought to improve the scoring system, the review criteria, the clustering of proposals, as well as the application forms (esp. to more clearly align the questions to the review and selection criteria). A December 2010 report on the most recent survey of key stakeholder groups—NIH grant applicants, NIH peer reviewers, Scientific Review Officers, Program Officers and Advisory Council members—indicated higher ratings of the system in terms of "satisfaction" and "fairness".[22]

15.  What is clear from policy review work such as by the NIH, the Australian Research Council's consultations on "Peer Review Processes" (2009-10),[23] the RCUK Efficiency and Effectiveness of Peer Review Project[24] and the Parliamentary Office of Science and Technology paper on Peer Review, (2002),[25] is that peer review is held to be beneficial to the scientific community and has become central to the process by which science is conducted.

16.  Yes there are strains and tensions. For example, from time to time one hears that the peer review system is "close to breaking point" given the demands being placed on researchers to act as reviewers and serve on grant review panels and awarding committees. It is true that workload is an issue. However, the vast majority of researchers accept that there is a price to pay to enable the best selection method, viz. peer review, to operate fairly and effectively, and are willing to pay it.

17.  Policy analysis by major research funders and academic research on peer review (incl. from sociological, psychological, economic and other perspectives) has helped to identify some of the difficulties with peer review. Even more importantly, there is increasing experience of ways in which these can be addressed. Many of the issues apply both to reviewing work for publication and assessing grant/funding proposals and include:

—  Potential gender bias.

—  Conflicts of interest.

—  Recruiting and retaining referees.

—  Training.

—  Clarity of review criteria.

—  Opportunities to rebut criticisms.

—  Assessing multi- and inter-disciplinary research.

18.  The experiences of the NIH community of reviewers[26] echo those of many Oxford colleagues who express frustrations with, eg:

—  "Clunky" on-line systems.

—  Reviewers being asked questions that, at least to their thinking, bear little or no obvious relationship to the selection criteria.

—  Reviewers not being clear why they had been asked to review certain proposals.

—  Mismatches in the allocation of referees, where it appears that the information about the potential "pool of referees" has not been sufficiently detailed or accurate to select the right referees.

—  Reviewers not receiving any training (on-line or other).

—  Reviewers not receiving any feedback on their reviews.

19.  Practical issues such as these can be addressed by funding agencies, and improvements can be made.

Peer Review of Work for Publication

20.  From their beginnings in the mid-17th century, scientific journals were subjected to criticism about the quality of what they put into print. Thus from the outset they began to develop referee systems for the express purpose of controlling the quality of the papers accepted. The result was an institutionalized mechanism for the application of standards to scientific work, which has changed little in the ensuing centuries.[27]

21.  As with peer review and funding decisions, there is now quite a lot of evidence as to the practical issues which need to be tackled to make the publication system fairer, and more effective and efficient.

22.  It is important that all reviewers uphold the highest ethical standards.

23.  The World Association of Medical Editors (WAME) has proposed a comprehensive policy which addresses all the major areas of publication ethics that contemporary science journals should consider, including:

—  Conflict of Interest.

—  Study Design and Ethics.

—  Authorship.

—  Peer Review.

—  Editorial Decisions.

—  Originality, Prior Publication, and Media Relations.

—  Plagiarism.

—  Advertising.

24.  As WAME observes, peer reviewers are experts chosen by editors to provide written assessment of the strengths and weaknesses of written research, with the aim of:

—  improving the reporting of research, and

—  identifying the most appropriate and highest quality material for the journal.

25.  Reviewers should be required to meet minimum standards (as determined and promulgated by each journal) regarding their background in original research, publication of articles, formal training, and previous critical appraisal of manuscripts. Reviewers should be selected for their objectivity and scientific knowledge and their reviews professional, honest, courteous, prompt, and constructive.

26.  Whilst peer review is commonly accepted as an essential part of scientific publication, practices vary across journals and disciplines. Debates continue as to the best method(s) of peer review, the value-added, the ethics of the review process and how can new technology be used to improve traditional models.[28] And new approaches are emerging. The journal Atmospheric Chemistry and Physics (ACP) is an example of "interactive open access peer review" based on a two-stage process of publication and peer review combined with interactive public discussion. One sees the original manuscripts, any background papers, comments by peer reviewers and editors, dialogue with authors and revisions (if any), and there is a distinct section for open, public reaction and comment.

27.  What remains essential is that the "users" of research have confidence in the quality and integrity of both the research and the peer review process.

Professor Ian A Walmsley
Pro-Vice-Chancellor
(Research, Academic Services and University Collections), University of Oxford

10 March 2011


21   http://www.rcuk.ac.uk/research/peers/Pages/home.aspx Back

22   http://enhancing-peer-review.nih.gov/docs/Enhancing_Peer_Review_Report.pdf Back

23   See eg http://www.arc.gov.au/general/peer_consultation.htm Back

24   http://www.rcuk.ac.uk/research/peers/Pages/vfmpeer.aspx Back

25   http://www.parliament.uk/documents/post/pn182.pdf Back

26   See http://enhancing-peer-review.nih.gov/continuous_review.html Back

27   http://www.the-asci.org/addresses/asci1978.pdf Back

28   See Nature's Peer Review Debate - http://www.nature.com/nature/peerreview/debate/index.html Back


 
previous page contents next page


© Parliamentary copyright 2011
Prepared 28 July 2011