Peer review in scientific publications - Science and Technology Committee Contents


Written evidence submitted by Professor Roger Jones (PR 23)

1.  INTRODUCTION

I am writing this response to the Inquiry as a primary health care journal editor with over 30 years' experience of the peer review system as a researcher, author, reviewer and editor and also someone who has extensive experience of research grant funding through peer review and the organisation of scientific conferences, through peer review of submitted abstracts. I have seen drafts of one or two other responses to the Inquiry and, rather than re-iterate points that have already been made by others, I offer a few additional comments which I hope will be useful.

2.  The Place of Peer Review

2.1  Peer review has to be seen as part of a selection process—an important part, but only one of the factors that journal editors take into account when selecting papers for publication, such as timeliness/novelty, "fit" with other papers/themes in forthcoming issues of the journal, demands on space and editorial resources (notwithstanding the opportunities afforded by web publication—on line papers still require subeditorial work), newsworthiness and also commercial considerations—perhaps sometimes implicit and inappropriate, and related to the needs to attract advertising and sponsorship. "Hanging committees" are often used to make final reject/accept decisions on groups of high-quality papers which all have a claim to publication.

2.2  Judiciously-chosen reviewers are essential to journal editors, who cannot be expected to possess the expertise in depth to make selection decisions on everything sent to them; editors also frequently depend on their reviewers to be sure of originality and novelty, as well as the veracity and robustness of the study and the paper, and also to act as monitors of dual/duplicate and salami publication and sometimes the inappropriate inclusion of certain authors (gift/ghost authorship).

3.  THE CONDUCT OF PEER REVIEW

3.1  The way that peer review is carried out varies widely across journals in relation to almost all it's components—including blinding of reviewers to the identity and institutions of submitting authors, the instructions given by the journal to reviewers in terms of their reports and advice to editors, blinding authors to the identity of people who have reviewed their work, the availability and conduct of an appeals procedure for authors who feel hard done by or misunderstood, the incentives (eg free online journal access, money or a gift such as a CD) offered to reviewers, feedback to reviewers about the outcome of submissions and also feedback to them about the quality/utility of their reviews. As I discuss later, the evidence base on which editors are able to make decisions about some of these aspects of peer review is decidedly thin.

3.2  Useful resources for medical journal editors include the guidance offered by the International Committee of Medical Journal Editors (aka the Vancouver Group) and COPE, the Committee on Publication Ethics.

4.  PROBLEMS WITH PEER REVIEW

4.1  Perhaps the most difficult problem with peer review is that it is not "peer", in the sense that there is often a big difference between age, experience and seniority of the reviewer and the reviewed, which immediately introduces some potentially distorting forces eg when, in an open review system a junior reviewer is asked to review a bad paper by a senior with whom he may have an indirect but potentially career-relevant relationship. In open review, "peerness" itself may turn out to be a problem, as in the almost inevitable scenario of competitive experts reviewing each other's papers while wanting to be first in print. Other perverse incentives to be excessively hawkish or dove-ish are not hard to conjure up.

4.2  The selection and training of and feedback to reviewers have been mentioned—knowing how best to frame the review request to get the best out of reviewers and to avoid disparate advice is not easy—and variation between reviewers in their accept/reject, definitely/possibly recommendations is commonplace; this does not necessarily represent differences in the quality of the reviews, but can be related to the precision or otherwise of the editor's question.

4.3  Peer review of abstracts submitted to (international) scientific conferences is a particular problem. Abstracts accepted only as posters may appear in print and be citeable, so quality assurance is important. Yet conferences are big money-spinners and there is a considerable commercial incentive to maximise acceptance of submitted abstracts to maximise attendance—many institutions will only cover costs of staff going to conferences when their abstract has been accepted.

5.  THE PEER REVIEW RESEARCH AGENDA

5.1  There is some research—peer reviewed of course—in the literature about the effectiveness and problems of peer review, but much more needs to be done—by journals themselves, learned societies and government—for which funding should be identified.

5.2  Some topics from which pertinent research questions could be developed include:

—  The benefits and dis-benefits of open and blinded review processes.

—  The value of feedback on their performance to reviewers to improve the quality and usefulness of their reviews in future.

—  The value of training reviewers: clarification of the mission of the journal and the characteristics of the "ideal" review.

—  The use of structured/unstructured review inventories and structured/unstructured/categorical advice to editors on quality and suitability for publication.

—  The fate of papers rejected after peer review—do reviewers' comments help authors get published elsewhere and what does that say about the original "reject" decision?

6.  DECLARATION OF INTERESTS

I am the Editor of the British Journal of General Practice and have an interest in improving the quality of the peer review process. I am interested in conducting research in to peer review.

Professor Roger Jones MA DM FRCP FRCGP FMedSci FHEA
Editor, British Journal of General Practice
Emeritus Professor of general Practice, King's College London

3 March 2011


 
previous page contents next page


© Parliamentary copyright 2011
Prepared 28 July 2011