Peer review

Written evidence submitted by the
UK Computing Research Committee (PR 86)

The UK Computing Research Committee (UKCRC), an Expert Panel of the British Computer Society, the Institution of Engineering and Technology and the Council of Professors and Heads of Computing, was formed in November 2000 as a policy committee for computing research in the UK. Its members are leading computing researchers from UK academia and industry. Our evidence reflects the experience of researchers who each have an established international reputation in computing.

Measures to strengthen peer review

1. Good peer review requires good reviewers. However, the increasing volume of research results and the growing pressure for rapid turnaround for paper reviews places strain on the system. The best peer reviewers may be overwhelmed with reviewing obligations, especially in "hot" areas. This leads to situations where the community must sacrifice the volume of results subjected to top-quality review (with strong filters on entry to the reviewing process and/or longer turnaround times) or sacrifice the quality of reviews (with more of the reviewing delegated to less experienced subordinates and/or with superficial reviewing due to time constraints). Many measures may be taken to reduce this problem: paying reviewers/editors so they can afford more time; automating routine tasks such as plagiarism checking; developing different review processes that rely less on single opinions. Nevertheless, the tension between volume of results versus quality of review remains.

2. Computing technology enables publication of results more rapidly, more broadly and (sometimes) more openly. With this, however, comes new demands on the process of peer review. We need better ways of understanding and recording the provenance of published results and (in more open systems of reviewing) of authenticating authors and reviewers. This is not an area where a simple technical fix suffices because peer review is a social process, driven by social incentives.

3. Related to issues of provenance and authentication is the issue of anonymity within the peer review process. Traditional peer review is often based on the idea of a fair and knowledgeable chairperson who ensures that peer reviews supplied anonymously are handled fairly and with appropriate inspection/discussion during the review process. One can challenge this approach, since anonymity can act against open scrutiny of the review process. Alternative methods of open peer review, often made possible through computing technology, are therefore being promoted. However, there remain strong differences of opinion across the computing research community on the comparative merits and effectiveness of differing approaches.

4. IT systems now underpin much of the peer review process so the design of these systems becomes a key issue. Where IT systems are introduced to support peer review, they should be developed to ISO 13407:1999, Human-centred design processes for interactive systems.

The value and use of peer reviewed science on advancing and testing scientific knowledge

5. Peer review is a recognised approach to ensuring quality, with recognised limitations. New approaches have unknown limitations, and may be more susceptible to undetected fraud and manipulation.

6. A challenge encountered in all peer reviewing is in reviewing proposals for unconventional or controversial research. Peer review, being essentially a consensus process, can tend to be naturally conservative. This effect is particularly important in peer review of proposed research (which is outside the scope of this consultation) but also is important when reviewing the results of research since these influence future work.

7. It is common for conferences and journals to refuse resubmissions after a peer review rejection. This reduces load on reviewers but makes the dialogue between reviewers and authors short. It also naively enforces the assumption that science arrives fully-formed and correct.

The value and use of peer reviewed science in informing public debate

8. Conventional peer review, when rigorous, can provide a gold standard in public debate. There is a balance to be struck, however, between conservative reviewing and reviewing with the aim of stimulating speculative research as described at point 6, above. Peer review is not based upon a monolithic model and material likely to be controversial or of public interest should be reviewed with that in mind.

The processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases

9. Reviewers normally are experienced in their specialisms but rarely are trained, let alone certified, in effective reviewing. Although it may be impractical to engage in wholesale training of reviewers, it would be helpful to raise awareness of good practice and potential pitfalls. For example, few reviewers may be sensitive to the issue of confirmation bias (a psychological effect whereby once a decision has been formed, the reviewer selectively seeks out confirmation of that view).

10. Conventional peer review cannot operate without editors who need to be on average more knowledgeable than the referees: they select the referees and then interpret the comments and recommendations, synthesise the potentially divergent views of the referees, propose a decision and recommendation, and communicate it to the authors of the research. Editors also mitigate the potentially competitive nature of the relationship between the referee and the author of the material that is being refereed.

11. Some simple but effective ways of improving peer review processes deserve wider use. For example, a simple way to improve traditional reviewing is to distribute reviews after submission between all reviewers, exposing each reviewer to the thoughts and insights of others; then, the author’s rebuttal is distributed to the reviewers and the reviewers given opportunities to change their minds.

12. A related issue is the impact of peer review on research careers. Many institutions heavily weight career progress on the output of peer-reviewed work, which can lead to over publishing of small advances. While publication is the foundation of scientific research, career development should also give equal weight to factors such as the influence of previous work (did it actually change anything and by what degree) and the level of ambition in the researcher’s portfolio.

13. Computing research is inherently diverse because it now overlaps with many other disciplines and it is expanding rapidly. Although this is a sign of the vigour of the field, it sometimes can make it difficult to identify and maintain large and stable peer groups from which to draw reviewers.

The impact of IT and greater use of online resources on the peer review process

14. IT does several important things. (1) For publishers, it externalises costs. For example, peer reviewers now print documents at their own cost while authors may be required to do most/all of the typesetting of articles. (2) It loses all "affordance" of physical objects. Physical (paper) page limits are less of a constraint so "size" of a publication and "structure" of presentation of results can be understood differently. This also can make peer review obligations less tangible, since the physical reminders of a paper-based process are removed. (3) It enables publishers to make rapid assessments of the suitability of reviewers (via registers of expertise, etc). Some publishers (e.g. Nature) e-mail many potential reviewers soliciting peer review. All invest time in making decisions, but the journal only uses a few of them. Thus the journal gains at the expense of wasting unused peer review time. (4) It provides the potential for anyone to "publish" results, creating demand for forms of post-publication peer review and, alongside this, greater pressure for transparency in the review process.

15. Much published work is not fully reproducible because results may depend on computations performed on unpublished programs operating within computational systems that change over time. It would be difficult to eliminate this issue but more can be done to reduce it. Online resources should include programs, not just data. Authors should make increasing use of public domain resources. This is an issue not only for computer science but across the traditional sciences, which increasingly rely on results generated via computation.

16. Computer science continues to advance and, as it does so, it is likely to continue to provide opportunities to change the shape of peer review, sometimes in fundamental ways. There is a need for greater understanding of the means by which advanced computer science could change peer review and of the ways this could interact with human and social aspects of review processes.

Possible alternatives to peer review

17. Science advances particularly by correcting and developing from prior mistakes, misunderstandings, lack of generality and error, and by converting early inspiration (even if initially misguided) into rigorous ideas. Obviously, "correct" landmark papers are highly desirable, but they are rare. Peer review should not attempt to make landmarks even better and harder to criticise, but attempt to make the process more effective.

18. Pressure on career-minded academics to publish in highly rated journals places disproportionate strain on review panels of top-ranked journals, whereas less well-known but open access journals may in some areas produce greater post-publication feedback to authors. If, as seems likely, this leads to a more heterogeneous mixture of styles of publication and reviewing, we will have a greater need for those assessing the contribution of academics to scrutinise the content of the contribution as well as the reputation of the journal in which it was published.

19. Attempting to make peer review better in isolation reinforces the view that science aims towards perfection. In fact, encouraging peer review to be part of a fallible process might (arguably) improve science. For example, some web journals (e.g., WebmedCentral, Public Library of Science, etc) have started publishing prior to peer review, but with an on-going obligation on the author to revise in response to reviewer’s comments.

20. Occasionally some conventional paper journals "retract" problematic papers. IT can help to avoid this, as it makes retraction just part of a wider spectrum of possibilities. If papers were more often followed up with criticism, whether peer reviews or further work from peers, then perhaps the spectrum of generalising, fixing weaknesses, to indentifying flaws could be used more fully.

Declaration of interests

21. UKCRC is an expert panel of the Institution of Engineering and Technology and the BCS for computing research in the UK. Its membership and constitution is described at www.ukcrc.org.uk.

22. All contributors to the UKCRC response to the Select Committee are researchers who publish in (at least) IT and computer science. Some are editors of journals. Most are on review panels and are funded by agencies that use peer review.

UK Computing Research Committee

14 March 2011