Peer review

Written evidence submitted by Jonathan Sturgess (PR 80)

General comments

1. I review papers in my own time – my employer does not allocate time for this in my working week. I do not receive payment for reviewing papers. In one case (COMPEL) I receive a free subscription to the journal. For the rest, I have to subscribe to the journals whose papers I review. In all cases, I have to purchase myself any papers that are referred to in a paper that I am reviewing that I might wish to check out.

2. I will confine my comments to personal experience, i.e. reviewing for engineering journals especially in computational electromagnetics. Papers in this field are usually concerned with developing a method for solving a particular engineering problem and/or are validating a theory against measurement, experiment or an alternative method of calculation.

3. In many cases, no validation is presented. Where it is, it is often against a single example. I have almost never seen a paper that compares theory against a large sample of measurements and which applies statistical methods to assess the accuracy to which the theory fits the data.

4. The peer reviews that I undertake have at least three reviewers looking at the paper simultaneously.

The strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

Strengths

5. Peer review draws on the broad experience of practitioners in the field. In many cases they will have addressed similar problems, will know the difficulties and pitfalls and will be able to see whether the paper addresses these satisfactorily.

6. Having several reviewers for each paper helps stop plausible-sounding rubbish getting through. It provides a broader appreciation and a wider perspective on the use and merits of a paper than could be provided by a single reviewer.

7. Journal publication is a competitive business. Journals care about their reputation for publishing useful, timely papers of good quality. Peer review is part of ensuring this.

8. Peer reviewers can look at papers from the readers’ point of view (is it well written, well explained, useful; does it contain useful information/data?)

8. The review process includes "Comments to the authors". By this means reviewers can make constructive comments for improvements to the paper prior to publication. It is quite common for a paper to be marked as "Publish after revision" on a first review. From time to time a paper will be substantially re-written (at the request of the reviewers) before it is published.

Weaknesses

9. The honorary editor selects the reviewers to use for a particular paper and makes the final decision about publication. This places a lot of power and responsibility in the honorary editor’s hands. The honorary editor is an amateur, doing the job part-time. He or she may not be trained for the job and may be under time pressure both from the publisher and from the "day job". This could lead to hasty, ill-considered decisions based more on the reputation of the authors and their institution than on the merits of the particular work.

10. Journal publication is a competitive business – there is pressure to keep the time from paper submission to publication to a minimum. Authors want to see their work in print a.s.a.p. and will select a journal to submit their paper to partly on this basis. This can rush the review process.

11. Just because a paper is peer-reviewed doesn’t mean that it is good, or correct. Peer review cannot validate the work presented, although it can spot obvious errors.

The value and use of peer-reviewed science in advancing and testing scientific knowledge

12. "A day in the lab. can often save an hour in the library". Peer-reviewed science and engineering is essential for the development and dissemination of ideas, methods and techniques. It is one of the principal means of passing knowledge from academia to industry. Peer review gives a degree of assurance of quality, thoroughness and independence to the work.

13. Peer review is a process through which the quality of the communication of knowledge is improved. This is an important aspect. Accurate and efficient communication of an idea is as important to the advancement and testing of knowledge as the idea itself.

14. Peer review is a process that can identify weaknesses in a piece of work and flaws in an argument. It provides an opportunity for these to be addressed before publication, thus strengthening the work.

The value and use of peer-reviewed science in informing public debate

15. The general public often assumes that peer review is peer verification. This error gives a false confidence is reports based on peer-reviewed papers.

16. The public is (rightly) concerned by the close relationship between authors and reviewers. How can they be independent when they all know each other? Doesn’t that present a conflict of interest? [1]

17. Peer-reviewed science and engineering helps provide scientific thought and opinion that is independent of special interest groups (government, business etc.)

The impact of IT and greater use of online resources on the peer review process

18. Papers are sent out electronically and reviews returned electronically. This speeds the review process.

19. Online resources might trap plagiarism, but not incorrect analysis or inappropriate conclusions.

20. The instant availability on-line (at a cost) of many of the papers that are listed as references in a paper helps the review. The ability to search the internet allows the reviewer to obtain a wider perspective on the topic under review.

Possible alternatives to peer review

21. One option is a "A publish everything and let the reader decide" approach. This is a problem for the novice in the field. What’s good and what’s rubbish? It is more of a problem for industry who wants to research a topic without becoming an expert in the subject. They rely on the peer review process to guarantee a degree of quality. I would be unlikely to subscribe to, or take much notice of, a journal that was not peer-reviewed.

22. "Cloud Reviewing" – where a paper is published on line and rated by its readers in much the same way as goods and services are reviewed and rated. The problem with this approach is that not all reviews would be equally valuable. If anyone can post a review, then the reader is left to work out for themselves whether the review is made from a position of deep understanding or just passing (possibly un-informed) interest. A further drawback would be the question of moderating the reviews. Who would do that? Such a system would also waste a lot of a researcher’s time wading through reviews of a paper attempting to build up a picture of the quality of the paper from many partial opinions of variable and unknown quality.

Jonathan Sturgess

10 March 2011