Peer review

Written evidence submitted by Dr David Taylor (PR 11)

Declaration of Interest

1 I have 40 years experience of the evaluation, management and resolution of environmental issues in the heavy chemical, specialty chemical, agrochemical and pharmaceutical industries. Until recently (2008) I was the Global Director of Environment & Sustainability for the pharmaceutical company AstraZeneca, responsible, amongst other things for overseeing the direction of the company research programme in environmental science. I am a Fellow of the Royal Society of Chemistry, the Institute of Water and Environmental Management and a non-executive director of wca Environment Ltd.

2 I have written over 40 peer reviewed papers and have acted as a peer reviewer for a number of journals including the Analyst, Water Research and the Science of the Total Environment.

"Review by Peers" and "Peer Review"

3 Scientists make their work available to each other by public presentation, publication in the so-called "grey literature" and/or publication in scientific journals. Their colleagues then criticise, duplicate and/or overturn these conclusions and theories, subsequently making their own conclusions public and thus permitting the never ending cycle to continue.

4 This is the way science advances, often in fits and starts and frequently after having been diverted into blind alleys. It is this process of review by peers that is the basis of the quotation by Newton that what he had achieved had been done by "standing on the shoulders of giants". It is relatively rare for individual scientific papers to be of such seminal importance that they lead on their own to subsequent major advances. Most papers represent relatively modest improvements in scientific knowledge and the majority of scientific papers receive relatively few citations by other scientists. Thus new work builds on work produced by several previous workers.

5 It is important to recognise that in the long term it matters little if published material is inaccurate, incompetent or even fraudulent since the advance of the scientific canon only uses that material which turns out to fit the gradually emerging jigsaw. Incorrect pieces may find a place for a time, but will always be eventually replaced by the correct piece [1] .

6 In the short term however it is prudent to try to ensure that the scientific work that is published is accurate and many scientific journals endeavour to achieve this by using the relatively recent technique called "peer review" to evaluate drafts of scientific papers before they are published. There is nothing altruistic about this, journal publishing is a cut-throat business and it is in the self interest of journal editors to ensure that what is published under their name is of high quality.

7 However it needs to be borne in mind at all times that this type of peer review is merely a minor subset of the much more important review by peers that actually leads to the advancement of science.

What is "Peer Review" intended to achieve?

8 The journal editor usually has very specific requirements for the reviewer: the reviewer should determine if the work is novel, i.e. has it been published previously by others (or indeed by the present authors [2] )? , is the work described in sufficient detail for another scientist to be able to duplicate it and can the conclusions of the paper be logically derived from the data presented, including whether the statistical treatment of the data is appropriate?

9 Note that the reviewer is not asked to affirm that any theories or opinions presented in the paper are correct. The purpose of peer review is NOT to establish the validity of the science. This is the function of the much more extensive review by peers which follows publication

10 Also note that the reviewer is not expected to repeat the authors work. This is, in any case, completely impractical where experimental work is concerned but is not expected even where the study is simply related to the manipulation of data. The function of a peer reviewer is not the same as an auditor. (See later comments related to Good Laboratory Practice)

11 Publication in a Peer Reviewed journal is therefore not a guarantee that the theory is correct, or even that the information is valuable. The reverse is also true, publication in a non peer reviewed journal or the grey literature does not mean that the science is untrustworthy. Dr Wakefield’s paper on MMR and Autism, for example, was published in the highly prestigious and peer reviewed Lancet and the paper on cold fusion by Fleischman & Pons appeared in the similarly peer reviewed Journal of Electro analytical Chemistry. The paper by Crick and Watson on the structure of DNA did appear in Nature, but was not peer reviewed prior to publication. In contrast, the influential Stern Report on the economics of climate change is merely part of the ‘grey literature’ as indeed are the 29 reports of the Royal Commission on Environmental Pollution.

12 Peer Review should therefore lead to the reduction in the number of duplicate publications and to the improvement of the presentation of scientific work. However, it is not a mechanism for determining scientific "truth".

Problems with the Current Peer Review Process

13 Peer Review is an unpaid activity often undertaken in their spare time by busy scientists. In many cases the reviewer is only able to give a short and often cursory overview of the work in question. As a consequence the most able person capable of reviewing a new piece of work may be too busy or simply disinterested. Obvious problems are likely, but not certainly, to be detected. Subtle issues may be overlooked. This problem is becoming greater as the number of journals continues to increase.

14 Science is objective but scientists are human. Consequently all reviewers are to some degree affected by publication bias: the reviewers are likely to be much less critical of a paper which supports their own theory than one which conflicts with it. Similarly just because a paper is rejected by the reviewers (and editors) does not mean that the paper was incorrect. For example the journal Nature [3] has admitted that it had rejected papers that actually marked major scientific advances including work by Hawking on black hole radiation.

15 The editor needs to identify reviewers that are conversant with the scientific field in question. This has led to some journals, including Nature, asking the author of the proposed publication to recommend appropriate reviewers for their submission. This is not conducive to the production of objective reviews. The ideal reviewer is one who is knowledgeable and disinterested.

16 The reviewer does not have a veto on publication, only the editor has this sanction. On the one hand, this is a safeguard to prevent blackballing of papers by recalcitrant reviewers, but conversely it means that sometimes editors can overrule reviewers for purely commercial reasons. I terminated my own reviewing for one journal because the editor insisted on publishing inadequate papers from scientists in a particular developing country because the journal was trying to increase its readership in that country.

17 The peer review process is usually opaque; the author is unaware of the identity of the reviewer(s). This anonymity protects the reviewer from any potential reprisals, e.g. where the author was a senior figure in the scientific world and the reviewer was much junior. However, it also enables a reviewer to indulge in unfair and self interested criticism.

18 There is some anecdotal evidence of sharp practice during Peer Review, where a reviewer has held up or prevented publication of information pertaining to their own field of work in order to produce a pre-emptive publication of their own.

Consensus and Peer Review: Stepping Stone or Roadblock?

19 Where a consensus has developed in the scientific community both reviewers and editors will be very unlikely to accept any new material which seeks to challenge that viewpoint. Thus the peer review process is an excellent method for guarding the consensus view against attack. This is a positive benefit where the consensus view is actually correct e.g. as seems to be the case with HIV as the causal agent of AIDS.

20 However scientific endeavours are littered with examples of where a strongly held consensus has eventually been overturned, often after a considerable struggle e.g. the continental drift theory of Wegener and the Helicobacter pylori theory of ulcer formation which eventually led to a Nobel Prize for Marshall and Warren. In such cases the gate keeping activities of the Peer Review system can be seen to have had a very serious negative impact on scientific advance.

The changing face of scientific publishing

21 The current systems by which scientists make their work available to other scientists date back centuries. In Newton’s day the scientific journal was in its infancy and much communication between scientists took place directly by slow and laborious letter writing. The rise of the scientific journal has presented scientists over the last two centuries with a valuable mechanism to share their ideas with the wider community. However, in the 21st century we are beginning to see the emergence of new technologies that will once again enable scientists to talk to each other directly without the need to wait 12 months or longer to go through the painful process of submitting a paper to a journal.

22 The World Wide Web is now being extensively used by the publishers of scientific journals to distribute material more rapidly and in a user friendly format. However this medium also allows any scientist to instantly share his data and opinions with the world wide community on his own website. Some have suggested that this could become a free for all with no consideration of quality. There is some truth in that. However we are now beginning to see scientists posting draft papers on their web sites, seeking feedback and comment and subsequently revising the document. The possibility is now there for instantaneous challenge and debate potentially leading to acceleration in scientific progress.

A final note on regulatory science

23 A large amount of scientific information is generated in order to satisfy regulatory requirements, for example for agrochemicals, biocides and pharmaceuticals. This is not usually published in scientific journals due to concerns about commercial confidentiality but the detailed reports form part of the grey literature and are available to the regulatory agencies who evaluate the information in a process which combines review by peers with peer review.

24 In addition, another process, archiving and audit, operates in this area that simply does not occur in academic work. Under the internationally agreed Good Laboratory Practice (GLP) regulations the scientists must retain, and archive, all the "raw data" from their experimental work. The retained information should be comprehensive enough for an auditor to completely reconstruct the study from beginning to end. This information is subject to random detailed inspection and auditing by government inspectors as a mechanism to ensure quality is maintained and fraud eliminated.

25 As a consequence much of this grey literature is likely to have a much higher degree of quality assurance and reproducibility than much in the peer reviewed literature. However GLP procedures provide quality assurance for the data and procedural aspects of the work they do not deal with the quality of the science and should not be used to imply that any theories resulting from the observations have any special validity.

26 Summary & Conclusion

Ø In general the peer review of journal submissions can have a beneficial effect on the overall quality of the published literature.

Ø The current practice of Peer Review has a number of problems: it is a secondary unpaid activity carried out in their spare time by overworked scientists. It is subject to a number of both positive and negative biases and in extremis can choke off or severely delay scientific advances that do not fit the current paradigm despite being correct.

Ø Peer review is not guaranteed to detect fraudulent work and because of its narrow focus is not a suitable mechanism to guarantee the validity or otherwise of scientific theories. Unfortunately, in recent years, the impression that journal peer review confers the stamp of authority on a paper, has gained wide acceptance. This is completely erroneous. A peer reviewed paper is more likely to be better constructed, freer from egregious errors and present new information than one that has not been peer reviewed but the opinions expressed may very well be incorrect.

Dr D Taylor

25 February 2011