Peer review

Written evidence submitted by Jonathan Cowie (PR 45)

Summary

1. Peer review is a very powerful driver ensuring quality publication. Identifying appropriate reviewers can be a problem especially with multi- and interdisciplinary research. Improvements may be made: by keeping the author anonymous to at least most, but not all, of the referee panel; by making public referees' identities (which has demonstrated increases diligence although can increase reluctance to referee); and encouraging scholarship activities with academia (as scholarship activities are arguably discouraged). Such modifications to peer review, along with attitude within the scientific community, could encourage scientists outside of academia (such as in industry, monitoring, education and policy work) to contribute papers. This would also have other benefits.

Main body of response

Q1. the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

2. Peer review has demonstrably proven to be the most powerful tool in assessing and promulgating scientific knowledge. Only those with the expertise necessary to assess scientific understanding – peers within a scientific discipline – are in the best position to make such assessments.

3. Its weaknesses include that some areas of scientific endeavour are multidisciplinary (such as some clinical research that draw on both medical and social science), others are interdisciplinary (that biophysics that straddles – falls between – biology and physics), and others still are both multi- and interdisciplinary (such as environmental science that both covers many disciplines but is also often a synthesis of them). Here a problem that can arise is that the review panel may not contain the necessary range of expertise to properly assess what is presented to them. Some might say that an example of such a failure might be the Research Assessment Exercise (RAE) 2001's Unit of Assessment panel that dealt with environmental science research (cf. the Institute of Biology's response to the Science & Technology's 2002 consultation on the RAE) as this panel, being composed mainly of either geologists or climate/Earth system scientists, was not equipped to assess many areas of environmental research and so did not score as highly as much of the research assessed by nearly all the other panels.

4. Another weakness is that psychology can cloud the assessment. Here a key example is that it is the 'science' being presented that should be assessed and not the presenter (scientist). Not all peer review is double-blind (with reviewers and the science paper's authors kept anonymous). There is evidence to suggest that keeping the scientists' identities anonymous while their work is being reviewed prevents (for example) female researchers being discriminated against. It may also help reduce discrimination due to race, nationality, being a young researcher, or even from being a scientist from another discipline (who may nonetheless have something of value to contribute) or even a scientist who is not based within the academic research community (but possibly from industry, agency monitoring, education, policy etc.).

5. Of course, it is possible to guess the identity of an anonymous piece of work if the author as overtly self-cited their previous work within the work being assessed, or if there is a pre-publication on the net it can be Googled. However, this is only true because we are operating under the current system. If authors knew for certain that their work (at leas in part) was to be assessed anonymously then they could take the appropriate precautions.

6. Paradoxically, there is some evidence to suggest that open peer review – where the reviewers are not anonymous – encourages reviewers to be more diligent as they are open to greater and personal scrutiny. The argument against this is that it inhibits researchers from raising their head above the parapet volunteering to review. However this last is an argument of marginal pragmatism and not one to do with the quality of peer review.

7. Conversely, the counter argument – that submitting scientists' identities be known to reviewers – prevents duplication of work in that the reviewers (being familiar with scientists' work or who are then able to check scientists' previous publications) can ensure that the work submitted is a genuine advance and also ensures that technical questions arising out of personal knowledge specific to the researcher's methodology/circumstances be investigated through referee questions.

8. A final weakness is that reviewers are in effect unpaid volunteers. This means that there is an element (only an element) of self-selection among reviewers. It also means (especially if the reviewers are anonymous) that they potentially can get away with being less diligent.

Q2. measures to strengthen peer review

9. Encouraging reviewers' identities to be known may help.

10. Ensuring that the researcher's identity is kept anonymous to half (or two-thirds) of the reviewers to ensure that it the science only that is being reviewed. And the researcher's identity is known to the remaining half (or a third), to ensure that technical and circumstantial questions can be asked, may also be an improvement. However editors would need to be made aware of what (different) grounds from which (different) grouping of reviewers being presented would be acceptable for rejection.

Q3. the value and use of peer reviewed science on advancing and testing scientific knowledge

11. The value and use of peer review for advancing science has been referred to in answer to question 1. The null question, as to the hindrance of peer review to the advancement of scientific knowledge is also perhaps worth considering. Currently peer review in publicly available learned journals applies almost exclusively to researchers in academia and is conducted by the same. (Industrial research is far less open due to commercial reasons over competition.) While there is little demand for publicly available learned journals to publish work submitted by those from outside academia (say in industry, environmental monitoring, education, policy sectors) this does not mean that such contributions would not contribute to the advancement and testing of scientific knowledge. Indeed because it is difficult for such non-academic (if you excuse the phrase) scientists to submit work to academic journals is itself one reason why such work is not submitted. Yet those with scientific training who work outside of academia may have useful insights on primary research or have a different (even broader) perspective to provide review or comment papers. Today it would be difficult for a practicing monk, a college janitor, or a patents clerk, to submit work for publication and yet in the past understanding of genetics, glacial timing, and particle physics famously all came from those who were not themselves working as academics. (I am referring in the afore examples to Mendel, Croll and Einstein.) This raises the question as to what new innovation and understanding are we currently missing out on that could come from non-academe scientists? Given one question that seems to continually raise its head is that of translating British academic research into British wealth creation, it does suggest that making the divide between academe and non-academe science more permeable would have advantages. Indeed (and perhaps I am an idealist) non-academe scientists might be encouraged to contribute more to academia and might welcome it in terms of: 'continuing professional development' (a.k.a. by clinicians as 'continuing medical education'); developing a second string to their science bow so possibly facilitate potential future mobility between professional sectors and so forth; or even out of good old-fashioned interest in the science in which they have been trained. To do this will require both a change of attitude by the broader scientific community and the way peer review in journals is carried out.

Q4. the value and use of peer reviewed science in informing public debate

12. Peer reviewed science is invaluable in informing public debate. Conversely poor peer reviewed research undermines the quality of public debate and here Ben Goldacre's 'Bad Science' column in a national newspaper provides many examples that relate to issues of public debate that have not been properly subject to the rigours of peer review as used by the best practice scientific journals.

13. One difficulty of peer reviewed science in informing public debate is that (and I must qualify this by saying that this is in my unscientific – I have not formally surveyed the public – and personally biased perception) some of the public do not appear to understand what peer review is, or if they do they may suspect it is a bit of a closed shop. Here, opening up work from scientists outside scientific academia to the peer review process and publication, as suggested in my answer to question '3', may help a little.

Q5. the extent to which peer review varies between scientific disciplines and between countries across the world

14. It will be interesting to see the answers others give to this and whether or not this Committee inquiry will ascertain as to how much of the variation between disciplines and countries compares with variation in peer review practice within scientific disciplines and countries.

6. the processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases

15. I alluded to this problem in my answer to question '1', paragraph 3. Again, it might be useful in part to turn this question on its head and note that not only has multi-disciplinary (and inter-disciplinary) research increased over the years, but there has also been a trend over the decades of greater specialization within disciplines. From my own experience (limited and personal biased as it is) as someone involved in supporting (as secretariat) the review process, and as someone authoring a multi and interdisciplinary undergraduate university textbook among other scientific ventures who has been peer-reviewed, I have wondered whether or not science is too silo-like.

16. The best way to combat this is (obviously) to have scientists available for peer review who are well read across a number of disciplines.

17. However, it has been argued that the system (with components such as the RAE) does not encourage scholarship activities as it might. (With regards to the RAE this has been raised before such as by the Institute of Biology and several of the biological Affiliated Societies in its responses to the HEFC RAE consultation (1998) and House of Commons S&T Select Committee RAE inquiry (2002) as well as the Commons S&T enquiry on the 2000 White Paper Excellence and Opportunity (2001).)

18. One also wonders whether the mantra that university staff should combine both research and teaching is also conducive to scholarship activities? Notwithstanding that surely everyone who is, or has been, a university student knows that there are some lecturers who are better than others, some are arguably better placed to spend most of their time focussed on specialist research and others to teach, and here the latter could arguably afford more time to take in a broader picture (including an appreciation of neighbouring disciplines) and so who may be also be more suited to making a contribution to refereeing papers.

19. Having said all of this, many may well be resistant to some of the changes I have suggested. These may with some justification include the argument that making the peer review process more diligent will add to peer review costs: but this would be true of virtually all diligence-improving changes.

Jonathan Cowie has spent a number of decades working in science communication (policy, publishing and promotion) with scientific learned societies. In the course of this he: has serviced the editorial board of two peer-reviewed journals as well as an academic books committee; been a committee member peer reviewing grant applications; had his own review papers peer reviewed; and had his proposals for science events peer-reviewed. He has also served a term of office as a member of council of the Association of Learned and Professional Society Publishers. He is not currently serving on any editorial board or employed by any academic publisher. Some of his science activities are summarised on the Concatenation Science Communication website – http://www.science-com.concatenation.org.

Jonathan Cowie

9 March 2011