Peer review

Written evidence submitted by Professor Thomas Ward,
Pro-Vice-Chancellor (Academic), University of East Anglia (PR 42)

Declaration of interest: As a research mathematician I participate routinely in all aspects of peer review, and have carried out several editorial roles. I am currently an Editor for the London Mathematical Society Bulletin, and serve on the London Mathematical Society publications committee (though I am not writing on their behalf). I have a clear interest in the health and credibility of the peer review process. My University, like any research-intensive University, uses and contributes to peer review across all disciplines, with research outputs being subject to peer review and academic staff carrying out peer review, participating on panels, and serving in multiple editorial roles.

1) Recent events and discussions, both in the public sphere and in the research community [1] , have cast a spotlight onto peer review, and we welcome the opportunity to take a fresh look at this important ingredient in the maintenance of standards in research publication. "Peer review" is here mainly taken to be the conventional, largely anonymous, system of expert review of research papers. Research proposals are more usually judged by a panel of non-anonymous expert reviewers, and for these the anonymous review is replaced by a collective judgement.

Understanding what peer review is and is not

2) Much of the confused discussions surrounding peer review start from the wrong assumptions – sometimes deliberately for rhetorical effect. Peer review is a process where a small group (sometimes a single person) is asked to form a judgement on the academic rigour, intellectual content, and significance of a submitted research output or proposal. It has been the central tool in sorting wheat from chaff in the production of scientific and scholarly publications, and lies behind much of our confidence in scientific decision-making. It is important however to understand that it is only one ingredient in how our understanding of complex problems emerges.

3) Thus a reasonable view of peer review is that it acts as a filter, selecting research that seems to the reviewers sound, significant, and of value. This is not the same as a guarantee of truth, nor does it imply an independent replication of the work.

4) The other ingredients in how we come to a shared understanding of something complex include reproducibility, different approaches providing confirmatory evidence, coherence of the conclusions with a wider and independently arrived at picture, and so on.

5) Peer review is sometimes attacked for not being something it never could be – an absolute arbiter of truth. Any judgement on the health and role of peer review needs to start from a reasonable understanding of what it can do.

Is peer review too cosy?

6) Peer review is sometimes attacked for reflecting a closed world in which a small group of researchers have closed their eyes to other possibilities and seek to confirm each other’s work. At one level this must inevitably happen to some extent: researchers are humans who operate in social groups.

7) The strongest defence we have against the inevitable effect of this is the web of understanding. There are very few scientific assertions of any significance that hang entirely on one lengthy chain of reasoning. They typically rest on a complex web of work, with many independent chains of reasoning confirming a similar conclusion. They also fit into a larger picture and are coherent with other established knowledge. Thus scientific research is generally exposed to repeated multiple testing and retesting in the strongest possible sense: not by repeating a possibly flawed experiment, but by independent approaches to the same question.

8) In order to accurately weigh the extent to which this is a problem, one needs to have a realistic view of the behaviours involved. If you carry a prior absolute conviction that scientific research into Anthropogenic Global Warming – for example - is fundamentally flawed, and that its practitioners are a sinister cabal of people determined to close their minds to all other voices, then you will weight the cosiness problem very highly. If you carry a prior assumption that scientific research is predominantly carried out honestly if imperfectly, and that the combined force of the inter-connected web of confirmatory work and the constant desire to find new and contrarian ways of viewing a problem act as powerful incentives to accuracy and to new thinking, then you will weight the cosiness problem as neither trivial nor overwhelming.

Does a UK view on peer review in isolation make sense?

9) Peer review is an international not a national mechanism. It underpins the global production of scientific knowledge, and is one of the reasons we can trust and use research carried out elsewhere – and in turn have research carried out in the UK trusted and used elsewhere.

10) If damage is done to the peer review mechanism in the UK, we risk losing our ability to interact as a trusted partner in international research collaborations.

Does the citizen scientist provide a replacement?

11) The last year or two has seen much public debate of the merits of the "citizen scientist" and "peer-to-peer review". While it has never been completely clear what either of these phrases mean, we must be alert both to the opportunities and the dangers this thinking may pose. Peer-to-peer review often appears to be nearly instantaneous and as a result compelling, in marked contrast to standard peer review where reviewers are asked for a considered review over a longer period of time.

12) The greater emphasis on openness, transparency, and the responsibility on scientific researchers to explain their work to a wider audience is to be welcomed. So too is the reverse flow, where the public help to shape the scientific research agenda. We face such large problems – climate change, food security, water security, disease, ageing, GM, energy security – that the public must be involved in the decisions about what questions should be asked, what are the priority research themes, and so on.

13) Approaches need to be found to lessen the distance between the general public and the research literature wherever possible. This might involve more public engagement on the part of authors, more use of web-based discussions attached to published papers, and so on. It would be hazardous however to compromise the researcher’s responsibility to present the evidence as they find it, whether it is popular or unpopular. We often face difficult public decisions in contested territory (Is the MMR vaccine safe? How can the balance of risks between nuclear power and controlling carbon emissions be judged?) and it is important that well-founded, peer-reviewed, research outcomes continue to be published and heeded even if they are under attack from vocal parts of the blogosphere or the press.

The impact of the web

14) Research publication has been changed fundamentally by electronic communication and the web in several different ways.

15) Electronic communication facilitates and speeds the cycle of peer review and eventual publication – though the real constraint is the willingness and speed of the reviewers.

16) The web has enabled the proliferation of a vast literature that is not peer-reviewed. This can lead to confusion and anxiety, with the distinction between genuine research and speculative opinion pieces becoming blurred.

17) The web has enabled the discussions often grouped together as the "blogosphere". Ideally this is a powerful mechanism bringing expert researchers and the general public closer to each other, and there are many highly successful interactions of this sort. However it cannot act as a substitute for review by experts. Nor should its volume – both in numbers of entries and in ferocity of rhetoric – be allowed to distort the calmer perpetual debate and revisiting and discussion of scientific ideas, research outcomes and policy formulation. This is no small matter: giving the blogosphere too great a weight in public policy debates informed by scientific research on, for example, vaccinations, might lead to avoidable fatalities.

18) The web makes much greater transparency on underlying data sets possible. It is now possible to make huge data sets available in this way, supporting another pillar of knowledge: reproducibility.

Recommendations

19) The committee should endorse the broad principles underlying expert peer review. It is the established international mechanism for filtering research publication and it continues to serve us well. It is not and can never be an all-knowing oracle, but it is a robust and honest process. In conjunction with reproducibility and the confirmatory web of knowledge, it plays a vital role in ensuring that scientific debate is soundly based, and that policy is informed by accurate research.

20) The committee should recognise the importance of anonymous peer review. This may seem antithetical to scientific openness, but in fact works well at encouraging radical new ideas to come through, enabling reviewers to challenge accepted ideas and speak candidly. It is still possible for reviewers to sign their reviews if they wish to do so, and there are mechanisms for making the process less anonymous. One interesting suggestion is to list the editors and referees involved in accepted papers. The experiments in some disciplines [2] with non-anonymous peer review should be welcomed but not hastily replicated in other disciplines until we have a much greater understanding of the impact.

21) It would be naive to imagine that moving to non-anonymous reviewing across the board would promote greater accountability. It would become much more difficult for editors to find reviewers, and reviewers might become less willing to express the very strong judgements sometimes needed. In some disciplines the importance of anonymous peer review has been recently endorsed [3] .

22) The committee may wish to reflect on the dangers posed by any blurring of the distinction between genuinely peer reviewed research literature and non-reviewed opinion pieces. Peer-reviewed publications should be clearly identified as such, and the non-reviewed literature should also be clearly identified.

23) The Research Excellence Framework assessing UK Universities is seeking to apply metrics to some aspects of the periodic assessments of research quality. Some of these metrics depend on peer-reviewed publications and citation counting of the articles cited. This illustrates the importance of the peer-review system to our shared decision-making in research policy, and the committee may wish to consider how peer-review is used as a benchmark of quality in the REF.

24) The committee may also wish to encourage policy leaders and politicians to be a little more discerning in how much attention is paid to the blogosphere and some campaigns led by newspapers. These have included at various times a lengthy campaign denying the link between HIV and AIDS, a campaign claiming a link between the MMR vaccine and autism, another claiming that GM potatoes were dangerous, etc. In each case we might have been better served if the normal expectation in public debate was that a scientific consensus where one exists – and the peer-reviewed literature – had been given greater relative emphasis.

25) The greatest volume of interaction between scientific statements and the public occurs not through the research literature but through newspapers and television pieces. Society’s ability to engage with difficult decisions would be strengthened if newspapers were required to make clear whether the sources of stories with scientific content have relevant scientific training, and whether the source materials come from the peer-reviewed literature or not.

Professor Thomas Ward

Pro-Vice-Chancellor (Academic), on behalf of the University of East Anglia

9 March 2011