Peer review in scientific publications - Science and Technology Committee Contents

Written evidence submitted by Sense About Science (PR 51)


1.  This response is based on observations drawn from the work we have done to popularise the role of peer review for the public, policy-makers, the media and civic groups.

2.  In November 2002, Sense About Science established a working party, under the Chairmanship of Professor Sir Brian Heap, to look at how an understanding of peer review might help the public to weigh the relative merits of claims on scientific and medical issues. The report of the working party, "Peer Review and the Acceptance of New Scientific Ideas" was published in June 2004 (Appendix A).

3.  Following the recommendations of the working party, in April 2005 Sense About Science undertook a series of workshops with educational bodies, patient groups and information providers to produce a user-friendly short guide to the peer review process. This guide, "I don't know what to believe" was published in November 2005 (Appendix B). It does not present peer review as a guarantee—although it draws comparisons with kite marking—but rather uses an outline of the scrutiny in the process to explain that not all claims have the same status. Hundreds of thousands of copies have been downloaded and we have noted particularly high demand from medical and health information charities, patient groups and teachers.

4.  Our activities, including the publication of "I Don't Know What to Believe", had shown us that while an advanced science education helps a relatively small number of citizens to make sense of specific debates, an understanding of how scientific findings are assessed can help everyone. We also received many requests for the short guide to peer review from teachers and schools and decided it would be helpful to produce an online resource about peer review. The aim of the resource was to make the reasoning of science publishers and scientific researchers available to 13-18 year olds, with particular emphasis on Key Stage 4 (Appendix C). This project built upon our previous work on popularising an understanding of peer review, which has succeeded in influencing the way scientific research is discussed in a range of civic forums; and aimed to introduce the idea of scientific scrutiny to the generation of citizens that has just begun to think about social debates and sources of information.

5.  The online resource took advantage of two significant changes in the education system. Firstly, from 2006 all pupils were required to take at least one GCSE in a science subject. Secondly, in September 2006 the role of ideas and evidence in science became an integral part of science teaching at Key Stage 4, and testing at GCSE. Whether pupils take a particular interest in science or not, the resource was intended to equip them to understand how scientific knowledge develops and how to make sense of popular stories about science and medicine.

6.  Sense About Science has held a series of ten Peer Review workshops in the UK, elsewhere in Europe, in the US and in South Africa. During the workshops early career researchers hear from leading journal editors, researchers and journalists about how peer review works, the challenges to the process and the role of peer review in helping the public to evaluate research claims. The popularity of these workshops and the positive comments from participants shows that there is a big demand for this information and we have plans to develop discussions about the social and scientific value of peer review in the US and China.

7.  In September 2009 the preliminary findings of one of the largest ever international surveys of authors and reviewers, the Peer Review Survey 2009, were released. Sense About Science developed the Peer Review Survey 2009, in consultation with editors and publishers and administered with a grant from Elsevier; the survey included some questions from the Peer Review Survey 2007 (commissioned and funded by the Publishing Research Consortium) for comparison, and new questions about future improvements, public awareness and pressures on the system. The Peer Review Survey was an electronic survey; 40,000 researchers were randomly selected from the ISI author database, which contains published researchers from over 10,000 journals. Altogether 4,037 researchers completed our survey.

8.  Broadly, our work has promoted the understanding of peer review as a tool to help people to make sense of science and evidence, and work out whether research claims have been independently scrutinised. Specific outcomes from this work have been:

(a)  getting people to ask "is it peer reviewed?" when faced with scientific claims.

(b)  getting politicians to move away from weighing up evidence without considering the status of the evidence, and encouraging them to ask about the quality of the evidence.

(c)  getting the media to be accountable for the source of the material they use by showing where the material came from and citing the journal in articles.


9.  We have found, through the work we do in responding to public discussions and questions, that people can get very worried and frustrated by conflicting claims and misleading information. It is not possible (nor desirable) to prevent people from encountering a wide range of information about science and health on the internet and in the news media.

10.  It is not feasible to quality-assure such information and would likely be counter-productive to try. There is already something better than a quality assurance scheme in place: the system of critical scrutiny that is the peer review of research results.

11.  Peer review is a dividing line within academic research: it indicates that work has been scrutinised for its validity, significance and originality. The ultimate test of scientific data, however, comes through its independent replication by others; peer review is the system which allows publication of data so that it can be both criticised and replicated. It is a system which encourages people to ask questions about scientific data.

12.  "Is it peer reviewed?" is the first question anyone can ask to determine the status of the evidence, and one that can help the public weigh-up the claims they are presented with. Understanding the process through which scientific research starts to be scrutinised and evaluated can be a helpful tool for the public to sift information and understand its status. If a piece of research is peer reviewed, individuals can then look for more information on what other scientists say about it, the size and approach of the study, and whether it is part of a body of evidence pointing towards the same conclusions. The central role of peer review in the selection of the scientific data upon which scientific conclusions are based, makes it extremely significant when accounting for and explaining those conclusions, or policies based on them, to the wider public.

13.  Across science, it is widely accepted that there is no better system, although improvements could be made in aspects of its execution and there are limitations to the peer review process that need to be considered:

(a)  There is variation in the quality of peer review. Some journals are of a higher calibre than others or draw on expertise more specifically in the field of inquiry. The rigour of reviewers can vary.

(b)  Peer review checks for validity, significance and originality. It does not guarantee that the results can be repeated, nor does it provide a guarantee against all mistakes or fraud.

(c)  The endurance of findings over time and under wider scrutiny by all scientists in the field is more important.


14.  Peer review is a topical subject. Stories in the press about the problems with the peer review process include "Climategate"; the recent reporting in Science that NASA scientists had found a bacterium which—unlike any other known organism—lived off arsenic, and the subsequent questioning of these claims; and stem cell scientists speaking out about work being rejected or delayed from publication.

15.  Our work in this area and the Peer Review Survey 2009 suggest that in spite of increased pressures on the peer review system, the process remains critical to effective scholarly communication and continues to perform the critical functions: filtering and improving manuscripts.

16.  Drawing on from the results of the Peer Review Survey 2009 Mulligan et al write in Serials (Appendix D): "It is clear that there is no desire to replace [peer review] with the 'wisdom of the crowd' via metrics such as usage statistics, but instead to augment it or to subtly change its approach."

17.  The survey draws out comparisons on how peer review varies between disciplines, and how it is viewed by the scientists within these disciplines. It provides views from the research community on the strengths and weaknesses of peer review as a quality control mechanism for scientists. Possible alternatives to peer review are discussed, as well as measures to strengthen peer review, and details such as anonymity and incentives for reviewers.

18.   Summarising results on the topical issues of fraud and plagiarism, Mulligan et al write:

"Fraud continues to attract attention in the media, but within the community is not perceived as a critical issue. Nonetheless, there is a desire on behalf of the community for preventive measures to be taken, but exactly what those measures should be is unclear. It is difficult to develop a system that guarantees fraudulent papers are never published—such, it could be argued, is the wider role of science. Repeating the experiment is perhaps the most effective way, but experimental outcomes may genuinely vary, especially in the life sciences. Reviewers can only do what they do best; identify if research is new, interesting, correctly conducted, acknowledges previous work, and is appropriately summarized. Preventing fraud is most likely to be successful when done by the institute at which the research is being conducted. It is the institute that will have access to the laboratory notes and the raw research files."

19.  The results of the survey are included in Appendix E. However, the full results have not been published yet and are pending peer review. Some preliminary results have been peer reviewed and were published in an article in Serials (Appendix D).


20.  Sense About Science is a UK registered charity that works to equip people to make sense of science and evidence. We work with over 4,000 scientists, from Nobel prize winners to our Voice of Young Science network of postdoctoral researchers, to help civic groups, community organisations, media and commentators to weigh up claims about evidence.


21.  Sense About Science's funding includes donations from a wide range of scientific publishers and learned societies who have publishing arms.


Appendix A: Peer Review and the Acceptance of New Scientific Ideas (2004). Available online at:

Appendix B: "I Don't Know What to Believe" (2005) Available online at:

Appendix C: Peer Review Education Resource. Online at:

Appendix D: Mulligan, A and E Raphael (2009). Peer review in a changing world—preliminary findings of a global study, Serials, 23(1), 25-34

Appendix E: Peer Review Survey 2009.[1]

March 2011

1   Not printed Back

previous page contents next page

© Parliamentary copyright 2011
Prepared 28 July 2011