Written evidence submitted by Sense About
Science (PR 51)
POPULARISING THE
ROLE OF
PEER REVIEW
1. This response is based on observations drawn
from the work we have done to popularise the role of peer review
for the public, policy-makers, the media and civic groups.
2. In November 2002, Sense About Science established
a working party, under the Chairmanship of Professor Sir Brian
Heap, to look at how an understanding of peer review might help
the public to weigh the relative merits of claims on scientific
and medical issues. The report of the working party, "Peer
Review and the Acceptance of New Scientific Ideas" was published
in June 2004 (Appendix A).
3. Following the recommendations of the working
party, in April 2005 Sense About Science undertook a series of
workshops with educational bodies, patient groups and information
providers to produce a user-friendly short guide to the peer review
process. This guide, "I don't know what to believe"
was published in November 2005 (Appendix B). It does not present
peer review as a guarantee—although it draws comparisons
with kite marking—but rather uses an outline of the scrutiny
in the process to explain that not all claims have the same status.
Hundreds of thousands of copies have been downloaded and we have
noted particularly high demand from medical and health information
charities, patient groups and teachers.
4. Our activities, including the publication
of "I Don't Know What to Believe", had shown us that
while an advanced science education helps a relatively small number
of citizens to make sense of specific debates, an understanding
of how scientific findings are assessed can help everyone. We
also received many requests for the short guide to peer review
from teachers and schools and decided it would be helpful to produce
an online resource about peer review. The aim of the resource
was to make the reasoning of science publishers and scientific
researchers available to 13-18 year olds, with particular emphasis
on Key Stage 4 (Appendix C). This project built upon our previous
work on popularising an understanding of peer review, which has
succeeded in influencing the way scientific research is discussed
in a range of civic forums; and aimed to introduce the idea of
scientific scrutiny to the generation of citizens that has just
begun to think about social debates and sources of information.
5. The online resource took advantage of two
significant changes in the education system. Firstly, from 2006
all pupils were required to take at least one GCSE in a science
subject. Secondly, in September 2006 the role of ideas and evidence
in science became an integral part of science teaching at Key
Stage 4, and testing at GCSE. Whether pupils take a particular
interest in science or not, the resource was intended to equip
them to understand how scientific knowledge develops and how to
make sense of popular stories about science and medicine.
6. Sense About Science has held a series of ten
Peer Review workshops in the UK, elsewhere in Europe, in the US
and in South Africa. During the workshops early career researchers
hear from leading journal editors, researchers and journalists
about how peer review works, the challenges to the process and
the role of peer review in helping the public to evaluate research
claims. The popularity of these workshops and the positive comments
from participants shows that there is a big demand for this information
and we have plans to develop discussions about the social and
scientific value of peer review in the US and China.
7. In September 2009 the preliminary findings
of one of the largest ever international surveys of authors and
reviewers, the Peer Review Survey 2009, were released. Sense About
Science developed the Peer Review Survey 2009, in consultation
with editors and publishers and administered with a grant from
Elsevier; the survey included some questions from the Peer Review
Survey 2007 (commissioned and funded by the Publishing Research
Consortium) for comparison, and new questions about future improvements,
public awareness and pressures on the system. The Peer Review
Survey was an electronic survey; 40,000 researchers were randomly
selected from the ISI author database, which contains published
researchers from over 10,000 journals. Altogether 4,037 researchers
completed our survey.
8. Broadly, our work has promoted the understanding
of peer review as a tool to help people to make sense of science
and evidence, and work out whether research claims have been independently
scrutinised. Specific outcomes from this work have been:
(a) getting people to ask "is it peer reviewed?"
when faced with scientific claims.
(b) getting politicians to move away from weighing
up evidence without considering the status of the evidence,
and encouraging them to ask about the quality of the evidence.
(c) getting the media to be accountable for the
source of the material they use by showing where the material
came from and citing the journal in articles.
THE IMPORTANCE
OF STATUS
OF EVIDENCE
FOR THE
PUBLIC AND
THE ROLE
OF PEER
REVIEW
9. We have found, through the work we do in responding
to public discussions and questions, that people can get very
worried and frustrated by conflicting claims and misleading information.
It is not possible (nor desirable) to prevent people from encountering
a wide range of information about science and health on the internet
and in the news media.
10. It is not feasible to quality-assure such
information and would likely be counter-productive to try. There
is already something better than a quality assurance scheme in
place: the system of critical scrutiny that is the peer review
of research results.
11. Peer review is a dividing line within academic
research: it indicates that work has been scrutinised for its
validity, significance and originality. The ultimate test of scientific
data, however, comes through its independent replication by others;
peer review is the system which allows publication of data so
that it can be both criticised and replicated. It is a system
which encourages people to ask questions about scientific data.
12. "Is it peer reviewed?" is the first
question anyone can ask to determine the status of the evidence,
and one that can help the public weigh-up the claims they are
presented with. Understanding the process through which scientific
research starts to be scrutinised and evaluated can be a helpful
tool for the public to sift information and understand its status.
If a piece of research is peer reviewed, individuals can then
look for more information on what other scientists say about it,
the size and approach of the study, and whether it is part of
a body of evidence pointing towards the same conclusions. The
central role of peer review in the selection of the scientific
data upon which scientific conclusions are based, makes it extremely
significant when accounting for and explaining those conclusions,
or policies based on them, to the wider public.
13. Across science, it is widely accepted that
there is no better system, although improvements could be made
in aspects of its execution and there are limitations to the peer
review process that need to be considered:
(a) There is variation in the quality of peer
review. Some journals are of a higher calibre than others or draw
on expertise more specifically in the field of inquiry. The rigour
of reviewers can vary.
(b) Peer review checks for validity, significance
and originality. It does not guarantee that the results can be
repeated, nor does it provide a guarantee against all mistakes
or fraud.
(c) The endurance of findings over time and under
wider scrutiny by all scientists in the field is more important.
CRITICISMS OF
PEER REVIEW
14. Peer review is a topical subject. Stories
in the press about the problems with the peer review process include
"Climategate"; the recent reporting in Science
that NASA scientists had found a bacterium which—unlike
any other known organism—lived off arsenic, and the subsequent
questioning of these claims; and stem cell scientists speaking
out about work being rejected or delayed from publication.
15. Our work in this area and the Peer Review
Survey 2009 suggest that in spite of increased pressures on the
peer review system, the process remains critical to effective
scholarly communication and continues to perform the critical
functions: filtering and improving manuscripts.
16. Drawing on from the results of the Peer Review
Survey 2009 Mulligan et al write in Serials (Appendix
D): "It is clear that there is no desire to replace [peer
review] with the 'wisdom of the crowd' via metrics such as usage
statistics, but instead to augment it or to subtly change its
approach."
17. The survey draws out comparisons on how peer
review varies between disciplines, and how it is viewed by the
scientists within these disciplines. It provides views from the
research community on the strengths and weaknesses of peer review
as a quality control mechanism for scientists. Possible alternatives
to peer review are discussed, as well as measures to strengthen
peer review, and details such as anonymity and incentives for
reviewers.
18. Summarising results on the topical issues
of fraud and plagiarism, Mulligan et al write:
"Fraud continues to attract attention in the
media, but within the community is not perceived as a critical
issue. Nonetheless, there is a desire on behalf of the community
for preventive measures to be taken, but exactly what those measures
should be is unclear. It is difficult to develop a system that
guarantees fraudulent papers are never published—such, it
could be argued, is the wider role of science. Repeating the experiment
is perhaps the most effective way, but experimental outcomes may
genuinely vary, especially in the life sciences. Reviewers can
only do what they do best; identify if research is new, interesting,
correctly conducted, acknowledges previous work, and is appropriately
summarized. Preventing fraud is most likely to be successful when
done by the institute at which the research is being conducted.
It is the institute that will have access to the laboratory notes
and the raw research files."
19. The results of the survey are included in
Appendix E. However, the full results have not been published
yet and are pending peer review. Some preliminary results have
been peer reviewed and were published in an article in Serials
(Appendix D).
ABOUT SENSE
ABOUT SCIENCE
20. Sense About Science is a UK registered charity
that works to equip people to make sense of science and evidence.
We work with over 4,000 scientists, from Nobel prize winners to
our Voice of Young Science network of postdoctoral researchers,
to help civic groups, community organisations, media and commentators
to weigh up claims about evidence.
DECLARATION OF
INTERESTS
21. Sense About Science's funding includes donations
from a wide range of scientific publishers and learned societies
who have publishing arms.
APPENDICES/REFERENCES
Appendix A: Peer Review
and the Acceptance of New Scientific Ideas
(2004). Available online at: http://www.senseaboutscience.org.uk/index.php/site/project/33
Appendix B: "I
Don't Know What to Believe" (2005)
Available online at: http://www.senseaboutscience.org.uk/index.php/site/project/30
Appendix C: Peer Review
Education Resource. Online at: http://www.senseaboutscience.net/
Appendix D: Mulligan,
A and E Raphael (2009). Peer review in a changing world—preliminary
findings of a global study, Serials, 23(1), 25-34
Appendix E: Peer Review
Survey 2009.[1]
March 2011
1 Not printed Back
| 
