Examination of Witnesses (Question numbers
LASKEY CBE FRS FMEDSCI,
Good morning to you all. Some of you may not be aware that Parliament
was sitting until the early hours this morning, so some of my
colleagues are a tad on the tired side. Please bear with us. May
I, first of all, ask the four witnesses to introduce themselves?
Dr Gulley: I am
Dr Nicola Gulley. I am the Editorial Director at IoP Publishing.
I am Ron Laskey. I am here as Vice-President of the Academy of
Dr Parker: I am
Robert Parker. I am Interim Chief Executive of the Royal Society
I am John Pethica, Physical Sciences Secretary and Vice-President
of the Royal Society.
Welcome. You are familiar with the nature of this inquiry. Let
me start off with some basic questions. If you feel at the end
of the session that you would have liked to have responded in
more detail to one of the questions, please feel free to drop
us a note with any additional comments. Peer review is perceived
to be "fundamental to scholarly communications". If
it disappeared tomorrow, what would the consequences be?
Dr Parker: You
would have to come up with something else with which to replace
it. There isn't anything very obvious to replace peer review with
currently. The danger would be to the scientific record, really.
The importance of it is laid out in the evidence that has been
submitted with great clarity from most people who have submitted
evidence in writing to this review. The value and quality of that
scientific record is paramount, and peer review helps to keep
that in place.
In the biomedical sciences there would be a particular problem
of sorting the wheat from the chaff and knowing what information
could be depended on. This, I think, would corrupt the public
understanding of science where a firm basis of trust in scientists
is something that we could do with more of.
Dr Gulley: I would
add that there is the aspect of the time that it would take for
scientists to be able to find the information that they really
wished to read, because at the moment peer review also adds value
in providing that filter. There is also evidence that many authors
feel that the peer review does improve the quality of the articles
that they publish as well.
To add a historical perspective, of course, this has been going
on for a very long time. You asked the question of what would
happen if it disappeared. Its primary function is to improve the
process and the coherence of scientific knowledge and its utility.
Taking Professor Laskey's observation about sorting the wheat
from the chaff, in a sense, the opposite of that is something
that we have been told in the evidence, that peer review has a
tendency towards producing conservative judgments. How big a problem
is that for the progression of science and what can be done about
Some journals have a tendency to believe that things that are
already well known to be important have a higher impact. It can
be more difficult to establish a novel and completely unexpected
new branch of science if editors of journals are not alert to
the fact that it is coming. There are one or two recent examples.
One that springs to mind is a study in plant sciences which concerned
resistance to viral infection in plants. That has given rise to
a completely new area of understanding of a group of molecules
that turn out to be important in all cells, not just in viral
defence mechanisms against plants but because they change fundamentally
in certain types of cancer. That was a small niche of advance
that has suddenly become a front-line subject, but it would have
been very difficult to publish that in a front-line journal at
the time the work was being done.
To add to that a little, there is always a risk in this process
that new ideas may be impeded in the way I have described. That
is a risk. It has to be balanced against the fact that the likelihood
of radical breakthroughs is, unfortunately, rather smaller than
exotic ideas that don't actually work. It is that balance that
is difficult to achieve because there is a tension between those
Dr Parker: Knowing
the right people to ask about research that looks slightly different
is one of the most important things. Having professional people
overseeing the peer review process is absolutely paramount, because
it ensures that, if something is there that is very different,
you could get somebody to look at it who will look at it in an
open way. There are different outlets for different sorts of science.
Sometimes you can get things published that are a little odd or
seem a little odd at the time.
As Professor Laskey said, you don't always know very
quickly what is going to be important in areas that are far away
from chemistry. You could have some mathematical proof that is
not found to be terribly important until 50 years or 100 years
later, and suddenly it is important in finding out something else.
Dr Gulley: There
is also a cultural difference in certain research areas as well.
There is more conservatism in some research areas than there is
in other areas. Speaking from the journals' point of view, some
journals like to have articles that they feel are cutting edge.
That is, partly, how they approach it. Also, different things
need to be taken into consideration. There is peer review within,
for example, the research conferences where you do get a feel
for some of the new areas that are coming up before they go into
Q4 Graham Stringer:
Which areas are conservative and which are bold and radical?
Dr Gulley: It varies
considerably. I can only speak from the physics side in which
I am involved. There are certain very well-established areas where
there would be slightly more conservatism because they are very
well established and they want to be sure that before something
goes in to be the article of record it is correct. There could
be some areas, particularly where you have multidisciplinary areas,
where there are more differences of opinion. There is then less
conservatism about what gets recorded.
You have all acknowledged in different ways that there is a risk
or a problem there, but none of you really responded to the second
part of the question, which is, what can be done about it?
I alluded to that a little bit in my response. Given that there
is no perfect system, we have to devise a system which optimises
the process, that is to say, one that minimises the risks that
have been alluded to but also retains the key advantages of the
peer review process in establishing a coherent record. A variety
of models have been alluded to in other places. For example, different
kinds of publishing models are being evolved all the time. For
example, there is the arXiv record in high energy physics which
stems from the way that high energy physics actually works. That
is another way of establishing the record.
It is also important to recall that peer review,
as we are describing it here, is about generating a coherent scientific
record efficiently as far as possible, but often it is used for
other proxy purposes and assessment. That can, potentially, influence
how it is carried out.
Following on from that, is something like the PLoS ONE
model of publishing anything that is scientifically sound, regardless
of potential impact or perceived scientific interest, a better
way of doing things?
It is an alternative that solves some of the problems. At the
moment it is an evolving landscape. The attitude of PLoS ONE
to publish irrespective of impact but based solely on the criterion
of the quality of the science can prevent a trap that, in biomedical
sciences, is becoming increasingly troublesome, namely, that a
high proportion of time is spent fending off criticisms from reviewers
that may not be on the main theme of the work. The reviews are
beginning to dictate the agenda of the science in a way that is
not fully productive. That can be frustrating, a waste of time
and resource. PLoS ONE provides an alternative to that.
The downside, as Professor Pethica has already said,
is that there is now a proxy use of peer review, namely, to judge
careers by the calibre of the journals in which people have published
and to judge institutions by the Research Excellence Framework,
again based on the quality of journals in which people have published.
PLoS ONE, of course, is not a major front-line high impact
However, that has been compensated for by the ease
of electronic searches of the literature. Now you no longer have
to depend on readers of a small number of widely read journals
seeing your paper, because your paper will be noticed by electronic
search routes as an alternative. Against that changing model,
there is an increasing value of archival journals, such as PLoS
ONE, which ignore impact.
There are potential downsides to it because of the
proxy use of peer review data, but they do offer an alternative.
One trend that has been emerging, and it has been a surprise to
many people, is that, initially, people envisaged PLoS ONE
as a journal they would submit to only if their paper was
having severe criticism from other higher impact journals. Now,
important research has been submitted to get it on the record
quickly before it is scooped by someone else who has a smoother
path through the refereeing jungle.
Dr Parker: The
PloS ONE-type model or the cascading model also has another
advantage in that it can reduce the factor of articles being multiply
peer reviewed by different journals. It can save time in the peer
review process there. Without those cascade journals, you often
have the case where papers that are rejected by one journal are
then submitted to another and they are reviewed again completely.
They could be, and very often are, scientifically acceptable but
they just don't reach the impact criterion for that particular
journal. So the process goes on until they find a home. The cascading-type
journal does away with that. We found, from doing studies on the
articles that we reject, that most of them end up being published
somewhere else. There are very few articles that we receive that
are scientifically completely wrong. Usually, there is some merit
Q7 Stephen Mosley:
Dr Gulley, Dr Parker and Professor Pethica, could you summarise
for the record the peer review methodology that you use in the
journals that you publish?
Dr Gulley: For
the IoP Publishing journals, we use single-blind refereeing where
the referees know who the author is but the referees' names are
Dr Parker: We are
the same at the RSC. We also use, as I suspect others do, pre-screening
as well, so not all papers are sent to referees. Some of them
are rejected before they are sent to referees. That is either
by internal, qualified editorial staff or by external associate
It is the same process for the Royal Society's journals, which
Q8 Stephen Mosley:
The three organisations represented here all use the same methodologies,
but there are other organisations that use different methodologies.
Why are different methodologies used across the journals?
I have two background points to make. One is, of course, that
the subject areas vary very strongly. We should keep in mind that
we are discussing areas from pure mathematics through to biomedical
research. As a result, the review process in those cases needs
to be quite different simply because of the nature of the subject
and what they are trying to establish. As a result, the various
experiments in the forms that you have described vary quite strongly
across subjects. What you have heard about is physics and chemistry,
which are what I might call fairly traditional mainstream subjects.
As you move to the more trials-based medical work, you get a different
structure. Also, I have mentioned arXiv already and high energy
physics. This is a very large-scale collaborative exercise in
which that kind of model of communication is quite important.
There is some variability across the subjects and I suspect that
is the demand of the users. The need to establish fairly rapidly
a sensible view of what works in the science affects the methods
used. It is fair to say that all of these things are in a state
of steady evolution. There are core principles, but the actual
method by which it is used varies. You have evidence from a variety
of people submitted to you about things like completely open or
double-blind processes. You have some responses about that. I
don't want to add to those because they are broadly correct.
Dr Parker: I would
add that open review pre-printing, particularly, is not popular
with chemistry because there is very often the possibility that
an author will take out a patent on what they are producing. Putting
your results out there in a pre-printed form is risking losing
priority on them.
Another aspect that is different across disciplines
is the amount of experimental refereeing that is done. In mathematics
there might be very few experimental results, but in chemistry
and physics there are a vast amount of experimental results that
sometimes need specialist refereeing.
Dr Gulley: Some
of the research communities that I work with particularly are
very small, so doing double-blind refereeing where neither the
author nor the referee knows who each other is defeats the object
because, generally, the referees will know who the author is from
the subject area that they are working in or from the references
and things like that. It varies very much between different subject
With regard to pre-print, this is something within
physics that we have worked with for many years since the arXiv
was set up. That form of open refereeing on the pre-print side
is entered into for a lot of the subject areas with which I work.
However, when it comes to submitting to journals, we find that
people are happier to referee if they are kept anonymous because
they feel they have more options to be able to criticise openly.
As a result, there is a mixture within the physics area now.
Where you have these very small communities, isn't there a tendency
for the referee to be somebody who is more advanced in their career
than the person whose paper they are refereeing? Doesn't that
inhibit the evolution of some of the science because you would
feel a little reluctant in criticising previous work?
Dr Gulley: It also
works the other way in that you have some of the more junior researchers
assessing senior researchers work and the anonymity offered as
a referee enables the junior researchers to feel a bit more comfortable
when they have to criticise work in particularly well-established
research areas, particularly when the research area is small.
Then there is less worry about it impacting on grant applications
and things like that.
There is also an element of exactly what information
the editor or the editorial office of the journal has to take
into consideration making sure that there is a balanced opinion.
Generally, for IoP Publishing we don't just have one referee on
an article, for example. It is generally balanced with at least
two referee reports and occasionally more.
Q10 Stephen Mosley:
Professor Pethica, I noticed that you were nodding when the Chairman
asked his question. Perhaps you have a slightly different view.
No, on the contrary. It is quite commonplace for research students
to be trained by asking them to review papers. The question of
senior people reviewing junior people is very often reversed.
Q11 Stephen Mosley:
Dr Gulley, we have heard from Dr Parker that the pre-print server,
arXiv, isn't really appropriate for the chemistry community. However,
it is widely used within the physics community. Could you give
us an explanation as to how it works and how authors interact
with the system? Do they maintain their own records, etcetera?
Dr Gulley: The
arXiv was set up as a pre-print server so it is the authors' work
at a preliminary stage. Large collaborations can use that process
to be able to discuss and comment before finalising the paper.
It originated from the high energy physics area where they had
a need to be able to discuss the results across the international
collaborations. A lot of the work that is posted, particularly
from areas such as high energy physics, also goes through internal
peer review within the research facilities as well before it is
posted on the arXiv. There are a number of different stages it
has to go through.
As far as linking in with publishing is concerned,
a high percentage of articles that are pre-prints are eventually
submitted to journals and get published in journals as well, so
there is still that requirement for that independent peer review.
We link in with that. We make it very easy for authors to be able
to submit from the arXiv into our journals, for example, and this
is common across many physics publishers, where the arXiv number
can be used when submitting the article to a journal. Authors
are encouraged to update their versions as well. From the publishing
side, we encourage them to update the citations so that the link
goes back to the final version of record once it has been peer
reviewed and published.
Q12 Stephen Mosley:
From what Dr Parker said earlier, it is probably not appropriate
across the entire science community, but are there areas other
than physics that might benefit from it?
Dr Gulley: I am
not sure that I am the best person to comment on that since my
area is very much within physics.
Pure mathematics is a good example of an area where the numbers
are fairly small. It can take a very long time for the assessment
of theorems to become correct. Therefore, effectively, the process
is hybrid like that. I would like to draw your attention to the
contrast with engineering, for example, where you have an immediate
technological impact on what you are doing. Then the question
of publishing that Dr Parker raised is rather important.
Are the extremes the areas where there is a much more collegiate
approach to helping solve a global high energy physics problem
working in some of the big science projects versus things that
are much closer to potential commercialisation? Are those the
There will be a difference. The reason I raised it is that it
is not as if there is a lack of collaboration in that area too;
it is very extensive in large technological projects. As you know,
collaboration is not just in high energy physics. It is the other
factors that Dr Parker raised, such as patentability, exploitation
and all the issues that concern companies in research that involves
collaborative work and pre-exploitation. This boundary is an interesting
area that affects what you publish, when and how, and the role
of patents as distinct from peer review publication. There is
a continuum across the board.
Dr Parker: Also,
the speed of reproducibility of results is an issue where you
are sharing big resources like synchrotrons or various other things
such as in the area of astronomy. It is very good for peer review.
There are small numbers of people using big pieces of equipment
that are very expensive. It is good for them to share and work
Even in those areas you get some quite innovative commercial projects
emerging. Charge-coupled devices spring to mind and things like
Dr Parker: Yes.
Q15 Stephen Mosley:
Professor Laskey, within the Academy of Medical Science evidence
you suggest that the dissemination of non-peer reviewed information
may be potentially unhelpful, as you describe it. You also go
on to say that even things that are kite-marked may not be totally
appropriate. Would a kite-mark work or do you still have some
Two worries were voiced in the Academy's submission on this topic.
One is that biomedical sciences are more prone to inaccurate interpretations.
Measurements in biology tend to be, by the very nature of biological
material, more scattered than the more precise measurements that
can be made in the physical sciences. Although I don't like the
terms from the perspective of a biological scientist, I have to
admit that there is some truth in the description of hard sciences,
meaning the physical sciences, and the soft sciences, meaning
the biomedical sciences, in which it can be more difficult to
get precise and incontrovertible evidence. There is a worry that,
if you extended the pre-publication model to the biomedical sciences
without any attempt to peer review, a lot of half-truths would
creep into the literature.
The second problem is the appetite of the media for
some aspects of biomedical science. Without peer review we would
get a storm, frankly, of incorrect headlines. That is something
that would also worry us very much.
Chair: The media don't
do badly at doing that, anyway.
The scientific community tries to prevent that but not always
successfully, I am afraid.
Q16 Stephen Metcalfe:
Three of you publish journals. How do you ensure that your editors
are selecting the most appropriate reviewers? What process do
you go through? Is that process complicated when you are looking
at multidisciplinary work which covers a number of different areas?
Dr Parker: The
process we go through is that we have internal and external editors.
The internal editors are chemists who work within the RSC. The
external editors are people who work out in the community, who
are largely academics. We ensure that they choose the right referees
by having a long period of training for people who do things like
that. Building up a knowledge of the community is very important.
There are people who work on general journals that cover broad
subject areas, but most people will have a specialism within them
even if they do work on those broad journals. People do get to
know a particular area and the interactions between certain authors
and referees very well. You do get to know your community and
you get a feel for whether there are any issues between particular
people. We also do quite a lot of training of referees. We have
a feedback loop where referees always get the feedback on the
outcome of the articles that they have refereed so that they can
learn whether their refereeing activity is generally in line with
what is accepted and what is rejected. We also do straight face-to-face
training as well, particularly in China and India, but also elsewhere.
Dr Gulley: It is
very similar for IoP Publishing as well in that we have a combination
of in-house editors and external editors. We also do some training
of our referees, particularly within China. We have different
programmes across our international offices as well.
Recently, as a result of requests from some post-docs
and graduates, we have given them some initial training on what
peer review means. We are teaching them about what refereeing
means and what we are expecting. There is a lot of literature
as well that people are not always aware of so we have been trying
to raise the visibility of that. Internally, we also try and match
the interests of the referees to the papers as much as possible.
Again, that comes from the extensive training that is required
for the internal editors, as Dr Parker has mentioned.
To add to that, you specifically asked about multidisciplinary
situations, which are very broad. The process in the Society is,
essentially, to increase greatly the number of referees and reviewers.
Six or seven would be common, whereas two or three might be the
number you would have within a well-defined subject, to try and
ensure you get that coverage for a number of broad views.
Q17 Stephen Metcalfe:
Rather than the individuals having a broad knowledge, you expand
the number within their speciality and they would look at a part
of the particular subject.
In general, one is obliged to do that simply because there may
be a few people who have the vast and broad knowledge required,
but in truly interdisciplinary areas, which really span gaps,
you have to get a broad perspective and that means using more
people, including from a variety of countries, environments and
so forth. What we are describing here is a totally international
Q18 Stephen Metcalfe:
How do you keep that networking, that knowledge of who is in the
community, up to date? How is that managed?
Dr Parker: The
editors, whether they are internal or external, are out in the
community a lot. They are going to conferences, seminars, doing
university visits and industry visits. From the RSC, our editors
regularly attend up to 200 conferences a year overall. Our external
editors will certainly be attending quite a number of conferences
in their own subject areas.
Dr Gulley: Again,
it is the same for IoP Publishing in that we attend a number of
conferences each year. It is about 300 because of the broad range
of subject areas. The editors are encouraged to go along to become
part of the community and to update their understanding of the
subject area. Equally, we also track the trends internally from
various data sources. So we look at what sort of subject areas
are coming through and work with the researchers to look at how
we can make sure that the journals represent that as well.
One is looking at the process of the reviewers as well as the
editor. Of course, one can keep a record of how effective various
reviewers are, which is done by most journals. Some people are
more effective than others and are used correspondingly. Also
one uses the community to suggest future names of reviewers. It
is very common, for example, if a senior scientist is asked to
review something and they can't do for whatever reason, for them
to suggest other names of people. This is a productive, rapid
and efficient way of connecting the network of scientists. Since
you have multiple reviewers in most cases, then of course you
can test out the reviewers a little and build up a track record
Q19 Stephen Metcalfe:
How do you ensure that the whole system is impartial and that
bias does not creep in at any point? The Chairman touched upon
this issue. Particularly where you have very small groups, who
ensures that that is an impartial process?
Dr Parker: It is
synoptic, really. You have editorial boards that oversee the quality
of the journals. They review the quality of the decisions that
have been made and they oversee the content of the journals in
a retrospective sense. There is always the possibility with all
journals of appealing any decision. Appeals are dealt with very
seriously. They are taken to fresh referees. Usually, you try
and pick out particularly senior referees who you really respect.
You respect all your referees, hopefully, but there are certain
senior referees who you would particularly respect. Sometimes
they go to the editorial boards.
Q20 Stephen Metcalfe:
Dr Parker, do you allow all appeals through? If people don't like
the outcome of the review, can they just keep sending it back?
Dr Parker: All
appeals are dealt with, yes. We would always deal with appeals
Q21 Stephen Metcalfe:
For how many times would you allow that process to go on?
Dr Parker: Not
American journals have fixed rules. The Physical Review,
for example, has fixed rules about that. They have two layers
through which you can go. If you fail at the top, with the editorial
board having thrashed it out firmly, then the decision would be
no, we are not going to take this any further.
Dr Gulley: We have
some similar processes. Having a combination of the internal editors
as well as the external editors helps with impartiality. There
is also the option for appeals, as Dr Parker has said. We also
have an external science advisor that we can call on as well if
we need somebody to assess that we have actually followed the
procedures correctly. There are other options that we can look
at as well.
Q22 Stephen Metcalfe:
Obviously, training is a key tool in this. Can you describe the
training that the editors, the referees and the reviewers receive?
Is that a continuous process? Is there a continuous professional
development, not only just keeping up to date, but is that a structured
Dr Parker: We don't
have any structured training programme for that. You don't know
how often you are going to use a referee. Some referees get used
a lot. Some we will use more than 100 times a year, for example.
Some you might only use once a year because of their specific
subject area, but you look at their results over a period of time
and how accurate their responses have been. If referees need any
specific feedback, our editorial staff will give that.
Q23 Stephen Metcalfe:
You mentioned that earlier. Again, is that structured? Is it
a formal process?
Dr Parker: No.
It is ad hoc.
We should not lose sight of the fact that we have a large scientific
community doing this. I alluded to the fact that PhD students,
for example, are trained, as part of their learning process, to
understand how to criticise and to find out what is right and
wrong with the scientific literature. That process is something
they go through and it carries on with post-docs.
I wanted to raise one other issue related to this.
It is important to recall that the ultimate test is the data at
the end. If a journal repeatedly publishes very unadventurous
things, it will soon be left behind by those who are rather keener
to publish more exciting things. But those who go too far in that
direction, of course, run risks, too. It is a question of how
you get that balance right.
Dr Gulley: In regard
to training, most journals will offer referees guidelines to which
they can refer. IoP Publishing has referee reports where we try
and guide the referees through some of the things that they should
be looking at, such as the quality, if it is correct, and the
methodology. Depending again, on the subject area, it is very
much tailored to the research area that we are working with. This
is from overall feedback from the editorial boards. Again, at
conferences we try and run workshops where we would offer basic
training in refereeing. We explain what it is and what is expected.
That is internationally as well as within the UK.
From a referee's point of view, something that I found extremely
educational is to be sent back the referee reports of the other
referees. There are several times when I have wanted to kick myself
for missing something that the other reviewer spotted that I had
not. Equally, it is not uncommon to find that you are in complete
agreement. It can be an educational benefit.
Is that a standard practice?
It varies. Access online to the views of other referees is quite
widespread in my own field. The policy, back in the pre-electronic
era, was that you were sent the hard copy from some of the better
journals, but not all of them did it.
What about in other disciplines?
It is fairly common in the physics journals, for example, and
certainly in the American ones that I have been involved with,
that, if there is some dispute or argument, referees will be circulated
with the other people's views. That is most instructive and rather
Q26 Stephen Metcalfe:
It does sound like the system around selection is quite ad hoc
at the moment. Do you think that any of it should be more formalisedthat
there should be a standard set of guidelines around which you
work rather than allowing it all to grow around what feels right?
You have mentioned about PhD students taking some training in
peer review. Do you think that that should be a prerequisite of
gaining a PhD rather than something that is nice to have done?
Dr Parker: It could
be. Being a referee is often used as one of the criteria for tenure
in the US. We deal with a lot of requests from US referees, young
academics, wanting a letter of endorsement saying that they have
acted as a referee for the RSC and that they have been reasonably
good at it. It will help them to gain tenure. It is also used
within RSC potentially as part of the criteria for becoming a
chartered chemist through working in academia. So there is a CPD-type
Broadly speakingyou are referring to the training in general
the questions which arise around that are, first of all, the international
aspect. This is a process that is, essentially, identical across
all countries. Arranging for something like that is an international
exercise. The other issue is the question that I raised before,
which depends very strongly on the subject area in which you are
working and the process that they learn how to do. I have referred
to the extremes of, say, pure maths, and the technology of silicon
and biomedical. They have their own areas. Of course, there is
a continuum between those things. They are all interlocking and
are interconnected. One can image a process in a journal, for
example, on a specific subject area, where you could set out rules
like that, but of course we are constantly raising the question
of the boundaries between these things. It is more important that
the training of researchers in general includes the understanding
that they should participate in this process in an open way as
an expectation of being a good scientist.
Q27 Pamela Nash:
From the evidence that we have received so far, it has been claimed
that "the peer review system is in crisis", that academics
and researchers have increasing burdens on their time and there
are few incentives to participate in the peer review system. Can
I ask each of you what your opinion is on that?
It is subject-dependent. I take a slightly different perspective
in answering, which, hopefully, addresses the point, which is
that the complexities and the duration of peer review can impede
the publication of science if it introduces too many distractions
from the principal research programme, but I wouldn't say it is
in crisis. I would say that the engine is misfiring rather than
it has stalled completely.
Dr Gulley: From
the surveys that have been conducted over the last few years,
most researchers have a very high opinion of peer review. In a
recent survey that was done by Sense About Science, about 86%
of researchers said they enjoyed reviewing and there are benefits
to it in that they get to see papers ahead of time and they get
to keep up to date.
From the publishing point of view, we can support
that by making the process as easy as possible.
Dr Parker: I do
not think it is in crisis particularly. One of the challenges
is in building up a core of referees in areas that have a huge
growth in the output of the subject area. For us, it is in chemistry.
It is also the same for Dr Gulley in physics. There is a massive
growth in output from China at the moment. We have been working
very hard to build up Chinese referees and the quality of the
Chinese referees' reports that we use. Building up the referee
base in a linear fashion with respect to their growth is quite
difficult at the moment, because the growth in output is quite
Our experience is that, as a publishing of scientific literature
exercise, it is not a serious problem at the moment. It is possible
to find referees in the way we described. It is not a crisis.
The point we made in our submission to you in paragraph 15 is,
of course, that, inasmuch as peer review is used as a proxy for
other kinds of assessment, that can introduce a pressure on it.
Q28 Pamela Nash:
Dr Gulley, you mentioned that it would be helpful to make the
process of peer reviewing as easy as possible. Do you think that
any incentives for reviewers are needed? What would you do to
encourage more reviewers? For example, would you advocate payment
to reviewers or a formal recognition of any peer review work?
Dr Gulley: We have
different ways of encouraging reviewers. Again, it depends on
the community. There are different views from the researchers
when we talk to them about this. There are different ways of being
able to recognise the work that they are doing. For some of the
communities, we publish the names of the referees in the journals,
for example, and they get recognition that way. Some journals
have different rewards that they give to their top referees, for
example. There are different ways of recognising what they do.
Coming back to what Dr Parker was saying earlier, another way,
particularly within the US, is that we get a lot of requests to
support younger researchers in their applications for green cards.
There is also recognition partly in being involved in the community.
Certainly a strong aspect that comes out when we talk to researchers
is that they feel it is something they do to become part of the
community and stay involved with the community as well.
At the Royal Society the referee is not paid, but we do publish
a list of the referees at the end of the year to formally thank
them for their input.
Dr Parker: We have
asked our boards often about the whole recognition of referees
and remuneration. Remuneration would be a difficult thing because,
if you gave any realistic payment for the time that is involved,
it would be a huge amount of money and it would have to be recovered
from somewhere. It is just moving a financial burden around the
whole system. The system relies on the benefits that people see
from being involved in peer review. There is a quid pro quo as
long as you are someone who publishes as well; you are an author
as well as a referee, which is not always the case.
There are some other advantages, some of which have
been mentioned, like seeing material in advance and, if you do
peer review for high quality journals, then you see some high
quality work and some less high quality work, of course. You get
a chance to be involved in shaping how a subject develops, which
is quite a powerful thing to do.
We should not forget that this is not by any means the only method
by which scientists communicate. The sort of processes that are
being described here happen at conferences all the time. Indeed,
as important as it ever was, is going to talk to somebody about
what is actually happening.
Q29 Pamela Nash:
Dr Parker, you spoke about young academics approaching the RSC
wanting experience of reviewing to further their careers. Do you
think that any formal accreditation for the peer review system
or a more formal definition of that work would be helpful to them?
Dr Parker: It might
be. It would be quite difficult to do, though, because we have
about 33,000 referees all around the world that we use routinely.
Doing something for that number of people could be quite challenging.
Q30 Pamela Nash:
We have heard evidence of some publications using the cascade
system to pass submissions between journals. Do any of you have
experience of this process?
Dr Parker: We do,
yes. We find it does save in peer review time. Authors are often
happy to go along that route. It reduces the time to publication
if the article is not publishable in their journal of first choice.
It gives them a quick route for publishing in what might be a
journal of second choice.
Dr Gulley: It is
exactly the same for IoP Publishing as well.
That does work reasonably well in biomedical sciences, too.
Likewise. You will see
in many laboratories, for example, in the eastern part of the
world that they have a long list on the wall of the journals they
want to publish in. They just go down the list until they get
to one that publishes the article.
Q31 Pamela Nash:
There is unanimous support, then.
Q32 Pamela Nash:
Finally, I would like to ask a question about the Research Excellence
Framework. How is peer review going to be used as a benchmark
of quality? Are there any takers?
Dr Parker: The
REF panel is a peer review panel itself, isn't it? When it was
the RAE before, they always said that they would look at the quality
of the papers themselves. They would read the papers themselves
and wouldn't rely on the impact factors of the journals in which
they had been published. That was stated publicly, at least, in
the chemistry area by the chemistry panel. How they are going
to be used in REF, if it changes, I don't know.
If one just describes peer reviewing as it is termed in the broad
sense as described here as something that is acceptable or not
within a certain journal, then, of course, to some extent one
is only looking at peer reviewed paper submissions as being relevant
to REF. I think you are asking a slightly different question,
which is that peer review for high impact journals, low impact
journals and so forth, as a proxy, as I have alluded to before,
for assessment of quality, is a slightly more complicated question.
It depends on the individual subject area, the journals, whether
certain journals have an assessment of what they consider newsworthy
and what others consider is needed to build on to the knowledge
base and so forth. That depends, to some extent, on individual
journals and how they see themselves and their role within transmitting
and building scientific information. It does vary. That is a complication.
There is also a problem of matching the expertise of the REF panel
to the spread of subjects that they have within that subject area.
That often means that there is no one on a particular panel who
is expert in the exact area of a particular individual being assessed.
So there is a genuine problem. You have to accept a certain amount
of breadth and imprecise match of the expertise to the area that
they are investigating. It is a difficult issue as to whether
or not you can do that by assessing bibliometric criteria or not.
They are a very shaky basis for such a fundamental decision. This
comes right back to Professor Pethica's earlier point about the
proxy use of peer review for other purposes for which it is not
Q33 Graham Stringer:
How can and can't the peer review process be used to guard against
fraud or misconduct by scientists?
In the current electronic age it has become much easier to detect
data manipulation. Initially, there was a problem that data manipulation
itself became much easier because of Photoshop-type programs,
but in practice many journals now routinely examine the data files
to see how the images were prepared. Certainly, in biological
sciences that is becoming increasingly common. That makes scientific
misconduct more difficult.
Errors of interpretation are still very much things
that a good peer reviewer has to sniff out fundamentally. There
has been an attempt, with which you are probably familiar, of
establishing a research integrity office within the biomedical
sciences. That has attempted to look at incidences of misconduct
and to draw up a national code of conduct and a national procedure
for investigation of misconduct which can run alongside the peer
review process. There has been a problem for that organisation
in that it was set up to look at biomedical sciences. Research
Councils UK has wanted to extend its remit to all sciences. One
of the major funding bodies in setting it up was not happy with
it being extended to other sciences. That organisation, which
could have an important role to play, is caught in the very uncomfortable
position between different remits of the bodies that initially
funded it. It could make a useful contribution in that subject
in addition to the standard peer review process.
Q34 Graham Stringer:
Just on that point and we can then come back to the fraud or misconduct
issue, do you think the Government should intervene and put it
on a statutory basis? The similar office in the States is on a
Yes, it is. It is a difficult subject because, if it is done in
too draconian a way, it gets into the difficulty that the initial
organisation in the States ran into, which led to very substantial
criticism in the courts. The current stance of UKRIO is a more
balanced one, but whether or not it should be put on a statutory
basis across sciences or just retained for the biomedical sciences,
which I believe was the wish of the Department of Health representative,
I can't judge. That is a difficult matter. It is something that
does deserve to be looked at. It could perform an important role
for British science.
Q35 Graham Stringer:
Can I go back to the fraud issue? Does somebody else want to contribute
Dr Gulley: I want
to add that, fundamentally, that responsibility lies with the
author but things can be done to help this situation, particularly
on the international setting. For example, we have ethical policies.
Most journals have an ethical policy that they will promote and
ask authors to abide by.
Q36 Graham Stringer:
Should it be mandatory to have an ethical policy?
Dr Gulley: In certain
subject areas there are parts that are mandatory, such as stating
a conflict of interest and certain medical procedures that have
to be stated. The ethical policy that we have is much more general.
It also takes into account, for example, what is ethical or what
is viewed as being ethical in terms of plagiarism, for example,
which is one aspect that I wouldn't say has become easier but
it is being picked up more frequently now that you have much more
electronic access availability.
Dr Parker: There
are also the Committee on Publication Ethics Guidelines that are
pretty much an industry standard now. The difficulty with fraud
is that the whole peer review system relies on people being ethical.
That has to be balanced with what happens when you find that someone
has not been ethical. In the relatively rare cases when someone
has not been ethical, it will usually be picked up by a reader.
If it is not picked up by a referee, if it actually gets through
to publication, it should be picked up by a reader and then it
is usually dealt with either by the reader coming to the editor
of the journal or the reader going directly to the author and
dealing with the matter.
Q37 Graham Stringer:
The most recent fraud-related case is that of Andrew Wakefield,
which took 10 years to sort out, even though the journal in which
the article had been published had been approached after two or
three years showing that there had been bioselection and that
some of the figures had been altered. Why did that take so long?
Should the process be altered in view of that experience?
I don't know the details of why it took so long. That is not something
I am competent to answer. It does illustrate a concern that we
expressed in the written submission about it often being more
difficult to firmly dismiss incorrect information in the biomedical
sciences. That is a further reason why peer review is crucially
important. How that can be addressed more rapidly is hard to know.
It becomes particularly difficult once the media are involved
and everything is scrutinised openly.
I believe there is a tendency, unfortunately, for
people to be more reluctant to come forward and speak openly when
they think it is something of a major public issue. That, I accept,
is not the position that we prefer to see, but there is a tendency
of people to be reluctant to enter a public storm.
I can give you an example from the physical sciences, which is
rather more obvious, which is the case of Schön and the proposals
he had for various solid state structures, which, of course, caused
a great storm of excitement until people started to do the kind
of analysis of the data that had been described, and it all fell
apart. That took about a year and a half before people were convinced
that it was a fraudulent process which people had tried to repeat.
Here we are faced with a number of diverse paths.
One is the question of ethics that institutions, indeed, should
have. Many of the research institutions, be it the universities,
national laboratories or whatever, do have expected ethical behaviour
of their staff. Then there is the question of whether you will
catch it by peer review. Of course, the peer review process is
designed to try and catch these things, but, by the nature of
things being imperfect, something will eventually get through.
The numbers are fairly small, as we have seen. Then the question
is what happens after that? To some extent, that will depend on
the nature of the subject, the complexity and so forth. In a way,
it is inevitable that a scientific fraud will eventually be uncovered,
as we have seen in other cases. The question is: how do we shorten
the time scale and prevent these things getting out before they
cause media damage?
Q38 Graham Stringer:
Do you think there should be a code? Just using the Andrew Wakefield
caseI am sorry if you are not familiar with it in great
detailit was a journalist who was pushing the issue that
there had been a fraud. He went to the journal and the journal,
effectively, got the co-author to review what was going on and
excluded the journalist. Should there be a code of ethics or a
process for dealing with external complaints where fraud is suspected
with at least some evidence?
The UKRIO is attempting to achieve that. It is attempting to provide
a first point of call for people who seek advice on how to proceed
in examples of suspicion of fraud. It has drawn up a national
procedure which has been widely published and distributed to
universities and other research institutions. Again, the very
nature of fraud is that it is inherently difficult to prove that
it has occurred.
Dr Parker: It is
something that is also very subject-dependent. The Wakefield fraud
relied on clinical trials and statistics. You can understand
why that might take a bit longer compared with something in the
physical sciences area that could be repeated by someone else
relatively quickly and might be right or wrong. It is a different
Q39 Graham Stringer:
You have given a fairly dry account of what peer review is like.
It sounds unexciting. The insight we got into peer review from
the leaked e-mails at the university of East Anglia made it look
like a pretty tough contact sport where people were taken out
on journals and careers were threatened. What is the accurate
scenario? Is it the fairly desiccated view that you have been
giving us or is peer review a street fighting business where careers
are threatened? Where is the better insight?
Dr Gulley: From
my experience, it is probably closer to what we have described
so far. There are instances where you do get the street fight-type
scenario, but that has been very rare in my experience over the
last 14 years.
I think the rather dry flavour that we have left you with is probably
a more accurate description of the majority of cases. There are
a minority of exceptions.
Dr Parker: I am
sorry to have to agree with that. Yes, it is a rather dry subject,
but exactly the number of cases you get that have a big and florid
excitement about them are relatively small.
Q40 Stephen Mosley: Does
the publication of fraudulent or incorrect papers that have been
through the peer review process damage the public perception of
peer review as a mark of quality?
It damages the public perception of science as a whole and I think
that is extremely unfortunate.
If a particular journal does that kind of thing, it affects that
journal's reputation within the scientific community, which is
a significant matter too.
Dr Parker: I doubt
that the general public has much of a perception of peer review.
They have a perception of science, that scientists do experiments
and that they publish them. They probably don't really care that
much about peer review, although the Wakefield incident and the
UEA climate data issue have brought peer review a bit more to
Not to entirely leave it as unexciting, if a lot is at stake,
then the peer review process will tend to be more exciting.
Q41 Stephen Mosley:
You can also have a situation where a peer reviewed article may
disagree with a previously published paper, and that is perfectly
legitimate. If you have a situation where there is some perception
of doubt against the peer process, I guess it makes it difficult
to judge whether this is a proper result or not. Is that the situation?
Are you more wary of research that contradicts previous research
now than you might have been previously?
If you divide it by subject areas, the paper that comes along
and tells you that they think thermodynamics is wrong is not likely
to get much of a listening. There are such papers, I should stress,
that still come in. At the other end, there are problems that
are sufficiently broad where the information, the types of experiments
and so forth are not sufficiently defined where it is rather difficult
to be sure. There is a continuum of those things.
Dr Gulley: On the
other side, you also have areas where it is still evolving and
different models are going forward. For example, modelling of
the universe is a good example of that, where you will have contradictory
models that will evolve until you start to get some of the data
that can back up some of the theoretical models. That is part
of science discussion.
That is what it is.
Q42 Stephen Mosley: When
it comes to public perception and to the perception by policy
makers, how do you think the perception of peer review and scientific
research can be improved?
Perhaps I could venture a comment. Peer review has worked in the
sense that the scientific literature we have is coherent and it
has effects on the world around us which everybody can see. As
to the notion that it is a substitute for getting things absolutely
right every time, it would be useful if the public becomes aware
of the fact that mistakes happen. It is just that we try and minimise
Q43 Pamela Nash:
I would like to move on to international issues regarding the
peer review system. Are there any perceived differences in the
quality of peer review dependent on the country where the publisher
of a journal is based?
No, basically, to cut a long story short.
There are serious attempts to minimise those differences.
Q44 Pamela Nash:
Do you see that there are differences to be minimised?
No. The harmony outweighs the differences.
More than that, it is becoming more coherent. In the past, foreign
academies would have certain rules. For example, papers had to
be submitted for approval by certain structures. It is unquestionably
the case that international competition in this sense, as a consequence
of impact of the science and technology, has driven a convergence.
It is hard to say that there is any real detectable change. That
is, of course, enforced by the fact that journal reviewers themselves
are now drawn from across the world, be the journal UK-based,
Chinese, Brazilian, in the US or whatever. The process is essentially
the same and they all participate internationally in that process.
Q45 Pamela Nash:
You mentioned new technology in use in the peer review system.
To what extent do different publishing organisations share best
practice in terms of using new technology and online systems?
There are standard IT packages now.
Dr Parker: A lot
of publishers use the same or very similar packages. Publishers
collaborate on various things like linking references, but CrossRef
as a collaborative publishing group of publishers is also working
on anti-plagiarism software and things like that. There are things
that are shared across.
Q46 Pamela Nash:
How effective are these tools? You have mentioned the anti-plagiarism
tools. How is that being developed?
Dr Parker: That
is being developed at the moment.
It is fair to say, of course, that IT technology advance is a
constant battle. People on one side are doing cunning things and
on the other side they are advancing the technology. So anti-plagiarism
works for certain kinds of things, but it is an arms race, almost,
if you like.
Q47 Pamela Nash:
Is the technology at a level yet that it is benefiting the peer
review system or is it more of a hindrance in that the technology
has not caught up yet?
Dr Gulley: My experience
on the plagiarism side of things is that we are finding that we
are picking things up more before they go out to the referees,
for example. It is minimising the burden on the referees. When
articles are submitted to us, we can check them against what has
already been published. It is definitely helping in that respect.
In data manipulation, the software is now picking up cases. You
rarely hear about those because the journal simply declines to
deal with that author in future. There are cases of data manipulation
being detected by software.
Dr Parker: We have
done quite a lot of work on, essentially, running macros on the
articles that we are going to publish which check the experimental
data for consistency. It is the technical detail. You can check
that the spectra and the data are consistent with the number of
hydrogen atoms that are in the molecule that you have in the reported
structure. We do things like that which help to pick things out.
Q48 Pamela Nash:
I will go back to my original question to look at review internationally
from a different angle. Are you aware of any differences in the
quality of peer review carried out through UK-based journals by
reviewers from different areas of the world? I realise that this
might be a bit of a sensitive question.
Dr Parker: Publishing
is so globalised now that there are very few journals that are
based within a particular nation or are very isolated. Most of
them are globalised. There is very little difference in quality.
We referred earlier on to the fact that you look at the review
referee's performance. Of course, I can't comment on specific
cases but that is a factor.
Q49 Pamela Nash:
I am not asking just about the skills of reviewers from another
country but perhaps the facilities and funding that is available
to them in the UK. Do we take that into consideration or should
we take that into consideration in choosing reviewers?
Dr Parker: That
is an interesting question, isn't it? There have to be sufficient
experimental data in the area that we publish in to justify the
conclusions that are being drawn. If there are sufficient experimental
data, that is okay. The difficulty comes where, if there is someone
working in a developing country somewhere, they don't have access
to specific sorts of technology that would give them definitive
experimental data. One of the things that we are able to do as
a society is to try and work with people in those areas, to try
and develop sustainable clinical research, which we do, for example,
in sub-Saharan Africa.
You might have some views working in a country with incredibly
poor facilities, but there is a glimmer of something special in
that person's work. If you just judged everything by the standards
of access to laboratory facilities that you take for granted,
doesn't that squeeze that group of people out of the publishing
Dr Parker: Sometimes
it is an advantage because you will get referees offering to work
with people and it can set up collaborations. If you have a referee
who really sees the merit in this bit of research that has come
out from someone working under very difficult circumstances, they
could offer to set up a collaboration potentially. Those very
often are supported through societies as well. They support collaborations.
It is fair to say that that is one of the reasons why people in
those circumstances are often involved in refereeing for highly
theoretical subjects where that disadvantage does not apply, and
that is certainly widely used in mathematical and theoretical
Q51 Pamela Nash:
Dr Parker, you mentioned earlier about the growth of scientific
research in China. How do you support and develop peer review
skills in China and other emerging regions of scientific strength?
Dr Parker: We do
a lot of interaction with the Chinese academic market, as it is.
We have two offices in Chinaone in Beijing and another
in Shanghai. We have staff out in China. We do regular visits.
We set up conferences in China now. We started off doing roadshows
of the top chemistry departments in China. All of our roadshows
include presentations on how to publish and how to referee. We
have built up quite a significant connection with the Chinese
academic market. We also involve them on our editorial boards.
We get them involved as associate editors on our journals.
Dr Gulley: It is
the same for us as well. We have been working with researchers
in China for the past 11 years. We have a member of staff who
visits universities and gives lectures on how to get published.
We run workshops and we visit regularly. Again, we build up those
links and liaisons. We also work with a number of publishing partners
in China who publish their own journals. We certainly work closely
with them on looking at peer review and internationalising their
journals as well.
As you might expect, we work closely with the Chinese Academy
of Sciences on this.
To what extent do you share best practice amongst publishing organisations,
particularly in terms of evolving software and so on?
Dr Parker: There
are trade associations for publishing. Publishers get together
at those trade associations and at events like the Frankfurt Book
Fair. They share non-competitive knowledge as much as possible.
Publishers are really quite collaborative these days, much more
so than they used to be 20 years ago. They work together a lot
more on common issues like anti-plagiarism, reference linking
and those sorts of things.
It is through the trade fairs and conferences.
Dr Parker: Yes.
Dr Gulley: There
are some shared guidelines and recommendations that come out of
My next question is a slightly amusing example that occurred to
me when I was congratulated for getting on the Booker shortlist
for publishing Oxygen, but it was another Andrew Miller.
How big a problem is ambiguity of names? Do you use systems like
ORCID to help track authors?
Dr Parker: We are
trying to work with ORCID at the moment. That is a developing
situation. There will be an author tracking ability in a relatively
short time. It is an issue, particularly in places like Korea,
where there are only five or six really common surnames. You get
an awful lot of people with the same name. For example, we had
two people with the same name both in the chemistry department
at the university of Oxford. They both had very much UK names.
We try very much to keep those people and their records separate.
A bigger problem is proliferation of records by the same person.
It can be an issue.
Dr Gulley: ORCID
is a very good example of the collaboration, where it is required,
and it will be a solution to that problem.
Q55 Stephen Metcalfe:
Can I ask for your views on post-publication peer review and commenting,
whether any of your journals do that and what your experiences
Dr Gulley: Currently
none of our journals do that. There are experiments within the
industry that are trialling this. It will be interesting to see
how they progress.
Dr Parker: We don't
do it. It is another layer. It is something in addition to pre-publication
peer review. Where there is an issue, you should hear pretty quickly
from readers or whoever, anyway, so it is a way of opening that
up, I suppose, more generally speaking.
We do use it. We also have a system that is called eLetters. Is
it useful? Not really, because remarkably few people choose to
use it. It is important to keep in mind that the implication that
once something is published in the peer review literature, that
is it, and it is set in stone. As I alluded to before, much of
a PhD student's training is the process of assimilating over a
long period of time the scientific literature, deciding what is
good about it and what is bad about it and then allowing them
to progress from there. This process is inherent in the entire
scientific enterprise, anyway, in the training of people getting
involved therein. Most PhD students, for at least a year or a
year and a half, try to figure out which way is up in the scientific
literature, which is that process.
Q56 Stephen Metcalfe:
So you don't see it growing as a trend?
It is already a central part of the enterprise.
Stephen Metcalfe: It is
It is nice to have it. It is implicit in the fact that people
publish subsequent papers saying, "X was right, Y was wrong,
and we did this and produced that." That is implicit in the
whole structure of scientific papers; they have a preamble about
what has happened so far. To some extent it exists already.
Q57 Stephen Metcalfe:
It is not going to change the value of the pre-publication review.
It is not going to take away from that because it already exists.
In post-publication terms, it is, effectively, the process. That
is why at this point scientific literature is supposed to be a
coherent structure rather than a series of random samples.
In biomedical sciences, the Faculty of 1000 does provide a post-publication
assessment of the value of papers, and, if there is a move towards
publication in journals such as PLoS ONE and where impact
is less important, then a subsequent impact assessment such as
the Faculty of 1000 could become increasingly important.
Q58 Stephen Metcalfe:
Is the Faculty of 1000 welcomed by the academic community? Is
it well supported?
I think so. Its use is patchy but it is recognised as providing
a valuable service.
Q59 Stephen Metcalfe:
Has social media, by which I mean blogs, etcetera, had an impact
on this process at all? Are they helpful, or is it just a proliferation
of unchecked views?
There will be a change of view depending upon the age of the person
to whom you are asking that question. With the research students
it is quite common. As one gets somewhat older, the utilisation
is probably less.
Chair: It is the same
in this building.
Dr Parker: People
are relatively reluctant to blog on things at the moment, but
they like to see what other people are reading. If there is some
way of seeing what other people find interesting, that is where
the Faculty of 1000 comes in. It is a positive thing. Everybody
wants to be read by the best people.
Q60 Stephen Metcalfe:
So you don't see that as having a significant impact at the moment.
Dr Parker: Not
at the moment.
Dr Gulley: It can
add to something in the future. It is also an aspect that people
are starting to explore around how they explain their science
as well to a much broader and more general audience.
Aren't blogs used to help promote a piece of work that is being
Chair: We find that modern
technology is a very useful way of getting out to the broader
scientific community what we are doing.
Dr Parker: We use
blogs for trying to promote particular articles that we think
would be newsworthy or interesting to a wider audience. Some are
more successful than others.
Dr Gulley: It is
certainly a way to raise visibility. Again, for some articles,
it is more successful than others.
The challenge is making it a two-way process, though.
Chair: Graham described
peer review as a dry subject. As I said to you at the beginning,
some of my colleagues were up into the early hours, but you have
kept us awake and interested. Thank you very much for an informative
1 Note by witness: Professor Pethica's response of
"Likewise" could be interpreted as an indication that
the Royal Society operate cascading peer review. We do not. Back