House of Commons - Peer review in scientific publications

Peer review in scientific publications - Science and Technology Committee Contents

3 Editors, authors and reviewers

90. At the heart of the peer-review process are the people involved: editors, authors and reviewers. Dr Robert Parker, Interim Chief Executive of the Royal Society of Chemistry (RSC) told us that "having professional people overseeing the peer review process is absolutely paramount".[159] We also heard that:

Peer review or expert review is as good as the people who do it. That is the key challenge. It has to be used wisely. It is about how the judgment of experts is used. It is about balancing one expert opinion against another. The challenge is not whether peer review is an essential aspect of scholarship because there is no alternative to having experts look at things and make judgments.[160]

91. Peer review is regarded as an integral part of a researcher's professional activity; it helps them become part of the research community. The International Association of Scientific, Technical and Medical Publishers explained that "as every active researcher expects to publish and through peer review receive constructive critical comments on their work, so they too must expect to act as a peer reviewer for others".[161] It is a reciprocal activity; most researchers acknowledge this. Dr Nicola Gulley, Editorial Director at Institute of Physics (IOP) Publishing Ltd, further explained that "in a recent survey that was done by Sense About Science, about 86% of researchers said they enjoyed reviewing and there are benefits to it in that they get to see papers ahead of time and they get to keep up to date".[162] However, others have reported that "for many the review process is perceived as a 'chore and not a pleasure'. Reviewers feel this way because they are not rewarded or recognised for their work".[163]

The role of the editor

92. There are currently approximately 6,000 publishers around the world managing somewhere in the region of 25,000 peer-reviewed journals; publishers have become "stewards of the peer review process on behalf of research communities".[164] Broadly speaking, there are two types of journal editor: internal staff editors and external (academic) editors who are active researchers (see paragraph 101). The role of the editor is "central to the quality of the peer-review process".[165] The RSC explained that:

It is the editor who will consider the information produced through the process and so ultimately decide what feedback is communicated to the author and which articles are published. The judgement applied by the editor to the information collected in the review process requires knowledge, skill, and care.[166]

93. The British Sociological Association also recognised the importance of the editor in safeguarding against problems in the peer-review process.[167] This could include monitoring and preventing bias, looking out for signs of research fraud or misconduct, and ensuring feedback and requests for further information from reviewers to authors are rational. The latter is becoming an "increasingly troublesome" problem.[168] Professor Ron Laskey of the Academy of Medical Sciences explained that in the biomedical sciences:

a high proportion of time is spent fending off criticisms from reviewers that may not be on the main theme of the work. The reviews are beginning to dictate the agenda of the science in a way that is not fully productive. That can be frustrating, a waste of time and resource.[169]

94. Reviewer-suggested experiments were the subject of a recent Nature article, which suggested that "the problem is made more acute by the unwillingness of editors to express their opinions".[170] Dr Philip Campbell, Editor-in-Chief of Nature and Nature Publishing Group, told us that as a result of the remarks made in this article he had questioned his editors to find examples of "recent publications which had had to be revised, but where we had made a judgment that in this particular case this request for extra work was not required".[171] Dr Sugden, Deputy Editor & International Managing Editor at Science, explained that:

Often you will get two or three referees' reports on a paper, but those referees may not agree with each other. It is the editor's job, if they consider the paper worth pursuing, to then make a recommendation as to which of those referees' revisions they should follow and which they should not.[172]

Mayur Amin, Senior Vice President of Research & Academic Relations at Elsevier, added that at Elsevier feedback was collected "from the researchers, authors, reviewers and the editors" so that as publishers they could "take that on board and present it to an editor or a journal and say, 'Look, a whole lot of authors are getting displeased about the way the process is working. We need to modify the process'".[173]

REVIEWER SELECTION

95. One of the core decisions made by an editor during the peer-review process is who reviews the manuscript. Professor John Pethica, from the Royal Society, described how this decision is taken:

One can keep a record of how effective various reviewers are, which is done by most journals. Some people are more effective than others and are used correspondingly. Also one uses the community to suggest future names of reviewers. It is very common, for example, if a senior scientist is asked to review something and they can't do for whatever reason, for them to suggest other names of people. This is a productive, rapid and efficient way of connecting the network of scientists. Since you have multiple reviewers in most cases, then of course you can test out the reviewers a little and build up a track record on them.[174]

Dr Parker, from the RSC, added that:

Building up a knowledge of the community is very important. […] People do get to know a particular area and the interactions between certain authors and referees very well. You do get to know your community and you get a feel for whether there are any issues between particular people.[175]

96. For journals with staff editors, building and maintaining that relationship with the research community is achieved through attending conferences and seminars, as well as visiting universities and industry.[176] Dr Parker told us that RSC editors "regularly attend up to 200 conferences a year overall".[177] Dr Gulley, from IOP Publishing Ltd, indicated that their editors also attended a large number of conferences, in the region of 300-400 a year.[178]

97. Selecting the right reviewers for the job is a particularly important way of combating bias in peer review. Dr Gulley explained that "having a combination of the internal editors as well as the external editors helps with impartiality".[179] She added that there is also the option for authors to appeal if they disagree with the final editorial decision.[180] In addition to this, authors might also choose to take up their concerns in a public arena. A recent example of this is the open letter by 14 leading stem cell researchers to senior editors of peer-reviewed journals publishing in their field (see paragraph 77).

98. Bias in reviewer selection does not always work against authors. In the past, there have been accusations that top journals, such as Science and Nature, "are locked in such fierce competition for prestige and publicity that they may be cutting corners to get 'hot' papers".[181] The UK Research Integrity Office Ltd (UKRIO) drew our attention to the fact that "the Nobel Laureate, Robert Laughlin, commenting on a series of retractions from these eminent journals said 'in this case the editors are definitely culpable […] they chose reviewers they knew would be positive'".[182]

99. Dr Philip Campbell defended Nature against these accusations:

That is completely wrong. I totally refute that statement [...] It is not in our interests to cut corners. […] we have one of the most critical audiences in the world, and any paper that makes a strong claim is going to be absolutely hammered in the form of testing in the laboratory or scrutinised in terms of discussions at journal clubs, within universities and so on. It is simply not in our interest, for our reputation in the long run, to publish papers that have any degree of cutting of corners in the assessment process.[183]

Dr Campbell added that after a "hot paper" is published, though there is "an immediate stream of interest", there is no "direct effect on sales".[184] He explained that "there is a big barrier of independence, institutionalised within the company, in fact, between the commercial side and the editorial side".[185]

100. The role of the editor is at the heart of the peer-review process. The judgement applied by the editor to the information collected in the review process requires knowledge, skill, and care; particularly, in respect of identifying the right reviewers for the job and critically assessing the feedback from reviewers to authors.

Training

EDITORS

101. Publishers use a variety of arrangements for editorial responsibility during the peer-review process. Broadly speaking, the two main approaches are to appoint staff editors as in-house professionals, or to use editorial boards consisting of active researchers. Regardless of whether journals opt for the use of staff editors, academic editors, or a combination of both, some form of editorial training is necessary—especially in the light of the central role of the editor (paragraph 92).

102. The RSC and the IOP use "a combination of in-house editors and external editors",[186] as does the journal, Science.[187] Dr Andrew Sugden told us that the initial filtering to identify "innovative" and "original" submissions at Science is carried out through consultation with a Board of Reviewing Editors.[188] This Board is appointed by the staff editors and consists of mid-career active research scientists. "The responsibility for managing the peer review process and for making decisions on rejection/revision/acceptance of submissions for publication rests with the staff editors".[189] In contrast, Dr Philip Campbell, Nature, explained that:

Nature and the Nature journals are untypical journals in that they do not have editorial boards of active researchers. All selection decisions are the responsibility of the fully independent and Chief Editors of each journal and their teams.[190]

103. During the course of this inquiry, we questioned a number of publishers about the type of training they provide to their editors, both in-house and external. On the whole, training for staff editors appears to be provided on the job.[191] Dr Philip Campbell explained the situation at Nature:

The training that takes place [happens] by [staff editors] participating fully in the process of selecting papers. Every new editor sits within a small team with a team leader who will initially track their every thought and action in respect of every paper they handle.

As months go by, this scrutiny gradually relaxes. We reckon that it takes about two years of handling papers and visiting many labs and conferences for our editors to gain the full experience of the various ways in which authors, editors and referees can interact and hence optimize the process. Also, over that time, an editor builds up extensive scientific and research-community knowledge and contacts.[192]

104. Training for academic editors and editorial boards—at those journals that use them—varies. The Public Library of Science (PLoS) told us that its:

[academic editors] and their editorial boards are supported by PLoS staff, who provide initial training and ongoing support in the use of the journal management system. PLoS staff also send occasional communications on best practice to the editorial boards […] The journals have an electronic discussion facility so that all submissions can be discussed with colleagues on the journal or with editors who work on other PLoS journals (on a confidential basis). The PLoS staff editors are occasionally brought in to discussions to provide support on specific content issues or matters pertaining to publishing ethics.[193]

105. A more structured approach is taken by Elsevier, which provides its new external editors with:

a Welcome Pack which, in some 50 pages, introduces new Editors to Elsevier, its policies, procedures, the editorial and publishing teams which support the journal, the peer review process including tools to find reviewers, ethical guidelines, as well as support tools.[194]

The journal, PLoS ONE, also provides newly recruited editorial board members with "a pack of information providing guidance about the editorial process and standards associated with PLoS ONE", as well as "videos explaining the operation of the journal management system". Additional support and ongoing advice are provided by PLoS ONE administrative staff.

106. Broadly speaking, training for editors and members of editorial boards is provided on the job. We have heard that some publishers opt for a more structured approach, and include, for example, comprehensive welcome packs for new editors that cover peer-review processes, support tools and ethical guidelines. We encourage publishers to work together to develop standards—which could be applied across the industry—to ensure that all editors, whether staff or academic, are fully equipped for the central role that they play in peer review.

AUTHORS AND REVIEWERS

107. In addition to training their editors, some publishers also provide feedback or training for authors and reviewers. Dr Robert Parker, from the RSC, told us:

We have a feedback loop where referees always get the feedback on the outcome of the articles that they have refereed so that they can learn whether their refereeing activity is generally in line with what is accepted and what is rejected.[195]

108. He acknowledged, however, that the RSC did not run a structured training programme and that the feedback was provided "ad-hoc".[196] Professor Ron Laskey, Vice President of the Academy of Medical Sciences, considered feedback to be very helpful. He told us:

From a referee's point of view, something that I found extremely educational is to be sent back the referee reports of the other referees. There are several times when I have wanted to kick myself for missing something that the publisher spotted that I had not. Equally, it is not uncommon to find that you are in complete agreement.[197]

However, while feedback is common in some disciplines, it is by no means standard practice across all journals.[198]

109. Publishers are increasingly offering more training opportunities to reviewers, albeit in a sporadic way. Dr Janet Metcalfe, from Vitae, explained that bringing early-career researchers into the peer-review system was particularly important:

How do you get into that system? How do you become a reviewer? It is very often by recommendation. There are journals that have open calls for reviewers, but becoming a reviewer is usually part of the apprenticeship of being nurtured as a researcher by your principal investigator or senior academic. There are issues in terms of how we support those researchers to become involved and good at peer reviewing on both sides of the fence, but also how we recognise it by acknowledging the broadness of a researcher's activities.[199]

110. We heard examples of how publishers are addressing this challenge. Dr Nicola Gulley, from IOP Publishing Ltd, told us that:

Recently, as a result of requests from some post-docs and graduates, we have given them some initial training on what peer review means. We are teaching them about what refereeing means and what we are expecting. There is a lot of literature as well that people are not always aware of so we have been trying to raise the visibility of that. Internally, we also try and match the interests of the referees to the papers as much as possible.[200]

111. Elsevier is also working with postdoctoral students on peer review. It has developed a "Reviewer Mentor Programme" whereby:

experienced editors employed at two universities mentor postdoctoral researchers who have authored papers but not yet served as peer reviewers. Each mentor runs training workshops for the postdocs and then the postdocs review real articles under supervision. Each postdoc is marked, and upon successful completion receives a certificate. We are exploring ways to provide formal certification and a reviewer kite mark to scale up this successful pilot.[201]

112. Professor John Pethica, Physical Secretary and Vice President of the Royal Society, explained that "PhD students […] are trained, as part of their learning process, to understand how to criticise and to find out what is right and wrong with the scientific literature".[202] He added that it was "important that the training of researchers in general includes the understanding that they should participate in [peer review] as an expectation of being a good scientist".[203] Some concerns had, however, been raised about the lack of training in best practice for new reviewers, with suggestions that this should form part of post-graduate training.[204] We, therefore, questioned whether peer-review training should be a formal part of gaining a PhD. Sir Mark Walport, Director of the Wellcome Trust, told us:

Part of the training of a scientist is peer review. For example, journal clubs, which are an almost ubiquitous part of the training of scientists, bring people together to criticise a piece of published work. That is a training in peer review. Can more be done to train peer reviewers? Yes, I think it probably can. PhD courses increasingly have a significant generic element to them. It is reasonable that peer review should be part of that.[205]

113. Professor Rick Rylance, Chair-elect of Research Councils UK (RCUK), was broadly in agreement with Sir Mark's comments. He added that "research is a collective enterprise and that anyone who wishes to enter that field either as an academic or in some other capacity needs to understand that".[206] Dr Janet Metcalfe, Chair of Vitae, provided more details about the current opportunities for new authors and reviewers in universities and research institutions:

The tradition is very much an apprenticeship model. You learn the system by doing it in terms of writing papers, submitting them and maybe getting feedback from your principal investigator [PI]. Where that works it is absolutely fantastic […] But, because we are a collective in terms of the academic community, there is opportunity for that process not to be as well supported throughout the whole of the academic community as it could be.[207]

When we asked Dr Metcalfe whether she was in favour of more formal training, she responded:

I think the opportunities to have training should be there. The process by which a researcher learns to become expert is very much up to their individual circumstances. If they are getting good individual nurturing and mentoring by their PI, that is great. But there should also be the opportunity, for those researchers who respond more to formal training, to have that available as well.[208]

114. Professor Ian Walmsley, from the University of Oxford, agreed that "a combination of both mentorship, which I think has a primary role, and some elements of non-mandated training would continue to be very helpful".[209]

115. Others were in favour of formalised training; for example, the British Medical Association (BMA) stated that:

It is remarkable that there is no formal training process in place for such an important mechanism to ensure scientific quality. Guidance from a publisher alone, who may have parallel but different priorities, is not adequate. The BMA favours a system that provides proper peer review training as an option within postgraduate training.[210]

116. Professor Sir Adrian Smith, Director General of Knowledge and Innovation in the Department for Business, Innovation and Skills (BIS), considered that it was not a "one size fits all" situation, he told us:

We have to allow a lot of scope for particular research organisations or supervisors to decide on what is appropriate. Peer review training is already part of the Research Councils' postgraduate training. There is a formal expectation that students […] "obtain an understanding of the processes for funding and evaluating research." The terms and conditions of training grants actually put some of this in. If you think about it, if you are doing a PhD, you are having to read and access a lot of literature and synthesise that literature. […] It is an inherent part of the scientific process itself that you are constantly peer reviewing in a way. […] The amount of effort that has gone on in recent years on the part of the research councils to better codify their expectations of what research training should consist of and making that part of the conditions when they give out either doctoral training grants or research grants takes us most of the way. I do not think there is much that we could do in going further.[211]

117. Professor Sir John Beddington, the Government Chief Scientific Adviser, added that:

a number of universities have exercises where PhD students and some academics examine individual papers. In that case, everybody goes away, reads a paper over the weekend and then they have a meeting and discuss and critically appraise that paper. That is part of the process. Obviously, that practice will differ between universities and subject areas.[212]

118. A relatively straightforward way of educating reviewers about the quality of their reports and helping them improve their feedback to editors is to send them the reports of other reviewers, done confidentially when necessary. This should be standard practice across all journals. This would be a useful educational tool to improve the quality of future reports from reviewers.

119. Training for the next generation of authors and reviewers is also important. Many PhD students and post-doctoral researchers are fortunate to have the opportunity to discuss scientific literature in journal clubs and other informal settings. Some are mentored well by their principal investigator and thereby receive informal training in peer review. Others are not. Given the importance of peer review across the research spectrum, from grant applications to publications, we consider that all early-career researchers should be given the option for training in peer review.

FUNDING FOR TRAINING

120. Training in peer review, whether ad-hoc or in a formalised setting is clearly desirable; we therefore examined where funding for this training would come from. Vitae, the UK organisation championing the personal, professional and career development of doctoral researchers and research staff, explained the current situation:

Until recently there were few opportunities for researchers to undertake formal training. The advent of Vitae and government funding through the UK Research Councils for implementing the recommendations of the Sir Gareth Roberts review[213] have significantly increased the opportunity for early career researchers to participate in professional development opportunities, including academic writing for publication and grant applications. These courses generally include experience of the peer review process. There are also examples of universities and other bodies providing structured development opportunities in being a peer reviewer, including encouraging early career researchers to set up and run journal clubs.[214] However, the numbers participating in these activities are fairly small and with the end of 'Roberts funding' in March 2011 even this level of provision may […] fall.[215]

121. Roberts funding of just under £150 million was provided to the Research Councils in the 2002 Spending Review to "increase stipends, length of doctoral programmes and provide training for their funded researchers".[216] We asked Professor Rick Rylance, from RCUK, how training in peer review would be funded in the absence of Roberts funding, he responded:

The amount we are giving to universities for training and developing postgraduate research will increase, and it will include components which replace part of the Roberts funding. The issue we have to think about is that, on average, around only 25% of the UK postgraduate population are funded through agencies like the research councils. The rest of it is coming through other sorts of routes. How are universities going to provide a system for three quarters of the population who are not getting money from us? There has to be a joined-up conversation about how we develop that.[217]

122. Some of the other funders that Professor Rylance referred to are also providing the opportunity for training to be incorporated into the PhD programme, for example:

The Wellcome Trust funds four-year PhD programmes, so we are providing funding for a longer period. […] the four-year model of the PhD is becoming well established and that gives universities the opportunity to provide that transferable skills training.[218]

We queried whether training in peer review was a part of this "transferable skills training", and were told that the Wellcome Trust was "not prescriptive in what universities teach" but that it would be "reasonable" for peer review to be a component of the training.[219]

123. Dr Janet Metcalfe, from Vitae, explained the need to share responsibility for the training of future generations of peer reviewers:

Collectively, we all have a responsibility for [peer review] to work. I think journals have a responsibility to support and provide more information about what is required and to contribute to the training of their reviewers. I think institutions have a responsibility, as signatories to the Concordat for the Career Development of Researchers, to ensure that those opportunities are there. I think research and funding councils and Government have an obligation to provide enough funding within the entire system to make available that kind of training for our early career researchers.[220]

She added that it was also the responsibility of the individual researcher "to take advantage of [training] opportunities and ensure that they are developing their own expertise and understanding of the entire system".[221]

124. Training for early-career researchers is important. We note that "Roberts Funding" is coming to an end and that the Research Councils will therefore be increasing the amount they give to universities "for training and developing postgraduate research". We invite the Research Councils to set out further details of how and where this money will be allocated and what proportion of it will be dedicated to training in peer review, including academic writing and publication ethics (discussed later in this report). We also ask for further details of how this will be "joined up" across different research funders.

INTERNATIONAL CHALLENGES AND OPPORTUNITIES

125. Earlier we highlighted that significant changes are taking place in scientific publishing, including the fact that the share of publications by countries which are not traditional scientific leaders, such as China and India, is rising (paragraph 6). Mayur Amin, from Elsevier, described the current situation:

If you take somewhere like the USA, which produces about 20% of the output of papers, it conducts something like 32% of the reviews in the world, whereas China is producing something like 12% to 15% of the output of papers but is probably only conducting about 4% to 5% of the reviews. This is just a transitionary thing. China and India have grown very fast in the last few years; there are a lot of young researchers who will come up and take their place in peer review and start peer reviewing papers.[222]

126. This was widely recognised, for example, the Publishers Association told us that:

There remain considerable geographical imbalances between those who benefit from peer review and those who contribute, most starkly between the US, the most prolific peer reviewer, and China, whose output of papers in certain disciplines has risen exponentially since 2000 but whose participation in peer review is increasing much less quickly. It is expected however that these imbalances will even out over time and within the UK there is more of a balance between publication output and participation in peer review. Publishers active in India and China are appointing editors and establishing editorial offices from where they run workshops on peer review, journal publication practices, and publication ethics.[223]

127. Mayur Amin explained that:

It is incumbent upon publishers to help out here, both in terms of technical infrastructure to help editors find a broader pool of reviewers, and also in terms of training needs, appointing editorial board members in those developing countries as well as running workshops and providing literature to help train new and young reviewers to come on to the system.[224]

128. We discussed these international activities with a range of publishers. Dr Robert Parker, from the RSC, and Dr Nicola Gulley, from IOP Publishing Ltd, explained that both organisations carry out face-to-face training in peer review, particularly in China and India.[225] Dr Parker told us:

We do a lot of interaction with the Chinese academic market, as it is. We have two offices in China—one in Beijing and another in Shanghai. We have staff out in China. We do regular visits. We set up conferences in China now. We started off doing roadshows of the top chemistry departments in China. All of our roadshows include presentations on how to publish and how to referee. We have built up quite a significant connection with the Chinese academic market. We also involve them on our editorial boards. We get them involved as associate editors on our journals.[226]

129. Dr Gulley added that IOP Publishing had "been working with researchers in China for the past 11 years. We have a member of staff who visits universities and gives lectures on how to get published. We run workshops and we visit regularly".[227] Robert Campbell, Senior Publisher, Wiley-Blackwell informed us that they had "been carrying out a lot of training since 2005 in China, particularly in chemistry. We are increasing the percentage of peer reviewing from China now. It is still not parity but it is moving towards 20% of our papers".[228] Dr Fiona Godlee added that the BMJ Group was also "involved closely in training in Africa, China and India at the moment".[229]

130. We welcome the fact that the publishers we have heard from are training authors and reviewers on an international level, particularly those from countries which are not traditional scientific leaders, and we encourage others to do the same. This should help alleviate the current imbalance between publication output and participation in peer review.

Finding reviewers

131. In part as a result of the growth of scientific output, both at home and abroad, there have been expressions of concern about the state of the peer-review system, including claims that the peer-review system is in crisis.[230] In particular, claims that there is an increasing burden on reviewers and that "scientists face strong incentives to submit papers, but little incentive to review".[231] Professor Ron Laskey, of the Academy of Medical Sciences, stated that he "wouldn't say [peer review] is in crisis. I would say that the engine is misfiring rather than it has stalled completely".[232]

132. The Society for General Microbiology told us that "with the rise in research that is multidisciplinary and becoming increasingly specialized it is sometimes difficult to find reviewers with sufficient expertise".[233] Robert Campbell, Senior Publisher at Wiley-Blackwell, was of the opinion that there was "no quantitative evidence that [peer review] is in crisis".[234] He explained:

I think the peer review system, as a whole, is more robust than ever. […] in 2010 we had about 12% more submissions. There was no impact on publishing schedules and no added delays, although we only published 2% more articles, so the rate of rejection was higher. A study has been published in Nature by Tim Vines and colleagues where they did try to quantify this issue and tracked all the reviewers. They found that the population of reviewers is increasing with the 3% to 4% increase in the research community, as you would expect. Therefore the load on each reviewer is, if anything, slightly less than 10 years ago.[235]

133. The study by Dr Tim Vines, Managing Editor of the journal, Molecular Ecology, and colleagues analysed—at that journal—the number of requests required in 2001-10 to obtain a review; compared the number of submissions in 2001-07 with the number of unique reviewer names in each year; and calculated the mean number of reviews per reviewer in 2001-07.[236] They reported that it was slightly harder to recruit reviewers in 2010 than it was in 2001; editors had to send out more than two requests, on average, for every one acceptance, compared to 1.4 in 2001.[237] This increase, however, coincided with the journal's move from sending personal reviewer e-mail requests to an automated editorial system, leading to suggestions that requests might not be reaching their intended target because they were being tagged as spam.[238] They also found no increase in average reviewer workload over that period, because the reviewer pool had increased in parallel with submissions. The study concluded that there was "no crisis" in the supply of peer reviewers.[239]

134. We are not convinced that there is a "crisis" in the supply of reviewers, especially as so little data are available. It appears that the current imbalance between publication output and participation in peer review may be a transitory phase. However, publishers should not be complacent and should continue actively to monitor the situation by collecting data.

The burden on reviewers

135. While peer review may not be in crisis, we previously explained that reviewers were feeling the "burden" of peer review (see paragraph 49). The view of the Wellcome Trust was that it "imposes a significant burden on the research community".[240] The Medical Schools Council agreed that "the high volume of peer review requests that members are exposed to in addition to their other demanding roles, is a cause for concern. It is felt that the current system places excessive burden on reviewers".[241]

136. Dr Janet Metcalfe, from Vitae, explained her views on the burden of peer review as part of a wider problem in academia:

I think many researchers would feel there is a personal cost in terms of the effort they put into peer review. They appreciate that it is a very important part of the system—it is partly about protecting academic discipline and contributing to the academic community—but there is an expectation, not just with peer review but other aspects of being an academic, that you have to put in very long hours and you are expected to work beyond your terms and conditions of employment to be successful. These are systemic issues within the academic community, and peer review falls very much within that. It is also rarely identified as a specific element in workload conversations or models within institutions, so we have no idea how much time is spent by the academic community on peer reviewing.[242]

137. Dr Malcolm Read, from the Joint Information Systems Committee (JISC), did not recognise academic working patterns as a big problem:

I don't know that many researchers particularly feel they have a nine-to-five existence anyway. So I am not sure to what extent they would particularly resent [peer reviewing manuscripts in their own time]. I don't think there is a nine-to-five mentality in the research community.[243]

138. We were keen to find out whether the burden of reviewing falls disproportionately on one group of researchers over another. Professor Grazia Ietto-Gillies, from Birkbeck, University of London, told us that:

The reviewers' workload is not distributed evenly among academics. Academic stars are unlikely to be available for reviewing; hearsay suggests that sometimes professors ask their assistants or PhD students to do reviews which they sign! Academics low down in the pecking order may not be asked to review. Most reviews are done by academics in the middle range of reputation and specifically by those known to editors and who have a record of punctuality and rigour in their reviews: the willing and conscientious horses are asked over and over again by overworked and—sometimes desperate—editors.[244]

139. The Academy of Social Sciences agreed that "a minority of willing scholars find themselves increasingly burdened by requests and gradually withdraw their goodwill in order to protect their time" for other activities.[245] Once again, this highlights the "importance of employing professional and properly qualified scientific editors", in this instance to make sure "that no one reviewer is overburdened".[246] Electronic databases are making this easier for journal editors to achieve. The International Association of Scientific, Technical and Medical Publishers said that:

in most cases now, each journal with the help of its publisher has developed an electronic database of experts with links to fields of interest. This usually includes details of all those who have reviewed for the journal before and can also be used as a management tool to ensure the same reviewer is not overburdened with requests. The identification of new reviewers for new fields has been significantly aided by the existence of abstracting and indexing databases that allow all those working in a field to be identified.[247]

140. Professor Ian Walmsley, from the University of Oxford, explained that it was necessary to look at the broader picture of how the burden of peer review falls on the research community:

peer review is pervasive throughout all aspects of the academic endeavour, not just publishing. For example, one may distinguish that senior people will have more to do with evaluation of others through promotion, tenure, awards or what have you and perhaps at the editorial end in publishing, and that younger people will have more of the burden of evaluating individual articles or specific research grants.[248]

141. There is a sense of give and take about the burden of peer review. Professor Rick Rylance, from RCUK, described it as a "collective enterprise".[249] The IOP told us that "it is felt to be an integrated part of the role of a researcher [and there is] an expectation that by refereeing a peer's work you would in turn expect your work to be reviewed".[250] The IOP considered that there was "a case for revisiting this tradition, as other professions generally do not proceed on this pro bono basis when offering a service" but acknowledged that the "majority of participants" supported the current arrangements.[251] Dr Malcolm Read, from JISC, explained that the situation would only become worrying if scientists had to spend more time on peer review proportionally to their scientific research.[252]

142. Professor Sir Adrian Smith did not:

regard peer review as a burden which is somehow additional and keeping fabulous researchers away from their day job. Peer review is an integral part of the scientific and research process and is part of the day job.[253]

He added that like peer review, science itself is "time-consuming and labour-intensive" and that peer review of journals was an "incredibly efficient way of divvying up the labour".[254]

REDUCING THE BURDEN

143. Dr Andrew Sugden, from Science, summarised his view of the current situation journal editors find themselves in when trying to find willing reviewers:

It is usually [difficult to find reviewers] because they are over-committed. It is not usually because of an underlying unwillingness to review or about not having an incentive to review. It is simply because they are doing too many other things at the time. It may take us a week or two to find the three referees that we need for a paper sometimes. It is rare that it takes much longer than that.[255]

144. Journal publishers are working on managing and reducing the burden felt by reviewers, and thereby encouraging researchers to get involved. Two specific examples of this are discussed below.

Cutting out re-review

145. BioMed Central is experimenting with new processes in peer review to help reduce the burden on reviewers, and indeed authors. In a recent experimental policy at its journal, BMC Biology, authors are given "more responsibility for ensuring the validity of the paper" by being given the option to opt-out of further peer review once the initial comments come back from the reviewers.[256] Dr Michaela Torkar, Editorial Director at BioMed Central, explained how it works:

Submissions are usually screened by the editorial team. There is quite a high rejection rate at that point. They will often consult with their editorial board to ask about the question of impact at that point. […] Of those manuscripts that go to peer reviewers about 60% are either rejected or require only minor revisions, so there wouldn't be a requirement for a re-review anyway. Of the remaining 40% of authors who are offered the option of [the experimental] peer review opt-out [policy], more than half will take it up. The editorial team will make a clear decision after the first round of peer review to make sure that they are very clear in their instructions to the authors about what needs to be done. They will then assess the revised manuscript when it comes back and they will usually go ahead with publication without rereview. I think there were only a couple of cases where that really wasn't possible for some reason. If the revisions aren't as extensive as they should be—say, some of the conclusions aren't put sufficiently into context to show there are some limitations to the study—they will commission a commentary which is published alongside the paper. That is written by an expert who will put it in context and point out those limitations just to make sure that non-expert readers understand that there might be some problems.[257]

BioMed Central told us that this policy "has the important effect of lessening the burden on expert reviewers, a scarce resource".[258]

The cascade system

146. The consensus that emerged at a recent workshop convened by the Wellcome Trust in partnership with the Howard Hughes Medical Institute and the Max Planck Society was that "the burden on researchers of reviewing papers is excessive, and we need to move away from the current system where the same paper is often reviewed multiple times by different journals".[259] One way around this is the "cascade" system, whereby if a manuscript is rejected by the author's journal of choice, it can be passed on to another journal, crucially, with the reviews from the first journal. This can occur in one of two ways: either, within one publishing organisation and between its "sister" journals; or, between journals from different publishers.

147. In our discussions with various publishing organisations, we learnt that publishers are, on the whole, happy to share reviews internally within their organisation, that is, between their own sister journals.[260] However, "some journals are a bit squeamish about the idea of acknowledging that the paper went somewhere else before it came on to them".[261] The internal cascading system is used extensively at BioMed Central and PLoS.[262] Dr Michaela Torkar told us that at BioMed Central:

Sometimes the transfers will happen before the peer review and sometimes with the reviewers' reports. That does save time for authors and reduces the burden on the peer reviewers who don't have to re-review manuscripts for multiple journals.[263]

148. Dr Mark Patterson, from PLoS, added that "about 10% to 15% of submissions to PLoS ONE come from other PLoS journals. It is pretty clear that, internally, that works quite well".[264] He explained, however, that "the much more problematic issue is the sharing of reviews from one publisher to another".[265]

149. A well-known example of publisher to publisher cascading is the Neuroscience Peer Review Consortium, which is "an alliance of neuroscience journals that have agreed to accept manuscript reviews from other members of the Consortium".[266]

150. Dr Philip Campbell, from Nature and Nature Publishing Group, explained that the journal, Nature Neuroscience, participated in this consortium, he told us:

We did it with some misgivings because […] we invest a lot in getting editors out into the field and using referees whom we value because of the relationships that we have developed with them. To hand on, as it were, the outcome of that relationship to a competing publisher is something that hurts slightly. At the same time, you do have this competing interest of the research community to save people work. We found that the uptake of this facility, where authors can elect to have the referees' reports of the rejecting journal handed on to the next publisher, is not very great.[267]

151. Dr Patterson, PLoS, agreed that it "was not terribly popular with authors" but questioned "how much publishers were really behind" the experiment. He was "not convinced" that the "sense of ownership", as alluded to by Dr Campbell, was in the best interests of science.[268] Mayur Amin told us that Elsevier also participated in the consortium and also felt that authors were "somewhat reluctant" to engage.[269]

152. Peer review is a burden on researchers but a necessary one, as it is an integral part of the scientific and research process and is part of the role of a researcher. However, we encourage publishers to work with their reviewers, to identify innovative new practices to minimise the burden.

Recognition

153. Despite the importance with which it is viewed, peer review is rarely acknowledged as part of the formal workload of an academic researcher.[270] Dr Fiona Godlee, from BMJ Group, told us that:

scientists are under a lot of pressure on a whole host of things, such as getting funding and the bureaucracy surrounding scientific research, and peer review is just one other thing. […] the more we can do to make it something that they gain proper recognition for, the better.[271]

154. Tracey Brown, Managing Director at Sense About Science, agreed that there were "very few incentives" to encourage peer review within the university system and that there was "no recognition" of it in a researcher's career.[272] This was particularly the case for reviewing manuscripts according to Dr Janet Metcalfe, from Vitae, who described peer review as an "invisible contribution to the academic community except when you get on to an editorial board or grant panel".[273]

155. Professor Rick Rylance, from RCUK, considered that "peer review should be part of professional development for researchers" and that it was "important that their employers recognise quite how much labour is put into it and how important it is in terms of not just their personal but their general benefit".[274] Indeed, the British Medical Association suggested some form of "professional recognition, accreditation or development of a reward system to encourage participation" in peer review.[275]

REWARDS AND ACCREDITATION

156. In the course of our inquiry we have questioned how carrying out peer review can be better recognised as a professional activity so that reviewers receive credit for their time and effort. Dr Gulley explained that some journals also give rewards "to their top referees".[276] Professor Ian Walmsley, University of Oxford, gave us an example:

the American Physical Society has an outstanding referee award. Every year it makes a big deal of naming people who have provided consistent, high quality and useful reviews. […] It is not a direct financial compensation for time. However, I think most people would say this is a contribution to the community which reaps values in other ways.[277]

157. Another way in which journals show their appreciation to reviewers was described by Dr Robert Parker, from the RSC:

Being a referee is often used as one of the criteria for tenure in the US. We deal with a lot of requests from US referees, young academics, wanting a letter of endorsement saying that they have acted as a referee for the RSC and that they have been reasonably good at it. It will help them to gain tenure.[278]

Dr Nicola Gulley told us that IOP Publishing also help with requests to "support younger researchers in their applications for green cards".[279]

158. It has also been suggested that payment could be used as an incentive for researchers to undertake the burden of peer review.[280] Dr Parker told us:

Remuneration would be a difficult thing because, if you gave any realistic payment for the time that is involved, it would be a huge amount of money and it would have to be recovered from somewhere. It is just moving a financial burden around the whole system. The system relies on the benefits that people see from being involved in peer review. There is a quid pro quo as long as you are someone who publishes as well; you are an author as well as a referee, which is not always the case.[281]

There are also concerns that financial remuneration might reduce the impartiality of reviewers.[282] Some have suggested "payment in kind" (such as a free subscription) or a virtual payment system.[283]

159. Another form of recognition for reviewers is through accreditation. Dr Parker considered that this "might be" helpful to reviewers but "it would be quite difficult to do" because the RSC has about 33,000 referees all around the world that it uses routinely.[284] Dr Philip Campbell, from Nature, disagreed:

In principle, I don't think it is [difficult to do]. A manuscript tracking system can be easily programmed. If what is needed is that the referees themselves get a proper statement of credit, that is fine. It is equally easy for a journal to decide to publish a list of everyone who has peer reviewed for them over a particular period.[285]

160. Professor Rick Rylance, from RCUK, considered that "there would have to be quite a complicated cost-benefit analysis" on whether peer review should be formally accredited.[286] His instinct was that it probably wouldn't be worth it.[287]

161. An easier and, currently, more commonly used approach is the annual publication by journals of a list of the reviewers they have used, or provision to reviewers of their reviewing service at the end of each year. Professor John Pethica explained that "at the Royal Society the referee is not paid, but we do publish a list of the referees at the end of the year to formally thank them for their input".[288] Dr Nicola Gulley told us that IOP Publishing also do this for some research communities.[289] The Nature journals are working on giving more credit privately to referees directly at the end of every year, letting them know what work they have done.[290] Dr Philip Campbell explained that "in a very competitive academic world, when you are going for tenure or for some other promotion, to be able to have something like that stated on the record is helpful".[291] Dr Malcolm Read, from JISC, suggested that "greater transparency in the peer review process" might improve the situation, ensuring that reviewers' work was known to their peers.[292] Dr Andrew Sugden, from Science, warned that there can be a "downside" to this approach, as some reviewers prefer to remain anonymous.[293]

162. In the future, Mayur Amin, from Elsevier, told us that it may become easier to set up accreditation systems in peer review:

the advent of ORCID, which is [a] unique author identifier [system] may give us an opportunity also to be able to track with [a] unique identifier those people who have refereed and acted as referees. That may help to provide a stronger accreditation platform than is currently possible.[294]

163. Dr Mark Patterson, from PLoS, agreed that ORCID would "help to identify who has done what peer review".[295] Accurate identification of researchers and their work is not only useful in terms of tracking reviewer and author contribution, it is also increasingly important because of the problems of name ambiguity. Dr Parker, from the RSC, told us that this was "an issue, particularly in places like Korea, where there are only five or six really common surnames".[296] However, it was not only an international problem, for example, there were "two people with the same name both in the chemistry department at the University of Oxford".[297] The ORCID Initiative aims to establish an open, independent registry that is adopted by the publishing industry. Its goal is to resolve the systemic name ambiguity problem, by means of assigning unique identifiers linkable to an individual's research output.[298]

164. In order to help research institutions recognise the work carried out by reviewers on peer review, publishers first need to have in place systems for recording and acknowledging it. A variety of approaches are in use, including rewards, awards and letters of endorsement and these should be encouraged. New initiatives for accurate author and reviewer identification may make it easier for publishers to track reviewer contribution to the peer-review process.

Assessment

165. Professor John Pethica, from the Royal Society, told us that in addition to assessing manuscripts for the purposes of "generating a coherent scientific record", peer review is often "used for other proxy purposes and assessment" and that this "can, potentially, influence how it is carried out".[299] The proxy use that Professor Pethica refers to is the perceived importance of a piece of published research, as assessed during the peer-review process. When research is published in a high-impact journal—generally taken as one with a high Impact Factor (see paragraph 59)—that traditionally signals to the rest of the academic community that the research is perceived to be important. This has led to the suggestion that scientists have become "increasingly desperate to publish in a few top journals".[300] However, as we have noted, the Impact Factor relates to the journal as a whole rather than the individual published articles. Nonetheless, publication in a high-impact journal is frequently used as a proxy measure for assessing both the work of individual researchers and research institutions.

166. We questioned the logic of using the Impact Factor as a measure of quality. Professor Sir Adrian Smith, from BIS, told us that:

It is a little circular, is it not, because why would a journal be designated as high impact? It will be related to the quality of the journal, which, in some sense, will be related to the selectivity of the journal, which will be related to the fact that it is sifting out, to some extent, the cream of the things that are submitted to it.[301]

167. Sir Mark Walport, from the Wellcome Trust, disagreed:

Impact factors are a rather lazy surrogate. We all know that papers are published in the "very best" journals that are never cited by anyone ever again. Equally, papers are published in journals that are viewed as less prestigious, which have a very large impact. We would always argue that there is no substitute for reading the publication and finding out what it says, rather than either reading the title of the paper or the title of the journal.[302]

Professor Rick Rylance, from RCUK, added that "there is no absolute correlation between quality and place of publication in both directions".[303]

168. Below we discuss the use of Impact Factor as a measure of quality in relation to assessing excellence in research institutions as well as assessing researchers and the influence on research careers.

ASSESSING RESEARCH EXCELLENCE

169. The Higher Education Funding Council for England (HEFCE) distributes public funds to higher education institutions (HEIs) in England for teaching, research, and related activities. There are similar funding councils in Northern Ireland, Scotland and Wales. HEFCE provides quality-related (QR) research funding, on the basis of periodic assessments of the performance of universities and institutions. The last was the Research Assessment Exercise (RAE) in 2008; the next will be the Research Excellence Framework (REF), scheduled for 2014. The criteria for assessment in the REF are currently being developed.

170. The Academy of Medical Sciences told us that "a strong publication record is a key determinant in the allocation of grant funding both to individual researchers and to their universities via processes such as the [REF]".[304] Professor Thomas Ward, Pro-Vice-Chancellor at the University of East Anglia added that:

The Research Excellence Framework assessing UK Universities is seeking to apply metrics to some aspects of the periodic assessments of research quality. Some of these metrics depend on peer-reviewed publications and citation counting of the articles cited.[305]

171. Dr Parker, from the RSC, told us that:

When it was the RAE before, [the panel members] always said that they would look at the quality of the papers themselves. They would read the papers themselves and wouldn't rely on the Impact Factors of the journals in which they had been published. […] How they are going to be used in REF, if it changes, I don't know.[306]

172. The proposed use of bibliometrics (that is, citation analysis, which includes counting how many times a particular piece of work has been cited by others), along with the inclusion of an impact measure, were the two major characteristics that were to differentiate the REF from the RAE. The International Association of Scienti?c, Technical and Medical Publishers told us that:

Metrics-based assessments have been around since the 1960s […] The literature on these approaches is large but the majority of academics tend to critique these initiatives along the lines of Einstein's quote "not everything that can be counted counts, and not everything that counts can be counted".[307]

173. In April 2010, an article in the Times Higher Education Supplement suggested that HEFCE might not be using citation data in the REF process.[308] HEFCE confirmed to us that it had "ruled out the systematic use of citation data as a key indicator of research quality at present".[309] David Sweeney, from HEFCE, also clarified the situation for the use of Impact Factors:

With regard to our assessment of research previously through the Research Assessment Exercise and the Research Excellence Framework, we are very clear that we do not use our journal Impact Factors as a proxy measure for assessing quality. Our assessment panels are banned from so doing. That is not a contentious issue at all.[310]

He added that "the [REF] panels are meeting now to develop their detailed criteria, but it is an underpinning element in the exercise that journal Impact Factors will not be used".[311]

INFLUENCING RESEARCH CAREERS

174. While, in the light of HEFCE's statement, the use of journal Impact Factors to assess research quality may prove not to be a contentious issue so far as the REF is concerned, the fact remains that researchers still feel under pressure to get their work published in the high-impact journals. When we asked Professor Ian Walmsley, from the University of Oxford, why this is the case, he responded that:

Perhaps a simple answer to that from a parochial view of a university person is that that is the way one's career advances. […] a lot of very good work gets published in journals that do not have such high visibility, and I think that is quite crucial. None the less, having a highly cited paper in a journal that people would regard as high profile is considered important as a way to raise your visibility and develop your career. […]when a CV comes across the desk of a head of department for a faculty post, as a first pass through it makes a difference where those papers are published.[312]

175. However, as we previously noted, decisions about which papers are accepted by high-impact journals "can seem rather random", as a result of decisions that "are often editorial ones based on topicality".[313] We also questioned whether a researcher's contribution to peer review, as a reviewer, should be formally recognised as part of their work and whether this could be taken account of when evaluating them for promotion. Professor Walmsley told us that:

in evaluating people for promotion one would look not only but primarily at the quality of the research undertaken and published but also at how they have contributed to the working of the community. […] One would normally expect to see, on a CV for evaluation, that somebody had undertaken reviewing for research councils or, in this sense, professional societies or other publishers for journals.

As to the extent one wishes to quantify that to a greater degree, I would be cautious about that. One doesn't want to be prescriptive. One wants to see some threshold of evidence that people are playing a role without being quantitative about exactly how much they ought to be doing.[314]

176. Sir Mark Walport, from the Wellcome Trust, added that:

I think this is one of those things where it is easy to say that you need to give people recognition for peer review. The reality is are you going to promote someone from a lectureship to a senior lectureship or from a senior lectureship to a readership on the basis of review? You are not going to do that. You are going to do it on the core scholarly activities which are education and the research itself. It is something that the community has to recognise. It is beneficial to do peer review. As I said before, it is part of your continuous professional development. It is about keeping up to date with the field.[315]

177. We have concerns about the use of journal Impact Factor as a proxy measure for the quality of an individual article. We have been reassured by the research funders that they do not consider that publication in a high-impact journal should be used as a proxy measure for assessing either the work of individual researchers or research institutions. We agree that there is no substitute for reading the article itself in assessing the worth of a piece of research. We consider that there is an element of chance involved in whether researchers are able to get their articles published in high-impact journals, depending on topicality and other factors. Research institutions should be cautious not to attach too much weight to publication in high-impact journals when assessing individuals for career progression.

159 Q 3 Back

160 Q 252 [Sir Mark Walport] Back

161 Ev w126 Back

162 Q 27 Back

163 A Mulligan, Is peer review in crisis?, Oral Oncology, 2005, vol 41, pp 135-41 Back

164 Ev 115, para 12 [Elsevier] Back

165 Ev 97, para 10 [Royal Society of Chemistry] Back

166 Ev 97, para 10 Back

167 Ev w112, para 7.7 Back

168 Q 6 [Professor Ron Laskey] Back

169 Q 6 Back

170 H Ploegh, End the wasteful tyranny of reviewer experiments, Nature, 2011, vol 472, p 391 Back