Peer review in scientific publications - Science and Technology Committee Contents

Conclusions and recommendations

The peer-review process

1.  We conclude that different types of peer review are suitable to different disciplines and research communities. We consider that publishers should ensure that the communities they serve are satisfied with their choice of peer-review methodology. Publishers should keep them updated on new developments and help them experiment with different systems they feel may be beneficial. (Paragraph 20)

2.  The importance of a pre-publication technical assessment is clear to us. It should be a fundamental aim of the peer-review process that all publications are scientifically sound. Assessing the impact or perceived importance of research before it is published will always require subjective judgement and mistakes will inevitably be made. We welcome new approaches that focus on carrying out a technical assessment prior to publication and making an assessment of impact after publication. (Paragraph 29)

3.  We recommend that publishers, research funders and the users of research outputs (such as industry and Government) work together to identify how best to evaluate current peer-review practices so that they can be optimised and innovations introduced, and the impact of the common criticisms of peer review minimised. We consider that this would also help address any differences in the quality of peer review that exist. We encourage increased recognition that peer-review quality is independent of journal business model, for example, there is a "misconception that open access somehow does not use peer review". (Paragraph 58)

Innovative approaches to peer review

4.  We conclude that pre-print servers can be an effective way of allowing researchers to share and get early feedback on preliminary research. The system is well established in the physics community, and works particularly well, co-existing with more traditional publication in journals. We encourage exploration in other fields. We note, however, that pre-print servers may not work in fields where commercialisation and patentability are issues, or in the biomedical sciences, where publication of badly performed studies could have harmful consequences and could be open to misinterpretation. (Paragraph 72)

5.  The principles of openness and transparency in open peer review are attractive, and it is clear that there is an increasing range of possibilities. There are mixed results in terms of acceptance amongst researchers and publishers, although some researchers are keen to see greater transparency in their fields. We encourage publishers to experiment with the various models of open peer review and transparency and actively engage researchers in taking part. (Paragraph 78)

6.  We are impressed by the success of PLoS ONE and welcome the wider growth of quality online repository journals. These will accelerate the pace of research communication and ensure that all work that is scientifically sound is published, regardless of its perceived importance. However, we recognise that this is a relatively new and rapidly evolving model, and potentially open to abuse because publication fees are involved. It is important that a high quality of peer review is maintained across all repository-style journals. (Paragraph 89)

Editors, authors and reviewers

7.  The role of the editor is at the heart of the peer-review process. The judgement applied by the editor to the information collected in the review process requires knowledge, skill, and care; particularly, in respect of identifying the right reviewers for the job and critically assessing the feedback from reviewers to authors. (Paragraph 100)

8.  Broadly speaking, training for editors and members of editorial boards is provided on the job. We have heard that some publishers opt for a more structured approach, and include, for example, comprehensive welcome packs for new editors that cover peer-review processes, support tools and ethical guidelines. We encourage publishers to work together to develop standards—which could be applied across the industry—to ensure that all editors, whether staff or academic, are fully equipped for the central role that they play in peer review. (Paragraph 106)

9.  A relatively straightforward way of educating reviewers about the quality of their reports and helping them improve their feedback to editors is to send them the reports of other reviewers, done confidentially when necessary. This should be standard practice across all journals. This would be a useful educational tool to improve the quality of future reports from reviewers. (Paragraph 118)

10.  Training for the next generation of authors and reviewers is also important. Many PhD students and post-doctoral researchers are fortunate to have the opportunity to discuss scientific literature in journal clubs and other informal settings. Some are mentored well by their principal investigator and thereby receive informal training in peer review. Others are not. Given the importance of peer review across the research spectrum, from grant applications to publications, we consider that all early-career researchers should be given the option for training in peer review. (Paragraph 119)

11.  Training for early-career researchers is important. We note that "Roberts Funding" is coming to an end and that the Research Councils will therefore be increasing the amount they give to universities "for training and developing postgraduate research". We invite the Research Councils to set out further details of how and where this money will be allocated and what proportion of it will be dedicated to training in peer review, including academic writing and publication ethics (discussed later in this report). We also ask for further details of how this will be "joined up" across different research funders. (Paragraph 124)

12.  We welcome the fact that the publishers we have heard from are training authors and reviewers on an international level, particularly those from countries which are not traditional scientific leaders, and we encourage others to do the same. This should help alleviate the current imbalance between publication output and participation in peer review. (Paragraph 130)

The burden of reviewing

13.  We are not convinced that there is a "crisis" in the supply of reviewers, especially as so little data are available. It appears that the current imbalance between publication output and participation in peer review may be a transitory phase. However, publishers should not be complacent and should continue actively to monitor the situation by collecting data. (Paragraph 134)

14.  Peer review is a burden on researchers but a necessary one, as it is an integral part of the scientific and research process and is part of the role of a researcher. However, we encourage publishers to work with their reviewers, to identify innovative new practices to minimise the burden. (Paragraph 152)

15.  In order to help research institutions recognise the work carried out by reviewers on peer review, publishers first need to have in place systems for recording and acknowledging it. A variety of approaches are in use, including rewards, awards and letters of endorsement and these should be encouraged. New initiatives for accurate author and reviewer identification may make it easier for publishers to track reviewer contribution to the peer-review process. (Paragraph 164)

The assessment of researchers and institutions

16.  We have concerns about the use of journal Impact Factor as a proxy measure for the quality of an individual article. We have been reassured by the research funders that they do not consider that publication in a high-impact journal should be used as a proxy measure for assessing either the work of individual researchers or research institutions. We agree that there is no substitute for reading the article itself in assessing the worth of a piece of research. We consider that there is an element of chance involved in whether researchers are able to get their articles published in high-impact journals, depending on topicality and other factors. Research institutions should be cautious not to attach too much weight to publication in high-impact journals when assessing individuals for career progression. (Paragraph 177)

Managing data

17.  We conclude that reproducibility should be the gold standard that all peer reviewers and editors aim for when assessing whether a manuscript has supplied sufficient information, about the underlying data and other materials, to allow others to repeat and build on the experiments. (Paragraph 184)

18.  If reviewers and editors are to assess whether authors of manuscripts are providing sufficient accompanying data, it is essential that they are given confidential access to relevant data associated with the work during the peer-review process. This can be problematical in the case of the large and complex datasets which are becoming increasingly common. The Dryad project is an initiative seeking to address this. If it proves successful, funding should be sought to expand it to other disciplines. Alternatively, we recommend that funders of research and publishers work together to develop similar repositories for other disciplines. (Paragraph 189)

19.  Access to data is fundamental if researchers are to reproduce, verify and build on results that are reported in the literature. We welcome the Government's recognition of the importance of openness and transparency. The presumption must be that, unless there is a strong reason otherwise, data should be fully disclosed and made publicly available. In line with this principle, where possible, data associated with all publicly funded research should be made widely and freely available. Funders of research must coordinate with publishers to ensure that researchers disclose their data in a timely manner. The work of researchers who expend time and effort adding value to their data, to make it usable by others, should be acknowledged as a valuable part of their role. Research funders and publishers should explore how researchers could be encouraged to add this value. (Paragraph 203)

Post-publication review and commentary

20.  Post-publication review in an era of new media and social networking tools, such as Twitter, is very powerful. The widespread sharing of links to articles ensures that research, both accurate and potentially misleading, is rapidly spread across the world. Failings in peer review can, rightly, be quickly exposed. However, there is no guarantee that false accusations of failings will not also be spread. Pre-publication peer review still has an important role to play, particularly in relation to assessing whether manuscripts are technically sound prior to publication. However, we encourage the prudent use of online tools for post-publication review and commentary as a means of supplementing pre-publication review. (Paragraph 211)

21.  While it is too early to make a judgement on post-publication filtering mechanisms, such as Faculty of 1000 Ltd, we recognise that such a system could offer a valuable service if widely used. It is likely that such services will become more important with the growth of repository-type journals. (Paragraph 223)

Publication ethics and research integrity

22.  The integrity of the peer-review process can only ever be as robust as the integrity of the people involved. Ethical misconduct damages peer review and science as a whole. Although peer review is not designed to identify systematically fraud or misconduct, it does, on occasion, identify suspicious cases. Where ethical misconduct is suspected, guidance for journal editors is in place, for example from the Committee on Publication Ethics, about how best to deal with it. In addition to relying on the vigilance of the people involved in the process, publishers must continue to invest in new technology that helps to identify wrongdoings. (Paragraph 244)

23.  Oversight of research integrity in the UK is in need of revision. The current situation is unsatisfactory. We are concerned that the UK does not seem to have an oversight body for research integrity that provides "advice and support to research employers and assurance to research funders", across all disciplines. The UK Research Integrity Futures Working Group report made sensible recommendations about the way forward for research integrity in the UK. Research Councils UK, Universities UK and the Government should revisit these recommendations and reassess how they can best be implemented. (Paragraph 262)

24.  Employers must take responsibility for the integrity of their employees' research. However, we question who would oversee the employer and make sure that they are doing the right thing. In the same way that there is an external regulator overseeing health and safety, we consider that there should be an external regulator overseeing research integrity. We recommend that the Government set out proposals on the scope and powers of such a regulator and consult with the research community and other relevant parties to develop them. (Paragraph 271)

25.  We also recommend that all UK research institutions have a specific member of staff leading on research integrity. Such a person would be a first point of call in case of an ethical breach. Where an investigation subsequently takes place within a research institution, it is essential that the outcome be published. (Paragraph 272)

26.  We recommend that the Research Councils, and other funders of research, reassess the robustness of their procedures for dealing with allegations of research fraud or misconduct, to ensure that they are not falling through the cracks. (Paragraph 276)

General conclusions

27.  Peer review in scholarly publishing, in one form or another, is crucial to the reputation and reliability of scientific research. Pre-publication peer review has evolved in a piecemeal and haphazard way to meet the needs of individual scientific communities. The process, as used by most traditional journals prior to publication, is not perfect, and it is clear that considerable differences in quality exist. However, despite the many criticisms and the little solid evidence on its efficacy, editorial peer review is considered by many as important and not something that can be dispensed with. (Paragraph 277)

28.  In order for current peer-review practices to be optimised and innovative approaches introduced, publishers, research funders and the users of research outputs (such as industry and government) must work together. There is much that can be done to improve the quality of pre-publication peer review across the board and to better equip the key players to carry out their roles. We note that new innovations in pre-publication review are being introduced that have the potential to accelerate the pace of research communication and avoid duplication of effort by the research community, with the consequent drain on resources. Publishers can learn much from one another and should share best practice where possible—particularly in relation to the ways in which data are managed and in terms of promoting publication ethics and research integrity. It is clear that breaches in the latter damage both the scientific record and public confidence in science. (Paragraph 278)

29.  The publication of peer-reviewed articles is not only important in terms of maintaining a robust scientific record, it also has an impact on the careers of researchers and the reputations of research institutions. We have been assured by research funders that they do not use journal Impact Factor as a proxy measure for the quality of research or of individual articles. However, representatives of research institutions have suggested that publication in a high-impact journal is still an important consideration when assessing individuals for career progression. We consider that research institutions should be cautious about this approach, because as we have previously noted, there is no substitute for reading the article itself in assessing the worth of a piece of research. (Paragraph 279)

30.  While pre-publication peer review continues to play an important role, the growth of post-publication peer review and commentary represents an enormous opportunity for experimentation with new media and social networking tools. Online communications allow the widespread sharing of links to articles, ensuring that interesting research is spread across the world, facilitating rapid commentary and review by the global audience. They also have a valuable role to play in alerting the community to deficiencies and problems with published work. We encourage the prudent use of online tools for post-publication review and commentary as a means of supplementing pre-publication review. (Paragraph 280)

previous page contents next page

© Parliamentary copyright 2011
Prepared 28 July 2011