Peer review

Written evidence submitted by Daniel Mietchen (PR 82)

Strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

This document is a response to the peer review inquiry by the UK Parliament's Science and Technology Committee and consists of two parts: Part I lists specific recommendations (along with brief comments) on how to improve the current system, whereas Part II comments on the issues specifically mentioned in the inquiry announcement.


Part I - recommendations


1. Test whether particular variants of peer review are fit for their intended purposes

a. The efficiency of the currently predominating non-public single-blind pre-publication review by a few has never been convincingly demonstrated.

b. A 2007 Cochrane review, stated: "Experimental studies assessing the effects of grant giving peer review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile."

c. The costs of peer review are high (cf. RIN report "Activities, costs and funding flows in the scholarly communications system", and RCUK Project "Efficiency and Value for Money of Peer Review" for grants).

d. Pre-publication and pre-funding peer review are slow.


2. Peer reviews should be public by default

a. And published along with the manuscript/ grant proposal.

b. This would increase the transparency of the process and facilitate the application of expertise and reason.

c. Exceptions may be granted upon request in cases where the preservation of personal privacy or of any objects of study or other valued goods would justify that.

d. Requiring the reviews to be made public does not mean to require the identity of the reviewers to be made public, though the two modes are often confused in discussions.


3. Provide room to experiment with new variants of peer review

a. If peer review does not perform significantly better than chance under certain circumstances, then introducing some random elements into the decision-making process could achieve similar effects at higher speed and lower cost.

b. Allow feedback between reviewers and reviewees. The lack of such a possibility in many funding schemes is at the heart of the disadvantage for innovative and cross-disciplinary projects. In journals, the lack of interaction is less common but still affects much of what is known as editorial (rather than peer) review.

c. For some further details on the degree of experimentation, see the scenario sketched out in "Processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases" below.


4. Pre-publication peer review should only concern technical soundness, not impact guesstimation

a. The impact of papers in any journal is heavily skewed, even in the so-called top-tier journals. In other words, the quality filter that journals supposedly provide does not work coherently.

b. Consequently, do not let impact guesstimation be part of the decision to publish scholarly work. Such a system has been introduced by PLoS ONE in 2006 and now reached a point where multiple other publishers are copying the model.

c. Post-publication filtering can be highly automated, pre-publication peer review cannot.

5. Pre-funding peer review should focus on the achievements of the applicants rather than on research proposals

a. Grant recipients should be held to very high standards of reporting on the progress of the project throughout its duration, with encouragements to perform their research in the open, and severe punishments for proven cases of misconduct.

b. This would increase the speed and lower the costs of the process and save enormous amounts of time for everyone, especially the majority of those whose submissions do not result in funding.

c. Such a system would be much fairer to researchers early in their career.


6. Diversify evaluation criteria of scholarly impact

a. This would allow, for instance, to take into account reviewer activities, including in post-publication peer review.

b. On the author side, it would help to provide incentives to make data available via public repositories, or to participate in collaborative knowledge environments like wikis.

c. This requires a standardized system for author identification, e.g. the one due to be released by ORCID later this year.


Part II - comments


0. Clarification of terminology: Peer review is not monolithic

The objects of peer review are typically either scientific manuscripts or grant proposals, but in principle, any outputs of scientific activities could be subjected to peer review, and they often are.

No clear criteria exist for who is or is not a peer in regard to the review of a given scholarly work.

The purposes that peer review serves for the different communities involved - scientists (in their roles as authors, readers, reviewers, editors, grant applicants), science funders, publishers and the public - are not always aligned.

The dominant dimensions along which scientific manuscripts are being peer reviewed are journal scope, novelty, prospective impact, and technical soundness. Of these, the first is mainly administrative in nature, the second is often not easy to judge, the third can hardly be foreseen, and only the fourth can be determined with reasonable accuracy. For research proposals, some other dimensions typically come into play: Feasibility of the proposed research, and appropriateness of the budget. Both of these can be judged easily if the given project follows well-trodden paths, but if that is not the case, proper judgements are often impossible, and in case of doubt, projects perceived as too risky are often rejected, especially if they span different fields, which often have only partial overlap with the expertise of the reviewers.

The information that authors and reviewers (and the public) have about the review process may vary considerably, and so can the communication channels used between any of the parties involved, if such interaction is possible.

1. Purposes of peer review

1.1 Both manuscripts and grant proposals

Assessing methodological soundness

Assessing proper contextualization with respect to prior research

Impact guesstimation

1.2 Manuscripts

Assessing validity of conclusions based on reported data

Selecting a publication venue that is appropriate according to a perceived hierarchy of journals

1.3 Grant proposals

Assessing appropriateness of budget and other resource use

Assessing feasibility of the proposed project, considering the requested resources and the applying team

Deciding whether to fund a project, under what funding scheme, and under which conditions


2. Measures to strengthen peer review

Make reviews public (possibly with an option for reviewers to remain anonymous)

Allow interaction between reviewers and reviewees

Reduce the length of proposals (cf. Gates Foundation) or papers



3. Value and use of peer reviewed science on advancing and testing scientific knowledge

Post-publication public peer review: tremendously important; the essential ingredient of scientific progress. Key to success of this system: replicability.

Pre-publication peer review: much less essential, though more widely used.


4. Value and use of peer reviewed science in informing public debate

Post-publication peer review: Highly valued, but so may be popular beliefs

Pre-publication peer review: much less so, and only a small area in the medial landscape

Pre-funding peer review: typically does not see public debate (exceptions: ITER and other large projects), but could benefit from it

Post-funding peer review: typically does not see public debate (exceptions: ITER and other large projects, research awarded prestigious prizes), but could benefit from it


5. Extent to which peer review varies between scientific disciplines and between countries across the world

There certainly is considerable variation, but any such effects can be reduced by bringing the process online and into the public sphere.


6. Processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases

Multiple possibilities - typically with an open component. To provide an idea about the range of possibilities, I will sketch out one scenario (that of finding reviewers for grant assessment) in detail.

Assuming the funding agency want to have three reviewers per grant proposal, they could demand that submitters arrange for two of them to be posted in public along with the proposal, with the names of the reviewers being public. A third reviewer would be chosen by the funder to review the proposal and to comment on the two existing reviews. The reviews and comments by this third reviewer would be public as well but she could choose to remain anonymous to the public (she will always be known to the funder, much like in the current system). In addition to that, the funder could invite comments from the submitters and the public, and add a reputation scheme (of the kind implemented at sites like Stack Overflow) that allows to evaluate contributions to the system and aggregates these evaluations into some measure of reputation.

Such a scenario would have the following effects (amongst others), compared to the current system:

The quality of submissions would be higher, as submitters would know that if they submit low-quality items, the whole relevant community could find out (this effect is known from journals which practice public peer review, e.g. Atmospheric Chemistry and Physics). Furthermore, the need to get two (public) reviews acts as an additional filter.

The quality of reviews generally rises when they are to be published (again, this effect is known from journals, e.g. at BioMed Central). This effect is somewhat modulated by the reviewer's identity being known or not: If it is, reviewers tend to be more kind and less critical than if they can hide behind anonymity. Reviewer 3 as described above would balance that out.

The funder would get an initial (though possibly somewhat biased) review along with each submission, and could then select reviewer 3 (or 4 even), so as to cover those aspects of the grant that need further evaluation.

The funding organization have to spend less work on finding reviewers and could divert some of their efforts into making the conversations around these proposals useful to the community, thereby increasing their pool of potential reviewers on the fly.

Other parameters that could be considered for adjusting peer review approaches to the purpose at hand: Person-centric versus project-centric funding, or the communication channels (e.g. interviews, or video abstracts) used for interaction between applicants (or authors), funders (or publishers), peer reviewers and possibly the community at large.


7. Impact of IT and greater use of online resources on the peer review process

Online resources can help peer review in a multitude of ways.

In traditional peer review, they could (and already do) help publishers (or funders) to identify reviewers, reviewers to identify and locate relevant references. They also facilitate checks for plagiarism.

Less used today but still possibly quite useful are online tools to bring together scientists (authors, grant applicants, and reviewers of both), funders and readers interested in a particular topic. Here, both person-centric approaches (social networks) and topic-centric approaches (e.g. databases and collaborative platforms like wikis) appear to have potential that has yet to be fully explored.

One example for such enhanced exchange by way of online tools would be blog-based post-publication public peer review, which has in many cases helped to identify flaws in published studies that had not been detected during pre-publication peer review.


8. Possible alternatives to peer review

Again, terminology is key. If the currently prevailing peer-review model can be described as pre-publication single-blind closed review by a few, all of the parameters used in this description could in principle be modified. The largest improvements are to be expected from making the reviews public and from facilitating post-publication peer review by the community at large.

Declaration of interests

I am a scientist and as such engaged in peer review on both ends of both manuscripts and grant proposals. I am also frequently blogging about peer review, peer-reviewed research and systemic properties of science, as well as about open approaches to it.

Daniel Mietchen

10 March 2011