Peer review

Written evidence submitted by Nikolaus Kriegeskorte (PR 14)

Open evaluation (OE): post-publication peer review and rating

Executive summary

A scientific publication system needs to provide two basic services: access and evaluation. Access means we can read anything, evaluation means we don't have to read everything. The traditional publication system restricts the access to papers by requiring payment, and it restricts the evaluation of papers by relying on just 2-4 pre-publication peer reviews and by keeping the reviews secret. As a result, the current system suffers from a lack of quality and transparency of the peer-review evaluation process, and the only immediately available indication of a new paper’s quality is the prestige of the journal it appeared in.

Open access (OA) is now widely accepted as desirable and is slowly beginning to become a reality. However, the second essential element, evaluation, has received less attention. The evaluation system steers the attention of the scientific community and, thus, the very course of science. For better or worse, the most visible papers determine the direction of each field and guide funding and public policy decisions. Evaluation, therefore, is at the heart of the entire endeavour of science. As the number of scientific publications explodes, evaluation and selection will only gain importance. A grand challenge of our time, therefore, is to design the future system, by which we evaluate papers and decide which ones deserve broad attention. Open evaluation (OE), an ongoing post-publication process of transparent peer evaluation (including written reviews and ratings of papers), promises to address the problems of the current system.

Here I outline a vision for an open publication and evaluation system with the following key features: Papers are evaluated in an ongoing fashion after publication by means of reviews and ratings. Reviews and ratings are public judgements from other scientists and can be signed or anonymous. More important papers are more deeply evaluated as they will receive more evaluations. Scientists are more motivated to perform reviews, because it helps build their reputation. Reviewers will want to look smart and objective when openly evaluating a paper; open reviews that are low-quality or appear to serve the reviewer's own interest will negatively impact the reviewer's reputation. Average ratings will come with confidence intervals, bringing science and statistics to the evaluation process. The weighting of different reviewers (e.g. by h-factor) and different rating scales (e.g. by relevance to a decision process) can be defined post-hoc in multiple paper evaluation functions (PEFs). These PEFs defined by individuals or groups (e.g. scientific societies of governmental organisations) provide a plurality of perspectives for prioritising the scientific literature. The transition toward a future system of instant publication can be achieved by providing an OE system that will initially serve to more deeply evaluate important papers published under the current system of pre-publication peer review. When the OE system has proven its superiority to the current system of peer review, it will replace the current system.

(1) The current system of scientific publishing

The current system of scientific publishing is not generally open access, provides only journal prestige as an indication of the quality of new papers, relies on a secretive, intransparent, and inherently noisy paper evaluation process, delays paper publication by many months on average, incurs excessive costs, and is privately and intransparently controlled.

The current system serves two positive functions: (1) It administers peer review and provides an evaluative signal that helps readers choose papers, namely journal prestige. This function is critical to scientific progress. However, the drawbacks listed above suggest that the current system does not serve this function satisfactorily. (2) The current system provides an appealing layout for scientific papers. This function is desirable, but not critical to scientific progress.

Recent positive developments in scientific publishing include the Public Library of Science (PLoS) and other open-access journals, the Frontiers journals, and Faculty of 1000, which provides very brief post-publication recommendations of papers with a simple rating ("Recommended", "Must-read", "Exceptional"). ResearchBlogging.org collects blog-based responses to peer-reviewed papers – a big advance as it allows anyone to participate and provide evaluative information. However, as of yet these responses lack numerical ratings, are not digitally signed, and not visible when viewing the target paper itself.

While these developments represent important steps in the right direction, they do not go far enough to fully address all problems related to the way the current system utilizes peer review, namely the lack of quality and transparency of the peer review process, the substantial publication delays, the lack of availability of evaluative information about papers to the public, and the excessive costs incurred by a system, in which private publishers are the sole administrators of peer review.

Fig. 1: The current system

(2) The crucial innovation: open post-publication peer review

The problems of the current system can all be addressed by open post-publication peer review.

Open: Any scientist can instantly publish a peer review on any published paper. The scientist will submit the review to a public repository. Reviews can include written text, figures, and numerical quality ratings on multiple scales. The repository will link each paper to all its reviews, such that readers are automatically presented the evaluative meta-information. In addition, the repository allows anyone to rank papers according to a personal objective function computed on the basis of the public reviews and their numerical quality ratings. Peer review is open in both directions: (1) Any scientist can freely submit a review on any paper. (2) Anyone can freely access any review.

Post-publication: Reviews are submitted after publication, because the paper needs to be publicly accessible in order for any scientist to be able to review it. Post-publication reviews can add evaluative information to papers published in the current system (which have already been secretly reviewed before publication). For example, a highly controversial paper appearing in Science may motivate a number of supportive and critical post-publication reviews. The overall evaluation from these public reviews will affect the attention given to the paper by potential readers. The actual text of the reviews may help readers understand and judge the details of the paper.

Peer review: Like the current system of pre-publication evaluation, the new system relies on peer review. For all of its faults, peer review is the best mechanism available for evaluation of scientific papers. Note however, that public post-publication reviews differ in two crucial respects:

(1) They do not decide about publication – as the papers reviewed are already published.

(2) They are public communications to the community at large, not secret communications to editors and authors.

This makes the peer reviews the equivalent of getting up to comment on a talk presented at a conference. Because these reviews do not decide about publication, they are less affected by politics. Because they are communications to the community, their power depends on how compelling their arguments are to the community. This is in contrast to secret peer review, where uncompelling arguments can prevent publication because editors largely rely on reviewers’ judgments, because there is too little time and no formal mechanism for a judgment of the reviewers’ judgments.

Merging review and reception: Currently review is a time-limited pre-publication process and reception of a paper by the community occurs later and over a much longer period, providing a very delayed – but ultimately important – evaluative signal: the number of citations a paper receives. Open post-publication peer review will remove the artificial and unnecessary separation of review and reception. It will provide for a single integrated process of open-ended reception and review of each paper by the community.

Signed or anonymous reviews: The open peer reviews can be signed or anonymous. In analyzing the review information to rank papers, signed reviews can be given greater weight if there is evidence that they are more reliable.

Fig. 2: The future system

Digitally authenticated: Reviewers can digitally sign their reviews by public-key cryptography (http://en.wikipedia.org/wiki/Public-key_cryptography). The idea of digitally signed public reviews has been developed here (http://code.google.com/p/gpeerreview/), where further discussion and a basic software tool that implements this function can be found.

Reviews as open letters to the community: Reviews will no longer be secretive communications deciding about publication. They will be open letters to the community with numerical quality ratings that will influence paper visibility on webportals (see below). Open post-publication review will build on the current system by providing a forum for comments and evaluations of papers.

Open posting of private pre-publication reviews: The new system will allow the original pre-publication reviewers of a paper to make their reviews public, so that their work in reviewing the paper can be of benefit to the readers of the paper and to the community at large.

Community control of the critical function of paper evaluation: Open post-publication peer review allows the scientific community to organize the evaluation of papers, thus taking control of this critical function, which is currently administered by publishers.

Improving evaluation quality: The quality of the evaluative signals will be improved by post-publication review for a number of reasons:

(1) Since reviews are open letters to the community, their power is dependent on how compelling they are to the community. (In the present system, a scientist can reject a paper with no good arguments at all – for a high-impact journal perhaps he’ll say that the paper is good, but claim that the finding not sufficiently surprising.)

(2) Many reviews will be signed, so the reviewer’s reputation is on the line: he or she will want to look smart and reasonable. (Anonymous reviews can be down-weighted in assessment functions if they are thought to be less reliable.)

(3) Important papers will accumulate more reviews over time as the review phase is open ended, thus providing an increasingly reliable evaluative signal.

Eventually, journal prestige will no longer be needed as an evaluative signal.

Fig. 3: Open publication and evaluation process

Multiple lenses onto the literature: The necessary selection of papers for reading can be based on the reviews and their associated numerical judgments. Any reader can define a paper evaluation function (PEF) based on content and quality criteria and will automatically be informed about papers best conforming to his or her criteria. The evaluative function could for example, exclude anonymous reviews, exclude certain reviewers, weight evidence for central claims over potential impact of the results etc.

Fig. 4: A plurality of paper evaluation functions (PEFs) provide multiple lenses onto the literature

Webportals as entry points to the literature: Webportals will serve as entry points to the literature, analyzing the numerical judgments in the reviews by different criteria of quality and content (including the use of meta-information about the scientists that submitted the reviews). There will be many competing definitions of quality – a unique one for each webportal or each individual defining his or her own paper evaluation function. Webportals can define such evaluation functions for subcommunities – for scientists too busy (or too lazy) to define their own. A webportal can be established cheaply by individuals or groups whose members share a common set of criteria for paper prioritization.

(3) The ultimate goal: free, instant scientific publishing

Free instant publishing: Once open post-publication peer review provides the critical evaluation function, papers themselves will no longer strictly need journals in order to become part of the scientific literature. They can be published like the reviews: as digitally signed documents that are instantly publicly available. Post-publication review will provide evaluative information for any sufficiently important publication. With post-publication review in place, there is no strong argument for pre-publication review. Publication on the internet can, thus, be instant and reviews will follow as part of the integrated post-publication process of reception and evaluation.

Peer-to-peer editor choice: After publication, the author asks a senior scientist in his or her field to edit the paper. If the senior scientist accepts, an acknowledgment of his or her role as editor will be added to the paper. The editor’s job is to select two to four reviewers and to email them with the request to publicly review the paper. If they decline, the editor has to find replacements.

However, anyone else is allowed to review the paper as well. In particular, the author may also inform other scientists of the publication and ask them to review the paper. Author- and editor-requested reviews will be marked as such. Requested as well as unrequested reviews can be signed or unsigned.

Editors must not have been at the same institution or on any paper with the authors. Reciprocal or within-clique review editing is monitored and discouraged. Such information, will in any case be publicly available after the fact and may be used to weight the reviews in any automatic quality assessment.

Fig. 5: The nature of a review in the current and future systems

Revisions: If the weight of the criticism in the accumulated reviews and the importance of the paper justify it, the authors have the option to revise their paper. The revision will then be the first thing the reader sees when accessing the paper and the authors’ response to the reviews may render the criticism obsolete. However, the history of revisions of the paper, starting with the originally published version, will always be available – along with the complete history of reviews and author responses.

Reviewing-activity statistics: Reviewers must be registered as professional scientists whether they sign a given review or not. The level of reviewing activity of a scientist is a public piece of information. Similarly, it is public information, how many reviews were written anonymously by each scientist. (However, it is not public information, of course, which papers a scientist reviewed anonymously.) In general, each scientist is expected to write about as many reviews as he or she receives.

The process of reception and review: Good papers will accumulate many and positive judgments over time and bubble up in the process – some after 4 reviews and 2 weeks past publication, others after years.

High-prestige publications such as Nature and Science could take note of independently published studies that have turned out to be important. Based on the broader and more reliable evidence of public review, they could decide to showcase, i.e. to republish, these projects – perhaps in a modified version suited for their more general audience. These high-prestige journals would, thus, benefit from the greater quality of the broader and deeper public evaluation of the papers, which would contribute to the quality of their product.

(4) Additional information on these ideas

The ideas in this short paper are explored in greater depth in my blog at:

http://futureofscipub.wordpress.com/.

In addition, I am currently editing a Special Topic on the challenge of developing an open evaluation system for all of science in Frontiers in Computational Neuroscience. This will be a collection of 10-20 papers with visions for a future system of open evaluation and peer review. The call for papers is here:

http://www.frontiersin.org/computational%20neuroscience/specialtopics/beyond_open_access__visions_fo/137

(5) Declaration of interests

I am a brain scientist interested in contributing to an improved system for scientific publication and evaluation. I have no personal financial stake in this matter or any other conflicts of interest.

Nikolaus Kriegeskorte

Medical Research Council

Cognition and Brain Sciences Unit

Cambridge

28 February 2011