Peer review

1.0 The strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public?

Key

Strengths:

1.1 It is a well established, understood and accepted quality control system (particularly in academia) which should not be abandoned unless and until something which produces as good or better/faster outcomes is devised and proven.

1.2 It helps avoid the taint of ‘vanity publishing’ often ascribed to ‘open access’ content.

1.3 Findings published in peer reviewed journals are considerably more likely to be respected than findings published by the authors/organisation itself.

1.4 It attempts to ‘normalise’ submissions by applying anonymity to ‘independent’ referees in the discipline, and thereby validate the work i.e. its accuracy, potential for repetition and worthy of open disclosure.

1.5 The process should help eliminate incorrect/erroneous material appearing in the subject literature.

1.6 It should establish valid interpretation of results and formulation of appropriate conclusions.

1.7 It is a ‘free’ to authors service (publisher funds the process - but rarely pays the Reviewers!).

1.8 It should benefit authors and help dissemination of ‘good’ science through input from those with most expertise.

1.9 Peer review represents a crucial democratisation of the editorial process, incorporating and educating large numbers of the scientific community, and lessening the impression that editorial decisions are arbitrary.

1.10 The process has become a means for assessing the productivity of individual academics and university departments in terms of the number of papers they have published in peer reviewed journals or presented at conferences. This has merit so long as the process does no encourage publication for the sake of publication which dilutes quality.

1.11 Through the process the high quality of the discipline ‘literature’ should be sustained and its new ‘knowledge’ advanced with confidence.

Weaknesses:

1.12 It can be slow – it relies on the goodwill of the referees (who are time constrained) and requires a process of referral/revision and resubmission.

1.13 It relies on the comprehensive expertise of the independent referees, which may not be readily available particularly if the research is cross disciplinary.

1.14 It is resource constrained – there is an exponential growth in research output without a concomitant growth in available ‘experts’ to provide review.

1.15 There is the potential for good quality groundbreaking research to be marginalised in favour of established views.

1.16 Normalising the subject knowledge can mean that radical advances and new hypothesise can be dismissed (see Scientific Method below).

1.17 Even good journals get it wrong sometimes e.g. MMR/autism.

1.18 Scientific publishing is lucrative, more and more journals are being produced, so increasing numbers of papers are published. This can result in the quality threshold for publication in some journals being so low that peer review in these instances does not carry an adequate assurance of quality.

1.19 The size of datasets behind some research reports constrain the degree of scrutiny that is practical.

1.20 It is not transparent – it can be influenced by the position/opinion of referees especially if the authorship or provenance of the research is revealed.

1.21 Vested interests of even "independent reviewers" can result in overly supportive or too negative reviews – it is impossible to achieve an absolute peer review standard.

1.22 Reviewers may be reluctant to overly criticise a piece of work for fear of repercussions in the narrow realms of specialist disciplines.

1.23 It is rare for ‘failed’ experiments to be reported in peer reviewed journals – yet these can be informative, particularly in replicating key findings or assessing public health issues.

1.24 Peer review cannot usually detect intentional fraud or artefactual results caused by experimental methodology that has not been described by the authors.

1.25 Peer review (and open literature) is now almost the exclusive territory of the academic research sector which the public cannot access (cost) which results in the public being more dependent on un-reviewed material.

1.26 For researchers it increases the time taken to get work published or grant applications accepted.

1.27 The Scientific Method can be characterised as:

i. Pose a subject question.

ii. Research the subject background.

iii. Construct a hypothesis that may answer the question.

iv. Test the hypothesis with an experiment.

v. Analyze the data from the experiment, draw supported conclusions.

vi. Determine whether the hypothesis was ‘True’ ‘Partially True’ or ‘False’.

vii. Publish the results to inform others working in the subject.

viii. If the original hypothesise was ‘Partially True’ or ‘False’ the work can be continued by developing a revised hypothesis, new experiment etc as part of an iterative process.

A

A

A

B

B

B

C

C

D

D

2.0 Measures to strengthen peer review?

2.1 Recognise more explicitly the hierarchy of journal status: good journals can be trusted to have operated good peer review processes, whereas publication in poor journals, even if peer reviewed, should not have the same status i.e. move away from a pass/fail peer review dichotomy to more of a continuous scale; the citation index is one measure that could be used however, for this to work the difference in quality between journals needs to be recognised not just by scientists but by journalists, politicians, and ultimately the public; before that can happen, there has to be greater honesty and maturity at understanding that not all science is equal or even good.

2.2 Requiring inclusion with the publication any salient version points on which the authors and individual reviewers disagree might help inform the sector; the reviewers and authors would need to agree a review statement. This would unfortunately further extend the process.

2.3 Encourage extended commentary by the sector after a paper has been published; this has been successfully trialled by a number of journals which publish on line e.g. BMJ.

2.4 Use of peers from within the subject community is paramount and if the subject is cross-disciplinary then peers from all involved disciplines should be used to focus on their specialist subject areas of the paper, or grant application.

2.5 Peer review of conference papers should take into account that papers submitted by entrepreneurs (as opposed to scientists) may not always deserve rejection for lack of novelty (academic purity) but acceptance that iteration and improvement for business benefit has social value too.

2.6 Journals and conference organisers might require independent peer reviewers to produce a synopsis of the submitted paper as a means of ensuring true review, and as a performance check.

2.7 Use of electronic communication and relevant e-Submission tools.

2.8 Use of plagiarism detection software to reject unoriginal papers, freeing time for more deserving cases.

2.9 Some journals offer a paid for service for those organisations wishing a rapid independent peer review and publication, but this is open to criticism as being vanity publication under another guise.

2.10 Remunerate referees to encourage them to devote more time to the review process and produce a superior quality of feedback.

D

3.0 The value and use of peer reviewed science on advancing and testing scientific knowledge?

3.1 It tends to favour the status quo, potentially limiting speed of innovation.

3.2 Radical theories and those that do not conform to conventional wisdom are unlikely to be accepted for publication; thus science knowledge and technical know-how tends to advance incrementally in small steps.

3.3 It theoretically provides a means of managing the volume of published material and limits the output of meretricious work.

3.4 Science needs some form of quality control because some science, particularly in some controversial public health areas, is poor quality (see 4.1 and 4.3.). All science could be published without peer review and the quality control take place by a collective review process – only good science would survive the test of time, poor science would naturally wither away. This has two disadvantages: it would push the responsibility for quality control onto the reader, resulting in massive duplication of effort, and the inevitable increase in publications would make keeping abreast of a given subject harder still.

3.5 The Internet is creating a new spirit of openness of information which requires society to recognise that trying to control the information that reaches the public (which is what peer review does) is, to an extent, an outdated concept. This is recognised in some subject areas, such as physics, where most papers are placed on a preprint server before traditional publication.

C

C

4.0 The value and use of peer reviewed science in informing public debate?

4.1 Notably for research which has a public health implication peer review remains essential i.e. the new model (see 3.5) does not work because:

i. people would feel the need to act without waiting for a professional consensus to emerge.

ii. not everyone, especially not lay people or even some doctors, is equipped to make their own judgement of scientific data.

iii. groups with a vested interest can flood the literature with their views, masquerading as research, and potentially mislead readers (who do not have the specialist background to assess the merits of the work) and thus society at large.

4.2 There has to be some level of confidence in the quality of the science for dissemination to the public; multiple peer review is the proven means to date.

4.3 The MMR/autism issue was an example of the problems caused by incorrect science in the public domain and their significant public health impact.

4.4 The majority of the public does not ever access peer reviewed scientific papers, and for the majority of the few that do (save as requirement of their profession) it is in journals such as Nature and thus only reaches as very narrow sector of the general public.

A

A

5.0 The extent to which peer review varies between scientific disciplines and between countries across the world?

5.1 There are concerns that peer review is not applied evenly throughout the world especially in cultures where deference is given to senior/established scientists.

5.2 The cost of peer reviewed journals means that their circulation and availability to the scientific community is largely constrained to the developed world.

5.3 Peer review is a quality control means adopted by the academic and research communities, and not by the rest of society and its organs e.g. business and commerce.

5.4 Some specialist journals promulgate a particular set of beliefs in controversial areas and will use referees who are known to be sympathetic to their views.

6.0 The process by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases?

6.1 Reviewers are commonly chosen via three routes:

i. known contacts of the Editorial Board.

ii. recommendation from the author (open to nepotism).

iii. auto-generated recommendations via e-Submission tools.

6.2 A reviewer always has the option to decline to review a paper outside their expertise, or when they fear there might be a conflict of interests. They also increasingly decline due to the pressure of their ‘day job’. Often the leaders in a field fall into this category. Recompensing the referees and/or their employer could ease this problem. Inevitably the increased costs would fall upon the journal reader.

6.3 Journal editors need to maintain up-to-date information on the competences of their reviewers and should be required to periodically seek an update from the reviewer to include a listing of their qualifications to fulfil the role; a spot check audit of this material should be undertaken. But anything which involves reviewers in extra work will make them more prone to decline review work!

6.4 Journals that aim to achieve quick peer review and publication use a small pool of reviewers with broad subject knowledge, as opposed to specific expertise in a particular subject field.

6.5 The differences in choice of reviewers and in acceptance rates illustrate the difficulty of comparing the quality of peer review between different journals.

6.6 Open peer review (see 3.5) is where authors’ agree to have their papers posted on a scientific journal website for open comment whilst the papers is being sent for traditional peer review i.e. before acceptance or rejection.

7.0 The Impact of IT and greater use of online resources on the peer review process?

7.1 IT can assist by:

i. speeding the process of communication (and thereby time to publication)

ii. standardising roles and activities, thus helping to standardise the process

iii. providing transparency (a holistic view and record of who has done what).

7.2 There is the potential for providing access (Internet) to academic literature formerly only available in academic libraries and those able to subscribe to journals.

7.3 Internet only journals are emerging; peer review usually only has a limited place in these publications.

7.4 It enables an online questionnaire style review which makes the review slightly more rigorous thereby enhancing the interchange of information between the authors and reviewer.

7.5 The advent of open source information on the Internet may perversely actually increase the desire of many readers for systems such as peer review to filter information and assess its quality, and reliability, prior to becoming available!

8.0 Possible alternatives to peer review?

8.1 Peer review must not be jettisoned in favour of an alternative without extensive piloting and demonstrably better outcomes first being proven.

8.2 The traditional paper review requires multiple reviewers and review stages and provides the reader with some assurance of valid content, with the publisher charging for access to the ‘definitive’ article. Some alternative models proposed/in existence include:

i. Pay to Publish

Review process costs are covered by authors’ research funders, offering potential for publishers to pay for reviews – this is open to charge of ‘vanity publishing’ and does not of itself increase the number of available referees. Equally, renowned authors may be allowed to publish for free (i.e. skewing the output).

ii. Pay for Submission

The author pays for the reviews, the publisher reserves the right not to publish. Open to similar abuse as Pay to Publish.

iii. Referee ranking

The regular review process is used but as a ‘reward’ referees are allocated star rankings (both by their publisher and the authors of submitted papers) along the ‘eBay’ model. This would be recognition of reviewers’ efforts and could be used as evidence of their expertise so long as it can be built into their career advancement.

iv. Post-publication Review

The Wikipedia model uses the less rigorous open review model whereby the paper is made available subject to some minimal constraints, and interested parties are invited to review/edit/comment on it. This leads to a large amount of submission which in turn creates a large amount of information. The reader must make a decision regarding the validity and maturity of the information seen. Equally it will be difficult to manage the process of (de)selection since anyone would be able to offer an opinion, however poorly informed.

v. Closed Access Forum to Open Access

A dynamic group collaboration of experts provided with tools to manage the submission process to a closed forum on a topic. If the content became a ‘hot’ topic, the publisher would have the content reviewed and make the forum ‘open’ to view.

8.3 Consumers and users of information will increasingly challenge the view of a "peer", since the commercial and business world does not use peer review. Perhaps open publication using freely accessible media can be relied upon with the value being judged over time by all users – material of value will gain prominence and respect by "popular demand" with the inevitable converse. But will that which does not gain prominence actually get removed from availability (Internet)? This would however, lead to a major danger of short/medium term instability in science/medicine in which fashionable bandwagons are jumped upon, or extremist views promulgated before being subsequently discarded perhaps decades down the line. However, an exception to this approach: ‘Health Issues’ was given in 4.1.

8.4 The use of the same reviewers for papers as research grant proposals may be a practice that is in need of change; it can be argued this is an incestuous vicious circle which requires breaking. But is this practical?

8.5 Alternative methods such as publishing research results solely on websites for open inspection and comment (as being done by some medical journals) may increase in popularity and challenge the dominance of traditional peer review. This may make it less likely that peer review will be used effectively and honestly. However, the over all benefit of such methods to the scientific process has yet to be proven given that the comments will still require review (moderation) by an independent (not author) knowledgeable authority to distinguish the ‘valid’ from the ‘invalid’! In effect they will be tending to take on the role of a reviewer.

A