Peer review

Written evidence submitted by
Professor Daphne L McCulloch (PR 76)

Section 1 Authorship and declaration of interests

Submitted by: Daphne L McCulloch OD, PhD

Current post: Professor, Department of Vision Sciences,
Glasgow Caledonian University

Relevant Activities/interests:

· Personal research in the fields of visual development and specialist vision assessment strategies

· Regular authorship and co-authorship of scientific journal papers that are subject to peer review (14 peer reviewed publications since 2008)

· Regular reviewer for peer reviewed journals and for research grant applications to government and charitable organisations (approximately 10 reviews per annum)

· General Secretary, Board of Directors, International Society for Clinical Electrophysiology of Vision www.iscev.org

· Editorial Board member for two journals, Documenta Ophthlmologica and Ophthalmic and Physiological Optics

Capacity: This is the personal view of an individual academic, University teacher and clinical research scientist.

Section 2 Introduction

Peer review is the mainstay of quality control in science and technology. This process is so central to research and development that "peer reviewed" is almost synonymous with high-quality scientific work. As peer review is so central to science and technology it is important to review its implications, strengths, weaknesses and alternative strategies in order to make the most effective use of the peer review process.

Section 3 Requirements for quality review in science and technology

3.1 Quality review is necessary to support advances across a wide variety of fields. The applications relevant to the current inquiry are to examine and validate scientific results and papers prior to publication. Additional applications include evaluation of research proposals and awards to optimise the use of resources to achieve the greatest benefits. The specific goals of quality review for publication include assessment of:

Validity: Is the information sound and any conclusions valid based on the known reliability of the methods, statistical power and absence of confounding factors?

Importance: Does the information contribute to new knowledge or provide supportive evidence for an issue of current controversy? Is it important to the target audience of the publication?

Balance: Does the information properly take account of relevant work in the field and related fields and clearly identify any opinion as distinct from factual information.

Clarity: Are the language and illustrations clear and understandable to the target audience?

3.2 Evaluation of research potential includes specific research proposals, research potential of individuals and groups such as the appointment of individuals to panels and posts, research prizes and awards and future funding streams such as the REF requires that past outputs are judged collectively using the above goals. The quality review must then integrate the assessment of the past to predict the future potential. In the case of a specific research proposal the additional evaluation of the probability of successful completion and value for money are usually necessary.

Section 4 Definitions:

Expert review: Evaluation by selected authorities in the relevant field or fields.

Peer review: Evaluation by people in the same occupation, profession, or industry (usually implying that the evaluators are at a similar level to those being evaluated).

Lay review: Evaluation by people who are from a not authorities or members of the same occupation or profession.

Often the terms "peer review" is used when and "expert review" is actually undertaken, for example a proposal or manuscript from a junior scientist will invariably be referred for expert reviewer. When the applicant is an expert, this same expert review is also a peer review. I will use the phrase "expert peer review’ to cover both situations.

Section 5 Strengths and weaknesses of expert peer review

5.1 Efficiency: The expert peer review system has evolved based on a very high degree of good will. In most cases the reviewers act as volunteers and contribute their expertise solely for the enhancement and improvement of science and technology. Where reviewers receive a payment, this is usually nominal, and not commiserate with time required or with the cost of expert advice.

The disadvantages of the expert peer review system for efficiency is that selected reviewers may well decline requests to review, particularly as the burden of such work is escalating. In addition, the hidden cost of review impacts on employers, Universities in particular, that provide the time and resource to support widespread expert peer review. Looking to the future, unlimited expert peer review cannot be presumed.

5.2 Balance and bias: Whilst there are exceptions, on the whole expert peer review provides a balanced quality review. This depends on both the integrity and the range of knowledge of the reviewer. The system has evolved such that single opinions require affirmation and differing opinions require further review. The system of expert peer review is fallible; erroneous or deceptive information may pass review and sound proposals or research may be rejected. Although there usually are not formal appeal processes, science and technology allows for further review – alternative publications and alternative funding streams for those who are harshly judged, rebuttal and presentation of further evidence when weak science is passed through the peer reviewed system.

Section 6 Alternatives to expert peer review:

6.1 Expert review: The implication of expert review is that such review would be by selection of appropriate authoritative reviewers who would be paid for their opinions. This system has the advantages that greater rigor could be required and that the reviewers may be more targeted authorities. The obvious disadvantage is that costs would be very high; think of the costs of expert legal advice and apply this across the fields of science and technology!

6.2 Lay review: Intellegent and informed lay people have valuable opinions. However, they lack the specific knowledge necessary to assess aspects such as validity, originality, importance and balance of a proposal of manuscript. Training of lay people would require expert input in the specific field, a prohibitive prospect. I suggest that the value of lay opinion lies in providing input to such areas as research priority rather than research quality.

6.3 Research metrics: In some situations metrics are available to reduce the requirement for peer review. For example the quality of a research team and their potential to complete a proposal or generate innovation from a funding stream is reflected in metrics such as the research outputs or citation rates. Metrics alone though do not ensure that a current proposal or piece of research is of high quality. This is particularly the case when assessing contributions by teams of scientists.

Section 7 Summary and recommendations
7.1 Expert peer review in all of its applications is the indispensable mechanism for quality control across a very wide range of scientific endeavour. Expert peer review is highly efficient and delivers a high degree of accuracy for the evaluation of the quality of science and technology.

The expert peer review system requires support and development. At present ‘good will’ is straining. Experts are finding that the requests for such review are increasing and in these times of economic restraint, employers are rightly asking for an account of activities and ‘outputs’. The system is mutual and therefore the time volunteered by scientific experts around the world benefits not only science and technology but also the reviewer (and his employer) who obtain regular quality feedback with little or no cost.

7.2 Support for expert peer review could include.

· Greater publicity and acknowledgement of the peer review contributions would go a long way to encouraging and improving expert peer review reviewers. Specific acknowledgement of the volume of peer review in the REF.

· Funding directly to reviewers and employers (approximately 50-50 split to acknowledge the extra time as well as the institutional resource. Even token amounts would build on the good will and encourage participation of selected experts.

· Development of metrics in support of the expert peer review system. These could include provision of summary statistics for authors and applicants (numbers of previous publications, number of time these have been cited and global indices such as the H-index)

Please feel free to contact me for further information regarding my opinions or to provide feedback regarding whether this input has been useful

Prof essor Daphne L. McCulloch

Vision Sciences

Glasgow Caledonian University

10 March 2011