Peer review


Written evidence submitted by Martin Hill (PR 72)

1 Abstract


1.1. Peer review is commonly perceived by academics and related institutions as a 'quality control with issues'. Corrections to those issues tend to be directed at modifying existing peer review practices.

1.2. I suggest directing corrections to the ends – quality suitable to inform government policy – rather than the traditional means. Work practices from commerce and some academic disciplines could be more widely applied to policy-informing research, regardless of journal peer review practices.

2 Introduction


2.1. This paper is a summary extract of research into knowledge distribution systems by Martin Hill, at the Department of Information and Systems Engineering, Cranfield University, Defence Academy UK. This research models and simulates how reliable and trustworthy expert opinion can be disseminated in three general cases: academic research, tactical military operations, and civilian systems engineering.

2.2. Specifically and relevantly, part of the research compares academic research practices and those of commercial systems engineering projects. This is not yet complete, but provisional findings may be suitable for this inquiry.

3 Peer Review


3.1. ‘Peer Review’ is an umbrella term used for a number of activities, usually:

3.1.1 Journal Paper Submission: papers are submitted to journals for publication, the editor identifies reviewers and forwards the paper to them for comment. This is "very slow, takes up a lot of academic time, is highly selective, is very variable, arbitrary, prone to bias, open to abuse, patchy at detecting important methodological defects, and almost useless at detecting fraud and misconduct" [1]

3.1.2 Funding Allocations: experts in the field sit on the research council funding boards, and advise which research to direct funds to. This paper does not consider this further.

3.1.3 Post-publication assessments, responses and replications: once a paper is published, it is read by other experts who can comment on it and carry out further work to publish in response. This ‘discourse by document’ has typical transaction times of years, with the delays of journal peer review typically contributing one of those years.

3.2. Trivially, any useful review of the conclusions by experts has to be by other experts, who are therefore the ‘peers’ of the expert under review. In academic research however, ‘peers’ tend to refer only to those who share the same research discipline, rather than experts from related relevant fields. For example, it is rare to see expert software engineers in astronomy review panels, despite the heavy dependence of modern astronomy on bespoke software toolsets.

4 Assuring Quality


4.1. The background intent behind this inquiry appears to be [1] how can we improve ways of extracting reliable understanding (not necessarily reliable truth) from academic research in order to inform policy, the public, and other research. That is, we do not need certainty from scientists, but we need to know that the uncertainties and gaps are identified and that the evidence, conclusions and opinions are suitably scoped.

4.2. This reliability and scoping of what is known and how well it is known is vital in several human activities, particularly in safety and mission critical research, development and engineering. The activities, reviews, organisation structures, skills, processes and so on that assure this are typically bundled under the term "Quality Assurance".

4.3. Some traditional inertia asserts that academic research is ‘special’, and therefore that quality assurance practices from other human activities cannot be applied. However research disciplines such as medicine have seen considerably strict quality controls introduced over the last few decades, research disciplines such as astronomy have valued quality controls such as audited data archiving for hundreds of years, and research disciplines such as agrochemistry have long had oversight regulations derived from quality controls.

5 Peer Review & Quality


5.1. The key issue with peer review as a quality control – and attempting to use it as a vehicle for improving quality – is that it occurs far too late. Quality has to be assured from the beginning of the work, not in retrospect. You cannot check that equipment has been operated correctly after the event if logs have not been kept, you cannot check that the equipment has been cleaned correctly if there is no inspection, that adjustments to data have been applied suitably if there is no audit, that inconvenient data has not been discarded or convenient data fabricated.

5.2. It has been argued that professional pride and fear of rejection encourage researchers to rigour, but such incentives have to be weighed against ambition, personal and community biases, and the drive for high citation rates.

5.3. Journal peer review would remain vital to disseminating knowledge in academic research. The vast range of work produced cannot all be read by everyone, and journal peer review provides a mechanism for selecting from all the work those items that are interesting and relevant.

5.4. By separating quality assurance from peer review, we can expect to see properly assured published work that journals would not be interested in, which is vital for reliable metastudies. That is, journals can continue to select a subset for reading, but studies that are ‘boring but important’ and not seen as suitable for journal publication can still be made available with useful appropriate quality assurances.

6 Science, Quality and Policy


6.1. ‘Science’ is a large umbrella term that includes a huge range of fields, practices and communities. Research into the ‘hard’ sciences such as physics would necessarily have very different quality assurance requirements to those ‘soft’ sciences such as the psychology of learning.

6.2. The aggregation and assembly of knowledge to inform government policy has different requirements to that used to inform cutting-edge researchers. Attempting to keep that activity combined into one practice will result in the continued tension between the speculative work and opinion necessary to develop new understanding, and the more reliable scoped understanding necessary to implement workable, appropriate, safe, improving policy.

6.3. All the same, quality controls that are built into ordinary research should improve the "speed of reliable discourse" of academic researchers as the quality of the paper is better by the time it leaves the institution, rather than much later after replication.

6.4. Similarly quality assurance reduces the attention consumed by studying, the time and resources consumed by replicating, and the public trust consumed by contradictions, due to fraudulent and mistaken work. These savings in time and resources have to be weighed against those spent on quality controls, and vary from discipline to discipline.

7 Responsibility and Funding


7.1. Quality controls are generally identified as part of project planning, in consultation between the project managers, working staff and quality assurance experts. They are funded, run and managed by the project, but monitored and certified externally to ensure independence.

7.2. Institutions are responsible for the quality controls of their projects just as they are responsible for other aspects of their work, from IT support to power to training. Universities have already introduced such controls, so developing and using them to certify contributions to policy for example will not require starting from scratch.

7.3. The extra time, effort and funding required to include quality controls are a barrier to adoption. For small projects, this barrier can be considerable compared to the time, effort and funding of the research work, and has to be weighed against the benefits to the wider scientific community and policy makers of more reliable research outputs. For large projects, such practices can (indeed, should) benefit the collaboration and are sometimes necessary for the project to succeed at all.

7.4. Another barrier is the perception of the clipboard tyrant interfering with science. Research can be attractively seen as unfettered investigations into the workings of people, the world, the universe. However when government policy needs to be informed by that work, then that work needs a sound footing.

8 Conclusion


8.1. Peer Review is not a suitable mechanism to build quality assurance on, largely because it occurs far too late – after the work is completed.

8.2. Quality controls are widely used in many human activities that require the aggregation of expert knowledge. There is no one-size-fits-all, but there are many experts in the field.

8.3. Academic institutions are already introducing quality controls in some research disciplines. Supporting this with independent inspections, national standards and certificates would encourage recognition of valued work independently of journal publication.

Martin Hill

10 March 2011

[1] “Peer Review in the e-environment”, Fiona Godlin @ Freedom of information conference 2000, . A review of widespread dissatisfaction with journal peer review, and the documented widespread failures to assure quality, is available if requested.