Peer review

Written evidence submitted by
Sir Iain Chalmers, James Lind Initiative (PR 47)

1 What’s wrong with the current situation?

1.1 Billions of pounds are wasted each year in the steps from research question to publication: much research is unnecessarily duplicated, poorly designed, unpublished or unusable (1). To reduce this waste, proposals for new research and reports of new research should begin by using or doing systematic reviews of all the existing evidence relevant to the research questions being addressed. These systematic reviews must use explicit, scientifically-defensible methods, both to reduce biases, for example, biased reporting of research (2), and to reduce the likelihood of being misled by the play of chance (3, 4). Reports of new research should also indicate in updated systematic reviews what contributions new evidence has made to the totality of evidence addressing particular questions (5).

1.2 These principles continue to be widely ignored, both by researchers and by the peer reviewers who judge proposals for new research and the reports of new research submitted for publication (5, 6, 7). As a result, people have suffered and died unnecessarily (8), and resources for health services and research have been wasted (1). Furthermore, current regulation of biomedical and health research has done very little to reduce these serious problems. Indeed, because of the delays it has caused in addressing uncertainties about the effects of treatments, research regulation has resulted in avoidable suffering and deaths of millions of patients, most of whom have not been participants in research (9).

2 What can be done to address the problems outlined above?

2.1 By research funders: Some research funding organisations have begun to address the problems outlined above. The National Institute for Health Research (NIHR), for example, is one of a few research funding bodies which have addressed these dangerous and wasteful deficiencies systematically. Researchers applying for research funding support are now required to show by reference to relevant systematic reviews of existing evidence that the proposed new research is necessary, and that it has been designed to take account of the lessons learned from previous research (http://www.netscc.ac.uk/funding/calls.asp). Since around half of all research is never published (1), the NIHR withholds 10 per cent of the value of each grant until the researchers have submitted a report of their findings for publication. Other research funders should implement similar policies.

2.2 By the researchers: To make clear why research is being done, and to reduce wasteful duplication, researchers should publish protocols for their research, whether these are for additional primary studies (10), or for systematic reviews of existing evidence (11).

2.3 By medical journals: Of the five principal general medical journals (Lancet, BMJ, JAMA, New Eng J Med, and Ann Intern Med) only The Lancet has made clear to potential contributors that, regardless of the type of research being reported, the journal will expect reports of research to begin and end with systematic reviews of other research addressing the same questions (12). The Lancet’s example should be emulated by other journals.

2.4 By government: Mandatory elements of current research regulations in the UK have done nothing to address the problems outlined above (9, 13). Peer review cannot take account of what cannot be seen. To make clear to the public that government rates the interests of patients above those of other interests, it should require registration of research and publication of protocols (10, 14), and take steps to outlaw the unethical, unscientific, wasteful and often biased failure to report research (15).

Declaration: My work is funded by the NIHR, but my principal interest now is as a patient who wishes choices in his health care to be informed by scientifically reliable evidence.

References

1 Chalmers I, Glasziou P (2009). Avoidable waste in the production and reporting of research evidence. Lancet 374:86-89. doi:10.1016/S0140-6736(09)60329-9.

2 House of Commons Health Committee (2005). The Influence of the Pharmaceutical Industry. Fourth Report of the Session 2004-2005, Vol. II. London: Stationery Office.

3 Sheldon T, Chalmers I (1995). The NHS Centre for Reviews and Dissemination and the UK Cochrane Centre. In: Select Committee on Science and Technology, Sub-Committee I. Medical Research and the NHS Reforms. London: HM Stationery Office, 1995:151-167.

4 Sense about Science (2009). Systematic reviews. http://www.senseaboutscience.org.uk/PDF/SenseAboutSystematicReviews.pdf.

5 Clarke M, Hopewell S, Chalmers I (2010). Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. Lancet 2:20-21.

6 Jefferson T, Deeks J (1999). The use of systematic reviews for editorial peer-reviewing: a population approach. In: Godlee F, Jefferson T, eds. Peer review in health sciences. London: BMJ Publishing Group, pp 224-234.

7 Robinson KA, Goodman SN (2011). A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med January 4, 2011 154:50-55.

8 Chalmers I (2007). The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the need for systematic reviews. In: Rothwell P, ed. Treating individuals: from randomised trials to personalised medicine. London: Lancet, pp 37-58.

9 Chalmers I (2007). Regulation of therapeutic research is compromising the interests of patients. International Journal of Pharmaceutical Medicine 2007;21: 395-404.

10 Chan A-W (2008) Bias, spin, and misreporting: time for full access to trial protocols and results. PLoS Med 5(11): e230. doi:10.1371/journal.pmed.0050230

11 PLoS Medicine Editors (2011) Best Practice in Systematic Reviews: The Importance of Protocols and Registration. PLoS Med 8(2): e1001009. doi:10.1371/journal.pmed.1001009

12 Clark S, Horton R (2010). Putting research into context-revisited. Lancet 376:10-11.

13 Academy of Medical Sciences. A new pathway for the regulation and governance of health research. http://www.acmedsci.ac.uk/index.php?pid=99.

14 Chalmers I (2004). Government regulation is needed to prevent biased under-reporting of clinical trials. BMJ;329:462.

15 Chalmers I (2007). When does clinical science cease to exist? Science in Parliament

2007;64:19-20.

Sir Iain Chalmers

James Lind Initiative

9 March 2011