Science and Technology CommitteeJoint written evidence submitted by The Cochrane Collaboration & the Centre for Reviews and Dissemination

Executive Summary

1. Systematic reviews1 are increasingly used as best evidence to inform decisions in health care. Researchers preparing systematic reviews aim to include all relevant data irrespective of whether they have been published in the scientific literature or not, and irrespective of whether it is favourable or unfavourable to the intervention, to provide reliable estimates of the benefits and harms of any given intervention.

2. There is substantial evidence that incomplete disclosure of data from clinical trials is widespread and impacts adversely on patient care and public health. Decision-makers, including health professionals, patients, carers, and clinical guideline developers, may be led to believe that treatments are more effective and less harmful than they really are. The effects of non-disclosure include:

2.1Patients are harmed by misinformed treatment decisions.

2.2Public and private resources are wasted on ineffective or harmful treatments.

2.3Clinical trial participants trust that their involvement in research will lead to better care for others, and when the data they provide are withheld, this trust is abused.

3. This submission, focusing on the Committee’s questions three to five, from The Cochrane Collaboration and the Centre for Reviews and Dissemination shows how withholding of clinical trial data can cause harm to public health. It also describes some lessons that can be learned from experience outside the UK and presents our proposals for action.

Introduction to The Cochrane Collaboration and the Centre for Reviews and Dissemination

4. The Cochrane Collaboration (, established in 1993, is an international network of more than 27,000 people from over 100 countries including health professionals, researchers, methodological experts, policy-makers, and consumers such as patients and their advocates and carers. The primary purpose of The Cochrane Collaboration is to prepare, update, and promote the accessibility of high-quality systematic reviews of the effects of interventions in clinical care, health policy, and other aspects of health and social care. Over 5,000 systematic reviews (“Cochrane Reviews”) have so far been published in the Cochrane Database of Systematic Reviews, part of The Cochrane Library ( The Cochrane Collaboration is the world’s largest organization preparing and maintaining systematic reviews in health care. The Cochrane Collaboration’s work is internationally recognized as a benchmark for high-quality information about the benefits and harms of healthcare interventions and has strong representation from the UK. The Cochrane Collaboration receives funding from a variety of public sources, with the National Institute for Health Research (NIHR) being a major contributor, as well as from royalties from sales of The Cochrane Library, and has a commercial sponsorship policy for its research.2

5. The Centre for Reviews and Dissemination (CRD) is part of the NIHR and a department of the University of York. Established in 1994 to support NHS decision-making, CRD produces freely available databases ( of systematic reviews, economic evaluations, and health technology assessments based on the worldwide research literature, and maintains an international prospective register of systematic review protocols. CRD also undertakes systematic reviews and economic evaluations of health and public health questions and carries out underpinning methodological development.

6. Neither The Cochrane Collaboration nor the Centre for Reviews and Dissemination (CRD) receive funding from the pharmaceutical industry.

Factual Information

Question 3: What evidence is there that pharmaceutical companies withhold clinical trial data and what impact does this have on public health?

7. Trials that appear to demonstrate beneficial effects of an intervention being tested are more likely to be published,3 , 4 published more quickly,5 and disclosed in greater detail6 than those that fail to show such benefit or which identify harm.

8. Full reports of drug studies identified by searching regulatory agency databases have often not been published, or contain far more information than published papers.7 , 8 , 9 Legal settlements with drug companies have identified many unpublished trials or data.10 , 11 , 12 Researchers have also identified unpublished trials and data by searching for trial protocols, such as records registered in a clinical trials database.13 , 14

9. An international team of Cochrane researchers evaluating the effects of neuraminidase inhibitors (such as Tamiflu and Relenza) discovered that many more trials of these drugs had been conducted than had been published when they reconstructed the hidden trial programmes by cross-referencing publication bibliography, correspondence, conference abstracts, pharmaceutical, and regulatory sources.15 Eight studies were available from published sources in the Cochrane Review published in 2006,16 compared with over 120 unpublished studies identified subsequently.17 Insufficient data were available from these studies to evaluate the effects of the agents on preventing severe complications of influenza. Oseltamivir (Tamiflu) was used widely within the UK during the 2009 influenza epidemic. The Cochrane team, led by Dr Tom Jefferson, has prepared a separate response for the Committee describing this research in more detail.

10. Researchers from the Nordic Cochrane Centre planned to assess the benefits and harms of two slimming pills, orlistat and rimonabant, but the European Medicines Agency (EMA) denied access to data they held on the unpublished trials. After the researchers complained to the European Ombudsman, the EMA reversed its position and has now taken steps to ensure greater transparency in the disclosing trial data it receives during the licensing of drugs and devices.18 Their success in persuading the EMA to release the data is well-described in Ben Goldacre’s book, Bad Pharma.19

11. An analysis of 164 efficacy trials submitted to the US Food and Drug Administration (FDA) in 33 approved new drug applications found that a quarter of trials submitted remain unpublished five years after FDA approval of the drug.20 All these trials were industry-sponsored. Among those trials published, unexplained discrepancies between the trials and their corresponding publications—the addition or deletion of outcomes, changes in the statistical significance of reported outcomes, and changes in overall trial conclusions—tended to lead to more favourable presentations of the drugs in the medical literature available to healthcare professionals. A further report from the same research team, again studying FDA data, showed that including unpublished studies in two meta-analyses led to changes in the overall results.21

12. A landmark review published in the New England Journal of Medicine in 2008 used reviews from the US FDA to investigate studies covering 12 antidepressant drugs.22 All of the studies were industry-funded. They demonstrated that about a third of studies, involving 3449 participants, were unpublished. Studies that demonstrated an overall beneficial effect of the drug in question were far more likely to be published. Limiting the analysis to published data only resulted in an overall effect that was 32% more favourable than if the analysis had included both published and unpublished data. This highlights the considerable risk of decision-makers relying solely on published reports; in this example these provided substantially over-optimistic results.

13. A systematic review of the effect of a class of antidepressants, selective serotonin reuptake inhibitors (SSRIs), in children showed that including unpublished trials changed a favourable risk-benefit profile to an unfavourable one for several of the drugs.23 The drug companies concealed for many years unpublished trial data suggesting that these drugs increase the risk of suicide in children and adolescents.24 This was the subject of BBC Panorama programmes.

14. Even when trial reports are published the quality and completeness of the information presented may be inadequate and misleading. In an article from the Journal of the American Medical Association in 2004, researchers studied the results of 102 trials covering 3736 trial outcomes.25 They found that only half of the positive outcomes and 35% of the harms were reported adequately, with a bias towards reporting findings that demonstrate a beneficial effect over those that do not. As the article authors report, “published articles as well as reviews that incorporate them, may be unreliable and overestimate the benefits of an intervention”.

15. There have been a number of high-profile court cases in the USA in which pharmaceutical companies have been found guilty of withholding data from clinical trials, resulting in settlements against them:

15.1In 2010, Forest Pharmaceuticals pleaded guilty to charges relating to three drugs—levothyroxin (Levothyroid), citalopram (Celexa), and escitalopram (Lexapro)—and paid costs of over $313 million USD in the legal case against them. The company publicized positive results of a study of citalopram (Celexa) in adolescents but failed to disclose the negative results of a contemporaneous European study in a similar population.26

15.2In July 2012, GlaxoSmithKline was the subject of the largest healthcare fraud settlement in US history. The company pleaded guilty to one count of failing to report safety data about the diabetes drug rosiglitazone (Avandia) to the FDA. The US also alleged that GlaxoSmithKline did not disclose data from two studies on the use of paroxetine (Paxil) for adolescents, both of which failed to demonstrate efficacy in treating depression in this age group.27

16. Publication and outcome reporting bias is not limited to industry-sponsored research. For example, Cochrane researchers investigating the effects of vitamin A supplementation in low- and middle- income countries concluded cautiously, on the basis of evidence available to them at the time, that vitamin A probably saved lives when given to children.28 However, they were aware of a publicly funded trial involving one million children completed at least four years earlier that was not available in any published form. They included preliminary results from the study based on information contained in a set of PowerPoint slides posted on the Internet.29 This reduced the size of the effect of vitamin A supplementation on mortality. With only limited information about the trial available, they could not appraise the study fully and await full publication of the study.

17. The events outlined above demonstrate that clinical trial data have been withheld. The impact on public health is considerable, including inappropriately optimistic assumptions about the benefits of treatments leading to misinformed healthcare decisions, harm to patients, and waste of resources. This can occur at the level of the individual patient consultation or at a population level when, for example, guidance provided by trusted bodies may be unknowingly founded on an incomplete and biased evidence base.

Question 4: How could the occurrence and results of clinical trials be made more open to scrutiny? Who should be responsible?

18. Governments should ensure that:

18.1Central repositories or databases of results from trials are made available, so that data can be stored with appropriate safeguards. Collective responsibility for trial data, which is shared by sponsors, investigators, research ethics committees, trial participants, and the wider public, requires governments to ensure that adequate mechanisms, resources, and infrastructure are provided to facilitate access to the protocols, results, and data.

18.2Government agencies should consider introducing legislation that makes it a requirement to register all trials and to provide the results from all trials to the public. Such legislation should make trial sponsors and researchers responsible for ensuring that:

18.2.1All clinical trials are registered, in advance of recruitment beginning, on a publicly available database.

18.2.2Full trial protocols are made publicly available free of charge and in easily accessible electronic formats, preferably before recruitment begins but certainly within 12 months following completion of the trial.

18.2.3Results for all protocol specified outcomes, with analyses based on all participants, are made publicly available free of charge and in easily accessible electronic formats within 12 months after completion of all trials.

18.2.4Anonymized, individual participant data are made available without restriction and free of charge, with appropriate safeguards to ensure ethical and scientific integrity and standards, and to protect participant privacy.

19. Research ethics committees should be responsible for ensuring that all clinical trials and their protocols that they approve are registered (as approved—not necessarily as undertaken) on a publicly available database.

20. Sponsors of trials should be responsible for ensuring that:

20.1Full trial protocols are publicly available free of charge and in an easily accessible electronic form from the beginning of the study.

20.2Sponsored research is made publicly available irrespective of study results.

21. Researchers and journal editors should be responsible for ensuring that:

21.1Any trial report submitted for consideration for publication in a journal is registered in a public trials registry, according to the “Obligation to Register Clinical Trials” statement from the International Committee of Medical Journal Editors (ICMJE).30

21.2The results for randomized controlled trials that are reported in a journal meet the relevant reporting guidance, Consolidated Standards of Reporting Trials (CONSORT), which includes items on outcomes, changes to outcomes, and numerical results.31 The ICMJE refers journal editors and researchers to the CONSORT statement in its guidance.32

21.3Members of the public should be encouraged to participate only in trials where full disclosure of the trial protocol and its results are stipulated as a condition of participation on the informed consent form.

Question 5: Can lessons about transparency and disclosure of clinical data be learned from other countries?

22. The Food and Drug Administration Amendments Act (FDAAA) of 2007 mandated that trials conducted in the USA, or conducted with the aim of pursuing an FDA new drug application, or of a drug manufactured in the USA needed to fulfil the following requirements:

22.1Registration of the trial not later than 21 days after enrolment of the first participant.

22.2Results to be submitted to within 12 months of the study completion date.

23. This Act also expanded the remit of the database to incorporate clinical trial results. A review of the extent and quality of disclosure of trials has, however, demonstrated that registration remains incomplete, and that the quality of entries and completeness of reporting of results are inconsistent.33

24. In Germany, the Institute for Quality and Efficiency in Health Care (IQWiG) abandoned its analysis of study data based on published reports relating to the antidepressant reboxetine, because it suspected that the manufacturer, Pfizer, was concealing a substantial volume of data from the trials that had been conducted.34

25. Reboxetine is currently licensed as an antidepressant by the Medicines and Healthcare products Regulatory Agency (MHRA) for use in England and Wales, and the drug has been widely prescribed worldwide. When researchers at IQWiG were finally granted access to data from unpublished studies they found that data for two-thirds of the patients were missing in the published reports. While published data suggested the drug was beneficial, the complete data set did not. In particular, reboxetine performed poorly compared to other known selective serotonin reuptake inhibitors (SSRIs). The analysis also confirmed that there were harms associated with the drug. The agency concluded that “no proof of benefit from treatment with reboxetine could be deduced” from the data available at that time, either for acute therapy, or to prevent relapse.35 The drug continues to be displayed in the British National Formulary without any comment on the findings of the IQWiG researchers.36 Data from the NHS in England in 2010 show that there have been over 50,000 prescriptions written for reboxetine at a cost of over £838,000 GBP.37

26. The European Medicines Agency (EMA) held a workshop in November 2012 to discuss access to clinical-trial data and transparency. This led to the agency establishing working groups to develop policies in five key areas: protecting patient confidentiality; clinical-trial-data formats; rules of engagement; good analysis practice; and legal aspects. Recommendations from these groups are expected in April 2013. Guido Rasi, the Executive Director of the EMA, stated that “We are not here to decide if we publish clinical-trial data, but how.”38

27. Each of the above examples indicates a growing international awareness of harm caused by incomplete disclosure of the results of clinical trials. Withholding results breaches the trust of trial participants who might reasonably assume that their participation in the research will be used to develop scientific understanding and improve the care of future patients. Also, the wider public expects policy-makers and clinicians to have access to the totality of the evidence in making decisions about health interventions. When trial data are withheld, these expectations are undermined.

Recommendations for Action

28. Many trials are international in their scope, which means that governments need to work together to ensure co-ordination of measures to improve the accessibility of trial data.

29. Government agencies should introduce legislation to ensure that:

29.1All clinical trials are registered at their inception, before recruitment of the first participant; see The Cochrane Collaboration’s statement.39

29.2Full trial protocols become publicly available free of charge and in easily accessible electronic formats before recruitment begins; and that updates are made available with changes clearly documented.

29.3Results for all protocol-specified outcomes, with analyses based on all participants, become publicly available free of charge and in easily accessible electronic formats within 12 months after completion of all trials.

29.4Anonymized, individual participant data are made available without undue restriction and free of charge; with appropriate safeguards to ensure ethical and scientific integrity and standards, and to protect participant privacy.

29.5It is a requirement for results from all trials to be made publicly available.

30. Government agencies should recognize collective responsibility for trial data, which includes sponsors, investigators, research ethics committees, trial participants, and the wider public; and to ensure that adequate mechanisms, resources, and infrastructure are provided to facilitate access to the protocols, results, and data.

February 2013



Contributor Details

Prof Lisa Bero
Director, US Cochrane Center, San Francisco Branch
Department of Clinical Pharmacy and Institute for Health Policy Studies
University of California San Francisco
Suite 420, Box 0613
3333 California Street
San Francisco California 94143–0613

Mr Martin J Burton
UK Cochrane Centre
Summertown Pavilion
18–24 Middle Way
Oxford OX2 7LG


Prof Mike J Clarke
Director, All-Ireland Hub for Trials Methodology Research
Centre for Public Health
Queens University Belfast
Institute of Clinical Sciences, Block B, Royal Victoria Hospital
Grosvenor Road
Northern Ireland BT12 6BJ

Ruth Foxlee, MSc
Information Specialist
Cochrane Editorial Unit
The Cochrane Collaboration
11–13 Cavendish Square
London W1G 0AN


Prof Peter C Gøtzsche
The Nordic Cochrane Centre
Blegdamsvej 9, 7811
DK-2100 Copenhagen

Prof Jeremy Grimshaw
Senior Scientist
Clinical Epidemiology Program
Ottawa Hospital Research Institute
The Ottawa Hospital—General Campus
501 Smyth Road, Box 711
Ottawa ON K1H 8L6


Toby J Lasserson, MSc
Senior Editor
Cochrane Editorial Unit
The Cochrane Collaboration
11–13 Cavendish Square
London W1G 0AN

Carol Lefebvre, MSc
Independent Information Consultant
Lefebvre Associates
Oxford OX5 1JY


Harriet MacLehose, PhD
Senior Editor
Cochrane Editorial Unit
The Cochrane Collaboration
11–13 Cavendish Square
London W1G 0AN

Prof Lesley Stewart
Director, Centre for Reviews and Dissemination
University of York
York YO10 5DD


Dr David Tovey
Editor in Chief, The Cochrane Library
Cochrane Editorial Unit
The Cochrane Collaboration
11–13 Cavendish Square
London W1G 0AN

L. Susan Wieland, PhD
Research Associate
Center for Integrative Medicine
University of Maryland School of Medicine
East Hall
520 W. Lombard Street
Baltimore Maryland 21201

Declarations of Interest

All contributors are members of The Cochrane Collaboration.

Ruth Foxlee, Toby Lasserson, Harriet MacLehose, and David Tovey are employees of The Cochrane Collaboration.

Martin Burton is employed by the Oxford University Hospitals NHS Trust as Director of the UK Cochrane Centre, funded by the National Institute for Health Research.

Mike Clarke is employed as Director of the All Ireland Hub for Trials Methodology Research, which includes a variety of work on the design, conduct, interpretation, and use of clinical trials. Attitudes, regulations, and legislation relating to clinical trials may have an impact on this work.

Lesley Stewart is Director of the NIHR Centre for Reviews and Dissemination (CRD) at the University of York. She is responsible for delivering programmes of work that include systematic reviews of both aggregate and individual participant data on which legislation relating to clinical trials may impact. She is Co-Editor in Chief of a journal that publishes systematic review protocols.

According to the ICMJE conflicts of interest form,40 all contributors declare, that apart from those declarations stated above: neither they nor their institutions received payment or services from a third party for any aspect of this response; (2) no financial relationships with entities that could be perceived to influence, or that give the appearance of potentially influencing, this response; and (3) no other relationships or activities that could be perceived to have influenced, or give the appearance of potentially influencing, this response.

1 A systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarize the results of the included studies.

2 The Cochrane Collaboration. Commercial sponsorship policy. [cited 21 Feb 2013]; Available from:

3 Hopewell S, Loudon K, Clarke M J, Oxman A D, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews. 2009(1):MR000006.

4 Scherer R W, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database of Systematic Reviews. 2007(2):MR000005

5 Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews. 2007(2):MR000011

6 Dwan K, Altman D G, Arnaiz J A, Bloom J, Chan A W, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3(8):e3081.

7 Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Medicine. 2008;5(11):e217; discussion e217.

8 Turner E , Matthews A M, Linardatos E, Tell R A, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine. 2008;358(3):252-60.

9 Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003;326(7400):1171-3.

10 Steinman M A, Bero LA, Chren M M, Landefeld C S. Narrative review: the promotion of gabapentin: an analysis of internal industry documents. Annals of Internal Medicine. 2006;145(4):284-93.

11 Vedula S S, Bero L, Scherer R W, Dickersin K. Outcome reporting in industry-sponsored trials of gabapentin for off-label use. New England Journal of Medicine. 2009;361(20):1963-71.

12 Nissen S E, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. New England Journal of Medicine. 2007;356(24):2457-71.

13 Dickersin K, Chan S, Chalmers T C, Sacks H S, Smith H, Jr. Publication bias and clinical trials. Controlled Clinical Trials. 1987;8(4):343-53.

14 Dwan K, Altman D G, Cresswell L, Blundell M, Gamble C L, Williamson P R. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database of Systematic Reviews. 2011(1):MR000031.

15 Jefferson T, Jones M A, Doshi P, Del Mar C B, Heneghan C J, Hama R, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Systematic Reviews. 2012;1:CD008965.

16 Jefferson T O, Demicheli V, Di Pietrantonj C, Jones M, Rivetti D. Neuraminidase inhibitors for preventing and treating influenza in healthy adults. Cochrane Database of Systematic Reviews. 2006(3):CD001265.

17 See Jefferson et al, 2012.

18 Gøtzsche P C, Jorgensen A W. Opening up data at the European Medicines Agency. BMJ. 2011;342:d2686.

19 Goldacre B. Bad Pharma: How drug companies mislead doctors and harm patients. London: Fourth Estate; 2012.

20 See Rising et al, 2008.

21 Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202.

22 See Turner et al, 2008.

23 Whittington C J, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet. 2004;363(9418):1341-5.

24 Healy D. Let them eat Prozac. New York: New York University Press; 2004.

25 Chan A W, Hrobjartsson A, Haahr M T, Gøtzsche P C, Altman D G. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457-65.

26 United States Department of Justice. Drug Maker Forest Pleads Guilty; To Pay More Than $313 Million to Resolve Criminal Charges and False Claims Act Allegations. Wednesday, 15 September 2010. 2010 [cited 15 Feb 2013]; Available from:

27 United States Department of Justice. GlaxoSmithKline to Plead Guilty and Pay $3 Billion to Resolve Fraud Allegations and Failure to Report Safety Data. Monday, 2 July 2012. 2012 [cited 15 Feb 2013]; Available from:

28 Imdad A, Herzer K, Mayo-Wilson E, Yakoob M Y, Bhutta Z A. Vitamin A supplementation for preventing morbidity and mortality in children from 6 months to 5 years of age. Cochrane Database of Systematic Reviews. 2010(12):CD008524.

29 Clinical Trial Service Unit and Epidemiological Studies Unit (CSTU). DEVTA (1999 - 2004; first results 2007). 2013 [cited 18 Feb 2013]; Available from:

30 International Committee of Medical Journal Editors (ICMJE). Uniform requirements for manuscripts submitted to biomedical journals: publishing and editorial issues related to publication in biomedical journals: obligation to register clinical trials. 2013 [cited 18 Feb 2013]; Available from:

31 CONSORT. Consolidated Standards of Reporting Trials. 2013 [cited 18 Feb 2013]; Available from:

32 International Committee of Medical Journal Editors (ICMJE). Uniform requirements for manuscripts submitted to biomedical journals: writing and editing for biomedical publication. 2007 [cited 18 Feb 2013]; Available from:

33 Zarin D A, Tse T, Williams R J, Califf R M, Ide N C. The results database—update and key issues. New England Journal of Medicine. 2011;364(9):852-60.

34 Institute for Quality and Efficiency in Health Care (IQWiG). Pfizer conceals study data. 2009 [cited 15 Feb 2013]; Available from:

35 See IQWiG 2009.

36 Joint Formulary Committee. 4.3.4 Other antidepressant drugs. Reboxetine. British National Formulary (online). London: BMJ Group and Pharmaceutical Press; 2013.

37 National Health Service (NHS). Prescription Cost Analysis—England, 2010. 2010 [cited 18 Feb 2013]; Available from:

38 European Medicines Agency (EMA). Workshop on access to clinical-trial data and transparency kicks off process towards proactive publication of data. Press release. 2012 [cited 18 Feb 2013]; Available from:

39 The Cochrane Collaboration. The Cochrane Collaboration Supports Prospective Registration of Clinical Trials. 2004 [cited 19 Feb 2013]; Available from:

40 International Committee of Medical Journal Editors (ICMJE). ICMJE uniform disclosure form for potential conflicts of interest. 2013 [cited 20 Feb 2013]; Available from:

Prepared 16th September 2013