Science and Technology CommitteeWritten evidence submitted by the Cochrane NI Review Group

We are responding to the third and fourth questions.

3. What evidence is there that pharmaceutical companies withhold clinical trial data and what impact does this have on public health?

1. We started working on a Cochrane review of neuraminidase inhibitors in 1998. Cochrane reviews are studies summing up what is known of the effects of an intervention in healthcare. In this case the “intervention” was the class of drugs called neuraminidase inhibitors or NIs. NIs are supposed to alleviate influenza, either by shortening duration of illness and diminishing dangerous complications such as pneumonia or by acting prophylactically (prevention) in contacts of people with influenza or on whole populations exposed to the threat. At the time NIs comprised two anti-influenza compounds: zanamivir (Relenza, GW now GSK) and oseltamivir (Tamiflu, Roche).1,2

2. The Cochrane Collaboration (CC) is an international network of volunteer scientists who carry out reviews of evidence on interventions in health care according to highly structured and reproducible methods. Cochrane reviews are considered as the gold standard in evidence-based decision making for interventions (

3. Cochrane reviews are widely cited by governments and health departments worldwide. The CC receives considerable funding from the UK Department of Health, and is completely independent from pharmaceutical companies or other potentially conflicting influences. The Cochrane group conducting the review of NIs consists of researchers at the University of Oxford, University of Queensland, Bond University, Johns Hopkins University, and independent scientists in Osaka and Rome. Comments by any reader can be posted on the Cochrane protocol or the full review at any time. The comments are taken seriously, as this account shows.

4. In 2009, our review was in its third update,3 the world was in the middle of an influenza pandemic (or so WHO was telling us) and we received a letter from a Japanese paediatrician, Dr Keiji Hayashi. Dr Hayashi wanted to know how it was possible that in our 2005 update we had included eight unpublished Tamiflu trials contained in extreme summary form within another review funded by Roche and carried out by Roche staff and consultants. How could we possibly have done that as we had not seen the original studies? We asked the two Roche consultants for the data. They told us to go and ask Roche. We did. Roche asked us to sign a confidentiality agreement with a secrecy clause. We declined. We cannot publish a Cochrane review subjected to secret and restrictive clauses. This would compromise independence, transparency and reproducibility of the review.4

Once the British Medical Journal (BMJ) got involved with Channel 4 News5,6,7 and brought media pressure to bear, Roche publicly promised full study reports.8 However, Roche gave us only the first of the 4–5 parts of the 10 trials. They told us that was all we needed for our review. We now know that this is not the case. By reading the Roche material and some NICE documents leaked to the BMJ, we discovered that the reports that a pharmaceutical company produces for each drug trial that it conducts (called clinical study reports) are massively complex documents, containing hundreds or thousands of pages of information with minute details about trials, their planning and execution. This represents a major shift in the level of detail that Cochrane reviewers such as ourselves have been used to, since in the past we have relied on journal articles that are typically only a few pages in length.9

5. We discovered many more Tamiflu trials. The list has grown from the 26 we had originally identified to 123—the vast majority Roche sponsored. We asked for all the Roche completed clinical study reports so we could assess them for our review. In the three years since this request, we have had a lengthy negotiation with Roche regarding the release of these trial details, documented publically in and To date we have had no success in obtaining the trial data from Roche. Without these trial data it is not possible to conduct a review of the evidence for this drug.

6. At the end of 2010 the European Regulator EMA accepted a ruling by the European Ombudsman that trial data for drugs on which a regulatory decision had been made should be accessible. They opened their archives. We received incomplete reports for 16 Tamiflu trials, all they had. Because of timing constraints, our 2012 update of our Cochrane review is based on over half of the evidence provided by EMA and approximately 2000 pages of FDA comments on Tamiflu10. We are in the process of reviewing the other half of the Roche Tamiflu trial data.

7. One consequence of our access to this bonanza of regulatory material has been a comparison between the details and broad message of the few published trials and their regulatory much more detailed reports. There are discrepancies in reporting harms and some important aspects of study design between publications and regulatory reports. We also think that the drug interferes with natural antibody production.10 If confirmed, this finding would suggest that use of Tamiflu weakens natural host defences and may weaken response to any antigen stimulating interventions such as vaccines.

8. On the basis of regulatory evidence released from EMA and FDA we have also found that the positive effects of the drug are not as marked as those claimed by the manufacturer and its consultants in industry-sponsored publications. Like FDA, we found the effect of Tamiflu on influenza complications (eg pneumonia) and person-to-person transmission unproven.10

9. As these effects were at the basis of the scientific rationale for stockpiling Tamiflu, we wonder whether access to all trial data would have avoided stockpiling at huge public expense. But we do not know for sure because we do not have all the data.

10. The practical result of all this is our refusal to consider published trials (either on their own or as part of reviews) for inclusion in our Cochrane reviews. There is growing international concern regarding the limitations of relying solely on the very short versions of drug trials.11,12,13 So now we have asked EMA to do a more thorough job by requesting the remainder of the missing sections of the trials they originally looked at and all the other missing trials. The idea is that EMA asks Roche for the data and then has to release it following its new policy. This is documented at

11. Meanwhile what started as a comment from a Japanese colleague has turned into a global campaign for access to data on trials.12,13,14 You can read about that here The BMJ set up a Tamiflu micro site on with our correspondence with Roche, WHO and CDC The latter two continue to recommend the use of Tamiflu, seemingly disregarding the lack of evidence for their effects that we and others have documented. They also refuse to answer our questions (see and

12. As independent medical scientists we are deeply disturbed that despite serious concerns by ourselves and many other independent scientists in this field regarding the effectiveness of Tamiflu and the secrecy surrounding its trials, it appears that recommendations for the use of this drug continue to be at odds with what trial evidence shows. The financial consequences of these recommendations are ongoing, notwithstanding the costs of stockpiling this drug during the swine flu outbreak in 2009, are considerable for the NHS at a time of tight financial constraints. Most of all, no one seems to know exactly how much has been spent on a drug for which no one (apart possibly the manufacturer) has seen and analysed the full data set. We can say this, as we have analysed documents, reviews and Health Technology Assessment reports produced by WHO, CDC and NICE. These are based on journal articles reports of published trials and in some cases on additional data furnished by the manufacturer on an ad hoc basis. We assume these data were subject to the same controls Roche tried to impose on us.

13. We have also engaged GSK, asking them for clinical study reports and individual participant level data for their drug Relenza. Some of the press coverage recently suggested that GSK after its record fine in the US courts of justice for fraud ( would open its archives to researchers. GSK has told the world that requests for data would be handled with the intermediary of an independent committee scrutinizing the worthiness of the analysis plans in the application. Despite the plaudits, our group is yet to receive any data from GSK and we remain unconvinced, as we want reproducibility of our Cochrane analyses. We recognize that we do not hold a monopoly on truth.

14. Obstacles and conditions (such as exclusivity, secrecy or contractual bans on sharing) attached to data release make reproducibility of results harder or impossible because they constrain our ability to share the data underlying our analysis with third parties seeking to reproduce or verify our analysis.

15. We are disappointed that the DH has not intervened to require Roche to make the missing trial data publicly available after the amount of tax payers’ money that was spent on it. We also find it hard to comprehend how the CMO and NICE have not been held accountable for their decisions.

16. Our attempts to independently assess the evidence for the effectiveness of the NI drugs highlights three major problems that we believe are generalisable to other drugs.

17. First, there is clear evidence that full reports of trials are not available for public scrutiny, even by well known researchers such as our Cochrane group. This means that vital evidence for the safety and effectiveness of Tamiflu is simply not available. This does not allow individual clinicians to make rational decisions for their patients. The second problem is that our reading of EMA, Japan’s PMDA and the US FDA’s reports (see our Cochrane review) suggests that their scrutiny does not encompass the full dataset but a pre-agreed selected subset of toxicology and pharmacodynamics studies plus a few (usually two) trials per indication. In the United States, these trials are dubbed “pivotal”. Although the US FDA appeared to have done a much more thorough job than EMA with data re-analysis and trial site visits, the regulatory perspective is different from ours. Our job as Cochrane researchers is to look at all the relevant evidence, not judge whether a product is worthy of a market authorization or license on the basis of a pre-arranged set of studies.

18. The third problem is decision-makers who make policy on the basis of short journal publications and expert advice and are unwilling to revise their policies when new evidence emerges that journal publications and expert advice have mislead. In the case of Tamiflu this may have had damaging effects to public and clinician confidence in the rigour of how medications are assessed in the UK for safety and effectiveness, and risk that NHS funds have not been used for interventions which offer the best value for money. The individuals responsible for making these decisions do not appear to be accountable, even when evidence shows that their decisions may not stand up to scrutiny.

This is how three front-line public health physicians in the Midlands see the effect of current anti-viral policies on their role:

“Many of us in PCTs considered our role to have been transformed from front line public health to that of an NHS delivery system for the pharmaceutical industry.”


19. Trials are experiments conducted on human beings. Full reporting of their results (anonymized to prevent individuals being identified) should be a right, not a gift. It is ethically wrong not to make their results public. Your doctor should be in possession of all the facts or be able to access a source that does. Think about that next time he prescribes something for you.

4. How could the occurrence and results of clinical trials be made more open to scrutiny? Who should be responsible?

1. All past, current and future trials should be registered in one of the trial registers as soon as their protocol is finalized. For old trials retrospective registration should be allowed. Registration provides consumers and/or taxpayers with a complete overview of what is going on. On its own however, registration is not enough because there is evidence of reporting bias of register entries, including failure to update them.

2. Clinical study reports should be made available with minimal redactions in PDF format in a central website. This should be run by a publicly funded body and governed autonomously. Only this type of availability should be considered “publication” ie making public.

3. The Committee should not consider “publication” to mean either journal articles of a few pages’ length, or similar-length summaries posted on sponsors’ websites. In both cases the potential for introducing reporting biases and consequent distortions of the evidence has been shown to be very high.

4. Sponsors’ failure to publish and maintain the trials entries should be considered unethical. The medical director of the sponsor and principal investigators should be routinely reported to the GMC.

5. The relevant individual participant level data should be made available in anonymized form from the central resource. The body should require a reason for the request and apply a level of scrutiny to requests which deters frivolous requests.

6. The costs of regulating the new system would be offset by preventing the use of drugs for which there is no or little evidence of effectiveness or that cause harm.

Cochrane Neuraminidase Review Group

Tom Jefferson MD, The Cochrane Collaboration, Roma, Italy

Peter Doshi PhD, Division of General Internal Medicine, Johns Hopkins University, Baltimore, Maryland, USA

Matthew Thompson MD, Department of Primary Care Health Sciences, University of Oxford, Oxford, UK

Mark Jones PhD, University of Queensland School of Population Health, Brisbane, Australia

Rokuro Hama MD, Japan Institute of Pharmacovigilance, Osaka, Japan

Chris Del Mar, Centre for Research in Evidence Based Practice, Bond University, Gold Coast, Australia

Disclosure Statement

All authors have applied for and received competitive research grants. All authors are co-recipients of a UK National Institute for Health Research grant to carry out a Cochrane review of neuraminidase inhibitors ( which used as its basis more than 25,000 pages of Clinical Study Reports for oseltamivir

In addition:

Tom Jefferson was an ad hoc consultant for F. Hoffman-La Roche Ltd in 1998–99. He receives royalties from his books published by Blackwells and Il Pensiero Scientifico Editore (Italy), none of which are on Clinical Study Reports. He is occasionally interviewed by market research companies for anonymous interviews about Phase 1 or 2 products. In 2011–12 Tom acted as an expert witness in a US litigation case related to Tamiflu.

Peter Doshi received €1,500 from the European Respiratory Society in support of his travel to the society’s September 2012 annual congress where he gave an invited talk on Tamiflu. He is funded by an institutional training grant from the Agency for Healthcare Research and Quality (AHRQ) #T32HS019488. AHRQ had no role in study design, data collection and analysis, decision to publish, or preparation of the submission.

Matthew Thompson received payment for running educational courses at the University of Oxford and University of Oxford ISIS consulting services for external teaching and training.

Mark Jones has no conflicts of interest to declare.

Rokuro Hama has written the following books:

Published in January 2008: “Tamiflu: harmful as feared” (Kin-yobi Publishing Co). Royalties were split between his institution and the Tamiflu sufferers group 7%–1%.

Published in November 2008: “In order to escape from drug-induced encephalopathy”. NPOJIP(Kusuri-no-Check). Royalties to his institution.

Dr Hama provided scientific opinions and expert testimony on:

11 adverse reaction cases related to oseltamivir where applications were made by their families for adverse reaction relief by PMDA (Pharmaceuticals and Medical Devices Agency). This is reported in: IJRSM 2008:20:5–36. Two cases were paid in May 2005 and others were not.

A law suit on the fatal adverse reactions to gefitinib against AstraZeneca and the Japanese Minister of Health Labor and Welfare. Dr Hama argued that gefitinib’s fatal toxicity was known before approval in Japan as shown in “Gefitinib story”: and in other articles: Paid by the plaintiff’s lawyers.

Chris Del Mar provided expert advice to GlaxoSmithKline about vaccination against acute otitis media in 2008–09. He receives royalties from books published through Blackwell BMJ Books and Elsevier.

Chris Del Mar and Tom Jefferson have recently updated their Cochrane review on physical interventions to prevent the spread of acute respiratory infections with World Health Organization (WHO) funds.

January 2013

Main Related Publications (Items Marked * are Attached)

1. Jefferson T, Demicheli V, Rivetti D, Jones M, Di Pietrantonj C, Rivetti A. Antivirals for influenza in healthy adults: systematic review. Lancet 2006;367:303–13*

2. Jefferson T, Demicheli V, Di Pietrantonj C, Jones M, Rivetti D. Neuraminidase inhibitors for preventing and treating influenza in healthy adults. Cochrane Database of Systematic Reviews 2006;(3):CD001265

3. Jefferson T, Jones M, Doshi P, Del Mar C. Neuraminidase inhibitors for preventing and treating influenza in healthy adults: systematic review and meta-analysis. BMJ 2009;339:b5106*

4. Doshi P. Neuraminidase inhibitors: the story behind the Cochrane review. BMJ 2009;339:b5164*

5. Cohen D. Complications: tracking down the data on oseltamivir. BMJ 2009;339:b5387

6. Godlee F, Clarke M. Why don’t we have all the evidence on oseltamivir? BMJ 2009;339:b5351

7. Editor’s Choice: We want raw data, now. BMJ 2009; 339 doi: 10.1136/bmj.b5405

8. Smith J, on behalf of Roche. Point-by-point response from Roche to BMJ questions. BMJ 2009;339:b5374*

9. Doshi P, Jones M A, Jefferson T. Rethinking credible evidence synthesis. BMJ 2012;344:d7898 doi: 10.1136/bmj.d7898*

10. Jefferson T, Jones M A, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Systematic Reviews 2012, Issue 1. Art. No.: CD008965. DOI: 10.1002/14651858.CD008965.pub3*

11. Drug Data Shouldn’t Be Secret by Peter Doshi and Tom Jefferson
The New York Times, 10 April 2012. URL: Shortened URL:

12. Doshi P, Jefferson T, Del Mar C (2012). The Imperative to Share Clinical Study Reports: Recommendations from the Tamiflu Experience. PLoS Med 9(4): e1001201. doi:10.1371/journal.pmed.1001201
Short URL: PDF for printing:

13. Godlee F. Clinical trial data for all drugs in current use. BMJ 2012;345:e7304 doi: 10.1136/bmj.e7304 (Published 29 October 2012)

14. Payne D. Tamiflu: the battle for secret drug data. BMJ 2012;345:e7303 doi: 10.1136/bmj.e7303 (Published 29 October 2012)

15. Godlee F. Open letter to Roche about oseltamivir trial data. BMJ 2012;345:e7305 doi: 10.1136/bmj.e7305 (Published 29 October 2012).

Prepared 16th September 2013