Select Committee on Health Appendices to the Minutes of Evidence


Memorandum by Medical Research Council Clinical Trials Unit (NC 119)


  The validity of any clinical guideline or Health Technology Assessment (HTA) is dependent on the quality and completeness of the evidence upon which it is based. We are concerned that the requirement for rapidity may seriously compromise the quality of some HTAs. We consider it regrettable that, currently, it is not mandatory for industry submissions to disclose all relevant information, and therefore such submissions may be incomplete and unrepresentative. We are also concerned that plans to carry out NICE assessments at earlier stages may be counter-productive. The inevitable consequence will be to shift the basis of the assessment on to trials undertaken for licensing purposes, rather than those focussing on long-term risks and benefits. This would jeopardise both the quality of the review and the conduct of the primary clinical research upon which adequate assessment depends.


  The scientific staff of the Medical Research Council, Clinical Trials Unit have been carrying out systematic reviews since 1988 and have an established track record. We have published widely on both the results of the systematic reviews we have undertaken and on the methodology of systematic review and meta-analysis. We have also been actively involved in the work of the Cochrane Collaboration since 1993.

  We strongly support the rationale that the delivery of healthcare should be guided by good evidence and that there is a need for national guidance to be applied consistently across the UK. We endorse the principles of the National Institute of Clinical Excellence (NICE) that recommendations are based on the best available evidence and that all stakeholders are involved in a transparent and collaborative manner. However, we believe that there are some issues relating to methodology and quality that require consideration by the Select Committee.

  The statements made here reflect the opinions of the undersigned individuals and do not necessarily reflect the policy of the Medical Research Council.


  The critical foundation upon which NICE recommendations should be based is reliable information on whether an intervention is beneficial or harmful, together with the size of that benefit or harm. If this basic information is unclear or incorrect, then any further analysis or modelling that follow (however sophisticated) will be similarly incorrect or unjustified, and decisions based on those analyses are likely to be flawed. For example, we suggest that it may be seriously misleading to estimate cost effectiveness or relative toxicity in the absence of a reliable estimate of effectiveness. Estimation of treatment effect is best established through well-designed and carefully conducted systematic reviews. In most circumstances these will be based on the results of randomised controlled trials (RCTs). The conclusions of any systematic review will depend upon exactly which data are considered and it is therefore essential that the information assessed be unbiased and representative of the evidence base. There is empirical evidence that many types of bias (including publication, language, selective reporting and patient exclusion bias) can influence the results of systematic reviews and lead to unjustified conclusions. For example, there is a considerable body of research evidence demonstrating that trials with results that favour a new treatment are more likely to be published in the medical literature than trials that show the standard treatment to be better or are inconclusive (publication bias). Published reports may focus on positive outcomes (selective reporting bias) and do not always provide sufficient information needed for their results to be included in a quantitative analysis. Consequently, reviews based only on the data that can be extracted from published reports are likely to be based on only a proportion of the relevant evidence and subject to a number of biases, giving misleading estimates of treatment effect and conclusions that could potentially lead to inappropriate patient care.


  At the 2001 conference on Clinical Excellence, NICE staff suggested that full systematic reviews as done by the Cochrane Collaboration were not necessarily essential to future HTAs, nor always going to be possible given the requirement for rapid assessment. This is despite the fact that there is evidence that Cochrane reviews are of higher quality than non Cochrane systematic reviews, even although not all Cochrane reviews necessarily go beyond the published data. Given the possible profound influence of NICE HTAs on the Health Service, this implied shift towards potentially inferior reviews should be avoided. We appreciate the urgency of producing guidance, especially in situations relating to new drugs in previously untreatable diseases. However, if treatments enter practice based on a hasty review of incomplete evidence, it is likely to be difficult to withdraw them from practice if the treatment is subsequently shown ineffective or no better than standard therapies. In that case the appropriate action would be to support prescription in the context of a well-designed RCT (see also 2.0, 3.0, 4.0).

  Proposed Action:

    —  A document detailing the required standards and acceptable methods for HTA reviews should be prepared and made available for public scrutiny.

    —  Assurance should be sought that the systematic review elements of HTAs should be done to at least the standards of the Cochrane Collaboration, and that quality should not be compromised by the need for speed (within reason).


  Where companies submit data for consideration by NICE, it is of the utmost importance that information about all potentially relevant RCTs is disclosed. Potentially, companies could submit details only of those trials showing favourable results for their products and withhold data from those with less encouraging results.

  Proposed Action:

    —  Procedures should be implemented to ensure that this cannot happen and that information about all relevant trials worldwide (including those that have been discontinued, or those that may be perceived as not have the required data available) should be disclosed in any submission to NICE.


  Analyses done as part of the HTA should be formulated independently rather than being driven by the original trial analyses and should be detailed prospectively in the HTA systematic review protocol. (This is not always done at present). It is important that reviewers do not simply repeat the original analyses but address the underlying question of the HTA in an objective way. For example, if the number of participants in trials is insufficient for subgroup analysis then this should be stated and such analyses not included in the review.

  Proposed Action:

    —  Analyses should be planned prospectively and detailed proposals for these should be routinely included in HTA review protocols.


  Sometimes new treatments appear "good" because they have been compared with inappropriate or sub-optimal "standard" treatments.

  Proposed Action:

    —  Submissions made to NICE should include justification of the choice of comparator, including a description of the evidence-base upon which the choice was made.


  We question the need for submissions to remain confidential. Where elements of an HTA report are labelled as confidential (especially where there is blanket confidentiality on all aspects of a trial), it is impossible for those outside the review process to comment on its validity. This undermines the NICE stated objective of transparency. While we appreciate the issues of commercial sensitivity pre-licensing, most NICE submissions will be post-licensing and likely to be made as a route to facilitate purchase of a product by the public purse. Surely then, the evidence put forward should be available to public scrutiny.

  Proposed Action:

    —  Submissions should not be allowed to remain confidential (except in exceptional circumstances)


  Selection of topics for consideration in a HTA should be driven not only by the political/social need for an answer to a healthcare question, but also by whether there is in fact sufficient evidence available to give a reliable answer to the question posed. If it is known at the outset that there are only a very limited number of RCTs and/or patients available, and consequently that there are insufficient numbers and insufficient statistical power to obtain a reliable answer to the question posed, then further research should be encouraged before the assessment is undertaken. There have been recent examples of HTAs reviewing one or two trials with a total of less than 500 patients, when at least 1000 patients would be required to establish reliably whether the investigational drug is better or worse than its comparators. From the outset it is known that such a review will provide little, if any, additional information and that the review will be inconclusive.

  Proposed Action:

    —  As part of the NICE scoping process, explicit procedures to assess at an early stage whether a proposed review is reasonably likely to reach a reliable answer based on the amount of available evidence should be implemented. A judgement should then be made as to whether it is appropriate to carry out a full HTA at that point in time.

    —  If the question is of the utmost urgency, for example high-profile new drugs in previously untreatable diseases, the recommendation should be for interim prescribing in the context of a well-designed RCT (see 1.1) rather than a hasty review.


  Where reviews have been done and there is insufficient evidence to judge whether or not an investigated treatment is better or worse than standard care, a strong statement should be made to this effect. New treatments should not enter clinical practice when it is unclear whether they are more effective than standard therapy (unless there are overwhelming additional reasons for adopting the new treatment). It is important to realise that with further evidence these treatments might not be shown to be more effective than the standard, but could have greater risks and/or lesser benefits.

  Proposed Action:

    —  Reviews, HTA reports and NICE guidelines should state very clearly when there is insufficient research evidence available to determine whether an investigated intervention is better or worse than its comparators.

    —  NICE should consider whether it would be useful to adopt standard phrases to report this fact and that such phrases should be included in any product literature that refers to the NICE evaluation. This may help avoid any potential confusion in marketing literature, whereby readers could interpret NICE appraised as meaning NICE endorsed.


  The points discussed in 2.0 and 3.0 are particularly pertinent to proposals made at the recent Clinical Excellence conference that NICE should review interventions at an earlier stage. In considering such a move, the agency should be aware of the potential effect on primary research. For example, if an intervention is evaluated an early stage where only a limited amount of research evidence is available from completed RCTs, then this is likely to come from short-term results of trials, perhaps the minimal required for licensing. Even a very bland statement that the intervention could be considered as one of a range of treatment options could seriously jeopardise other planned or ongoing trials, which might use more appropriate comparators and examine long-term risks and benefits.

  Proposed Action:

    —  We propose that in such circumstances part of NICE responsibilities should be to explore what trials are ongoing and to actively encourage participation in high quality trials that may provide sufficient additional information to reach a definitive answer.

    —  If NICE does move towards early assessment, potential overlap and the interface with the Medicines Control Agency and the Committee on Safety of Medicines need to be clarified.

January 2002

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2002
Prepared 8 July 2002