APPENDIX 26
Memorandum by Medical Research Council
Clinical Trials Unit (NC 119)
SUMMARY
The validity of any clinical guideline or Health
Technology Assessment (HTA) is dependent on the quality and completeness
of the evidence upon which it is based. We are concerned that
the requirement for rapidity may seriously compromise the quality
of some HTAs. We consider it regrettable that, currently, it is
not mandatory for industry submissions to disclose all relevant
information, and therefore such submissions may be incomplete
and unrepresentative. We are also concerned that plans to carry
out NICE assessments at earlier stages may be counter-productive.
The inevitable consequence will be to shift the basis of the assessment
on to trials undertaken for licensing purposes, rather than those
focussing on long-term risks and benefits. This would jeopardise
both the quality of the review and the conduct of the primary
clinical research upon which adequate assessment depends.
INTRODUCTION
The scientific staff of the Medical Research
Council, Clinical Trials Unit have been carrying out systematic
reviews since 1988 and have an established track record. We have
published widely on both the results of the systematic reviews
we have undertaken and on the methodology of systematic review
and meta-analysis. We have also been actively involved in the
work of the Cochrane Collaboration since 1993.
We strongly support the rationale that the delivery
of healthcare should be guided by good evidence and that there
is a need for national guidance to be applied consistently across
the UK. We endorse the principles of the National Institute of
Clinical Excellence (NICE) that recommendations are based on the
best available evidence and that all stakeholders are involved
in a transparent and collaborative manner. However, we believe
that there are some issues relating to methodology and quality
that require consideration by the Select Committee.
The statements made here reflect the opinions
of the undersigned individuals and do not necessarily reflect
the policy of the Medical Research Council.
1.0 QUALITY OF
SYSTEMATIC REVIEW
IN HTA
The critical foundation upon which NICE recommendations
should be based is reliable information on whether an intervention
is beneficial or harmful, together with the size of that benefit
or harm. If this basic information is unclear or incorrect, then
any further analysis or modelling that follow (however sophisticated)
will be similarly incorrect or unjustified, and decisions based
on those analyses are likely to be flawed. For example, we suggest
that it may be seriously misleading to estimate cost effectiveness
or relative toxicity in the absence of a reliable estimate of
effectiveness. Estimation of treatment effect is best established
through well-designed and carefully conducted systematic reviews.
In most circumstances these will be based on the results of randomised
controlled trials (RCTs). The conclusions of any systematic review
will depend upon exactly which data are considered and it is therefore
essential that the information assessed be unbiased and representative
of the evidence base. There is empirical evidence that many types
of bias (including publication, language, selective reporting
and patient exclusion bias) can influence the results of systematic
reviews and lead to unjustified conclusions. For example, there
is a considerable body of research evidence demonstrating that
trials with results that favour a new treatment are more likely
to be published in the medical literature than trials that show
the standard treatment to be better or are inconclusive (publication
bias). Published reports may focus on positive outcomes (selective
reporting bias) and do not always provide sufficient information
needed for their results to be included in a quantitative analysis.
Consequently, reviews based only on the data that can be extracted
from published reports are likely to be based on only a proportion
of the relevant evidence and subject to a number of biases, giving
misleading estimates of treatment effect and conclusions that
could potentially lead to inappropriate patient care.
1.1 GENERAL APPROACH
At the 2001 conference on Clinical Excellence,
NICE staff suggested that full systematic reviews as done by the
Cochrane Collaboration were not necessarily essential to future
HTAs, nor always going to be possible given the requirement for
rapid assessment. This is despite the fact that there is evidence
that Cochrane reviews are of higher quality than non Cochrane
systematic reviews, even although not all Cochrane reviews necessarily
go beyond the published data. Given the possible profound influence
of NICE HTAs on the Health Service, this implied shift towards
potentially inferior reviews should be avoided. We appreciate
the urgency of producing guidance, especially in situations relating
to new drugs in previously untreatable diseases. However, if treatments
enter practice based on a hasty review of incomplete evidence,
it is likely to be difficult to withdraw them from practice if
the treatment is subsequently shown ineffective or no better than
standard therapies. In that case the appropriate action would
be to support prescription in the context of a well-designed RCT
(see also 2.0, 3.0, 4.0).
Proposed Action:
— A document detailing the required
standards and acceptable methods for HTA reviews should be prepared
and made available for public scrutiny.
— Assurance should be sought that the
systematic review elements of HTAs should be done to at least
the standards of the Cochrane Collaboration, and that quality
should not be compromised by the need for speed (within reason).
1.2 NEED FOR
COMPLETE INFORMATION
IN INDUSTRY
SUBMISSIONS
Where companies submit data for consideration
by NICE, it is of the utmost importance that information about
all potentially relevant RCTs is disclosed. Potentially, companies
could submit details only of those trials showing favourable results
for their products and withhold data from those with less encouraging
results.
Proposed Action:
— Procedures should be implemented
to ensure that this cannot happen and that information about all
relevant trials worldwide (including those that have been discontinued,
or those that may be perceived as not have the required data available)
should be disclosed in any submission to NICE.
1.3 ENSURING
APPROPRIATE ANALYSES
Analyses done as part of the HTA should be formulated
independently rather than being driven by the original trial analyses
and should be detailed prospectively in the HTA systematic review
protocol. (This is not always done at present). It is important
that reviewers do not simply repeat the original analyses but
address the underlying question of the HTA in an objective way.
For example, if the number of participants in trials is insufficient
for subgroup analysis then this should be stated and such analyses
not included in the review.
Proposed Action:
— Analyses should be planned prospectively
and detailed proposals for these should be routinely included
in HTA review protocols.
1.4 APPROPRIATE
COMPARATORS
Sometimes new treatments appear "good"
because they have been compared with inappropriate or sub-optimal
"standard" treatments.
Proposed Action:
— Submissions made to NICE should include
justification of the choice of comparator, including a description
of the evidence-base upon which the choice was made.
1.5 CONFIDENTIAL
SUBMISSIONS
We question the need for submissions to remain
confidential. Where elements of an HTA report are labelled as
confidential (especially where there is blanket confidentiality
on all aspects of a trial), it is impossible for those outside
the review process to comment on its validity. This undermines
the NICE stated objective of transparency. While we appreciate
the issues of commercial sensitivity pre-licensing, most NICE
submissions will be post-licensing and likely to be made as a
route to facilitate purchase of a product by the public purse.
Surely then, the evidence put forward should be available to public
scrutiny.
Proposed Action:
— Submissions should not be allowed
to remain confidential (except in exceptional circumstances)
2.0 SELECTION
OF TOPICS
Selection of topics for consideration in a HTA
should be driven not only by the political/social need for an
answer to a healthcare question, but also by whether there is
in fact sufficient evidence available to give a reliable answer
to the question posed. If it is known at the outset that there
are only a very limited number of RCTs and/or patients available,
and consequently that there are insufficient numbers and insufficient
statistical power to obtain a reliable answer to the question
posed, then further research should be encouraged before the assessment
is undertaken. There have been recent examples of HTAs reviewing
one or two trials with a total of less than 500 patients, when
at least 1000 patients would be required to establish reliably
whether the investigational drug is better or worse than its comparators.
From the outset it is known that such a review will provide little,
if any, additional information and that the review will be inconclusive.
Proposed Action:
— As part of the NICE scoping process,
explicit procedures to assess at an early stage whether a proposed
review is reasonably likely to reach a reliable answer based on
the amount of available evidence should be implemented. A judgement
should then be made as to whether it is appropriate to carry out
a full HTA at that point in time.
— If the question is of the utmost
urgency, for example high-profile new drugs in previously untreatable
diseases, the recommendation should be for interim prescribing
in the context of a well-designed RCT (see 1.1) rather than a
hasty review.
3.0 INCONCLUSIVE
REVIEWS
Where reviews have been done and there is insufficient
evidence to judge whether or not an investigated treatment is
better or worse than standard care, a strong statement should
be made to this effect. New treatments should not enter clinical
practice when it is unclear whether they are more effective than
standard therapy (unless there are overwhelming additional reasons
for adopting the new treatment). It is important to realise that
with further evidence these treatments might not be shown to be
more effective than the standard, but could have greater risks
and/or lesser benefits.
Proposed Action:
— Reviews, HTA reports and NICE guidelines
should state very clearly when there is insufficient research
evidence available to determine whether an investigated intervention
is better or worse than its comparators.
— NICE should consider whether it would
be useful to adopt standard phrases to report this fact and that
such phrases should be included in any product literature that
refers to the NICE evaluation. This may help avoid any potential
confusion in marketing literature, whereby readers could interpret
NICE appraised as meaning NICE endorsed.
4.0 TIMING OF
TOPICS
The points discussed in 2.0 and 3.0 are particularly
pertinent to proposals made at the recent Clinical Excellence
conference that NICE should review interventions at an earlier
stage. In considering such a move, the agency should be aware
of the potential effect on primary research. For example, if an
intervention is evaluated an early stage where only a limited
amount of research evidence is available from completed RCTs,
then this is likely to come from short-term results of trials,
perhaps the minimal required for licensing. Even a very bland
statement that the intervention could be considered as one of
a range of treatment options could seriously jeopardise other
planned or ongoing trials, which might use more appropriate comparators
and examine long-term risks and benefits.
Proposed Action:
— We propose that in such circumstances
part of NICE responsibilities should be to explore what trials
are ongoing and to actively encourage participation in high quality
trials that may provide sufficient additional information to reach
a definitive answer.
— If NICE does move towards early assessment,
potential overlap and the interface with the Medicines Control
Agency and the Committee on Safety of Medicines need to be clarified.
January 2002
| 
