Select Committee on Health Second Report


 "national guidance will mean that interventions with good evidence of clinical and

 cost-effectiveness will be actively promoted, so that patients have faster access to

 treatments known to work."

 [A First Class Service, Department of Health, 1998, 2.17.]

Faster access for patients?

82. NICE pointed out to us that it could not, on its own, "provide faster access to treatments", and that guidance clearly needs to be properly implemented before patients would feel its benefit. According to NICE, "it has been independently argued that, generally, where NICE recommends the use of a technology it will lead to faster and more uniform access to these technologies".[105] However, some witnesses contended that because of 'NICE blight' the existence of NICE has in fact considerably slowed access, with drugs that might previously have been available being vetoed pending a recommendation from NICE. It is to be hoped that the recent government directives will put an end to this practice, and that our recommendation that NICE move towards assessing new treatments at the time of launch will enable NICE to secure faster access for patients in future.

The issue of evidence and technical quality

83. For NICE's stakeholders the importance of ensuring that its work is carried out to the highest possible standard is clear: "For the final guidance (HTA [health technology appraisal] or [clinical] guidelines) to be credible, whether with the public, patients, or the NHS, the process of assessment or development needs to be highly robust, transparent, and if at all possible, above criticism".[106] This view was endorsed by Professor Maynard, who asserted that "it is essential that the science supporting NICE and other regulatory bodies be of the highest quality. Poor science and bad management can discredit the whole process".[107] While we are not expert in the methodology for assessing either clinical or cost-effectiveness, there are some generic issues relating to NICE's processes which have become clear during the course of our inquiry. Whether or not the treatments or interventions NICE promotes can be said to be supported by "good evidence of clinical and cost-effectiveness" is essentially dependent on three key factors: the quality of the evidence on which the assessment is based; the quality of the actual analysis and assessment; and the clarity with which NICE links its recommendations to its analysis and evidence.


84. The Alzheimer's Society pointed out that "the decisions that NICE makes can only be as good as the evidence it receives".[108] The Consumers' Association, which publishes the Drug and Therapeutics Bulletin, argued that it was "worrying that the group commissioned by NICE to prepare the Health Technology Assessment report on zanamivir indicated they did not have access to all the information needed to fulfill this brief".[109] A lack of information from manufacturers was also mentioned by Dr Sheila Bird, a member of NICE's Appraisal Committee, who noted that "NICE-commissioned research into cost-effectiveness of disease-modifying treatments of multiple sclerosis was not accorded full access [to individual patient data] by all pharmaceutical companies".[110]

85. As discussed in paragraph 40, the UK Cochrane Centre and the MRC Clinical Trials Unit both made the point that, unlike the situation which applies in respect of the medicines licensing system, there is no legal requirement for manufacturers to provide NICE with all the relevant information about a treatment or intervention under assessment, and they claimed that this left open the possibility of NICE decisions being based on a biased subset of data.[111] In their oral evidence, both NICE and the pharmaceutical industry strongly contested that this had ever happened in practice.[112] However, NICE felt that being able to back up requests for information with the force of statute would be helpful. We recommend that the Government should take steps to ensure the submission of all relevant clinical information to NICE. The definition of what types of information are deemed 'relevant' to NICE will obviously require careful consideration, and to this end we suggest that NICE should work with the ABPI to establish guidelines on which types of information must be routinely supplied to NICE, and which types must be made available on request. If the ABPI is prepared to require companies' compliance with these guidelines as a condition of continued membership, it may be possible to avoid the alternative of legislation in order to ensure that no important information is withheld from NICE. As we have recommended that all data submitted to NICE should be made public, it would clearly be necessary to put in place mechanisms to ensure that no information identifying individual patients is submitted to NICE, and neither would companies be required to submit efficacy data on the use of a particular drug or intervention in indications other that those under consideration by NICE, as these may be legitimately commercially confidential.

86. The Medicines Control Agency (MCA), working with the Committee on Safety of Medicines (CSM), conducts detailed assessments of the quality, safety and efficacy of new medicines as part of the licensing process, and the preliminary summary and committee judgements produced are likely to prove a valuable analytical resource for NICE. Improved regulation of submission of information to NICE should be supplemented by closer working relationships between the MCA and NICE, including the sharing of appropriate summary information prepared for the CSM, in order to prevent duplication and strengthen the quality of NICE's outputs.

87. At present, the information submitted to the CSM as part of the drug licensing process remains confidential, as do the CSM's preliminary summary and committee judgements. If this information were to be shared with NICE in confidence, this would run counter to the principle of transparency that NICE should publish all the evidence on which its decisions are made. Under the system we have proposed in section II, if pharmaceutical companies insist that their unpublished data remain confidential then they will be forced to withdraw from the NICE interim appraisal process. Although this carries the risk of the NHS being denied interim guidance on a treatment, we believe that pharmaceutical companies are unlikely to use this as an 'opt-out' clause, as their refusal to submit their evidence to public scrutiny would surely have a negative impact on commissioners' and clinicians' perception of the usefulness of their product.

88. There are, however, further issues relating to the strength of NICE's evidence base which relate to the availability rather than the accessibility of information. Much of the evidence NICE uses in its assessments comes from research conducted by drug manufacturers prior to the drug licensing process. Professor David Barnett, Chair of NICE's Appraisal Committee, described the shortcomings of this information: "Frequently the evidence base available for the appraisal is limited to little more than that which is used for licensing purposes and the important issues of clinical and cost-effectiveness in real life practice have not been investigated sufficiently".[113] Aventis explained the distinction between the types of appraisal:

"Clinical effectiveness differs from clinical efficacy as it incorporates the notion of how a drug works in real life, rather than the constrained observational environment of a clinical trial being run according to a strict protocol. It also includes the expanded notion of tolerability, which includes not only the classical side effect profile but also the broader, specific and general impact on a patient's quality of life."[114]

89. While the clinical effectiveness of treatments for short-duration illnesses such as influenza may be relatively easy to assess, for chronic relapsing conditions such as multiple sclerosis, it can take decades before the final value of the treatment is known. Equally, for cancer treatments a determinant of clinical effectiveness such as five-year survival rates will obviously not be available until five years after patients are first treated. This means that in assessing the clinical effectiveness of a drug before or at the time of licensing, NICE often has to rely on evidence gained via a range of modelling techniques where other more quickly measurable endpoints (for example, time to disease progression) are used as surrogate endpoints for clinical outcomes such as survival. This type of modelling is at best an imperfect science, and several of our memoranda questioned the appropriateness of approximations used by NICE in different appraisals.[115]

90. The ABPI raised concerns that "recommendations made by NICE in some circumstances are being made too early, on the basis of immature evidence", and went on to argue that "issuing definitive guidance on a medicine too early can have damaging consequences - it can deny opportunities for discovery of valuable therapeutic potential in the form of new uses".[116] A further problem with conducting technology appraisals solely on the basis of pre-licensing research is that, as most of this research is industry-sponsored, the full disadvantages of the use of the treatment in practice have rarely been revealed.

91. We accept that there are limitations on the information that can be gained prior to the launch of a treatment, and that there is a tension between the difficulties in assessing clinical effectiveness at an early stage, and the NHS's evident need for guidance at the time of launch to help it manage the introduction (or restriction) of new treatments in the NHS. The system of appraisals at the time of launch that we have recommended does not preclude the possibility of conducting fuller appraisals of a treatment's effectiveness when more information has been collected. Indeed, we recommend this should take place, but within the broader context of NICE's main work on clinical guidelines.

92. According to the definition given by Aventis, clinical effectiveness includes a treatment or intervention's "general impact on a patient's quality of life". As such measures are frequently not included in the efficacy trials conducted by pharmaceutical companies, patient organisations are often the only source of information on the impact of a treatment on quality of life. Such organisations vary considerably in their size and in their capacity to contribute to NICE's work, and less common conditions, acute conditions and those with short survival times may lack representative organisations. This means that the quality of evidence on quality of life and patient experience supplied to NICE may vary considerably between different appraisals and guidelines. This gap was identified by the Alzheimer's Society: "NICE must have high quality evidence of what patients think. If NICE cannot get that from the patient groups in a particular appraisal then it should be commissioning that research".[117] Patients and carers themselves may be even less able to contribute. While the MS Society argued that NICE's "attitude seems to be that 'real' people cannot be objective about their conditions and somehow pollute the Institute's scientific approach",[118] Professor Barnett implied that the problem may be one of evidence quality, describing some patient and carer submissions as "rather disparate and poorly focused". Professor Barnett did, however, agree that "often the patient perspective has not been, to date, given the credence it should and may seem disadvantaged relative to the weight of scientific, clinical and health economic data provided from other sources".[119]

93. NICE now makes a payment of £400 to relevant patient organisations who submit evidence, to cover their costs in contributing to the assessment process. This may go some way towards alleviating the financial burden of the NICE appraisal process, but it does not address the underlying issue, which is the lack of a systematic means of securing reliable evidence of patient experience and quality of life to feed into the NICE appraisal process. Professor Barnett told of plans to develop a 'Patient Impact Assessment Document' which will draw together the patient and carer submissions received by NICE, and which will run alongside the academic Health Technology Appraisal report. While this may be an improvement on existing systems, we do not believe that continuing to rely on proactive submissions from patients and their representative organisations is a satisfactory alternative to systematically obtaining high-quality data on patient experience and quality of life.

94. We recommend that NICE should consider options for improving its evidence base in respect of patient experience and quality of life, including the possibility of working with governments, at national and EU level, and with the pharmaceutical industry to promote the routine inclusion of condition-specific quality of life measures into controlled clinical trials carried out prior to licensing by the pharmaceutical industry. NICE may find it profitable to draw on the expertise of the new Commission for Patient and Public Involvement in Health established by the NHS Reform and Healthcare Professions Act, when this body is established. The Government should consider extending the remit of this body to include explicitly the securing of appropriate patient input into NICE processes.


95. How accurately and robustly NICE analyses the evidence it receives is also a key determinant of the overall quality of its appraisals. Many of our witnesses drew attention to problems in this area: Dr Duerden described "concerns about the technical integrity"[120] of some of NICE's guidance, as well as "technical mistakes or omissions". The Drug and Therapeutics Bulletin cited numerous examples of what they regarded as errors or discrepancies in individual pieces of NICE guidance, and told us that the scale and breadth of these made them feel that these were not isolated incidents, but "fairly fundamental problems".[121] Helen Marlow of Croydon Health Authority reported that "People who look at [NICE appraisals] in detail feel that the quality of some of them could be better",[122] and GlaxoSmithKline asserted that "Pretty much factual errors have occurred in all of the individual appraisals which we have been involved in".[123]

96. Crucial to the technology appraisal process is the Health Technology Assessment (HTA) report, the production of which NICE usually contracts out to one of six independent academic institutions. The Alzheimer's Society levelled particular criticism at the HTA report on drugs in its area, arguing that "it contained many assertions unaccompanied by evidence and conclusions not justified by the evidence presented. It was inconsistent in its treatment of different types of evidence citing unpublished and unreferenced observational data as well as properly conducted trials. References in the report were incomplete, some were incorrect, relevant publications were omitted".[124] GlaxoSmithKline also raised concerns about what they saw as "the great deal of variability in the assessment groups and the original assessments they provide for the Appraisal Committee to consider".[125]

97. The MRC Clinical Trials Unit and the NHS Centre for Reviews and Dissemination were concerned that the requirement for relative speed in making technology appraisals, might "seriously compromise [their] quality".[126] In the estimation of the NHS Centre for Reviews and Dissemination, a high quality systematic review of scientific evidence can take an average of 12-18 months, while NICE aims for a six month turnaround. CERT argued that "some of the variability in the appraisals may reflect insufficient resources at NICE. The health technology groups commissioned to develop the evaluation reports are paid only around £14,000".[127] NICE disputed this figure and asserted that the figure is approximately double this.[128] However, we would anticipate that a team of competent people (including four researchers and two administrators) working full time for six months with the necessary back up could cost as much as £150,000.

98. We recommend that NICE should ensure that the academic centres to which it contracts its assessment work are adequately resourced to enable them to conduct Health Technology Assessments to the highest possible standard. Increasing the resources NICE directs at this area of its activity may enable it to improve the quality of its work and recruit more scientists of the highest calibre and experience. In addition to this, we are concerned that with the considerable expansion in NICE's coverage we have suggested, there may be workforce as well as funding issues. There are at present only a limited pool of skilled academics available within the UK, especially in the field of health economics, and the pharmaceutical industry is competing to recruit from the same pool. NICE and the Government must work together to address this problem.

99. Several of our submissions also pointed to a current lack of quality audit of NICE's own appraisals.[129] Lord Hunt told us that in addition to workload and output targets, the annual accountability meeting held between NICE and its sponsor Minister considers the robustness of NICE's processes in general terms, but this arrangement clearly does not offer the scope for detailed audit of NICE's outputs.[130] We recommend that the Government institutes independent detailed peer review of a random selection of guidance prepared by NICE. This could be carried out by CHI/CHAI on a three-yearly basis.


100. A lack of understanding of NICE's analytical methodology was a common theme in our evidence. North Tyneside Health Authority argued that the credibility of guidance has been undermined where it is unclear how the Institute reached its decision.[131] Ealing Hammersmith and Hounslow Health Authority felt that "The 2001 Guide to the Technology Appraisal Process says very little about how the Appraisal Committee should come to its decision. Issues such as these could be given more prominence by making the process of weighting the strength of evidence, relative (cost) effectiveness and other factors more explicit".[132] Confusion about evidence-weighting seemed widespread: Clara McKay, principal policy adviser to the Consumers' Association, argued that a wide-ranging independent survey of patient organisations conducted by the Association revealed "a real concern across the board of patient organisations not having a sense of the impact they make on the outcome of the guidance", and this uncertainty was not confined to patient organisations. The NHS Centre for Reviews and Dissemination asserted that:

"Research findings at times appear to receive less weight than submissions from other stakeholders. The lack of clarity and transparency in the decision making process may lend itself to accusations of bias in favour of certain aspects of the process (e.g. input by pharmaceutical companies). The independence of NICE is threatened by this lack of clarity."[133]

101. NICE told us quite categorically that "we regard the evidence submitted by all of our stakeholders as of equal value, and equal value also with the assessment report that we receive ... from the independent academic organisations".[134] It had recognised a lack of clarity about how its decisions are reached, and its reports now included a "Consideration" section which sets out how the Appraisal Committee interpreted its evidence, and how different types of evidence influenced its conclusions.[135] However, we feel that this section is very brief, and has yet to offer the sort of information that will assure readers of the validity of NICE's processes.

102. Many concerns were raised about cost-effectiveness, and in particular NICE's use of cost per Quality Adjusted Life Years (QALYs), a method which attempts to provide a comparative measure of the benefits of a treatment against its cost, considering the extra years of life the treatment or intervention is likely to yield per person treated, adjusted to take account of quality of life. Dr Crayford of Croydon Health Authority was one of many who criticised NICE for its inconsistent approach to the use of Quality Adjusted Life Years, which he estimated NICE had employed in only about half of its assessments.[136] The ABPI argued that the true costs and benefits of treatments sometimes took years to establish, and several memoranda also made the point that NICE's calculations of cost and benefit failed to take account of the wider societal costs and advantages of particular treatments, including, for example, reductions in benefit dependency and improved ability to work, both for patients and their carers.[137] The difficulties surrounding the new and imprecise science of cost-effectiveness were recognised by NICE, who told us that "QALYs are not the answer, definitely not. They are, however, the only thing we presently have which is internationally recognised and gives us a handle on comparisons across technologies, across disease states and across individuals and age groups".[138] Whether or not QALYs are used, we recommend that NICE should consider the wider societal costs and advantages of particular treatments and in particular the wider costs and benefits to the public purse of reduced benefit dependency and improved ability to work both for patients and their carers.

103. Glyn McDonald of the MS Society raised concerns about NICE's decision-making process, describing the "creeping establishment of a threshold for cost-effectiveness".[139] Professor Barnett told us that "The Appraisal Committee does not consider the threshold [for cost per Quality Adjusted Life Year] and has not been given instructions about a threshold and has not discussed a threshold per se at all".[140] However, the King's Fund argued that, whether it is arrived at implicitly or explicitly, "NICE has a responsibility explicitly to justify the threshold that appears to have emerged as its decision rule".[141] The decision about how much money the NHS should be prepared to spend to improve an individual's health or save their life is fraught with difficult judgements which go beyond the technical, as illustrated by Professor Walley:

"... it may be that a new therapy is uniquely effective but constitutes very poor value for money, i.e. a conflict between clinical effectiveness and cost-effectiveness. Whether it should be funded is therefore a matter for judgement or appraisal, to include the financial, social, ethical and legal implications of its use or non-use. The rules or values for such appraisal are not clear. We require open debate about how we value such trade-offs."[142]

104. This was clearly understood by Professor Rawlins: "We certainly recognise that some of our decisions, or the basis upon which they are made, require value judgements. We fully recognise with those value judgements that we are no more competent than anybody else to make them".[143] We note NICE's plans to establish a Citizens' Council composed of 'ordinary men and women around the country' to advise on these value judgements. [144] We agree with the many witnesses who argued for a review of NICE's appraisal methodology, and the publication of clear criteria.[145] We therefore recommend that NICE, aided by the Department of Health, should conduct a review of its methodologies for assessing clinical and cost-effectiveness, which should result in the publication of a set of clear and consistent criteria for the assessment of both aspects. This should include a description of the weighting given to different types of evidence, a detailed argument for its use of QALYs, and the impact of both cost- and clinical effectiveness on the final determination, including any cost-effectiveness 'thresholds'. In tandem with this, NICE should work to strengthen its cost-effectiveness evidence base by encouraging pharmaceutical companies to collect this type of data routinely.



105. A further area of confusion and concern over NICE's remit and methodology relates to the issue of affordability. Although Lord Hunt and NICE were clear that affordability was not an issue for NICE, but was rather a matter for government, some witnesses felt that the boundary between cost-effectiveness and affordability was very blurred. Mr Cayton of the Alzheimers' Society argued that: "one of NICE's biggest problems was being given this second task, which in a sense people have referred to as really a political decision, of doing cost-effectiveness as well as clinical effectiveness".[146] Dr Cunningham of Lambeth, Southwark and Lewisham Health Authority suggested that:

"In a sense NICE was set up to give robust guidance on the basis of evidence, and it appears now to be almost being asked to do resource allocation by the Department of Health. It seems sensible to separate these two functions more clearly, particularly the selection of the portfolio which NICE considers and the decisions which are made when NICE has made its pronouncements."[147]

106. As the Government has now issued directions for mandatory implementation of all NICE guidance, the funding and provision of treatments would appear to "follow automatically"[148] from a positive NICE appraisal, or be automatically withheld following a negative one. This may create the impression that, because it is likely that NICE recommendations will be implemented whatever the cost, NICE is essentially making decisions about how money is spent in the NHS, or ministers are somehow making decisions through NICE. As Professor Walley put it, "because it is not apparent when the Minister has made a decision that the funding will be available, there is a perception, which I do not think is actually true, that at some point the Minister has influenced the process before it has got to the final stage".[149] This confusion may be consolidated by the wording of NICE's Framework Document, which states that NICE should have regard to "the effective use of available resources", and by the fact that NICE calculates and publishes the net budget impact of its recommendations, although NICE is clear that this plays no part in the appraisal process.[150]

107. Decisions about the affordability of NICE's recommendations are, we believe, properly a matter for government. We also believe that there is a need for decisions about the affordability of NICE's recommendations to be seen to be made entirely separately from NICE's decisions about clinical and cost-effectiveness. While we have received no compelling evidence that these decisions are not being taken independently, the widespread perception that the three elements are linked urgently needs to be dispelled. We therefore recommend that the Government should take steps to clarify its own role in taking decisions about whether or not individual pieces of NICE guidance will be funded. One solution would be for the Department of Health to issue a separate document alongside each piece of NICE guidance, stating the costs (both financial and staffing) of implementation of the guidance, and on the basis of this indicating whether or not, and in what circumstances, the guidance was to be implemented.

105   Ev 128, citing Raftery, J, Analysis of guidance on health technologies, BMJ 2001; 323:1300-1303. Back

106   Ev 47 (National Cancer Alliance). Back

107   Ev 236. Back

108   Ev 54. Back

109   Ev 5. Back

110   Ev 196. Back

111   Ev 199; Ev 244. Back

112   QQ290-93; QQ348-55. Back

113   Ev 131. Back

114   Ev 247. Back

115   For example, Ev 4 (Consumers' Association); Ev 248 (Aventis). Back

116   Ev 91. Back

117   Q134 . Back

118   Ev 44. Back

119   Ev 131. Back

120   Ev 17. Back

121   Q9. Back

122   Q165. Back

123   Q261. Back

124   Ev 54-55. Back

125   Q259. Back

126   Ev 222; Ev 243. Back

127   Ev 107. Back

128   Ev 166. Back

129   For example, Ev 239 (Professor Maynard); Ev 54-55, Ev 56 (The Alzheimer's Society). Back

130   Q450. Back

131   Ev 77. Back

132   Ev 203. Back

133   Q6; Ev 8; Ev 222. Back

134   Q356. Back

135   Q356. Back

136   Ev 72. Back

137   ABPI give the example of the medicine EPO, used to treat patients with kidney failure, which had an estimated QALY at launch in 1990 of £103,000 but which, following changes in the management of patients, now has a QALY of around £3-9,000 (Ev 92). For discussion of wider costs and benefits see, for example, Ev 233 (The Pituitary Foundation), Ev 231 (RCGP). Back

138   Q447. Back

139   Q110. Back

140   Q447. Back

141   Ev 210. Back

142   Ev 23. Back

143   Q315. Back

144   Q315. Back

145   Ev 2 (Consumers' Association); Ev 246 (Aventis). Back

146   Q123. Back

147   Q146. Back

148   Q82. Back

149   Q82. Back

150   Q312. Back

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2002
Prepared 3 July 2002