CHAPTER 4: EVIDENCE
4.1 There are several types of evidence that it is
desirable to obtain before a therapy is advocated:
- Evidence that the therapy is efficacious above
and beyond the placebo effect (see paras 3.19 - 3.34);
- Evidence that the therapy is safe;
- Evidence that the therapy is cost-effective;
- Evidence concerning the mechanism of action of
- Methods available for obtaining such evidence
will be discussed in Chapter 7 : Research.
Evidence for Efficacy?
4.2 The conclusions from research into the efficacy
of the various CAMs are outside the remit of this report. However,
it is necessary to understand the general state of the CAM evidence
base, in order to consider what type of evidence needs to be collected
and to understand why CAM's claims often cause controversy.
4.3 CAM has been criticised by some witnesses for
not having scientific evidence to back its claims. The Academy
of Medical Sciences (P 1) told us that they are concerned that
many CAM practitioners do not take a 'scientific' approach to
treatment. They suggest that whereas conventional medicine makes
efforts to conduct rigorous research, and changes its clinical
practice when new information is discovered, CAM practitioners
are more likely to stick to their belief systems despite any negative
evidence that may emerge. This is one reason the Academy of Medical
Sciences uses to explain why CAM lacks an adequate evidence base
to convince them, as conventional scientists, of its claims. This
is a controversial statement, and as discussed in para 2.19, Sir
Iain Chalmers, Director of the UK Cochrane Centre, suggested that
conventional medicine is biased against CAM and conventional medical
practitioners and scientists are likely to require lower standards
of proof for conventional medical treatments then they do for
CAM (P 225). Nevertheless, as we concluded in para 2.7, we are
satisfied that there is at present no credible evidence base to
support the value of any of the therapies that we list in our
4.4 The Department of Health summed up their opinion
of the evidence base for CAM by saying that "Evidence for
CAM in the form of research has been criticised as being inadequate,
and there is some justification in this claim" (P 101). This
is a controversial area, as the definition of an adequate evidence
base varied across our witnesses. Some CAM practitioners have
claimed that a history of safe and apparently successful traditional
use is enough evidence to justify advocating the use of their
particular therapy. However, most of our witnesses with a conventional
medical or scientific background have asserted that, in order
for CAM therapies to be more widely accepted, it is important
that they have a critical mass of scientifically-controlled evidence
to support their claims; and that at the moment most of CAM lacks
such evidence. Many of our conventional medical witnesses have
suggested that, since much of conventional medicine is required
to undergo rigorous trials to justify its use, no less should
be expected of CAM. But even this view is controversial as some
of the CAM advocates we have heard from have suggested that much
of conventional medicine lacks a rigorously tested evidence base,
and that to require one of CAM is to operate a double standard.
In fact, the Department of Health followed their statement that
there was some justification behind the claim that the CAM evidence
base was inadequate, by acknowledging that the same could be said
for some conventional medicine. It is our view that most modern
conventional therapies are backed by scientific evidence. It is
in the case of some of the older and traditional treatments surviving
from the past (such as cold 'cures' and 'tonics') where evidence
is, like the evidence for much of CAM, lacking.
4.5 There are two notable weaknesses of the evidence
base for CAM that at present exists. One is that in most of these
areas little research is being done, and the second is that the
few studies which have been completed are given disproportionate
weight. It is worth considering this second feature in some detail.
Some CAMs have embarked upon research in order to build up an
evidence base. All the therapies that we have included in Group
1 either have done, or are working on, rigorous trials to test
their claims. However, one or two studies with positive results
in support of their claims for efficacy are not enough. It must
be remembered that with a statistical significance of p<.05
(the commonly accepted level of significance), one in twenty studies
of any procedure will show a possible significant effect; hence
a few positive results with small effects are not yet enough to
prove a therapy's efficacy, nor to justify its wider provision.
4.6 Another problem with the way existing studies
are used is that many old studies are recycled again and again
through reviews and meta-analyses.
There are some doubts about the usefulness and validity of the
results of meta-analysis. Professor Peter Lachmann, on behalf
of the Academy of Medical Sciences, told us: "Meta-analysis
is a highly contentious but very important issue and is subject
to all sorts of problems, of which comparable design, selective
publication of positive against negative results and various other
problems are well-known. Not all meta-analyses should be afforded
the same weight" (Q 1411). This problem highlights the need
for more original work, involving well-designed clinical trials,
to be done on CAM disciplines.
4.7 The importance of evidence of efficacy is less
clear than the importance of evidence of safety. Many witnesses
have suggested that if a person feels that a therapy is helpful
to them and can be shown not to be harming them, then it is not
necessary for there to be statistically valid research supporting
its claims. But the question then arises as to whether such a
treatment should be made available at public expense. The role
of patient satisfaction in evaluating therapies will be considered
in paragraphs 4.24 - 4.27.
4.8 One argument that has been repeated to us is
that the existence of evidence which supports a therapy's claims
is of secondary importance, provided that patients are aware of
whether there is any evidence or not. Consumer bodies such as
Patient Concern (P 166) believe that treating a patient with a
therapy that lacks evidence of efficacy is not wrong if the patient
is happy with the treatment, as long as he or she knows that there
is no definitive proof of efficacy and has not been led to believe
that the treatment will definitely work. They call for strong
measures to be taken against practitioners who mislead patients
with false claims of evidence of efficacy.
4.9 Another issue to consider in this area is how
much evidence there is to support the claims of other healthcare
interventions so as to consider the position of CAM in context.
The Medicines Control Agency require evidence of efficacy (and
of quality and safety) before licensing any new pharmaceutical
product. However, the British Dental Association (P 35) gave evidence
suggesting that much of clinical dental practice has a weak evidence
base. We have also heard evidence concerning several commonly
used conventional medical treatments that have a long history
of use but little research evidence to support such use. Examples
include the use of electro-convulsive therapy for the treatment
of depression and cervical and uterine curettage for treating
dysfunctional uterine bleeding.
4.10 However, the Institute of Biology (P 125), suggest
that if health practitioners are to be held liable for their services
to their patients then the medicines they prescribe must be proven
to be efficacious.
4.11 There are complications in this area beyond
simply evaluating the importance of evidence of efficacy. Many
submissions from CAM representatives, as well as the submission
from the NHS Confederation (P 144), claim that, for some forms
of CAM in some situations, there is already evidence of efficacy.
Therefore they suggest that the lack of mainstream acceptance
and the slow NHS uptake must be due to other factors. The NHS
Confederation claims CAM has suffered from 'unscientific prejudice'
from the scientific orthodoxy; however most conventional medicine
submissions deny this, and reiterate the argument that a few positive
studies should not be given too much weight and do not constitute
a critical mass of evidence (para 4.5). In particular, positive
trials of homeopathic treatment in allergic disorders have not
yet convinced many conventional practitioners. Specifically, trials
at the Glasgow Homeopathic Hospital, demonstrating benefit in
the treatment of asthma and allergic rhinitis with homeopathic
remedies, are thought by some independent observers to need larger
and longer trials for confirmation of the perceived effects.
4.12 Beyond these general points the diversity of
CAM therapies is such that our comments must be related to the
three groups of disciplines that we have listed in Box 1.
4.13 Of the therapies in Group 1 we were made aware
of good evidence of the efficacy of osteopathy and chiropractic.
Indeed, they appear to be somewhat more effective than the manipulative
techniques employed by conventional physiotherapists. There is
also scientific evidence of the efficacy of acupuncture, notably
for pain relief and the treatment of nausea.
The evidence for the efficacy of herbal medicine is mixed. Many
herbs have established activities while others do not; among those
which are active many are claimed to have numerous other actions
for which evidence is lacking. Many powerful drugs used in conventional
medicine are of herbal origin, such as morphine derived from the
poppy, or digoxin from the foxglove. Problems sometimes arise
when mixtures of herbs are used. Even when these are of proven
efficacy it may be difficult to identify the active ingredient
or ingredients and some preparations may be difficult to standardise
and control. In the case of homeopathy, although it is covered
by a separate Act of Parliament, we were not able to find any
totally convincing evidence of its efficacy. Nevertheless, we
accept that there is anecdotal evidence of benefit from homeopathic
remedies in animals, where presumably a placebo effect is less
significant. Much more research is needed.
4.14 Of the therapies in Group 2 there are many claims
of efficacy, usually for a limited range of ailments. We have
not examined each in detail. We see many of these complementary
therapies as inducing relaxation and a sense of well-being so
as presumably to stimulate the immune response, as in the placebo
effect. Many are greatly appreciated for the comfort they provide
to terminally ill patients.
4.15 We find no convincing evidence of efficacy for
any of the remedies in Groups 3a or 3b, but we did not carry out
a detailed examination.
4.16 Evidence for the efficacy of the treatment itself
is not the only important factor. The Royal Society of Edinburgh
(P 212) makes the point that evidence of the validity of diagnostic
procedures is as important as evidence supporting efficacy of
a treatment. Diagnostic procedures must be reliable and reproducible
and more attention must be paid to whether CAM diagnostic procedures
as well as CAM therapies, have been scientifically validated.
We agree that this is an issue that should always be kept in
mind when doing research in this area.
4.17 More research is needed on the efficacy of most
CAMs. In the case of therapies which possess research evidence,
but whose practitioners believe that conventional scientific views
are standing in the way of their acceptance, it would be constructive
if a body such as the National Institute for Clinical Excellence
(NICE) could evaluate such evidence as exists. (NICE did point
out that topics they enquire into are determined by the Department
of Health and are selected against a framework of the State's
priorities for the NHS (Q 1839). However, they did acknowledge
that, in their view, such subjects may be suitable for appraisals).
It would also help if such bodies made sure that on their evaluation
committees were doctors and scientists who were aware of CAM's
intricacies, philosophy and research (see Chapter 7).
4.18 In our opinion any discipline whose practitioners
make specific claims for being able to treat specific conditions
should have evidence of being able to do this above and beyond
the placebo effect. This is especially true for therapies which
aim to be available on the NHS and aim to operate as an alternative
to conventional medicine, specifically therapies in Group 1. The
therapies in our Groups 3a and 3b also aim to operate as an alternative
to conventional medicine, and have sparse, or non-existent, evidence
bases. Those therapies in our Group 2 which aim to operate
as an adjunct to conventional medicine and mainly make claims
in the area of relaxation and stress management are in lesser
need of proof of treatment-specific effects but should control
their claims according to the evidence available to them.
Evidence for Safety
4.19 Evidence that a therapy has few, if any, significant
adverse effects and will not cause avoidable harm must be considered
important in all medicine, including CAM. However, there are two
potential complications which confound this seemingly simple statement:
- What level of safety should be demanded?
- What type of evidence of safety is acceptable?
4.20 In determining what level of safety should be
sought, the risk/benefit ratio of the therapy in question must
be considered. If the potential benefits of a therapy are likely
to be very significant, or even life-saving, then the level of
risk a patient may be willing to take with the therapy is likely
to be higher than the level of risk they are willing to accept
for the benefit of temporary symptom relief or the cure of a minor
complaint. Another consideration is whether the risks a therapy
may possess are inherent or can be minimised through proper regulation
of its practitioners. For example we received some evidence about
the risks of acupuncture causing pneumothorax due to a needle
being inserted into the pleural cavity; however if practitioners
are properly trained and well-regulated this risk is minimised.
In determining what evidence of safety is acceptable it is important
to consider what weight should be given to a history of safe traditional
use. Within CAM such evidence is common and is often given reasonable
weight by CAM advocates and to a certain extent by policy-makers.
For example there are exemptions from licensing in the Medicines
Acts for natural remedies of traditional use, and a third category
of medicines, which will include traditional-use herbal medicines,
is being examined by the European Union (see Chapter 5).
4.21 There is no doubt that many CAM therapies are
very safe, as compared to many new powerful conventional remedies.
This is often used as an argument for approving the increasing
use of CAM, but it must be remembered that the use of a "safe"
CAM remedy to treat a serious or potentially lethal disease, so
that the use of conventional preparations with proven efficacy
is denied, is of course a real danger.
4.22 Several submissions we received suggest that
minimum standards of safety need to be defined and widely disseminated
in order to protect the public. The British Holistic Medical Association
have suggested that such work should be carried out by NICE and
the Commission for Health Improvement (CHI) who should then issue
4.23 The evidence that we received from almost all
the different therapies indicated that at the point of diagnosis,
if the practitioners thought that their treatment would not work,
they would refer their patients to a conventional medical practitioner.
We were encouraged by this sentiment, even though it was not universal.
PATIENT SATISFACTION AS EVIDENCE
4.24 We have heard many conflicting opinions on the
idea that high levels of patient satisfaction could be used as
evidence for a therapy's efficacy. It has been argued by some
that such satisfaction is very important. The International Federation
of Reflexologists (P 129) suggest that evaluation of patient satisfaction
is particularly important in CAM because much of CAM emphasises
patients' participation in the therapy and evaluation of its effects.
Many other witnesses have asserted that although patient satisfaction
has its place it is not sufficient to justify accepting that a
therapy works so that objective rather than subjective evidence
is needed. The Academy of Medical Sciences explained why this
may be: "It needs to be emphasised that patient satisfaction
is not in itself a sufficient estimate of clinical benefit. While
it is very important that patients be satisfied with the efforts
made on their behalf, it is at least equally important that they
should obtain objective benefit. The two do not always go together.
For example, patients with peripheral vascular disease, if they
go to a practitioner who allows them to continue smoking will
show a high patient satisfaction although their outcome will be
poor. In contrast, if they are made to stop smoking they are likely
to be dissatisfied but their outcome will be much better"
4.25 NICE, who have been charged with the responsibility
of evaluating the evidence for different NHS treatments over the
coming years, also express concern about the validity of anecdotal
evidence such as patient satisfaction. Professor Sir Michael Rawlins,
Chairman of NICE, told us: "Anecdote, by and large, is not
a very reliable method for determining efficacy and 2000 years
of medicine demonstrate the fragility of anecdote as a basis for
practising medicine" (Q 1833).
4.26 One point that most of our witnesses have agreed
upon is that patient experience is important enough to warrant
patients being involved in the appraisal of therapies. NICE have
made moves towards incorporating patients' views into their appraisals.
Mr Andrew Dillon, Chief Executive of NICE, told us: "In the
process we have established we invited nationally-based patient
advocate groups to make submissions into our individual appraisals.
So we have a written statement of their assessment of, as
far as they understand it, the patient's perspective of the disease,
and if it is an intervention which is currently in use in the
NHS, their understanding of the patient's experience of using
that intervention in the management of their illness. We also
invite patient advocates to join the appraisal committee meetings
themselves" (Q 1843).
4.27 In conclusion, patient satisfaction has its
place as part of the evidence base for CAM but its position is
complicated, as Sir Michael Rawlins, explained: "The difficulty,
of course, is that very often the anecdotal evidence relates to
conditions where there is fluctuation in the clinical course and
people who start an intervention at a time when there is a natural
resolution of the disease, very understandably, are likely to
attribute cause and effect when it may not be. But, on the other
hand, there are some anecdotes that are quite clearly important."
Therefore, ideally studies should include patient satisfaction
as one of a number of measures in evaluating a treatment, but
it alone cannot be taken as a proof or otherwise of a treatment's
efficacy or as evidence to justify provision.
Evidence About Mechanisms
4.28 The position of therapies without a scientifically
plausible mechanism of action (e.g. healing and homeopathy) needs
to be considered. If there is no scientifically plausible mechanism
through which a treatment may work in the human body, can the
use of such a therapy be justified? Should such therapies be considered
a product of the placebo effect enhanced by "tender loving
care" or should consideration be given to the possibility
that they may have explanations not yet understood by modern science?
(The role of the placebo effect is discussed in paras 3.19-3.34.)
4.29 Many of our witnesses have argued that if there
is evidence for efficacy then it is not necessary to understand
exactly how the effect is achieved, and we agree. This is, indeed,
the position with several conventional therapies. Professor Sir
Michael Rawlins explained that in NICE's search for clinical excellence,
it is evidence of efficacy and not the mechanism of action that
is prioritised: "I do not mind and I do not think the Institute
minds whether it understands how a treatment works or not. I do
not understand how many treatments do work, and this is after
35 years as a pharmacologist, but what we do like is good evidence
that they do whatever they claim to do" (Q 1833).
4.30 However, despite these arguments, the opposite
view is that if a therapy has no plausible mechanism of action
then spending research money on it and providing patients with
access to it is likely to be a waste of resources. It is worth
considering this argument in more detail, by asking two distinct
- Should mechanisms of action be plausible before
research into the efficacy of a therapy is funded?
- Should mechanisms of action be understood before
access to a therapy is provided?
4.31 It is of course true that many treatments have
been used for a long time without understanding their mechanisms
of action and only now are possible explanations for how they
work coming to light. Professor Lesley Rees, a Trustee of FIM,
used acupuncture as a case in point. Acupuncture is traditionally
said to work through affecting energy meridians that according
to Traditional Chinese medicine circulate around each person.
This explanation is not congruent with current scientific thought
and if an understanding of mechanisms of action were considered
of paramount importance doctors should have shunned acupuncture
years ago. Now, however, other possible mechanisms, which are
more amenable to modern scientific thought (e.g. concerning the
effect on the central nervous system and the stimulation of endorphin
receptors), are being discovered and evidence for acupuncture's
efficacy is growing. As Professor Lesley Rees summed up: "
has been used for thousands of years, yet there was no real information
about how it might work and I think it would be fair to say that
it would have been terrible if the benefits of acupuncture had
not been appreciated and used over all the years because we did
not have any real understanding of perhaps some of the mechanisms
about how they work" (Q 77).
4.32 In terms of research Professor Tom Meade from
the Royal Society told us that "
the distinction between
the effect and the explanation for the effect is central, and
you do not need to believe in the explanation in order to believe
in the effect" (Q 181). Therefore, "...it would be perfectly
possible for a funding body to allow a bit of research to go forward
even though the theoretical backdrop is totally irrelevant to
whether the treatment works or not. I think probably what we will
see now, increasingly, is applications for funds which simply
say 'There is good reason to think there is an effect here and
we want to study that. We are going to use these methods which
are well attested.' And if everyone agrees that then, if this
works out, it will reduce the ambiguity of the effectiveness of
this particular treatment" (Q 181).
4.33 A reason for funding efficacy studies of therapies
without a plausible mechanism of action is that research into
that area can help elucidate routes through which mechanisms of
action might work. Professor Meade explained: "
it is possible, in some circumstances, that the result of the
trial - in other words that something is effective - will actually
then give clues as to studying the mechanisms. Equally, if it
is not effective then it is beginning to exclude possible explanations
as well" (Q 181).
4.34 However, there is an alternative view, articulated
by Professor Lewis Wolpert, a fellow of the Academy of Medical
Sciences, that: "It is not just efficacy that you should
be thinking about. Medicine aims to base itself upon science.
Let me tell you what I mean. If you have therapy which you can
in no plausible way relate to the behaviour of cells
could not support research in that field" (Q 1404). Based
on the limited amount of research funding available in the medical
sciences, he suggests that research into therapies such as homeopathy
should not be funded: "A liquid which contains no active
molecule, which no chemist could plausibly give an account of,
is not an area where I would want to invest money. I am sorry:
one cannot give up all of chemistry just because one believes
homeopathy works" (Q 1404 and 1406). Professor Patrick Bateson,
giving evidence on behalf of the Royal Society, summed up this
argument by saying the role of mechanisms of action comes into
importance because "... the critical thing here is going
to be whether there is enough evidence to justify us spending
more time and trouble testing the efficacy and safety of treatment"
4.35 The mechanism through which homeopathy may work
on the body is a specific case in point, about which we have heard
much. Samuel Hahnemann at the turn of the 19th century put forward
the "law of similars", claiming that any disease can
be treated successfully with minute amounts of a drug which in
larger doses gives rise to the same symptoms. Therefore, homeopathy
is based on the idea of treating 'like with like' by administering
hugely diluted versions of basically dangerous substances, such
that a dose given to a patient may not contain even a single molecule
of the active principle. Many conventional doctors and scientists
cannot accept that infinitesimal dilutions can have any effect
on the body.
4.36 The arguments about homeopathy illustrate the
weight given to understanding mechanisms of action. The Department
of Health explained their position on homeopathy which clearly
shows they prioritise safety before anything else and give less
weight to issues of scientific plausibility: "In relation
to homeopathic medicines, we very much agree that there is uncertainty,
or limited evidence, about the specific mechanism whereby homeopathy
works. The starting point is that homeopathic medicines as such
are very much at the safe end of the spectrum; they are very dilute.
Often these substances do not have a clearly measurable effect
on the body, which is why the simplified homeopathic registration
scheme introduced in 1994 concentrates specifically on safety
and quality and not efficacy. We have taken a fairly pragmatic
approach: if homeopathy does not harm then it is less important
to have an in-depth understanding of its mechanism for effectiveness"
4.37 In terms of research funding for therapies without
a scientifically plausible mechanism of action, it seems that
opinion within the world of conventional medicine is very divided.
However, we recommend that if a therapy whose mechanism
of action is unclear does gain sufficient evidence to support
its efficacy, then the NHS and the medical profession should ensure
that the public have access to it and its potential benefits.
4.38 The question of NHS provision for therapies
such as homeopathy was answered by the Department of Health by
prioritising safety together with consumer choice. On the other
hand, as evidence from the Academy of Medical Sciences suggests,
the only reason for using therapies such as homeopathy is as a
vehicle for the placebo effect to work safely (see paras 3.19
- 3.34). Professor Peter Lachmann told us: "Other effects
of homeopathy apropos the placebo effect have already been mentioned
and I personally am entirely happy with the idea that homeopathy
is a good way of administering a placebo because it is free from
harm. I am well aware of the fact that in conventional medicine
placebo effects are sometimes produced by the administration of
drugs. That is less harmless because all drugs have some side-effects.
If drugs are given not for a good purpose but just given to make
the patients feel that something is being done for them, then
I would entirely agree that a homeopathic preparation, which would
produce the same placebo effect without possible harmful side-effects,
is to be preferred" (Q 1410).
4.39 The intricate arguments concerning the use of
the placebo effect as a therapy were discussed in chapter 3; this
does not contradict the argument that safety is a priority, and
as long as a therapy is safe, use of any benefits it may bring
to patients is justifiable without necessarily understanding its
mechanisms. In an era when the Government are hoping that NHS
treatments will live up to a standard of evidence set by NICE,
we welcome the fact that, as the quotation from Professor Sir
Michael Rawlins shows, NICE are willing to accept that a therapy
can be efficacious and worth considering even when its mechanisms
of action are unclear (see para 4.29).
4.40 It is our opinion that as long as the treatments
are known to carry no, or few, adverse effects, it would be against
the principle of clinical freedom
to prevent patients from having access to therapies which fulfil
these criteria and have never been restricted. This is especially
the case if the patients believe that such therapies help them
and the only argument against them is that an adequate evidence
base, derived from controlled trials, does not exist. It is also
our opinion that mechanisms of action are of secondary importance
to efficacy, a view shared by NICE (Q 1833). We also believe that
the principle of clinical freedom should allow therapy with any
credible evidence of efficacy the opportunity of validation by
further research and the possibility of NHS provision. Any medicine
with credible, accepted evidence for efficacy should be available,
whatever the controversy over its underlying mechanisms.
29 A significance level of p<.05 means that there
is a probability of less than 5% that the results a trial has
produced could occur by chance. Back
Meta-analysis is the combination of data from several studies
to produce a single estimate. From the statistical point of view,
meta-analysis is a straightforward application of multi-factorial
methods. If there are several studies of the same thing with each
giving an estimate of an effect, the meta-analysis provides a
common estimate representative of all the work. Back
See: Vincent, C. & Furnham, A. (1997) (Op.cit.), 'The
quality of medical information and the evaluation of acupuncture,
osteopathy and chiropractic'. Back
British Medical Association. The evidence base of acupuncture
in: Acupuncture: efficacy, safety practice. Harwood Academic
Publishers, London (2000); pp 7 - 37. Back
By "the principle of clinical freedom" we mean the
ability of a medical practitioner to exercise freedom of choice
in preventing, diagnosing and treating disease within the limits
of his or her clinical competence, having regard solely to the
welfare and well-being of the individual, and casting all other
considerations aside. Back