3 Measurement and evaluation
34. The DH's policies for eliminating avoidable harm
to patients have to be judged by their effectiveness. The measure
of this is the overall amount of harm done to patients, and its
cost to the NHS, which diverts resources that could be spent on
other care. In this chapter we examine the available evidence.
How much harm is done to patients?Box
4: Types of evidence about the scale of harm done to patients,
and their respective strengths and weaknesses
Type of Evidence
| Strengths
| Weaknesses
|
Reported incidents
| Provides a minimum number
| Under-reporting
|
Case note reviews
| Provides an adverse event rate
| Only provides a snapshot of a particular time and place; methodologies differ (limiting value of comparisons) and may not be applied consistently by reviewers
|
Patient surveys
| Captures the voice of the patient
| Under-reporting (patient may not be aware an incident occurred, or may not appreciate what counts as an incident)
|
Reported data on specific problems (e.g. healthcare-associated infections)
| Provides a minimum number
| Under-reporting; only captures one facet of safety
|
35. When we asked Mr Fletcher how much harm the NHS in England
does to patients, he referred us to data on reported incidents.[26]
Of the roughly 850,000 incidents in the NHS in England reported
annually through the NRLS, over 65% are "No harm" incidents
(including "near misses", i.e. prevented incidents),
about 25% are "Low harm" incidents, around 5% involve
Moderate harm, less than 1% (about 7,500) involve serious harm
and less than 0.5% (around 3,500) involve the death of the patient.[27]
36. However, as the NPSA itself acknowledges, there
is very significant under-reporting of safety incidents (as is
discussed further below). So figures for reported incidents are
not a reliable indicator of how many incidents actually take place
in the NHS.
37. When we asked the Chief Nursing Officer (CNO),
Dame Christine Beasley, the same question, she could only refer
us to patient survey data collected by the Healthcare Commission
(HCC), showing that "92% of people say their care is very
good, good or excellent".[28]
This answer, however, bypasses the fact that patients may well
be unaware of failings in their treatment. On the same occasion,
the NHS Medical Director, Professor Sir Bruce Keogh, told us only
"I have nothing to add".[29]
38. We sought to clarify what is known about the
harm done to patients. Richard Thomson, Professor of Epidemiology
and Public Health at the University of Newcastle, told us that
the best available data on the overall extent of harm to NHS patients
in England consist of "hospital case note review studies,[30]
the studies that have looked at medical records of patients",
which "demonstrate that around 10% of admissions to hospital
suffer some form of harm".[31]
These data show that, in between a third and half of admissions
associated with harm, that harm was preventable. The DH's memorandum
of written evidence for this inquiry states that these findings
are consistent with those of similar studies in other developed
countries.[32]
39. There have only been two, very limited, such
studies in England (relating to admissions in 1998 and 2004, respectively),
each covering around 1,000 admissions in one and two hospitals
respectively.[33] Other
countries in which studies have been conducted include the USA,
Australia, Denmark, France, Canada, the Netherlands, Spain and
New Zealand; some of these involved up to 50 hospitals and 30,000
admissions (see Annex 2 for a more detailed analysis).
40. All such studies are associated with important
methodological problems. Firstly, it can be difficult to decide
whether or not an adverse event was avoidable. Making such a judgement
carries the danger of "hindsight bias": not making sufficient
allowance for the fact that those involved in the event will not
have had the advantage of perceiving the situation with full (retrospective)
knowledge or the luxury of time to reflect. (However, it is arguably
not "hindsight bias" to regard harm as avoidable when
it occurs due to lack of foresight, i.e. ignorance of factors
that are known to contribute to unsafe care.)
41. Secondly, the criteria used to determine avoidability
can vary between studies and consistency can be hard to achieve
even among researchers in the same study. One study concluded
that "preventability is in the eye of the reviewer".[34]
A study of a hospital, oriented towards discovering cases of potential
medical negligence, in Massachusetts found an adverse rate of
2.7%; when the same notes were examined using a different approach
to assess all forms of harm to patients the rate was put at 11%.
42. Thirdly, as well as the problems in determining
preventability, there are particular issues relating to deaths
associated with unsafe care. Often, these are the deaths of patients
who were very ill with an already poor prognosis regarding their
original illness or condition. In such circumstances, determining
whether a death counts as an "excess death" (one attributable
wholly to the adverse event) is difficult, and sometimes, impossible.[35]
43. At various times over the past decade, the NPSA,
the DH and others have cited estimates of the extent of harm to
NHS patients in England, extrapolated either from the case note
review studies in England or from similar studies conducted overseas.
These extrapolations have suggested that the NHS inadvertently
kills tens of thousands of patients each year. Figures generated
in this way are questionable given the problems discussed above.
44. Despite the limitations of case note review data,
and the Department's failure to ensure the production of better
data, it is clear that a great deal of harm is done to patients
in the NHSas it is in all healthcare systems. The international
studies mentioned above put the rate of adverse events in acute
care at between 2.7% and 16.6%. Even at the lower end of the range
this represents a huge amount of harm given the millions of patients
treated.
45. Case note reviews are also an important source
of data at the local level. During our visit to Luton and Dunstable
NHS Foundation Trust we learned how the Trust had made effective
use of case note review data to monitor its progress in improving
patient safety. Each month, the Trust scrutinises a random sample
of 20 case notes to compute an adverse event rate, which is then
reported to the Board and to staff as an indicator of how far
care is safe.
46. There are various streams of data on particular
aspects of patient safety in acute hospitals, for example, on
medication errors and healthcare-associated infections (HCAIs).
Cases of MRSA and Clostridium difficile are now reported
and the Office for National Statistics produces statistics regarding
deaths in which these two diseases are recorded causes. These
data do not provide a reliable, comprehensive picture of the problem
since there is likely to be underreporting of cases and under-registration
of deaths; they do, nevertheless, show that HCAIs are a major
concern.
47. Regarding harm to NHS patients that occurs outside
NHS acute hospitals, Professor Thomson told us that the available
evidence is even thinner. There have been no case note review
studies in primary care. A review of the international literature
on safety incidents in primary care found that studies showed
"there might be errors occurring in anywhere between five
and 80 in every 100,000 consultations". But this cannot be
taken as a reliable indication of harm to patients in primary
care in the NHS in England. According to Professor Thomson:
there are a number of other measures that indicate
that there is an issue in primary care. For example, about one
in 25 patients admitted to hospital in some studies are shown
to have been admitted because of a medication problem. That implies
that there are clearly issues around medication safety in primary
care that could be addressed. Surveys of patients by the Commonwealth
Fund have suggested that maybe as many as six in 100 report that
they have experienced some sort of error in medication over the
last two years.[36]
However, this information too is inadequate for determining
exactly how many patients are harmed and in what ways.
48. In respect of harm to NHS patients from treatment
by independent-sector providers, Professor Thomson told us that
nothing is known: "That is an area really where there is
very little reliable data at all at the moment. I do not think
we have anything that could give us an overarching figure for
that at all."[37]
The HCC told us, regarding data on quality that Independent Sector
Treatment Centres were expected to provide, that "the level
of returns and data quality were very poor" in 2007 although
"there has been some improvement since". Also, for reports
on adverse events, "data quality can be poor and the level
of reporting from different providers of the same type varies
greatly".[38]
What does harm to patients cost
the NHS?
49. When we asked about the cost to the NHS in England
of harm to patients, the NHS Medical Director referred us to the
sum of £633 million[39]
paid out by the NHS Litigation Authority (NHSLA) in settlement
of clinical negligence claims in 2007-8.[40]
It is clear that this figure does not indicate the true cost,
since it relates only to litigation over the NHSLA threshold and
omits so many of the costs relating to avoidable harmnot
least the cost of additional medical treatment.
50. Other figures which attempt to give truer estimates
of the cost to the NHS are also incomplete and unreliable but,
even so, leave little doubt that the cost is huge. Mr Fletcher
of the NPSA referred to work done by the NAO in which Trusts were
asked to estimate the cost of unsafe care; this had yielded figures
for individual Trusts ranging between £88,000 and £400,000
per year. He also cited work by the NPSA on patient falls, which
were estimated to cost the NHS in England and Wales around £15
million per year.[41]
51. Alastair Gray, Professor of Health Economics
and Director of the Health Economics Research Centre in the University
of Oxford, told us that, according to an extrapolation from the
first hospital case note review study in England, additional bed-days
due to preventable adverse events could be costing "approximately
£1bn" per year.[42]
Such data are subject to the limitations relating to case note
reviews referred to above, but are at least based on a systematic
assessment of medical notes.
52. The CNO informed us that the annual cost of additional
bed-days due to healthcare-associated infections was in the region
of £1 billion,[43]
although this is an extrapolation from a single-hospital study
that used data from 1994-5.[44]
53. Dr Alison Holmes, the Director of Infection Prevention
and Control, and a Consultant in Infectious Diseases, at Imperial
College Healthcare NHS Trust, spelt out to us the need for better
data on the cost of healthcare-associated infections:
I think we are struggling here. We really do
need some very good economic evaluations that are not just about
bed days but also the impact in primary care, and I think we really
do have to move towards thinking: this is 'healthcare' acquired
infection, it is across a whole patient journey, and we should
be working very closely across primary and acute care. So economic
evaluation needs to consider not just what is happening in the
hospital but the impact on people's lives and in primary care.
We also have to use data that is useful, that actually comes from
the UK and, particularly, that is relevant to whether you are
a large teaching hospital or a smaller [District General Hospital].
It would be useful to make economic models based on English or
UK data.[45]
54. Another area of cost is that relating to adverse
events due to medication errors, which the NPSA has estimated
at £774 million per year.[46]
Professor Gray explained that this too is an extrapolated figure
the validity of which is far from certain:
That is an estimate which came from two main
sources. One is trying to get some handle on how many medication
errors occur amongst people already in hospital and the other
on what proportion of people are admitted to hospital as a result
of medication errors. Based on systematic reviews of literature
to get estimates of these rates plus studies of individual hospitals,
the figures were attached to these. Also, a small amount are based
on litigation costsI think it was a very small part of
that totaland that is how we got the £750 million
for medication errors.[47]
Conclusion
55. The evidence, particularly that from case
note reviews, both in England and internationally, indicates that
the extent of medical harm is substantial, even on a conservative
estimate, and that much is avoidable. International studies suggest
that about 10% of all patients who are admitted to hospital suffer
some form of harm. Judging how far patient safety policy has been
successful requires more reliable data regarding how much harm
is done to patients. Unfortunately, neither the NPSA nor the DH
was able to provide us with that. Government estimates of avoidable
harm and the attendant financial costs are extrapolations from
old, very limited, data; and no attempt has been made to produce
reliable up-to-date figures.
56. We remind the Department of the value of the
random case note review that was a part of Royal College inspections
for accreditation for training of junior doctors. We commend to
all hospitals the practice of conducting regular sample case note
reviews, as is done at Luton and Dunstable Hospital, to provide
a clear indicator of local performance in making services safer.
We recommend that the NPSA monitor progress by the NHS in improving
patient safety, using local sample case note peer review data
and other sources of information on harm to patients.
26 Q 3 Back
27
National Reporting and Learning System quarterly data, Issue 12,
May 2009 Back
28
Q 4 Back
29
Q 5 Back
30
Hospital case note review studies are also referred to as "record
review" studies. They are usually retrospective, i.e.
based on a study of records relating to completed treatment episodes. Back
31
Q 102 Back
32
Ev 2 Back
33
Charles Vincent et al., "Adverse events in British
hospitals: preliminary retrospective record review", British
Medical Journal, vol 322 (2001), pp 517-519; Ali Baba-Akbari
Sari et al., "Extent, nature and consequences of adverse
events: results of a retrospective casenote review in a large
NHS hospital", Quality and Safety in Health Care,
vol 16 (2007), pp 434-439 Back
34
Rodney Hayward and Timothy Hofer, "Estimating hospital deaths
due to medical errors: preventability is in the eye of the reviewer",
Journal of the American Medical Association, vol 286 (2001),
pp 415-420 Back
35
PS 80 Back
36
Q 102 Back
37
Q 103 Back
38
Ev 196 Back
39
NHS Litigation Authority, Factsheet 2: financial information,
November 2008 Back
40
Qq 27-29 Back
41
Q 30. See also Ev 6. Back
42
PS 81 Back
43
Qq 31-32 Back
44
PS 81; Rosalind Plowman et al., The Socio-economic Burden
of Hospital Acquired Infection: Executive Summary (Public
Health Laboratory Service, 1999) Back
45
Q 541 Back
46
National Patient Safety Agency, Safety in doses: medication
safety incidents in the NHS: The fourth report from the Patient
Safety Observatory, March 2007, Appendix 5, p 59 Back
47
Q 148 Back
|