Memorandum submitted by the Royal Statistical
Society (SAGE 30)
The Inquiry by the Science & Technology
Committee (STC) has selected four case-studies, the first of which
(the swine flu pandemic in 2009) the Royal Statistical Society
took a close interest in. The President, Professor David Hand
FBA, wrote to the-then Chief Medical Officer, Sir Liam Donaldson,
on 30 July 2009 because the Royal Statistical Society had identified
a number of issues related to reporting standards and broad epidemics
that weekly updates on the pandemic needed to address. The correspondence
is at Annex A.[52]
The Royal Statistical Society noted that improvements were subsequently
made which we attribute to Sir Liam's intervention.
The Royal Statistical Society wishes to make
some general observations before addressing STC's specific questions
in relation to H1N1 (2009).
We are aware that the Government's Chief Scientist,
Professor Sir John Beddington FRS, has commissioned a review,
which includes statistical matters, of the Government's risk register,
what have come to be known as "reasonable worst case planning
assumptions" (vide infra), and their calculation. We expect
that the Royal Academy of Engineering will address the STC on
cyber attacks and the vulnerabilities that arise if much of a
country's critical infrastructure is in private hands or if a
government does not have significant information-technology expertise.
SWINE-FLU
EPIDEMIC IN
2009
A. General tenets
1. Attention must focus on quantifying the
actual impact of events, which may be quite different from the
impact that was planned for. The 2009 pandemic, instead of being
a highly lethal variant of H5N1, was low lethality H1N1.
2. Even if it is reasonable to plan for
a "worst-case scenario", which is strongly disputable,
planners (and public) should be have an operational understanding
of the "reasonable worst-case scenario": for intensive
care unit (ICU) admissions, say. The reasonable worst case scenarios
might carry meaning as follows: there is a 10% chance that actual
ICU admissions during the pandemic will exceed the I10 planning
level, a 1% chance that actual ICU admissions will exceed the
I100 planning threshold, and 1 chance in 1,000 that actual ICU
admissions will exceed the I1000 threshold. Governments have to
decide what operational meaning, affordably, to attach to "reasonable
worst-case scenario".
3. Even if it were reasonable to plan for
the I100 "worst-case scenario", this scenario should
not be computed by taking all contributing parameters (H1N1 clinical
attack rate * hospitalisation rate of clinically affected H1N1
cases * ICU-admission rate for hospitalised H1N1 cases) at their
"worst-cases"! From the standpoint of statistical science,
this could be argued to have been poor risk analysis, even without
the benefit of hindsight.
4. If the Government's prior pandemic assumption
was a 1% case fatality rate, and there have been 4,000 confirmed
cases to date with only 10 deaths (versus the 40 deaths expected
if the prior assumption had held true), then these few deaths
constitute compelling evidence that UK's actual case fatality
rate is substantially below 1%. Thus, low numbers of deaths can
be highly informative for rejecting a prior judgment about a relatively
high case fatality rate.
5. The Royal Statistical Society regrets
that, in the first pandemic of the 21st century, England's Chief
Medical Officer was obliged to make extraordinary personal effort
to establish the number of H1N1-related deaths because of a lacuna
in England's registration system. In England, unlike in Scotland,
there is no obligation for the fact, and date, of death to be
registered within so many days of a death having been ascertained
if the death is referred to coroners. This loop-hole needs to
be closed for there to be effective monitoring of the lethality
of epidemics—whether H1N1, heroin, cocaine or mephedrone.
6. The term "excess deaths", as
used by statisticians, is unhelpful when it comes to public understanding
of epidemics. Rightly, the public appreciates that if a new virus,
such as H1N1, causes deaths then these are "excess"
deaths in the sense that, but for H1N1, they would not have occurred.
Confusingly, statisticians use the term "excess deaths"
to refer to the difference between the actual number of deaths
(say, in week 40 of 2009) and the number that, by comparison with
the past decade (say), would have been expected in week 40. Whether
the weekly deviations between actual and historically-expected
deaths were consistent with year-to-year random fluctuation or
sufficiently extreme to alert to impact from the pandemic (for
example, deviant more than 1.6 standard deviations) was an aspect
of pandemic monitoring that the Royal Statistical Society had
expected to see conducted by age-group and in the public domain.
The above lacuna in the registration of deaths in England was,
of course, a complication but lack of transparency about age-specific
monitoring of deaths was both puzzling and disquieting.
7. For H1N1, the Scientific Advisory Group
in Emergencies (SAGE) in England had access to three modelling
teams within the Department of Health, the Health Protection Agency,
and the Medical Research Council/Imperial College; and in Scotland
to the modelling team within Health Protection Scotland/University
of Strathclyde. The four teams contained deep expertise at an
international level. However, data, methods and communications
of these groups were kept very tightly under control to an extent
that was undermining of them as apparently conflicting with expectations
on independent scientific advice. Future epidemic analysis for
a SAGE could be opened up to more external scientific scrutiny
(see also: Dame Deirdre's report), and particularly so when SAGE
has taken over from an independent scientific advisory committee
(Scientific Pandemic Influenza Advisory Committee).
8. The membership of SAGE is, of course,
itself constituted to offer scientific challenge. In particular,
statistical challenge to epidemic modelling requires transparency
about the extent—or paucity—of empirical data or expert
judgments which are used as inputs to infectious disease models.
In particular, it is never sufficient to be told a percentage
(eg 11%) without also knowing either the numerator/denominator
from which it was derived (eg 1/9 or 110/1,000) or the standard
error that qualifies the estimate. Only by being told explicitly
about data limitations can SAGE take action to remedy any deficiencies
in surveillance designs or other data capture.
9. In addition to adherence to robust statistical
reporting standards, the Royal Statistical Society commends to
the attention of STC the benefits of:
representative sampling: i) to reduce
burden, for example if a laboratory can test only 200 samples
per day, but 1,000 are referred in, then a suitable random sampling
scheme is needed by which to select 200 for testing; ii) for surveillance,
for example to estimate unbiasedly the weekly age-group-specific
proportion of patients who contact their doctor about an influenza-like-illness
who have H1N1.
sample size that is fit-for-purpose: precision
in estimation depends on sample size and, in general, to increase
precision by a factor of two, sample size has to increase by a
factor of four.
randomization: i) for fair allocation
of scare resources (if two, equally-eligible patients need the
last available ICU bed, determine the admission by randomization);
ii) for like-with-like comparison to learn efficiently and defendably
which of several possible treatments of a novel disease is the
best choice.
10. The Royal Statistical Society considers
that fast-track refereeing of research protocols, fast-track licensing
for pandemic vaccines, and fast-track ethical clearance for pandemic
research studies were excellent initiatives, to which fast-tracked
administrative clearances might be added in future to minimize
delays in initiating studies that had research and ethical clearances.
Although the European Medicines Authority heightened its licensing
barrier when the pandemic virus turned out not to be a highly
lethal variant of H5N1, uptake of H1N1 vaccines by healthcare
professionals and the general public was limited. Did expedition
in licensing on the basis of size-limited randomized controlled
trials of immune-reaction inadvertently undermine trust in the
vaccines' safety when individuals came to make their own risk-benefit
assessments?
B. Answers to specific questions by Science
and Technology Committee
Q1a Potential hazards and risks and how were
they identified? The UK experienced three pandemics in the
20th century, and so one or more pandemic influenzas in the 21st
century were to be expected. Worse, highly lethal H5N1(A) influenza
had been transmitted from birds to humans. Its human-to-human
transmissibility was low but that could change by mutation, or
if H5N1 assorted with a highly transmissible influenza virus in
birds, pigs or man. In fact, the pandemic arose from H1N1 not
H5N1, and in the West (Mexico) not the East, and had low, not
high, lethality. Characterisation of influenza—prior immunity,
transmissibility (Ro initially and consequent on social distancing/prophylaxis/treatment),
age-specific hospitalisation and death rates per 100 clinical
cases and prior risk factors for infection or hospitalisation
had been addressed. With hindsight, too little attention had been
paid to the apparently "known" risks to pregnant women—yet
international variation in whether seasonal influenza vaccine
is advocated for pregnant women suggests conflicted prior beliefs.
Q1b How prepared was the Government for the
emergency? By international judgement, the UK's preparedness
was highly acclaimed, and looked to. The UK's preparedness in
scientific, vaccine, clinical, and infectious disease modelling
terms was indeed very well done: provision had been made to ensure
that the UK would not be short of antivirals, antibiotics, or
vaccine. Provision was made that capacity for intensive care unit
admissions could be doubled and, by introduction of the National
Pandemic Flu Service, that general practitioners could concentrate
on the more seriously ill of their patients. Provision was made
for cancellation of elective surgery and modelling studies, notably
by the team at the Medical Research Council/Imperial College,
had about exhausted what could be learned from historical pandemics
and had investigated the impact of routine (that is: pre-planned)
school closures on influenza transmission in France.
The extent of the UK government's preparedness
for high impact pandemic may mean that the Government seemed to
react rather slowly to the more mild profile of H1N1, of which
its cleaving overlong to reasonable worst case planning assumptions
rather than evolving projections (with uncertainty) of the H1N1
epidemic seemed symptomatic.
Q2. How did the Government use scientific
advice and evidence to identify, prepare for and react to the
emergency? Please see answer to Q1b. In addition, the UK had
an independent Scientific Pandemic Influenza Advisory Committee
(SPI) and a predecessor committee, and research funders (notably:
Medical Research Council, Wellcome, and NIHR) had given strategic
priority to pandemic-related research. Unusually, however, the
SPI was, in effect, stood down on 4 May 2009 and did not meet
thereafter until 10 September 2010. Formerly SPI subcommittees
worked to SAGE but their remit as a subcommittee of an independent
scientific advisory committee was, in effect, in abeyance.
In terms of statistical science, some key surveillance
designs (such as the FF100 cases and contacts by which initial
Ro was to be determined) were not subject to independent peer-review
ahead of H1N1; other surveillances were scientifically reviewed
but sample sizes remained inadequate for their national purpose
despite review; and there was inconsistency across hospitals in
England about the virological testing of patients who were hospitalised
for suspect H1N1. Monitoring of mortality by week and age-group
(one to four years and five to 14 years) in 2009 against historical
expectations was not brought into public view until June 2010.
Q3a What were the obstacles to obtaining
reliable, timely, scientific advice and evidence to inform policy
decisions? Please see answer to Q2 and General Tenets above.
Q3b Has the government sufficient powers
and resources to overcome the obstacles? Please see General
Tenets.
Q3c Was there sufficient and timely scientific
evidence to inform policy decisions? The Royal Statistical
Society is unsighted on SAGE's decision-making. However, Dame
Deirdre's review suggests that key decisions re H1N1 had to be
taken in June 2009 on the basis of judgment rather than data.
Q4 How effective was strategic co-ordination?
Please see General Tenets. Timely steps were not taken to improve
representativeness of samples (and their number) in England on
which weekly estimates were based of age-group-specific H1N1 proportion
among those who consulted doctors about an influenza-like-illness.
Q5 How important was international co-ordination?
Statistical reporting standards needed to be improved internationally
to include when each nation ceased routine testing for H1N1. However,
the excellent documentation by intensive care specialists of the
impact on admissions to intensive care units in Australia and
New Zealand of their first winter wave of pre-vaccination H1N1
was crucial for calibrating the UK's expectations of its second
wave of H1N1.
Royal Statistical Society
14 September 2010
52 Not published. Back
|
