HC 499 - Scientific advice and evidence in emergencies

Memorandum submitted by the
Royal Statistical Society (SAGE 30)

The Inquiry by the Science & Technology Committee (STC) has selected four case-studies, the first of which (the swine flu pandemic in 2009) the Royal Statistical Society took a close interest in. The President, Professor David Hand FBA, wrote to the-then Chief Medical Officer, Sir Liam Donaldson, on 30 July 2009 because the Royal Statistical Society had identified a number of issues related to reporting standards and broad epidemics that weekly updates on the pandemic needed to address. The correspondence is at Annex A1. The Royal Statistical Society noted that improvements were subsequently made which we attribute to Sir Liam’s intervention.

The Royal Statistical Society wishes to make some general observations before addressing STC’s specific questions in relation to H1N1 (2009).

We are aware that the Government’s Chief Scientist, Professor Sir John Beddington FRS, has commissioned a review, which includes statistical matters, of the Government’s risk register, what have come to be known as ‘reasonable worst case planning assumptions’ (vide infra), and their calculation. We expect that the Royal Academy of Engineering will address the STC on cyber attacks and the vulnerabilities that arise if much of a country’s critical infrastructure is in private hands or if a government does not have significant information-technology expertise.

Swine-flu epidemic in 2009

A. General tenets

1. Attention must focus on quantifying the actual impact of events, which may be quite different from the impact that was planned for. The 2009 pandemic, instead of being a highly lethal variant of H5N1, was low lethality H1N1.

2. Even if it is reasonable to plan for a ‘worst-case scenario’, which is strongly disputable, planners (and public) should be have an operational understanding of the ‘reasonable worst-case scenario’: for intensive care unit (ICU) admissions, say. The reasonable worst case scenarios might carry meaning as follows: there is a 10% chance that actual ICU admissions during the pandemic will exceed the I10 planning level, a 1% chance that actual ICU admissions will exceed the I100 planning threshold, and 1 chance in 1,000 that actual ICU admissions will exceed the I1000 threshold. Governments have to decide what operational meaning, affordably, to attach to ‘reasonable worst-case scenario’.

3. Even if it were reasonable to plan for the I100 ‘worst-case scenario’, this scenario should not be computed by taking all contributing parameters (H1N1 clinical attack rate * hospitalisation rate of clinically affected H1N1 cases * ICU-admission rate for hospitalised H1N1 cases) at their ‘worst-cases’! From the standpoint of statistical science, this could be argued to have been poor risk analysis, even without the benefit of hindsight.

4. If the Government’s prior pandemic assumption was a 1% case fatality rate, and there have been 4,000 confirmed cases to date with only 10 deaths (versus the 40 deaths expected if the prior assumption had held true), then these few deaths constitute compelling evidence that UK’s actual case fatality rate is substantially below 1%. Thus, low numbers of deaths can be highly informative for rejecting a prior judgment about a relatively high case fatality rate.

5. The Royal Statistical Society regrets that, in the first pandemic of the 21st century, England’s Chief Medical Officer was obliged to make extraordinary personal effort to establish the number of H1N1-related deaths because of a lacuna in England’s registration system. In England, unlike in Scotland, there is no obligation for the fact, and date, of death to be registered within so many days of a death having been ascertained if the death is referred to coroners. This loop-hole needs to be closed for there to be effective monitoring of the lethality of epidemics – whether H1N1, heroin, cocaine or mephedrone.

6. The term ‘excess deaths’, as used by statisticians, is unhelpful when it comes to public understanding of epidemics. Rightly, the public appreciates that if a new virus, such as H1N1, causes deaths then these are ‘excess’ deaths in the sense that, but for H1N1, they would not have occurred. Confusingly, statisticians use the term ‘excess deaths’ to refer to the difference between the actual number of deaths (say, in week 40 of 2009) and the number that, by comparison with the past decade (say), would have been expected in week 40. Whether the weekly deviations between actual and historically-expected deaths were consistent with year-to-year random fluctuation or sufficiently extreme to alert to impact from the pandemic (for example, deviant more than 1.6 standard deviations) was an aspect of pandemic monitoring that the Royal Statistical Society had expected to see conducted by age-group and in the public domain. The above lacuna in the registration of deaths in England was, of course, a complication but lack of transparency about age-specific monitoring of deaths was both puzzling and disquieting.

7. For H1N1, the Scientific Advisory Group in Emergencies (SAGE) in England had access to three modelling teams within the Department of Health, the Health Protection Agency, and the Medical Research Council/Imperial College; and in Scotland to the modelling team within Health Protection Scotland/University of Strathclyde. The four teams contained deep expertise at an international level. However, data, methods and communications of these groups were kept very tightly under control to an extent that was undermining of them as apparently conflicting with expectations on independent scientific advice. Future epidemic analysis for a SAGE could be opened up to more external scientific scrutiny (see also: Dame Deirdre’s report), and particularly so when SAGE has taken over from an independent scientific advisory committee (Scientific Pandemic Influenza Advisory Committee).

8. The membership of SAGE is, of course, itself constituted to offer scientific challenge. In particular, statistical challenge to epidemic modelling requires transparency about the extent - or paucity - of empirical data or expert judgments which are used as inputs to infectious disease models. In particular, it is never sufficient to be told a percentage (e.g. 11%) without also knowing either the numerator/denominator from which it was derived (e.g. 1/9 or 110/1,000) or the standard error that qualifies the estimate. Only by being told explicitly about data limitations can SAGE take action to remedy any deficiencies in surveillance designs or other data capture.

9. In addition to adherence to robust statistical reporting standards, the Royal Statistical Society commends to the attention of STC the benefits of:

representative sampling: i) to reduce burden, for example if a laboratory can test only 200 samples per day, but 1,000 are referred in, then a suitable random sampling scheme is needed by which to select 200 for testing; ii) for surveillance, for example to estimate unbiasedly the weekly age-group-specific proportion of patients who contact their doctor about an influenza-like-illness who have H1N1.

sample size that is fit-for-purpose: precision in estimation depends on sample size and, in general, to increase precision by a factor of two, sample size has to increase by a factor of four.

randomization: i) for fair allocation of scare resources (if two, equally-eligible patients need the last available ICU bed, determine the admission by randomization); ii) for like-with-like comparison to learn efficiently and defendably which of several possible treatments of a novel disease is the best choice.

10. The Royal Statistical Society considers that fast-track refereeing of research protocols, fast-track licensing for pandemic vaccines, and fast-track ethical clearance for pandemic research studies were excellent initiatives, to which fast-tracked administrative clearances might be added in future to minimize delays in initiating studies that had research and ethical clearances. Although the European Medicines Authority heightened its licensing barrier when the pandemic virus turned out not to be a highly lethal variant of H5N1, uptake of H1N1 vaccines by healthcare professionals and the general public was limited. Did expedition in licensing on the basis of size-limited randomized controlled trials of immune-reaction inadvertently undermine trust in the vaccines’ safety when individuals came to make their own risk-benefit assessments?

B. Answers to specific questions by Science and Technology Committee

Q1a Potential hazards and risks and how were they identified? The UK experienced three pandemics in the 20th century, and so one or more pandemic influenzas in the 21st century were to be expected. Worse, highly lethal H5N1(A) influenza had been transmitted from birds to humans. Its human-to-human transmissibility was low but that could change by mutation, or if H5N1 assorted with a highly transmissible influenza virus in birds, pigs or man. In fact, the pandemic arose from H1N1 not H5N1, and in the West (Mexico) not the East, and had low, not high, lethality. Characterisation of influenza – prior immunity, transmissibility (Ro initially and consequent on social distancing/prophylaxis/treatment), age-specific hospitalisation and death rates per 100 clinical cases and prior risk factors for infection or hospitalisation had been addressed. With hindsight, too little attention had been paid to the apparently ‘known’ risks to pregnant women – yet international variation in whether seasonal influenza vaccine is advocated for pregnant women suggests conflicted prior beliefs.

Q1b How prepared was the Government for the emergency? By international judgement, the UK’s preparedness was highly acclaimed, and looked to. The UK’s preparedness in scientific, vaccine, clinical, and infectious disease modelling terms was indeed very well done: provision had been made to ensure that the UK would not be short of antivirals, antibiotics, or vaccine. Provision was made that capacity for intensive care unit admissions could be doubled and, by introduction of the National Pandemic Flu Service, that general practitioners could concentrate on the more seriously ill of their patients. Provision was made for cancellation of elective surgery and modelling studies, notably by the team at the Medical Research Council/Imperial College, had about exhausted what could be learned from historical pandemics and had investigated the impact of routine (that is: pre-planned) school closures on influenza transmission in France.

The extent of the UK government’s preparedness for high impact pandemic may mean that the Government seemed to react rather slowly to the more mild profile of H1N1, of which its cleaving overlong to reasonable worst case planning assumptions rather than evolving projections (with uncertainty) of the H1N1 epidemic seemed symptomatic.

Q2. How did the Government use scientific advice and evidence to identify, prepare for and react to the emergency? Please see answer to Q1b. In addition, the UK had an independent Scientific Pandemic Influenza Advisory Committee (SPI) and a predecessor committee, and research funders (notably: Medical Research Council, Wellcome, and NIHR) had given strategic priority to pandemic-related research. Unusually, however, the SPI was, in effect, stood down on 4 May 2009 and did not meet thereafter until 10 September 2010. Formerly SPI subcommittees worked to SAGE but their remit as a subcommittee of an independent scientific advisory committee was, in effect, in abeyance.

In terms of statistical science, some key surveillance designs (such as the FF100 cases and contacts by which initial Ro was to be determined) were not subject to independent peer-review ahead of H1N1; other surveillances were scientifically reviewed but sample sizes remained inadequate for their national purpose despite review; and there was inconsistency across hospitals in England about the virological testing of patients who were hospitalised for suspect H1N1. Monitoring of mortality by week and age-group (1-4 years and 5-14 years) in 2009 against historical expectations was not brought into public view until June 2010.

Q3a What were the obstacles to obtaining reliable, timely, scientific advice and evidence to inform policy decisions? Please see answer to Q2 and General Tenets above.

Q3b Has the government sufficient powers and resources to overcome the obstacles? Please see General Tenets.

Q3c Was there sufficient and timely scientific evidence to inform policy decisions? The Royal Statistical Society is unsighted on SAGE’s decision-making. However, Dame Deirdre’s review suggests that key decisions re H1N1 had to be taken in June 2009 on the basis of judgment rather than data.

Q4 How effective was strategic co-ordination? Please see General Tenets. Timely steps were not taken to improve representativeness of samples (and their number) in England on which weekly estimates were based of age-group-specific H1N1 proportion among those who consulted doctors about an influenza-like-illness.

Q5 How important was international co-ordination? Statistical reporting standards needed to be improved internationally to include when each nation ceased routine testing for H1N1. However, the excellent documentation by intensive care specialists of the impact on admissions to intensive care units in Australia and New Zealand of their first winter wave of pre-vaccination H1N1 was crucial for calibrating the UK’s expectations of its second wave of H1N1.

Royal Statistical Society

14 September 2010