Scientific advice and evidence in emergencies

Memorandum submitted by Sir Liam Donaldson (SAGE 44)

You asked me to write to elaborate my comments, made during evidence to the Committee, that modelling data were of less value in guiding the response than they had been in the advance planning phase.

During the pandemic, a team of clinical advisors under my direct supervision, carefully investigated and documented every death attributable to swine ‘flu. This was vital work. It allowed me to present accurate information to the public and media. It helped show that we had a good grasp on the evolving pandemic in the UK. The work has subsequently been published in the Lancet and the British Medical Journal.

Combining our emerging data with case estimates, produced by the Health Protection Agency, allowed my team to estimate a case fatality rate (the percentage of people who develop swine ‘flu and subsequently go on to die). From an early stage in the UK pandemic it was apparent that the case fatality rate was much lower than we had feared.

The first planning assumptions, released on 16 July 2009 from scientific modelling work, suggested a much higher case fatality rate than our own emerging data. At that time our data indicated a case fatality rate of around 0.05%, compared with the published figure of 0.1-0.35%. Extrapolating our figures to the UK population, using SPI’s assumptions around attack rates, would have led to a ‘best’ estimate of 9,000 deaths, and a ‘worst case’ scenario of 25,000 deaths (see attached table). Such estimates made intuitive sense. The virus was being described as ‘mild’, and the past two pandemics, prior to recent advances in intensive care medicine, had seen around 30,000 to 35,000 deaths in the UK.

It was important to make the right judgement on a sensible upper case fatality rate. It was also important not to give too much weight to the Mexico experience, with uncertainties about their data and a different healthcare system.

Modelling scientists will always quite properly say that their estimates depend on the nature and quality of the input data. Thus, they become more refined as time goes on and more real data accumulate. The problem is that the public and media perception is different. They take estimates at face value so that even with the caveats expressed, the failure of early estimates to match with later actual figures leads to criticism and sometimes ridicule of those communicating the data. In the pandemic I found the deaths data I gathered, used to provide insights into the behaviour of the pandemic, on a rule of thumb basis, was giving me a fairer idea about severity from quite early on. The very high deaths scenarios seemed to me implausible. Such deaths data would not normally be available since ONS works in arrears and death certificates are not reliable on their own so the approach was novel. Understandably the modelling methods that had been carefully worked out over many years in the preparatory phase held scientific sway.

None of this is a criticism of the distinguished modellers who did the work just something that I feel needs to be reflected on. That is why before I left as CMO I established a Statistical Legacy Group whose report might be available soon.

Yours sincerely

Sir Liam Donaldson

Chairman

National Patient Safety Agency

23 November 2010

APPENDIX: Comparing the CMO’s team’s emerging data with SPI estimates of mortality.

Predictions of overall mortality for the UK have been estimated by applying the SPI attack rate and the case fatality estimated from the CMO dataset to the UK population. These are shown in table 1.

The case fatality rate (CFR) for the CMO data set has been calculated from the ‘best data’ available at the time. The total deaths in England as published in the CMO media brief on the Thursdays of 16 July 2009, 3 September 2009 and 22 October 2009 were used. These data were combined with the Health Protection Agency cumulative case estimates for the previous week, to produce case fatality estimates. The mid- and low- case estimates give a best and worst case estimate for overall mortality respectively. These calculations are shown in table 2.

The attack rates taken from the SPI documents to produce overall population mortality estimates are shown in table 3.

Table 1: Comparison of the predicted overall mortality based on the case fatality rate from the CMO data set with the predictions published by SPI

Worst Case

Best Estimate

SPI

CMO

SPI

CMO

16 July

65,000

25,000

23,000

9,000

3 September

20,000

9,000

5,000

2,000

22 October*

1,000

3,500

80-500

800

* 22 October estimates refer to the second wave only

Table 2: Calculation of Case Fatality Rate from CMO dataset

Cases

Case Fatality Rate

Deaths

Low

Mid

Worst

Best

16 July

26

50,000

20,000

0.13

0.05

3 September

61

280,000

120,000

0.05

0.02

22 October

93

435,000

195,000

0.04

0.02

Table 3: Attack rates estimated by SPI

Attack Rate

Worst Case

Best Estimate

16 July

30%

25% (20-30%)

3 September

30%

15% (10-20%

22 October*

12%

6% (less than 12%)

* 22 October estimates refer to the second wave only