Memorandum submitted by Mr Trevor Lawson
As a freelance journalist I have been covering
the issue of bovine TB and the subsequent Krebs' experiment for
several years. I write for The Guardian, Daily Express
and other publications, and I am currently completing a feature
on the issue for BBC Wildlife magazine. I write to draw
the attention of the Committee to a number of concerns I have
been unearthing about the current Krebs' trial.
Last year, I wrote to draw the Committee's attention
to the problems I had had in securing adequate information about
the statistical validity of the trial. During the course of this
year, my concern has grown, but key questions remain unanswered.
Below, I recount for the Committee's interest some of my e-mail
exchanges with Dr Christl Donnelly and Sir David Cox who helped
develop the statistics of the current trial.
On 19 May 2000, Dr Donnelly stated that:
Our assumption is that the number of observed
cattle breakdowns follows a Poisson distribution. Such a distribution
would arise if the cattle breakdowns occurred totally at random.
I was struck by this. It is clear from the evidence
available that bovine TB incidents are anything but random. They
occur in the same placesTB hotspotsand consistently
strike certain farms over time. On 22 May, I asked Dr Donnelly
the following questions (this e-mail is edited down):
My impression of the bTB outbreaks is that
they are not random. Indeed, the trial focuses on areas known
as hotspots. Am I therefore right in thinking that this assumption
is the basis for your null hypothesis?
Second, if this is so, I am uncertain how
you address the issue of repeat breakdowns. I note you state that:
"the key quantities are the total number of TB breakdowns
in the survey only (control) areas". The Poisson distribution
seeks to show random (and therefore unrelated?) events. But in
hotspots breakdowns are not unrelated. They can occur repeatedly
on the same farms. They are anything but random. Do you treat
repeat breakdowns on the same farm as new events which are unconnected
with previous breakdowns, which add to this key "total number
of TB breakdowns"? If so, what effect does this have on the
power of the experiment?
It also occurs to me that the hotspots are
sometimes related spatially. Some trial areas are virtually adjacent
to one another, though they may be some distance from other trial
areas which are adjacent to one another. So, how do you deal with
the spatial relationship between adjacent trial areas in Cornwall/Devon
and Hereford/Gloucester, versus the non-clustered areas elsewhere.
The reason I ask is that clustering could reflect an underlying
factor, such as geology or soil type, and to discount it could
be discounting a significant contributor to the disease.
[Dr Donnelly had also said the statistics were
able to cope with "various types of non-compliance".
I asked what she meant by this:]
But what about other issues which I shall
call "non-compliance", such as significant shifts in
weather patterns, changes in husbandry practices (new feedstuffs,
grazing regimes, antibiotics, and changes in the number of cattle
movements (per week, month or yearpick a period)? What
are your underlying assumptions here? Do you assume they are constant
or simply not relevant?
Having asked these questions, I was then astonished
to receive this response from Sir David Cox in Oxford's Nuffield
College and Department of Statistics, via Dr Donnelly. I have
numbered his paragraphs. He wrote on 23 May 2000:
(1) Dear Christl, Thank you very much
for keeping me in touch with your correspondence with Trevor Lawson.
I don't know Mr Lawson. He asks some interesting questions but
I do wonder if he really understands the nature and purpose of
power calculations. As I see them they are important indeed essential
assessments beforehand of the size of study that seems sensible
and, especially as in the present case there was no comparable
study to work from, they are absolutely inevitably based on simplifying
working assumptions. The answer is in any case general guidance;
who is to say what level of power is appropriate anyway other
than by general judgement. The assumption of a Poisson distribution
is surely the correct one to make although to some extent probably
a best case scenario is unlikely to be sufficiently far off to
(2) The really crucial point is that the
calculation shows that 10 triplets will give a reasonable level
of precision, neither absurdly over precise implying a waste of
recourses nor something so imprecise as to be useless. Another
crucial aspect is that the power calculations are assessments
beforehand. The precision actually achieved will be found from
the data when we have them; the power assumptions and calculations
play no role in that. Maybe the initial calculations will prove
a shade pessimistic and higher precision than expected will be
achieved or maybe the contrary which will be disappointing but
not disastrous. It is a quite pointless waste of time fretting
now over details of the power calculations, important though they
were two years ago. No doubt the forthcoming statistical audit
will look at power but hopefully spend most time on analysis methods.
(3) The use of hot spots was essential
for reasoned economy; the Poisson assumption applies to the comparison
of the three regions forming a triplet not to comparisons across
a broader region and the purpose of the randomisation across the
three areas forming a triplet is precisely to exploit inter-area
differences to enhance precision.
(4) Non-compliance in these contexts has
a reasonably precise technical meaning any failure of an individual
trial area to follow the regime to which it has been allocated.
I am about to follow up these comments with
Sir David, but he was right to say that I did not understand the
nature and purpose of power calculations. My understanding now
gives me greater cause for concern.
It appears from Sir David's first paragraph
(1) the underlying assumptions on which the statistics are based
are simple. Let us take one of those assumptions, as specified
in the Krebs report: all badgers will be killed in proactive areas.
It is clear that not all badgers are being killed. It is also
apparent that there is no certain way of assessing badger density
before culling or afterwards, and therefore what impact culling
is making on badger numbers. I suggest that these errors in assumptions
will affect the statistical reliability of the trial.
Sir David's second paragraph reveals that (2)
the statisticians have no idea how accurate the trial's results
will be until they get the data. I find this incredible. From
the outset, members of the press and the public have been led
to believe that the trial would last five years and that would
be that. I suspect that Ministers have been told the same thing.
The trial had a budget and a finite lifespan. There has been some
slippage in delivery, but there has been no suggestion that the
trial might need to continue for many more years if the data gathered
are inadequate. If the assumptions weaken the data too much, MAFF
will either have to abandon the trial with no useful results,
or carry on culling for an unknown period of time until enough
data are gathered. The implications for the public purse are massive.
Sir David's third paragraph is interesting,
since it suggests (3) that the data gathered will only give us
meaningful answers about bovine TB in the triplet areas. Because
hot spots have been selected for reasons of economy, the data
are specifically skewed and, I imagine, will tell us nothing reliably
about the wider countryside. Given that a supposed fear is that
the TB "epidemic" will continue to grow, it is surprising
that the trial will not be able to confirm this.
I regret that Sir David's fourth paragraph still
fails to clarify what non-compliance means for the trial. However,
if we take the literal words of the Krebs report, with the aim
in proactive areas being to cull all badgers, then I suggest it
is certain that (4) the entire trial is non-compliant, because
not all badgers are being killed.
I apologise to members of the Select Committee
for this long presentation of evidence. However, if you are able
to throw any light on the true statistical reliability of the
trial it would be welcomed by member of the press and, no doubt,
the farmers whose future depends on a solution to bovine TB.
27 October 2000