Select Committee on Agriculture Appendices to the Minutes of Evidence



Dr Fiona Mathews, Royal Society Dorothy Hodgkin Research Fellow, Department of Zoology, University of Oxford


  1.  It is increasingly recognised that many ecological experiments suffer from inadequate statistical power. Real and biologically important differences between groups are therefore likely to go undetected. A "statistically insignificant" result could reflect either the true lack of a treatment effect, or the inability of the study to detect an effect of this magnitude.

  2.  Although the results of a priori power and sample size calculations are only estimates, they do act as useful guides. However, as with any statistical method, the results will be compromised if the underlying assumptions do not hold true. Ecological studies are often complex to execute and analyse. For example, sampling units frequently border each other. Power and sample size calculations must reflect this reality as far as possible. There is a danger that erroneous calculations give "scientific" credence to studies that are too small or poorly designed.

  3.  The badger culling trial aims to determine whether the removal of badgers can decrease the incidence of bovine tuberculosis in cattle. Although the trial is controversial, there has been little scrutiny of the project's design. In part this may be due to sample size calculations which appear to show that the project will be able to detect important reductions in the incidence of TB in cattle within five years (p = 0.05, power = 0.9). However, the sample size calculations may have seriously over-estimated the study's statistical power, by failing to account for important aspects of the project's design.

  4.  The trial uses cluster randomisation, whereby groups of adjacent farms are allocated to the same treatment. Each unit within the cluster therefore cannot be treated as independent: farms within groups are likely to be more similar than farms in different treatment groups. Calculations using simulated data illustrate the potential impact on sample size: even were the variation between clusters only moderate, the sample size would need to be doubled to maintain the same statistical power. If the sample size remains unchanged, the chance of detecting a 20 per cent reduction in TB incidence would only be 66 per cent. With greater between-cluster variation, the sample size increase required is even greater.

  5.  The trial also assumes that TB occurs in accordance with a Poisson distribution: repeated breakdowns on the same farm, and breakdowns on adjacent farms are treated as independent. This assumption is unlikely to be justified. Under more realistic assumptions, the effective sample size is further reduced by up to a third.

  6.  In summary, the original sample size calculations appear incorrect. Rather than having a power of 90 per cent to detect a reduction in TB incidence of 20 per cent within five years, the true power is likely to be nearer to 50 to 60 per cent. This means that if culling badgers reduces cattle TB by up to 20 per cent, there is a 40 to 50 per cent chance that it would not be detected. Even were the effect of the badger cull greater, and cattle TB was reduced by as much as 25 per cent, there would still be less than an 80 per cent chance that the trial would give a "positive" result within five years.

  7.  Given the economic importance of bovine tuberculosis, the ethical implications of large-scale removal of protected wildlife, and the cost of the study, the badger culling trial requires urgent review.

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2001
Prepared 10 January 2001