Appendix 5: The polling industry in the UK
What is the scale of the industry?
According to the Market Research Society, the UK is the second largest research market in the world, second only to the United States of America. In terms of research spend per head of population, the UK is the largest, with £61 per capita in 2015 (compared to £39 in the United States, £24 in Germany and £23 in France). The UK research industry is a £4bn market and has grown steadily over the last five years by an average of 6% per year. Based on its own assessment of the size and impact of the UK research and evidence market, the MRS stated that the UK ‘business of evidence’ market employs up to 73,000 people and generates £4.8 billion (in annual gross value added). It also told us that data analytics exhibits the highest growth rate at over 350% growth since 2012.296
Despite the size and scale of the UK research market, political opinion polling makes up only a fraction of the revenue taken by polling organisations. The Market Research Society told us that political opinion polling, although highly visible, represents only a small sub-set of the wider research sector, accounting for about 1% of work undertaken outside of a General Election.297 Election polling was described by many of the witnesses as a “shop front”298 for polling organisations—an activity aimed at increasing their public profiles and advertising their accuracy, but which brings in comparatively little money and is often done at a discount.299 Johnny Heald, Managing Director of ORB International, told us: “The notion that we are getting fat on political opinion polling is not true at all. The newspapers do not pay for opinion polling now.”300
How is polling conducted?
The key concern for polling organisations is to get a sample of people that is as representative as possible of the target population as a whole. The approach to acquiring the sample can have a significant impact on the poll’s accuracy. There are two main methods of sampling:
- Random sampling: where members of the target population are selected at random, such that all members of the target population have a known, non-zero probability of selection. Only randomly selected members of the target population can be included in the sample, substitution is not permitted.
- Quota sampling: This method involves ensuring that the sample matches the target population according to known characteristics of the population. For example, the census provides information on the general population distribution for age, sex, and region. Polling companies use this information to fill ‘quotas’ in the sample such that the sample matches the population on these characteristics. Quota sampling can be implemented in any interview mode, including online. YouGov, for instance, has a panel of volunteers who have agreed to complete surveys. The panel of volunteers complete detailed questionnaires about their demographic characteristics, which can be used to fill quotas of the desired characteristics to take part in particular polls.301
Both random and quota sampling are susceptible to sampling bias—failing to achieve a sample of people that is representative of the target population on the characteristics of interest.
In order to address sampling bias, polling organisations employ a technique known as weighting. Weighting works in the same way as quota sampling—the sample is made to ‘match’ the target population according to known characteristics of the population. The data are adjusted such that groups which are under-represented in the sample are weighted up to match their prevalence in the population, while over-represented groups in the sample are similarly down-weighted. Weights will be derived using characteristics like gender, age, region, social class, level of education, housing tenure, and work status.302
Most opinion polls report a ‘margin of error’ for their estimates that arise due to sampling variability. The margin of error expresses the range of plausible values in the population for the characteristics estimated in the poll, given the sample size of the poll. For polls conducted using random sampling, the margin of error can be calculated using standard statistical formulae. However, since the vast majority of UK published polls use quota sampling, the margin of error is usually a ‘rule of thumb’ of + or–3% of the point estimate. This rule of thumb is based on the assumption of a simple random sample of approximately 1,000 respondents. It can be (approximately) interpreted as indicating that, over a very large number of replications of the same sample design, it is expected that only 5% of samples will produce an estimate outside the margin of error of the estimate.
297 Ibid.
301 House of Lords Library, Understanding and Sourcing Political Opinion Polls, Library Note,
LLN 2014/028, August 2014
302 UK Polling Report, Weighting: http://ukpollingreport.co.uk/faq-weighting [accessed 20 March 2018]