Memorandum submitted by Philip Redfern
In the last 12 years of my 35 years in the Government
Statistical Service I was Deputy Director of the Office of Population
Censuses and Surveys (OPCS, now merged in the Office of National
Statistics (ONS)). In that post I was closely involved in the
censuses of 1971 and 1981. My book on censustaking based on a
study of practice in Europe and North America was published in
1987 by Eurostat (the Statistical Office of the EC). In July 2003
I read a paper to the Royal Statistical Society giving a critique
of the 2001 census.
1. The census of population serves many
ends but this memorandum focuses on its primary aim of measuring
the national population by sex and age and the analysis of the
national totals between Local Authority (LA) areas. These figures
are the base of a vast infrastructure of national statistics.
Reference throughout is to England and Wales (E&W).
2. The 2001 census successfully broke new
ground in several ways, but in terms of measuring the national
population it was probably the biggest statistical disaster since
the war, the scale of which has been played down by ONS. I reach
this conclusion on several grounds. First, the census is the most
important data collection exercise that any country undertakes,
costing us some £250 million at today's prices; it cannot
be repeated if it goes wrong. Second, in 2001 details of some
4 million people (7½% of the population) were not entered
on census forms a record number. Third, the 2001 census-based
population estimates have already been increased twice since first
publication in response to pressure from city authorities, but
further revision is needed, particularly to correct the sex balance
3. This memorandum identifies six major
failings in 2001 and considers what steps are being taken to rectify
them in 2011.
4. Three of the failings in 2001 were in
public view and one does not have to be a census expert to see
them. (1) The form asked many more questions than any previous
census, with adverse impact on public response. (2) In most places
postal delivery and collection of the forms replaced the enumerators
employed in previous censuses, so there was no one on the doorstep
to explain to householders what they had to do, to provide help
if needed or to check that the form had been reasonably completed
and handed back. (3) For the first time ever the form had to be
completed only for persons "usually living at this address".
This ambiguous instruction gave an opt-out to anyone claiming
to be usually resident somewhere elsean opt-out widely
exercised by young people. For this and other reasons some million
people placed themselves beyond the census reach. In previous
censuses the form had to be completed for everyone present at
the address on census night.
5. Looking to 2011, (1) the prospective
form will probably be as long and complex, and for some people
off-putting, as in 2001; (2) ONS seem content with postal delivery
and collection in most areas; and (3) the prospective form still
retains an opt-out to answer only 4 out of 27 questionsan
opt-out available to anyone present on census night who claims
not to be "usually living in this household". All three
factors will damage census quality.
6. The three other major failings in 2001
were "behind the scenes" and therefore hidden from the
public. (4) The population estimate based on a census has to embody
an adjustment for non-response, including dwellings, households
and persons missed. The adjustment process in 2001 inevitably
failed to make proper allowance for the million beyond the census
reach (4). And it introduced a sex bias a deficiency
of males relative to femaleswhich prima facie was
spread across all kinds of LA (32). If the 2011 form is cleansed
of all opt-outs, the task of adjusting for non-response will be
less but still very challenging.
7. (5) A census test and a dress rehearsal
before the 2001 census should have sensed the storms ahead, but
didn't. Moreover, several warning systems which ONS set up in
2001 to detect if things went wrong failed. ONS need to convince
us that 2011 will be different.
8. That leads to (6): demographic checks,
that is, checks comparing the census-based population estimates
with independent estimates usually built up from data on births,
deaths, migration and previous censuses. Such a check in some
form is an essential part of any census. In 2001 the demographic
check took the form of a comparison of the census-based population
estimate with the 1991 population estimate rolled forward from
year to year for 10 years. This did shine a red warning light:
it pointed to a shortfall in the 2001 census-based estimate of
1.1 million males. That was not far from the truth but ONS ignored
it: they decided that the "roll-forward" was in error
and that the 2001 census-based estimates were correct.
9. By rejecting the roll-forward to 2001,
ONS showed their lack of confidence in that form of demographic
check. I share that lack of confidence, particularly as we look
to 2011 and note the growing migratory flows that are components
of the roll-forward arithmetic.
10. Demographic checks are difficult territory
even for census experts. I have developed and put to ONS two kinds
of check that do not rely on uncertain figures of migratory flows
and on earlier population estimates. The simpler of the two is
a calculation that is probably as near to a proof as one ever
gets in demography. It "proves" that the sex balance
(the ratio of males:females) in our 2001-based population estimates
needs to be corrected by adding ½ million males (32).
This error permeates the whole of our national infrastructure
of statistics, and will continue to do so until 2012 unless ONS
take action. The calculation was part of my paper to the Royal
Statistical Society in July 2003 and has been put to ONS twice
11. Demographic checks are a vital part
of censustaking, but a weak part of ONS's armoury calling for
12. Unless ONS act to correct the failings
of the 2001 census (1-6 above), the 2011 census will be little
better. The rest of this note deals with the six points in more
detail and puts forward five Recommendations that the Committee
might wish to consider.
I QUESTION CONTENT
13. The non-response rate in 2001 was high
compared with previous censuses: 6% overall and as high as 26%
in the City of Westminster according to ONS's first published
estimates, even though public participation is compulsory by law.
Many factors contributed. One was a long questionnaire which asked
36 questions of each persongetting on for twice the number
asked in 1991 (20), which in turn was 25% more than in 1981 (16).
14. The prospects for 2011 are not encouraging.
The Test Census to be taken in May asks 27 questions, but they
occupy more pages per person than the 36 questions in 2001 because
of changes in page design and question detail. The May test asks
a new question on income which will be difficult to answer correctly
and will be off-putting to many people. A balance has to be struck
between, on the one hand, meeting users' demands for more information
and, on the other hand, (a) the willingness of the public to supply
it and (b) the damage to the quality of population statistics
when response to the census falls.
Recommendation 1: ONS should ensure that
complex and contentious questions are not added to (or retained
in) the census form if they materially damage the quality of population
15. In previous censuses forms had been
delivered and collected by enumerators calling at each dwelling.
The enumerators were able to explain to householders what was
required of them, to help them if needed and to check that the
forms had been filled in properly and handed back. In 2001 the
method was changed in most areas because ONS felt they could not
recruit the numbers of enumerators needed. So forms were delivered
by post (mailout) using a previously-compiled address list and
returned by post (mailback).
16. ONS plan to repeat mailout and mailback
in 2011. But I believe that, if a high level of response is required
combined with good quality answers, then the doorstep contribution
made by enumerators should not be jettisoned. The issue is whether
the Treasury wants a good census and is prepared to pay enumerators
the rate for the job. Multiple methods of return (eg using the
interent) pose new risks in 2011.
Recommendation 2: ONS with the Treasury
should review the possibility of putting enumerators back on the
doorstep in the 2011 census.
III WHO IN
17. This was a major cause of the 2001 difficulties.
18. Public response to the census in 2001
would probably have been poorer than before because of the changing
social environment: greater geographical mobility, weaker family
cohesion, more single-person households, more second homes, more
security devices limiting access to dwellings, perhaps less respect
for authority, and so on. The much longer census questionnaire
and the absence of enumerators on the doorstep were further causes
of poor response.
19. The "final straw" was ONS's
decision to limit the request for a census return to persons "usually
living at this address". In the censuses of 1961 to 1991
a hybrid system had been adopted in which the census form was
to be completed in respect of everyone present at the address
on census night as well as by anyone else usually living at the
address. The numbers present on census night were the starting
point for population estimates because they included visitors
with usual addresses elsewhere in E&W who had not been returned
anywhere as "usually living at this address" (these
numbered 0.5 million in 1991). Whilst the smaller numbers of persons
returned as "usually living at this address" provided
statistics on household composition.
20. By limiting the return in 2001 to persons
"usually living at this address", ONS had, it seemed,
safeguarded the interest in household composition. But the figures
of population were undermined for two reasons. First, the experience
of 1991 implied that some ½ million visitors with usual addresses
in E&W would not have been returned anywhere as "usually
living at this address". Second, the census instruction "List
all members of your household who usually live at this address"
was unclear even when accompanied by 7 guidance notes. For large
numbers of people, especially young people without a settled way
of life, the instruction offered an opt-out from the burden and
intrusiveness of a long census form. And there were no enumerators
on the doorstep to check what was happening. Add to this the fact
that the penalties for non-compliance that had been enforced in
previous censuses fell into disuse in 2001. We may speculate that
some million people resident in E&W were missed as a result
of this misjudgement of census design. They had put themselves
beyond the reach of the census.
21. ONS accept that this was a serious problem
in 2001 but are not responding correctly. In the forthcoming test
census a hybrid form puts the full set of questions to everyone
"usually living in this household" and just four questions
(name, sex, date of birth and usual address) to visitors. The
ambiguity of the words "usually living in this household"
offers an opt-out to (eg) John, aged 20 and sharing a flat, having
left home 18 months earlier. He can enter his old home as his
usual address and so escape 23 of the 27 questions. Instead, the
form ought to ask the full set of questions to everyone present
on census night, and perhaps a more limited set to anyone usually
living in the household but absent on census night. A bigger proportion
of the population would then answer the full set of questions,
so reducing the uncertainties that stem from imputing personal
Recommendation 3: ONS should ask the full
set of questions of everyone present in the household on census
night 2011, so eliminating opt-outs stemming from the ambiguity
of the words "usually living in this household".
22. In every census worldwide the results
have to be adjusted for non-response, including missed dwellings,
households and individuals. But ONS had an additional problem
in 2001 when, as a result of form design, some one million residents
had put themselves beyond the reach of the census (20).
23. ONS designed a complex system, or model,
for estimating non-response, whose working was not widely understood.
It centred on a large-scale follow-up survey, the Census Coverage
Survey, for which interviewers went into the field soon after
Census Day. But there was little prospect that the interviewers
would pick up the appropriate proportion of the million beyond
the census reach. Nonetheless the model's estimates of non-response
were added to the raw census counts to give the first population
estimates based on the 2001 census. They were published for both
E&W and LAs on 30 September 2002 with a fanfare of PR which
spoke of "the most accurate census ever" and of errors
in the total population estimate of <plusminus>0.2% (<plusminus>_100,000
24. ONS's faith in the accuracy of the census
results appeared unshakeable. ONS seemed unaware of the million
beyond reach and therefore beyond the capability of the model
calculations to estimate. Under pressure from city authorities,
ONS revised some LA populations in 2003-04, so adding 0.3 million
(mainly males) to the national population. But further additions
are required, including ½ million males to correct the sex
25. In 2011 the task of estimating non-response
should be more manageable provided the census form ensures no
opt-outs. But it will still be a challenging task and needs to
V WARNING SYSTEMS
26. Before the 2001 census was launched,
ONS carried out census tests and a dress rehearsal which should
have warned them of weaknesses in their plans. They also set up
a raft of systems to detect when things went wrong: "rigorous
quality assurance", risk management and a risk register,
and an expert panel that signed off census estimates area by area.
But no red warning lights shone forth.
27. Critics will want to be assured that
there can be no repetition of this in 2011.
28. Demographic checks on census-based population
estimates are standard practice in any census. The checks involve
independent evidence usually drawn from data on births, deaths,
migration and previous censuses.
29. In the 2001 census ONS's demographic
check took the form of a comparison of the census-based national
population estimate with the "roll-forward": that is,
the 1991 population estimate carried forward ten years to 2001
by adding (or subtracting) annual figures of births, deaths and
migratory flows. This comparison shone a red warning light: the
2001 population estimate was 1.1 million below what had been expected,
most of the deficiency being young men. ONS's reaction was not
to question the census results but instead to "rewire the
warning light so that it shone green". They did this in part
by subtracting 0.3 million males from the 1991 population estimate
(the base for the roll-forward) and in part by revising the estimates
of migratory flows. According to my estimates, the warning light
had correctly shone red: it was the 2001 population estimate that
needed revision (32).
30. I share ONS's lack of confidence in
the roll-forward because it uses questionable figures of migratory
flows drawn from the sample International Passenger Survey as
well as relying on the accuracy of the preceding census. The two
demographic checks that I've developed do not draw on these sources.
31. In 1999 I proposed a demographic check
that I called the ISR method. It is cheap, robust and gives some
guide to margins of error, and I applied it to our 1991 census.
Details have been published in two international journals. The
findings contradict ONS's revisions to the 1991 population estimates.
Though ONS have been unable to put on paper any criticisms of
the method that stand up to critical examination, they decline
to experiment with or improve the method by applying it to the
32. A second and simpler demographic check
focuses on natives (that is persons born in E&W) born in the
60 years to mid-2001 and resident anywhere in the world. I compare
the sex ratio of the survivors in 2001 (males/females), taken
only from figures of births and deaths, (namely 1.041) with the
sex ratio of those of them still living in E&W extracted from
the 2001 census (1.017). The gap of 2½ percentage points
implies that the 3 million natives living outside E&W (that
is, emigrants) have a sex ratio of more than 1.3 _ a figure totally
out of line with the sex ratios of our emigrants in the principal
receiving countries (whose censuses rarely record ratios exceeding
1.1). The gap can be closed only by adding ½ million males
to our 2001-based population estimates.
33. A reliable form of demographic check
is urgently needed to validate both the 2001 population estimates
and the estimates that will emerge from the 2011 census. Some
development of the two methods I've proposed (§§31,32)
could offer a way forward.
Recommendation 4: ONS should develop robust
demographic checks that can be applied to the population estimates
for 2001 and, in due course, 2011.
Recommendation 5: ONS should examine claims
that the present 1991-based and 2001-based population estimates
are deficient, and make corrections as appropriate.
34. Unless ONS take action on the six issues
in this memorandum, I believe there is a real risk that the 2011
census will not be significantly better than the 2001 census.