Select Committee on Treasury Minutes of Evidence


Memorandum submitted by Philip Redfern

In the last 12 years of my 35 years in the Government Statistical Service I was Deputy Director of the Office of Population Censuses and Surveys (OPCS, now merged in the Office of National Statistics (ONS)). In that post I was closely involved in the censuses of 1971 and 1981. My book on censustaking based on a study of practice in Europe and North America was published in 1987 by Eurostat (the Statistical Office of the EC). In July 2003 I read a paper to the Royal Statistical Society giving a critique of the 2001 census.

EXECUTIVE SUMMARY

  1.  The census of population serves many ends but this memorandum focuses on its primary aim of measuring the national population by sex and age and the analysis of the national totals between Local Authority (LA) areas. These figures are the base of a vast infrastructure of national statistics. Reference throughout is to England and Wales (E&W).

  2.   The 2001 census successfully broke new ground in several ways, but in terms of measuring the national population it was probably the biggest statistical disaster since the war, the scale of which has been played down by ONS. I reach this conclusion on several grounds. First, the census is the most important data collection exercise that any country undertakes, costing us some £250 million at today's prices; it cannot be repeated if it goes wrong. Second, in 2001 details of some 4 million people (7½% of the population) were not entered on census forms— a record number. Third, the 2001 census-based population estimates have already been increased twice since first publication in response to pressure from city authorities, but further revision is needed, particularly to correct the sex balance (§32).

  3.  This memorandum identifies six major failings in 2001 and considers what steps are being taken to rectify them in 2011.

  4.  Three of the failings in 2001 were in public view and one does not have to be a census expert to see them. (1) The form asked many more questions than any previous census, with adverse impact on public response. (2) In most places postal delivery and collection of the forms replaced the enumerators employed in previous censuses, so there was no one on the doorstep to explain to householders what they had to do, to provide help if needed or to check that the form had been reasonably completed and handed back. (3) For the first time ever the form had to be completed only for persons "usually living at this address". This ambiguous instruction gave an opt-out to anyone claiming to be usually resident somewhere else—an opt-out widely exercised by young people. For this and other reasons some million people placed themselves beyond the census reach. In previous censuses the form had to be completed for everyone present at the address on census night.

  5.  Looking to 2011, (1) the prospective form will probably be as long and complex, and for some people off-putting, as in 2001; (2) ONS seem content with postal delivery and collection in most areas; and (3) the prospective form still retains an opt-out to answer only 4 out of 27 questions—an opt-out available to anyone present on census night who claims not to be "usually living in this household". All three factors will damage census quality.

  6.  The three other major failings in 2001 were "behind the scenes" and therefore hidden from the public. (4) The population estimate based on a census has to embody an adjustment for non-response, including dwellings, households and persons missed. The adjustment process in 2001 inevitably failed to make proper allowance for the million beyond the census reach (4). And it introduced a sex bias— a deficiency of males relative to females—which prima facie was spread across all kinds of LA (32). If the 2011 form is cleansed of all opt-outs, the task of adjusting for non-response will be less but still very challenging.

  7.  (5) A census test and a dress rehearsal before the 2001 census should have sensed the storms ahead, but didn't. Moreover, several warning systems which ONS set up in 2001 to detect if things went wrong failed. ONS need to convince us that 2011 will be different.

  8.  That leads to (6): demographic checks, that is, checks comparing the census-based population estimates with independent estimates usually built up from data on births, deaths, migration and previous censuses. Such a check in some form is an essential part of any census. In 2001 the demographic check took the form of a comparison of the census-based population estimate with the 1991 population estimate rolled forward from year to year for 10 years. This did shine a red warning light: it pointed to a shortfall in the 2001 census-based estimate of 1.1 million males. That was not far from the truth but ONS ignored it: they decided that the "roll-forward" was in error and that the 2001 census-based estimates were correct.

  9.  By rejecting the roll-forward to 2001, ONS showed their lack of confidence in that form of demographic check. I share that lack of confidence, particularly as we look to 2011 and note the growing migratory flows that are components of the roll-forward arithmetic.

  10.  Demographic checks are difficult territory even for census experts. I have developed and put to ONS two kinds of check that do not rely on uncertain figures of migratory flows and on earlier population estimates. The simpler of the two is a calculation that is probably as near to a proof as one ever gets in demography. It "proves" that the sex balance (the ratio of males:females) in our 2001-based population estimates needs to be corrected by adding ½ million males (32). This error permeates the whole of our national infrastructure of statistics, and will continue to do so until 2012 unless ONS take action. The calculation was part of my paper to the Royal Statistical Society in July 2003 and has been put to ONS twice since.

  11.  Demographic checks are a vital part of censustaking, but a weak part of ONS's armoury calling for remedy.

  12.  Unless ONS act to correct the failings of the 2001 census (1-6 above), the 2011 census will be little better. The rest of this note deals with the six points in more detail and puts forward five Recommendations that the Committee might wish to consider.

I  QUESTION CONTENT

  13.  The non-response rate in 2001 was high compared with previous censuses: 6% overall and as high as 26% in the City of Westminster according to ONS's first published estimates, even though public participation is compulsory by law. Many factors contributed. One was a long questionnaire which asked 36 questions of each person—getting on for twice the number asked in 1991 (20), which in turn was 25% more than in 1981 (16).

  14.  The prospects for 2011 are not encouraging. The Test Census to be taken in May asks 27 questions, but they occupy more pages per person than the 36 questions in 2001 because of changes in page design and question detail. The May test asks a new question on income which will be difficult to answer correctly and will be off-putting to many people. A balance has to be struck between, on the one hand, meeting users' demands for more information and, on the other hand, (a) the willingness of the public to supply it and (b) the damage to the quality of population statistics when response to the census falls.

  Recommendation 1:  ONS should ensure that complex and contentious questions are not added to (or retained in) the census form if they materially damage the quality of population statistics.

II  FIELDWORK: FORM DELIVERY AND COLLECTION

  15.  In previous censuses forms had been delivered and collected by enumerators calling at each dwelling. The enumerators were able to explain to householders what was required of them, to help them if needed and to check that the forms had been filled in properly and handed back. In 2001 the method was changed in most areas because ONS felt they could not recruit the numbers of enumerators needed. So forms were delivered by post (mailout) using a previously-compiled address list and returned by post (mailback).

  16.  ONS plan to repeat mailout and mailback in 2011. But I believe that, if a high level of response is required combined with good quality answers, then the doorstep contribution made by enumerators should not be jettisoned. The issue is whether the Treasury wants a good census and is prepared to pay enumerators the rate for the job. Multiple methods of return (eg using the interent) pose new risks in 2011.

  Recommendation 2:  ONS with the Treasury should review the possibility of putting enumerators back on the doorstep in the 2011 census.

III  WHO IN THE HOUSEHOLD HAS TO BE ENTERED ON THE FORM?

  17.  This was a major cause of the 2001 difficulties.

  18.  Public response to the census in 2001 would probably have been poorer than before because of the changing social environment: greater geographical mobility, weaker family cohesion, more single-person households, more second homes, more security devices limiting access to dwellings, perhaps less respect for authority, and so on. The much longer census questionnaire and the absence of enumerators on the doorstep were further causes of poor response.

  19.  The "final straw" was ONS's decision to limit the request for a census return to persons "usually living at this address". In the censuses of 1961 to 1991 a hybrid system had been adopted in which the census form was to be completed in respect of everyone present at the address on census night as well as by anyone else usually living at the address. The numbers present on census night were the starting point for population estimates— because they included visitors with usual addresses elsewhere in E&W who had not been returned anywhere as "usually living at this address" (these numbered 0.5 million in 1991). Whilst the smaller numbers of persons returned as "usually living at this address" provided statistics on household composition.

  20.  By limiting the return in 2001 to persons "usually living at this address", ONS had, it seemed, safeguarded the interest in household composition. But the figures of population were undermined for two reasons. First, the experience of 1991 implied that some ½ million visitors with usual addresses in E&W would not have been returned anywhere as "usually living at this address". Second, the census instruction "List all members of your household who usually live at this address" was unclear even when accompanied by 7 guidance notes. For large numbers of people, especially young people without a settled way of life, the instruction offered an opt-out from the burden and intrusiveness of a long census form. And there were no enumerators on the doorstep to check what was happening. Add to this the fact that the penalties for non-compliance that had been enforced in previous censuses fell into disuse in 2001. We may speculate that some million people resident in E&W were missed as a result of this misjudgement of census design. They had put themselves beyond the reach of the census.

  21.  ONS accept that this was a serious problem in 2001 but are not responding correctly. In the forthcoming test census a hybrid form puts the full set of questions to everyone "usually living in this household" and just four questions (name, sex, date of birth and usual address) to visitors. The ambiguity of the words "usually living in this household" offers an opt-out to (eg) John, aged 20 and sharing a flat, having left home 18 months earlier. He can enter his old home as his usual address and so escape 23 of the 27 questions. Instead, the form ought to ask the full set of questions to everyone present on census night, and perhaps a more limited set to anyone usually living in the household but absent on census night. A bigger proportion of the population would then answer the full set of questions, so reducing the uncertainties that stem from imputing personal characteristics.

  Recommendation 3:  ONS should ask the full set of questions of everyone present in the household on census night 2011, so eliminating opt-outs stemming from the ambiguity of the words "usually living in this household".

IV  MEASURING NON-RESPONSE AS A STEP TO MAKING POPULATION ESTIMATES

  22.  In every census worldwide the results have to be adjusted for non-response, including missed dwellings, households and individuals. But ONS had an additional problem in 2001 when, as a result of form design, some one million residents had put themselves beyond the reach of the census (20).

  23.  ONS designed a complex system, or model, for estimating non-response, whose working was not widely understood. It centred on a large-scale follow-up survey, the Census Coverage Survey, for which interviewers went into the field soon after Census Day. But there was little prospect that the interviewers would pick up the appropriate proportion of the million beyond the census reach. Nonetheless the model's estimates of non-response were added to the raw census counts to give the first population estimates based on the 2001 census. They were published for both E&W and LAs on 30 September 2002 with a fanfare of PR which spoke of "the most accurate census ever" and of errors in the total population estimate of <plusminus>0.2% (<plusminus>_100,000 persons).

  24.  ONS's faith in the accuracy of the census results appeared unshakeable. ONS seemed unaware of the million beyond reach and therefore beyond the capability of the model calculations to estimate. Under pressure from city authorities, ONS revised some LA populations in 2003-04, so adding 0.3 million (mainly males) to the national population. But further additions are required, including ½ million males to correct the sex balance (§32).

  25.  In 2011 the task of estimating non-response should be more manageable provided the census form ensures no opt-outs. But it will still be a challenging task and needs to be transparent.

V  WARNING SYSTEMS TO DETECT FAILURES

  26.  Before the 2001 census was launched, ONS carried out census tests and a dress rehearsal which should have warned them of weaknesses in their plans. They also set up a raft of systems to detect when things went wrong: "rigorous quality assurance", risk management and a risk register, and an expert panel that signed off census estimates area by area. But no red warning lights shone forth.

  27.  Critics will want to be assured that there can be no repetition of this in 2011.

VI  DEMOGRAPHIC CHECKS ON THE CENSUS RESULTS

  28.  Demographic checks on census-based population estimates are standard practice in any census. The checks involve independent evidence usually drawn from data on births, deaths, migration and previous censuses.

  29.  In the 2001 census ONS's demographic check took the form of a comparison of the census-based national population estimate with the "roll-forward": that is, the 1991 population estimate carried forward ten years to 2001 by adding (or subtracting) annual figures of births, deaths and migratory flows. This comparison shone a red warning light: the 2001 population estimate was 1.1 million below what had been expected, most of the deficiency being young men. ONS's reaction was not to question the census results but instead to "rewire the warning light so that it shone green". They did this in part by subtracting 0.3 million males from the 1991 population estimate (the base for the roll-forward) and in part by revising the estimates of migratory flows. According to my estimates, the warning light had correctly shone red: it was the 2001 population estimate that needed revision (32).

  30.  I share ONS's lack of confidence in the roll-forward because it uses questionable figures of migratory flows drawn from the sample International Passenger Survey as well as relying on the accuracy of the preceding census. The two demographic checks that I've developed do not draw on these sources.

  31.  In 1999 I proposed a demographic check that I called the ISR method. It is cheap, robust and gives some guide to margins of error, and I applied it to our 1991 census. Details have been published in two international journals. The findings contradict ONS's revisions to the 1991 population estimates. Though ONS have been unable to put on paper any criticisms of the method that stand up to critical examination, they decline to experiment with or improve the method by applying it to the 2001 census.

  32.  A second and simpler demographic check focuses on natives (that is persons born in E&W) born in the 60 years to mid-2001 and resident anywhere in the world. I compare the sex ratio of the survivors in 2001 (males/females), taken only from figures of births and deaths, (namely 1.041) with the sex ratio of those of them still living in E&W extracted from the 2001 census (1.017). The gap of 2½ percentage points implies that the 3 million natives living outside E&W (that is, emigrants) have a sex ratio of more than 1.3 _ a figure totally out of line with the sex ratios of our emigrants in the principal receiving countries (whose censuses rarely record ratios exceeding 1.1). The gap can be closed only by adding ½ million males to our 2001-based population estimates.

  33.  A reliable form of demographic check is urgently needed to validate both the 2001 population estimates and the estimates that will emerge from the 2011 census. Some development of the two methods I've proposed (§§31,32) could offer a way forward.

  Recommendation 4:  ONS should develop robust demographic checks that can be applied to the population estimates for 2001 and, in due course, 2011.

  Recommendation 5:  ONS should examine claims that the present 1991-based and 2001-based population estimates are deficient, and make corrections as appropriate.

CONCLUSION

  34.  Unless ONS take action on the six issues in this memorandum, I believe there is a real risk that the 2011 census will not be significantly better than the 2001 census.

February 2007





 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2007
Prepared 9 May 2007