House of Commons - Science and Technology

Select Committee on Science and Technology Sixth Report

5 The evidence base

82. In oral evidence to the Committee, Katherine Courtney emphasised that trials would commence once Royal Assent had been granted and the procurement phase had begun. She explained that the specific solution for the identity cards scheme could not be trialled until the bidders were given an opportunity to propose their ideas.[168] Furthermore, she stated that "we will be putting information about the plans for trialling into the public domain once we are able to begin the procurement process".[169] However we received more detail from the Home Office in response to subsequent written questions. The Home Office said that it will undertake three main tests:

a) a competitive trial of bidders' proposed solutions involving the live enrolment and verification of approximately 3000 people;

b) a large-scale matching test using pre-recorded biometrics that will provide statistical information on the relative performance of different solutions; and

c) a large-scale live enrolment to confirm that the solution will be capable of performing when the National Identity Register is fully populated.[170]

The Home Office also told us that these trials would provide "vital new information on fingerprint performance, - large scale performance, verification performance, enrolment and image quality, spoof resistance, usability and inclusivity".[171]

83. We welcome the Home Office's commitment to publicising fully its plans for trialling once the procurement process has begun. In order to continue this move towards transparency and to build public confidence in the scheme, we recommend that the Home Office also makes public the results of these trials.

84. It is anticipated that the trials will take place during the procurement process, which is expected to last between 15 and 18 months. In oral evidence to the Committee, Katherine Courtney stated that "we have made the assumption that the process of procurement, including the trials, will take somewhere between 15 and 18 months in order to make sure that we are giving sufficient time to operate those trials as part of that procurement process".[172] Furthermore, we acknowledge that the Home Office has stated that it will not be rushed. Katherine Courtney stated that "from the very first policy announcement when the Secretary was quite clear that there would be no big bang implementation of this scheme. That gives us lots of opportunity to test and ensure that we are getting things right".[173] The Minister, Joan Ryan, also noted that the future timetable of the scheme would be determined by the procurement process.[174] She said that the timetable is relatively loose because the Home Office wants to be "very cautious on the basis of all the lessons we have learnt from good and bad projects".[175] We welcome the Home Office's cautious incremental approach and we encourage the Home Office, if necessary, to extend the procurement phase to ensure that enough time is taken to gather the necessary scientific evidence and to undertake all the appropriate trials. In view of the potential adverse impact on large numbers of people, it is better that the scheme is late and workable than on time but flawed.

85. It is not only important that the Home Office allows enough time for trialling the technology but that time is built into the programme to allow for the results to be fed back into the proposed solutions. Professor Anne Anderson from Glasgow University has noted that "sufficient time must be included to refine the design in the light of evidence from realistic trials of the system in operation. It will be important to ensure that the relevant expertise is available to gather and analyze this data on the whole system performance".[176] We recommend that the Home Office publicly outlines the ways in which the results of the trials have influenced and changed the programme.

86. The Home Office has provided us in confidence with the details of the budget that has been allocated to trialling the technology supporting the identity cards scheme. Whilst we note that resources have been allocated specifically to trials, we would be concerned if this budget created a costs ceiling that will limit the trials that can take place. If the trials deliver unexpected results, it is imperative that further trials can be undertaken to explore such results in more detail. We seek assurance that the Home Office will not limit the number, scope or quality of technology trials in order to stay within the allocated budget. We recommend that the Home Office ensures that sufficient funding is available to undertake the necessary technology trials for this scheme and that it retains flexibility regarding the trials that may be required.

BIOMETRICS

87. The Home Office has stated that it will use fingerprints, facial recognition and iris scanning in the identity cards scheme. As already mentioned, the specific solution has not been trialled and the Home Office intends to run operational testing of the technical systems during procurement. However, several trials have taken place to test the biometric technology. The Home Office states that trials have taken place on the performance of facial recognition systems, the capability of facial and iris systems to resist spoofing attempts and 'benchmarking' of a fingerprint system.[177] The facial recognition tests started in November 2005 and are still underway. The tests on iris systems took place in early 2006 at the National Physical Laboratory and the fingerprint system benchmarking was completed early in 2006.

88. In 2004, the identity cards programme team, the UK Passport Service (UKPS) and the Driver and Vehicle Licensing Agency (DVLA), commissioned Atos Origin to run a biometrics enrolment trial to gather evidence on public perceptions and attitudes towards biometrics. The Home Office has repeatedly asserted that this trial was not an assessment of the technological capabilities of biometrics. The report noted that "testing of the biometric technology itself was not one of the objectives of the Trial, rather the Trial aimed to test and measure the processes around recording and verification of biometrics".[178] As will be discussed in the chapter on public engagement, the status of the trial caused confusion and there were numerous press reports detailing the apparent problems with the technology (see paragraph 138). This confusion has perhaps been exacerbated by the Home Office's treatment of the results from the trial and their inconsistent use of it as evidence. When questioned in an oral evidence session about the false non-match rates that resulted from the Atos Origin trial, Katherine Courtney said that "I think it is important to reiterate that the enrolment trial was a trial of process and customer experience. It was not designed as a trial to look at performance of the technology per se".[179] However, the results of the trial have been used to provide information about technology performance. On 29 June 2005, despite noting that the Atos Origin trial was not intended as a test of technology, the then Parliamentary Under-Secretary for Immigration, Citizenship and Nationality, Andy Burnham used statistics from the trial in order to answer a question relating to the failure to acquire rate of the technology.[180] There is evidence that whilst trial plans were set out clearly the processes with which they were enacted lacked rigor. As a result, the Home Office has selectively used evidence from the biometrics enrolment trial to support its assertions. We believe that the Home Office has been inconsistent regarding the status of this trial and this has caused confusion in relation to the significance of the evidence gathered about biometric technologies. We recommend that the Home Office clarifies whether or not it accepts the validity of the results gained during the trial regarding the performance of biometric technologies.

89. Furthermore, even though the biometrics enrolment trial was not devised as a performance trial, it highlighted problems with the technology. For example for iris enrolment, success at the first attempt was higher for asian and white participants than black participants.[181] Tony Mansfield from the NPL said that "it illustrated that if you buy off-the-shelf systems and deploy them with no adaptation to the ID cards programme the performance would not be terribly good…clearly the performance was inadequate in that trial."[182] Given the findings of the biometrics enrolment report regarding the performance of current biometric systems, we seek reassurance from the Home Office that systems will be adapted as necessary to improve performance levels and that final performance levels will be verified by independent testing.

90. The Home Office may have turned to the biometrics enrolment trial report to support its assertions regarding facial recognition technology due to a lack of information from other sources. Given the uniquely ambitious nature of the UK identity cards scheme, there is no independently-validated evidence regarding performance levels either from existing systems or from large-scale trials for the biometric technology involved in the scheme. The written evidence submitted by the LSE highlights that "the biometric technology at the core of the scheme has been untested at the scale proposed by the Home Office".[183] Indeed, the Home Office acknowledges that for iris scanning "no independent testing on databases of millions has been undertaken to date".[184] Neither is there a large-scale database for multimodal biometrics.[185]

91. Despite this lack of information regarding the performance of biometric technologies, the Home Office provided us with performance levels from existing evidence that would be "consistent" with their requirements (see paragraph 18).[186] The performance levels were taken from five different reports produced by four organisations over a period of four years.[187] Data relating to the performance of iris scanning relied upon unpublished, unverified results from the Schipol airport trial and the UAE iris system. The Home Office did note that "the data were not collected under the same conditions so caution should be used in interpreting these figures".[188] We are surprised by the Home Office's unscientific approach and suggest that rather than collating figures merely to provide information regarding performance, the Home Office admits that it cannot release details until it has completed trials. We note the lack of independent evidence relating to the performance of iris scanning and welcome the Home Office's commitment to undertake a large-scale matching test using pre-recorded biometrics. Given the relative lack of information available publicly regarding the performance of biometrics in a national scheme, we recommend that once the scheme is established the Home Office publishes details of the performance levels of the technology.

92. In light of the lack of evidence relating to the large-scale use of multimodal biometrics, we are concerned that, although the exact biometrics are not specified in the Identity Cards Act 2006, the Home Office has already fixed the number and type of biometric that will be used. The Home Office apparently made the initial decision to use face, iris and fingerprint biometrics by assessing the available existing scientific evidence. Katherine Courtney stated in oral evidence that "when we reviewed all the literature, the research, also the practical experience from programmes around the world, we looked at systems that did use more than one biometric".[189] Qinetiq has questioned the breadth of this research stating that "a single report from the National Physical Laboratory—valid though it was—was the sole justification for using three biometrics".[190] This report, the Feasibility Study on the Use of Biometrics in an Entitlement Scheme, produced by Tony Mansfield from the NPL and Marek Rejman-Greene from BTExact Technologies stated that combining biometrics "can improve performance…however the performance improvement is unlikely to be commensurate with the increased costs".[191] Furthermore, the research underpinning the Home Office's decision appears to have been theoretical rather than practical and the identity cards programme is still gathering evidence regarding the impact of multiple biometrics. When giving oral evidence to the Committee, Katherine Courtney stated that "the function for us of the multiple biometrics…needs to be tested during the procurement process in order to ensure that we have gathered the evidence base, that those biometrics will enhance the performance of the system".[192]

93. We are surprised and concerned that the Home Office has already chosen the biometrics that it intends to use before finishing the process of gathering evidence. Given that the Identity Cards Act does not specify the biometrics to be used, we encourage the Home Office to be flexible about biometrics and to act on evidence rather than preference. We seek assurance that if there is no evidence that any particular biometric technology will enhance the overall performance of the system it will not be used.

INFORMATION AND COMMUNICATION TECHNOLOGY

94. With regard to ICT, we note that the ICT systems have not yet been trialled. Evidence taken by the Committee from the ICT community recommended that the ICT solution should be trialled. Nick Kalisperas from Intellect said that "it needs to be piloted and then rolled out gradually."[193] Professor Martyn Thomas from the UKCRC agreed and emphasised that "the purpose of that would be to discover the weaknesses, the things that had gone wrong, and therefore you would need to allow plenty of time and plenty of budget for backtracking, for making modifications, perhaps for radical revisions of the scheme".[194] Although the Home Office has asserted that it intends to roll out the scheme gradually, it is not clear how it will do this.

95. We note the lack of explicit commitment from the Home Office to trialling the ICT solution and strongly recommend that it take advice from the ICT Assurance Committee on trialling. We seek an assurance that time pressure and political demands will not make the Home Office forgo a trial period or change the purpose of the scheme.

Research and development

96. In written evidence the Home Office said it was not necessary to embark on publicly-funded scientific research to improve the capabilities of biometrics.[195] This claim was subsequently denied in oral evidence and the identity card team asserted that research was being undertaken into fingerprint biometric performance. Katherine Courtney said "I would not say that we have not commissioned research. We have commissioned research. We have a piece of research that the Home Office is funding right now into fingerprint biometric performance".[196] We regret the confusion at the Home Office regarding the research that it is funding and what research it requires. The Government Chief Scientific Adviser's Guidelines on Scientific Advice in Policy Making state that departments should "think ahead and identify early the issues on which they need scientific advice…and where the current evidence base is weak and should be strengthened".[197] The Home Office has not provided us with evidence either that they have identified areas where the evidence base is weak nor that they have commissioned research in order to strengthen it. On the basis of the evidence that we have seen, we conclude that the Home Office does not seem to have an effective mechanism for ensuring that the required research and development in the relevant scientific and technological areas is carried out. We recommend that the Home Office identifies the gaps in the evidence base underpinning the identity cards programme, that it commissions research to fill these gaps and that it feeds any new developments into the scheme where appropriate. This process should be overseen by the departmental Chief Scientific Adviser.

97. The technological fields that will support the identity cards programme are constantly developing. Katherine Courtney acknowledged that "The field is evolving all the time. I think one of the challenges has to be to design a system that is flexible enough possibly to accommodate advances in the technology later down the line".[198] In relation to horizon scanning, the Home Office said that it would encourage the consortium that won the contract to take advantage of any new knowledge in the area. Marek Rejman-Greene, Head of the Biometrics Centre of Expertise, stated that "During the course of the deployment and early years of the programme, we would certainly ensure and ask the consortium that was winning the project to take advantage of that knowledge and home in on it".[199] Even if horizon scanning activities were already embedded in the identity cards programme, we would seek assurance that these activities fed back into the scheme. We note that the departmental Chief Scientific Adviser, Professor Paul Wiles, stated that with regard to the Home Office in general, "getting an organisation to actually lift its head from immediate problems and think ten or twenty years ahead and use that horizon scanning is sometimes a challenge".[200] We welcome Professor Wiles' admission and emphasise that it is part of the departmental Chief Scientific Adviser's job to ensure that this challenge is met by Ministers. We will return to this issue of research and horizon scanning in the overarching report on scientific advice, risk and evidence. The Home Office cannot afford to delegate responsibility for horizon scanning to others. We recommend that the Home Office actively undertakes horizon scanning activities relevant to the technologies involved in the identity cards programme and that it develops mechanisms to feed this information back into the scheme.

98. During the inquiry, the identity cards programme team displayed an apparent reliance on existing research, which although broadly relevant to the programme does not necessarily deal with the specific challenges of the project. Whilst we identified earlier the ways in which existing research had been used in the field of biometrics (paragraph 92), the same issues occur in relation to ICT. For example, in oral evidence to the Committee, Nigel Seed claimed to have studied the Royal Academy of Engineering and British Computer Society report on The Challenges of Complex IT Projects (see paragraph 60).[201] We have several concerns regarding this statement. We would welcome it if the Home Office was following best practice as detailed in The Challenges of Complex IT Projects report. However, despite requesting it, we have not received detailed evidence from the Home Office that supports Nigel Seed's claim.[202] Secondly, this report does not in itself provide a detailed guide of how a major project should be managed. Finally, whilst it is vital that the Home Office learns the lessons outlined by this report, the report is also not a substitute for funding research specifically related to the identity cards project. We urge the Home Office to commission, and where appropriate fund, research focused on the specific requirements of the information technology systems in the identity cards scheme rather than relying on general existing study results.

Technology and operating costs

99. As already mentioned, the question of the cost of the identity cards scheme has caused fierce debate both within and outside Parliament. As a result, the Identity Cards Act 2006 requires a six monthly report on costs to be brought before Parliament.[203] The Home Office has repeatedly stated that the total year-on-year running costs of the scheme, primarily relating to people and services, would be £584 million. Katherine Courtney said to us that "We are quite confident in our cost estimates".[204]However, the Home Office has not released meaningful estimates within this figure. In December 2005, the-then Minister Andy Burnham said that "the estimates are…commercially sensitive and to release them may prejudice the procurement process and the Department's ability to obtain value for money from potential suppliers".[205]

100. The costs outlined by the Government were challenged by the LSE in its Identity Project Report. The LSE estimated that the scheme's implementation and running costs would be in the range of £10.6 billion to £19.2 billion during first ten years of the scheme.[206] The discrepancy between the LSE and Government figures caused prolonged, and at times hostile, debate. The written evidence submitted to us by the LSE acknowledges that the discrepancy in figures between their estimates and the Home Office estimates results in part from what is included. The LSE states that "Our figures always included set-up costs, running costs and costs of integration with other Departments. The Home Office figures…are 'The current best estimate for the total average running costs'".[207] Katherine Courtney also pointed out to the Committee that "the costs modelling behind the LSE made a number of fundamental assumptions which were very different to our own proposition".[208] However, in oral evidence Dr Edgar Whitley from the LSE still said that "On the basis of no technology trials or limited technology trials and specifications still being changed I just cannot see how they can be so clear that it is £584 million".[209] We have no wish to guess the true costs but it is difficult to believe that such a certain figure can be established when there are so many variables.

101. The Home Office figures were audited by KPMG.[210] The Home Office has interpreted this audit, which was published in November 2005, positively. In a report on the KPMG review, the Home Office stated that KPMG "confirmed that the majority of cost assumptions within their scope were based on appropriate benchmarks and analysis from the public sector and suppliers".[211] In oral evidence, Katherine Courtney stated that, "our cost assumptions have been independently audited by KPMG and so we can have quite a high degree of confidence in them at this point in the development of the scheme".[212]

102. However, the audit highlighted some potential problems with the scheme. Despite Government assertions that a 10-year card life would be feasible, KPMG found that supporting information from suppliers was inconclusive.[213] KPMG stated that "the durability of the cards over the ten year period is questionable" and it recommended that the Home Office revise its cost estimates accordingly. [214] KPMG also noted that:

"the performance of the biometric matching drives a significant amount of cost […] the IDCP [identity card programme] team should have further discussion with the USVISIT programme to gain detailed insight into the cost drivers for this area and the UAE [United Arab Emirates] to verify the cost and performance of the fingerprint and iris hardware matchers respectively".[215]

When questioned on 22 March 2006 about whether the identity cards team had followed this suggestion, Katherine Courtney admitted that they had not yet done so.[216] Our concern that the identity cards programme still has to gather evidence regarding the performance of multimodal biometrics is compounded by the fact that they are also not apparently exploring the relationship between performance levels and cost (paragraph 90). We also note that Intellect has said that "It is imperative that the Government selects a solution that is proven to work, and not one that is selected solely on the grounds of cost".[217] We recommend that the identity cards programme team returns to the KPMG audit report and implements its recommendations. Furthermore, we re-emphasise that the Home Office needs to work out how costs will impact on performance and we seek reassurance from Government that cost limitations will not compromise the level of performance that is accepted.

103. Despite the release of figures regarding the running costs of the project, there is still a lack of clarity concerning the start-up costs and specifically the costs of the technology within the project. We agree with Professor Angela Sasse's comment that "We have not been given enough detail to really check the validity" of costings.[218] We do not share the Home Office's belief in their costings given that the breakdown of technology costs provided to us in confidence only provided a broad overview and did not include any figures.[219] In the light of this lack of evidence, we can only conclude that the Home Office is not confident in its figures and as a result, we are incredulous that the Home Office is seemingly able to produce firm costings regarding the running costs of the scheme when the costs of the technology are not yet clear.

104. As well as assessing the costs of the scheme, the Home Office has undertaken work to assess the benefits of the identity card programme.[220] This research considered the strategic benefits for example in the delivery times of services, quantifiable benefits such as improved crime detection and non-quantifiable benefits like convenience. It found that "Most organisations have at present only been able to model more conservative, incremental changes that would result from ID cards. Even with these constraints, the quantified, financial benefits range from £650m to £1.1bn per annum when the scheme is fully rolled out".[221] As the scope of the scheme is finalised, we encourage the Home Office to update its research regarding the benefits, as well as the costs, of the scheme.

105. We are sceptical about the validity of costs produced at this early stage. We acknowledge that the release of firm overall costing has been driven by political imperatives but the Home Office could have credibly given a broad range instead of precise figures. We note the danger that a desire to keep below a costs ceiling might drive the choice of technology. We seek assurances that the costings are flexible. We strongly recommend that, once the procurement process has taken place, the Home Office publishes a breakdown of technology costs, including set-up costs, running costs and predicted savings as a result of the scheme in the Home Office and elsewhere.

Social science

106. The Home Office has undertaken nine pieces of social science research in 2004 and 2005. This research included:

a) omnibus research, which was carried out in February, April, October and December 2004;

b) qualitative research on "special needs issues" and "citizen's views on proposed customer propositions", which was completed in December 2004; and

c) two pieces of quantitative research on UK citizens' and user organisations' views on the scheme, and foreign nationals' views on the scheme. This work was published in October and December 2005.[222]

In oral evidence to us, Professor Angela Sasse said that she thought that the Home Office became aware of the societal impact of the scheme during the Home Affairs Select Committee investigation in 2004.[223]

107. We welcome the work that the Home Office has undertaken in the area of social science research. We have also received evidence that suggests that the scope of this research could be broadened. Professor Anne Anderson from Glasgow University has stated that "although this input from social science may well have been valuable to the…National Identity Scheme, it is a narrow perspective on social science and where the social sciences could be used to improve the scheme".[224] Furthermore, she notes that "the National Identity Scheme is a very challenging project. It is a complex socio-technical system and to be effective will require that the Home Office considers the social as well as the technical dimensions".[225]

108. Professor Anderson also notes that ICT systems often fail to deliver benefits because the systems have been designed without understanding about the context of use or users' needs. She further states that "the challenges of implementing the various biometric technologies have been the focus of concern, and it appears that less attention has been given to the challenges of how to design and implement the system in ways that are usable, useful and appropriate".[226] The ways in which the identity card scheme would function vary depending on whether members of the public or service providers are the prime users and beneficiaries of the scheme. If the former, then there might be recognition of the views of different users regarding the amount of information made available to the service provider. Professor Anderson highlights the positions of individuals such as celebrities, those being stalked or those leaving abusive relationships. The matter has also been raised regarding the ways in which an identity card might be used by those with mental health problems or those that are blind.[227] In general, Professor Anderson notes that "The key point I want to make is that the Home Office needs to be more sensitized to these social concerns".[228]

109. We have noted that the identity card proposals have been firmed up since the earliest studies.[229] For example, from 2010 the cards will be compulsory, the Identity and Passport Service has been created and the minimum age has been set at 16. Furthermore, we note the complexity of the social concerns regarding identity cards highlighted briefly by Professor Angela Sasse and Professor Anne Anderson. We recommend that the Home Office prioritise funding as necessary to ensure that required social science research is undertaken and if necessary commissioned. In particular, we emphasise the need to undertake work to understand the attitudes of prime users towards the current proposals.

110. In response to written questions, the Home Office explained that advice on social science is derived from different sources dependent upon its nature. Statistical advice is provided by the Research Development and Statistics (RDS) unit within the Home Office; advice on research requirements is given by the Marketing and Communications team within the Identity and Passport Service (IPS), and advice on commissioning research is provided by the Central Office of Information.[230]

111. Within the identity cards programme, large pieces of social science work have tended to be commissioned from private companies, rather than being undertaken in-house.[231] We recognise that commissioning externally may be more cost effective than maintaining an in-house capability, but there are disadvantages. As a result, social science work has seemed to focus upon one-off pieces of work rather than consistent monitoring. Furthermore, Professor Sasse has noted that this work is not necessarily followed up. She said in oral evidence that "there is a bit of a lack of depth and a lack of following-up on problems that have been discovered to see how they could be overcome".[232] We recommend that the Home Office establishes a clear process by which advice from external social science experts regarding future research and the social science aspects of the programme can feed into the scheme. Once research has been undertaken, we urge the Home Office to develop the expertise that will allow it to follow up the results.

168 Q 331 Back

169 Q 324 Back

170 Ev 113 Back

171 As above Back

172 Q 326 Back

173 Q 310 Back

174 Q 1144 Back

175 Q 1145 Back

176 Ev 120 Back

177 Ev 113 Back

178 UK Passport Service/Atos Origin, Biometrics Enrolment Trial, May 2005, p 8 Back

179 Q 299 Back