Select Committee on Social Security Seventh Report


The Evaluation Process

64. The evaluation of ONE has two main aims. First, to test the feasibility of delivering ONE in the different models. Second, to test the effectiveness of the different models in improving both the quality and quantity of the labour market participation of people of working age. Table 2 shows the main elements of the evaluation of the pilots.

Table 2: Main elements of the evaluation of the pilots


  • To what extent does ONE put more benefit recipients in touch with the labour market through the intervention of their personal adviser?

  • To what extent does ONE lead to an increase in the sustainable level of employment by getting more people into work?

  • To what extent does ONE ensure that more claimants experience an effective, efficient service which is tailored to their personal needs?

  • To what extent does ONE change the culture of the benefits system and the general

 public towards independence and work rather than payments and financial dependence?

  • How cost effective are the different ONE models?


  • Policy evaluation through a programme of quantitative and qualitative research to assess the impact of ONE in improving labour market participation

  • Delivery evaluation, involving social research to examine the experiences and views of ONE participants and staff and difficulties with delivering the service, and operational research examining the operational effectiveness and cost of the different variants

  • Cost-benefit analysis

  • Database

Source: Ev. p. 10.

The evaluation will seek to compare results in the pilot areas with their control areas (see paragraph 13), and to compare results from the non-compulsory stage of the pilots (until April 2000) with the compulsory stage thereafter.

65. The first qualitative information (on the non-compulsory basic model) will become available at the end of 1999, and the first quantitative information on the immediate labour market effects 2-3 months after joining ONE should be available for the non-compulsory basic model in Autumn 2000. The final evaluation results from the pilots will not be available until 2002.[125]

66. The pilots have been designed to evaluate each of the three models as a whole (basic, call centre and private/voluntary sector). The evaluation's unit of analysis is therefore each model, not the four areas that make up each model, or sub-areas below this.[126] The evaluation will consider each of the three models in comparison with each other and with the control areas (see Table 1).[127] The design of the evaluation is not intended to compare the different private and voluntary sector pilots against each other. This potentially poses problems, given that different private and voluntary sector partnerships may be operating in different pilot areas offering different arrangements for the delivery of ONE. The evaluation strand dealing with delivery (see Table 2) will assess each method of delivery used in the private and voluntary sector pilots but, overall, the evaluation process is not intended to assess which of the private and voluntary sector variants is best.

67. We have already expressed our concern that the choice of pilot areas is not representative, in that it fails to include sufficient inner city areas with the variety of problems which such areas have. Although Ministers were keen to reassure us that such areas did exist within the pilots,[128] we are concerned that the design of the evaluation will not enable the impact of ONE to be assessed in these sub-areas, and hence the lessons can not be learned for future national roll-out. We have been reassured to some extent by confirmation from Ministers that the evaluation will identify areas where there are difficulties in providing the ONE Service and the implications for its delivery; and in considering labour market outcomes across each variant, indicators such as the concentration of minority ethnic groups will be taken into account.[129] We recommend that the evaluation process should make every effort to assess the impact of ONE on the most deprived areas within the pilots.


  68. The questions which the evaluation process will address, and by implication the success criteria, are currently ill-defined. Officials told us that there would be no new targets set for the pilots; instead, staff would operate within the Agencies' existing targets, although there was a commitment that adviser meetings would take place within three days of the start-up meeting.[130] Existing targets of the Benefits Agency and Employment Service focus on service delivery issues and job placements. One question the evaluation process will examine is the extent to which ONE puts more benefit recipients in touch with the labour market through the intervention of their personal adviser. If ONE is to be a valuable initiative for all of the targeted client groups, it will be important to develop criteria for measuring the progress an individual makes in getting in touch with the labour market which are more relevant than simple job placements. Ministers confirmed that the intention was to develop methods of assessment which would measure whether people have become more employable or moved closer to the labour market.[131] As we have already noted, the Government is currently developing a diagnostic tool which will measure people's distance from the labour market and evaluate the obstacles which they face to finding work (paragraph 52). Such a tool could, eventually, be used to track clients' progress towards independence. However, it may be some time before it is available for use in ONE and in the meantime, we recommend that the Government should publish the clear measures which it intends to use in assessing the reduction in people's detachment from the labour market.

69. The ONE pilots are also intended to assess whether the service achieves an increase in the sustainable level of employment by getting more benefit recipients into work. A key question is how sustainable employment will be measured. Ministers have said that labour market outcomes will be measured at frequent intervals after a client enters ONE and that, in the longer term, the Evaluation Database will enable tracking of people who have moved into employment and who then return to benefit.[132] Such long-term tracking will be important, for example, in evaluating job sustainability for disabled people.

70. ONE aims to give claimants a more effective, efficient service which is tailored to their personal needs. Many of the existing targets for measuring service delivery will enable this aim to be evaluated. However, we are not satisfied that existing measures are sufficient to assess the quality of information and advice which claimants receive from ONE advisers. Ministers have advised that information on the quality of benefit advice will be gathered by monitoring the proportion of claims that are disallowed; by exploring the confidence of ONE advisers in giving advice; and by asking claimants whether they thought staff could answer their questions and whether they were confident in the information and advice received.[133] None of these methods of evaluation will test whether, objectively, the claimant was given accurate and appropriate advice and information relevant to their situation. We recommend that as part of the evaluation process independent benefit checks should be carried out on a selection of ONE participants to identify the quality of the benefits information and advice which they have been given and to identify the effect of ONE on the take-up of benefits.

71. ONE is also aimed at changing the culture of the benefits system and the general public "towards independence and work rather than payments and financial dependence".[134] The measures for evaluating whether a change of culture has been achieved are far from clear. Nevertheless, given the proportion of claimants participating in ONE who will not be available to look for work for various reasons, the development of criteria for judging whether a person has moved towards greater independence is important. We recommend that the Government publish the measures by which it intends to evaluate the success of ONE advisers in assisting people who are not able to work towards greater independence.

125   For further details on the timing of evaluation results see Ev. pp. 10, 11 & 36, and QQ. 235-236. Back

126   Ev. p. 37. Back

127   Ev. p. 36. Back

128   Q. 248. Back

129   Ev. pp. 118-119. Back

130   Q. 66. Back

131   Q. 239. Back

132   Ev. pp. 118-119. Back

133   Ev. pp. 118-119. Back

134   Ev. p. 2. Back

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries

© Parliamentary copyright 1999
Prepared 27 July 1999