Troubled families: progress review Contents

1Impact of the programme

1.On the basis of a report by the Comptroller and Auditor General, we took evidence from the Department for Communities and Local Government (the Department) as the department responsible for delivery of the Troubled Families programme.1

2.In 2011, following riots in some parts of England, the then Prime Minister announced a commitment to “turn around” the lives of 120,000 troubled families in England.2 The Government defined troubled families as those with problems including youth offending, truancy, anti-social behaviour, and worklessness.3 The Government estimated that these families cost the Exchequer up to £9 billion per year, £8 billion of which was spent in reacting to their problems, rather than solving them.4

3.To meet the Prime Minister’s commitment to turn around families’ lives, in the autumn of 2011 the Government established the cross-departmental Troubled Families programme. The Department led the team responsible for the design and oversight of the programme, and responsibility for its implementation at the local level sat with 152 upper-tier local authorities.5 The Government set a budget of £448 million for phase one of the programme, which lasted from 2012 to 2015.6 This was paid to local authorities on the basis of a payment by results framework set by the Department.7 In June 2013, the Department announced that the programme would be extended until 2020 at a cost of £920 million.

4.The Committee of Public Accounts previously reported on the programme in April 2014.8 We recalled the Department to give further evidence, following media coverage which suggested that an evaluation was not able to attribute improvements in families’ lives directly to the programme.

Findings of the programme evaluation

5.In March 2013, the Department announced that it had commissioned a research consortium, led by the firm Ecorys, to conduct an evaluation of the Troubled Families programme. The evaluation consisted of six separate strands. Each of these strands was in turn part of either a process assessment of changes in the delivery of services to troubled families: an evaluation of the financial benefits of the programme or of a study assessing the overall national impact of participation in the programme.9 Within this consortium, the National Institute of Economic and Social Research (NIESR) was the partner responsible for the national impact study.10 The Department published reports detailing the findings of the different strands of the programme evaluation on 17 October 2016, as well as an overarching synthesis report summarising the key findings of each of the different strands of evaluation.11

6.The findings of the Department’s process evaluation were nuanced. It found evidence of good practice and widespread local service transformation, such as that the programme’s funding had enabled local authorities to increase the support they could offer to families through means such as hiring employment advisers. However, the process evaluation also found a number of challenges that local authorities faced in working with troubled families, including variation in the level of support offered to them.12

7.The Department’s cost benefit report estimated that the amount spent per family participating in the Troubled Families programme by various public services had fallen by £7,050 one year after the families started on the programme. It also estimated that, on average, it cost £3,350 to deliver the programme to a family. The Department stated that this implied a gross saving of £2.11 for every £1 spent on families participating in the programme. However, the cost benefit report also made it clear that these are gross figures, meaning that any savings cannot be attributed directly to the Troubled Families programme.13

8.The national impact study could not find evidence of a net impact when programme participants were compared against a matched comparator group for outcomes in the data that included crime, employment and health measures.14 The sole area in which the impact evaluation found a statistically significant improvement was subjective: it concluded that, as a result of participating in the programme, troubled families felt better about their lives and the direction they were taking.15 The findings of the national impact study were supported, in turn, by those of a complementary survey of troubled families, constructed by another partner in the ECORYS consortium.16

9.The Department told us that it was clear that the families in the programme had improved outcomes but that it was simply not possible to attribute these to the programme specifically.17 NIESR agreed with this assessment.18 The Department claimed that the findings of the impact study had many caveats as a result of poor quality data.19 However, NIESR maintained that there is no reason to believe that the findings of the impact study are biased in the statistical sense.20 The Department told us that it was optimistic about making more progress in tackling issues with the data in the future.21

10.The Department explained that this was an ambitious piece of statistical analysis that had not previously been attempted.22 Again, NIESR disagreed, refuting this and stating that such impact evaluations are not uncommon when assessing the effectiveness of government policies.23 In addition, the Department suggested that it was too early in the programme life cycle for an evaluation to be carried out and that this was one of the reasons that the evaluators had to work with poor quality data.24

Payments under the programme

11.The Department used a payment by results framework for the Troubled Families programme, under which it set out that local authorities would receive an attachment fee for assigning families to the programme and receive reward payments when the outcome criteria describing a family as “turned around” had been met.25 Local authorities claimed reward payments for 116,654 families out of the maximum 117,910 for which they could claim.26

12.We questioned whether the payment by results framework had caused the Department to take an approach that was focused on achieving short-term rather than long-term goals. The Department accepted our assertion that the payment by results framework measurements of a “turned around” family were all short term measures.27 The Department told us that its commitment to wholesale system reform demonstrated that it was not merely focused on short term goals.28

13.The process evaluation showed how the short term outcome measures set under the payment by results framework encouraged perverse behaviour by authorities in some cases. It found evidence that some local authorities attempted to move families through the programme quickly in order to draw down attachment payments. It also found evidence of some local authorities claiming results for families at the margins of the programme in order to meet targets. In such cases, according to this evaluation, this led to a divergence from optimal caseload sizes and case duration, meaning that families did not always receive the support they required to tackle deep rooted problems. Local authorities also made attempts to activate outcome payments by using national administrative datasets to retrospectively identify families that had achieved a positive outcome but had not been one of the families originally prescribed to the programme. These issues are unlikely to have been universal, however. The process evaluation also concluded that the financial framework had benefitted local authorities by encouraging them to improve their local data management systems as they became more outcomes focused.29

1 C&AG’s Memorandum for the House of Commons Committee of Public Accounts, The Troubled Families programme: update, October 2016.

3 C&AG’s report, para 1.2

4 C&AG’s report, para 1.3

5 C&AG’s report, paras 1.4 & 1.12

6 C&AG’s report, para 1.14

7 C&AG’s report, para 1.15

8 Committee of Public Accounts, Programmes to help families facing multiple challenges, HC 668, Fifty-first Report of Session 2013–14, 4 April 2014

9 C&AG’s report, Figure 7

12 Department for Communities and Local Government, National Evaluation of the Troubled Families Programme: Process evaluation final report, October 2016.

13 Department for Communities and Local Government, Local Authority Data on the Cost and Potential Fiscal Benefits of the Troubled Families Programme, October 2016.

14 National Institute for Economic and Social Research (TFA0027), para 9; Department for Communities and Local Government, National Evaluation of the Troubled Families Programme: National Impact Study Report, October 2016.

16 Department for Communities and Local Government, National Evaluation of the Troubled Families Programme: Final Synthesis Report, October 2016.

18 National Institute for Economic and Social Research (TFA0027), para 9

20 National Institute for Economic and Social Research (TFA0027), para 6

23 National Institute for Economic and Social Research (TFA0027), para 10

25 C&AG’s report, paras 1.15–1.16

26 C&AG’s report para 1.21

28 Ibid

29 Department for Communities and Local Government, National Evaluation of the Troubled Families Programme: Process evaluation final report, October 2016.

16 December 2016