Troubled families: progress review Contents

2The Department’s presentation of the results

Publication of the programme evaluation

14.In August 2016, the BBC’s Newsnight programme reported that it had obtained a copy of the national evaluation of the Troubled Families programme.30 The BBC reported that this evaluation had concluded that the programme had “no discernible impact” on the lives of troubled families; claimed that there was a lack of transparency around the programme, and that the Department had suppressed the report since the autumn of the previous year due to these negative results. The National Audit Office had previously been informed by the Department that it planned to publish the national evaluation in late 2015.31

15.We wrote to the Secretary of State for Communities and Local Government on 5 October 2016 to request a copy of the evaluation reports in advance of the evidence session which took place on 19 October 2016.32 The Department did not send the evaluation reports to us in advance of our evidence session, although they did publish all evaluation reports on their website on the evening of 17 October 2016, less than 48 hours before the evidence session.33

16.We asked the Department for the reasons for the delay in the publication of the Troubled Families evaluation reports. The Department said the delay in finalising the report was due to concerns about the quality of data used in the evaluation. To remedy these issues, we were told that the Department commissioned an independent academic to quality assure the work, leading to delays in the report’s publication.34

17.The reasons for the delay in the publication were also a source of disagreement between the Department and The National Institute of Economic and Social Research (NIESR). NIESR told us that it provided a full draft report of the national impact evaluation report to the Department on 28 July 2015, after which its findings did not change.35 In addition, the head of NIESR’S impact evaluation, Jonathan Portes, claimed that a draft of the synthesis report was shared with the Department in October 2015.36

18.The Department informed us that it had finalised the reports on 10 October 2016. We asked the Department why it was not possible for them to share the evaluation reports in draft format in advance of their publication. The Department told us that ministers decided to publish the reports on 17 October and as such, the Department focused its efforts in getting the reports into the public domain as quickly as possible. The Department was evasive in addressing our queries about the delay in the publication of the evaluation reports, and did not address specifically our question regarding why the reports could not be shared with us earlier.37

19.The Department stated that it will be publishing a report to Parliament on the progress of the Troubled Families programme on an annual basis. The first of these is set to be published before the end of March 2017.38

Presentation of programme results

20.In June 2015, the Department published final data for each local authority on the number of families that they had worked with by December 2014 and the number of families “turned around” by May 2015. The published data indicated that 116,654 families had successfully met the outcomes criteria set by the Department out of 117,910 families that had been identified under the programme, representing a 99% success rate. The Department informed us that the fact that it made payments for 116,654 families under its results framework meant that these families had had their lives changed by the programme.39

21.However, the Department also acknowledged that these payments were made for “snapshots” on how families were performing at a certain point in time in areas such as attendance at school. There is therefore no guarantee that the improvements the Department has paid local authorities for, and which it uses as the basis of claiming that families are “turned around,” can be sustained over time.40

22.Additionally, in practice, the 99% success rate which the Department claimed did not include all of the families that local authorities worked with. The Department told us that they encouraged local authorities to work with more than the prescribed number of families, although did not pay them for it. The Department is not aware of how many families local authorities worked with in total.41

Financial benefits of the programme

23.In March 2015, the Department published Benefits of the Troubled Families programme to the taxpayer.42 The Department used the data in the report to claim that £1.2 billion had been saved through the programme. However, the Department had not accounted for the cost to local authorities of delivering the programme. It was also the case that these observed savings could have occurred for other reasons, such as the reduction in funding of public services, although equally, this cannot be proven.43 The Department built upon this work in a cost benefit report that was published in October 2016 as part of the national evaluation of the Troubled Families programme, and later developed a cost savings calculator for use by local authorities to enable them to estimate the cost of delivering the programme. The Department also estimated gross savings of the programme based upon the gross reduction in service use by each family one year after the intervention. The Department stressed, however, that as these were gross figures they could not be used to attribute financial benefits to the Troubled Families programme directly.44

24.The Department explained that there are challenges to building the cost savings calculator as it can be difficult to convince other national agencies to share the data that they need, such as the number of police callouts.45 We were told by the Department that it would work with local authorities to make further improvements to its cost savings calculator in order to develop a better approximation of fiscal benefits that can be attributed to the programme.46

31 C&AG’s report, para 1.24

35 National Institute for Economic and Social Research (TFA0027), para 1

36 Jonathan Portes (TFA0028), para 8

42 Department for Communities and Local Government, Benefits of the Troubled Families programme to the taxpayer, March 2015.

43 Qq 107–108, and C&AG’s report, para 1.32

44 Q 107; and Department for Communities and Local Government, Local Authority Data on the Cost and Potential Fiscal Benefits of the Troubled Families Programme, October 2016.

16 December 2016