Making it work: the European Social Fund - European Union Committee Contents

CHAPTER 3: Measuring the effectiveness of the ESF

How is effectiveness measured?

80.  In England, the Department for Work and Pensions (DWP) predicted that "the availability of individual level data for the 2007-13 programme, alongside a longitudinal survey of 10,000 participants ... will improve understanding of who takes up ESF, performance and longer term outcomes." We heard too that the ESF programme in England is regularly reviewed and monitored by national and regional ESF committees[26] and at annual review meetings with the European Commission (pp 2, 212).

81.  The Scottish Evaluation Plan (2007-13) is refreshed annually. The framework for enhanced monitoring and evaluation systems, as set out by the Scottish Government, includes political accountability and improved programme management and performance. The Scottish Government warned that the methodology for evaluations should not be prescriptive in order to ensure that specific programme priority issues could be addressed and efficiency and effectiveness maintained. It explained that for the Scottish programme there would be evaluations which would cover the overarching themes of the programme, in addition to the issues targeted by each of the priorities (p 165).

82.  We heard that in Wales, "evaluations and monitoring reports for the 2007-13 programmes are being designed to meet the information and analysis needs of WEFO [the Welsh European Funding Office] and key stakeholders. WEFO will provide the all-Wales Programme Monitoring Committee with the necessary reports to enable them to assess the effectiveness and quality of the implementation of the programmes." In addition, headline targets were being used to track programmes' progress in meeting their objectives and WEFO was commissioning annual longitudinal surveys of ESF participants in order to supplement the programme monitoring data to assess the effectiveness of the programmes (Q 279, pp 171-2). (See paragraph 90 and Box 8 for more about longitudinal surveys.)

83.  The Commission reported that the evaluation system for the current programming round had "been strengthened in order to support the monitoring of ESF interventions in terms of outputs, outcomes and impact". This included switching to an ongoing evaluation system (p 118).

84.  In addition to this, a database of Member States' evaluations was set up by the Commission in November 2009 and an "evaluation expert network" will be launched after the summer to synthesise the evaluations. The Commission suggested that this would "improve mutual learning between public authorities involved in evaluations" and "allow for sharing information regarding the effectiveness of the ESF across EU-27" (p 118).

How easy is it to assess effectiveness?

85.  Several of our witnesses highlighted a difficulty in assessing the effectiveness of the ESF. Criticism ranged from an apparent paucity of data to suggestions that what data was available was not the right kind or could not be used to assess efficacy at a local, as opposed to a regional, level (QQ 55-6, 97, pp 46, 47, 91, 191, 253). The DWP took a similar view, stating that "it has always been very difficult to estimate the effectiveness of the ESF." In particular, they identified a difficulty in isolating the impacts attributable to the ESF compared with other provision (p 211).

86.  While the Learning and Skills Council (LSC) recognised that data had previously come "very late in the day to be able to make any judgments on the performance of the programme as a whole", it believed that data monitoring had improved in the 2007-13 programme (Q 14). The Welsh Assembly Government was also more positive about assessing the effectiveness of the ESF during this programme, highlighting that figures on qualification achievements were already beginning to filter through (Q 279, pp 171-2).

87.  The Commission was the most positive about assessing the effectiveness of the ESF and considered that there was "a great wealth of information available through monitoring and evaluation studies that proves the effectiveness of the ESF" (p 114). Like the LSC, it felt that evaluation arrangements had improved in the 2007-13 programme (p 118).

Soft and hard outcomes


Soft and hard outcomes

Hard outcomes tend to be easy to measure and quantify, for example, the number of people participating in ESF-funded provision who enter into employment following that intervention; or the number who obtain a qualification.

Soft outcomes are somewhat harder to measure and to quantify and include acquiring a skill, developing participants' confidence, improving someone's ability to work with others, or securing an improvement in someone's attendance or timekeeping. That is, improving a person's employability. They can be described as the interim steps on someone's journey towards employment.

88.  Our witnesses had differing views about the use of soft and hard outcomes and their relative merit. Some stressed the importance of an increased focus on soft outcomes in measuring the effectiveness of the ESF, particularly in work undertaken with those furthest from the labour market, the "hardest to reach" (Q 346, pp 3, 142, 165). This was felt to be particularly important because ESF intervention could help people on their journey towards work, without that necessarily leading to a hard outcome such as a job, at any rate during the life of the programme. The Government Office for the North West stressed that even so, it did not mean such action was not valuable, or essential in helping participants to progress (Q 189).

89.  We heard from a participant at Step Up (see Box 3), which works with 14-18 year olds to help them gain new skills and prepare for employment, that the emotional support provided by the project was particularly welcome (Appendix 3). Similarly, we heard from Tomorrow's People Trust (see Box 3) that a large part of their work was motivational in nature and that the objective of the programme was not only to secure a job for people but to make them more employable (Appendix 3).

90.  Longitudinal cohort surveys (see Box 8) measure the progress of participants over time and were cited as a method of capturing and examining soft outcomes: sustainability and retention of employment; whether participants who have improved their qualifications and skills progress to higher level and higher paid work; and participants' views of the support they receive (p 221). Many witnesses, including the Government, recognised the value of longitudinal surveys in informing assessments of the ESF's effectiveness (Q 280, pp 172, 212). The Government Office for the North West in particular suggested that longitudinal research was the most effective way of measuring the impact of the ESF, though it regretted that the necessary measures were not currently in place to "measure the steps as opposed to the whole journey" (QQ 186, 189).


Longitudinal Cohort Surveys

A Longitudinal Cohort Survey is the name given to a survey used to record and examine the progress of a group of participants (cohort) over time. This progression can include interim steps on a participant's journey towards employment, such as beginning to actively seek employment or undertaking a work placement, and can follow participants for the lifetime of the programme (QQ 189, 280).

91.  Nevertheless, we heard that many providers already used soft outcomes in measuring effectiveness, such as the Scottish Government and the Welsh Assembly Government. Lincolnshire County Council had an index of well being and stressed the importance of an effectiveness assessment that looked at "how a community has changed and how a sector has changed just as much as about the individual" (QQ 276-7, 278, 329).

92.  The Commission supported the current soft outcome indicators in the Welsh and English programmes, which include showing the background of those on the programmes and results indicators used to identify soft outcomes such as the number of people engaged in job search activity (p 142). The Commission welcomed further debate and decisions in the UK about whether there could be an evaluation of distance travelled towards employment in future programmes (Q 241).

93.  However, the Minister and the Commission both thought that the application of soft outcomes was limited (Q 360, p 142). The Minister in particular was quite critical about their use, stating that they were "all very well … but in the end getting close to the labour market does not mean much until you have a job and that is what we want to get people into" (Q 360). He highlighted that the Government were currently "focusing on what the outcome is right at the end" (Q 361).

94.  Other witnesses also had concerns about the use of soft outcomes to assess the effectiveness of the ESF. The LSC agreed that they were difficult to define and hard to capture (p 15). Higher Education European Funding Services Limited felt that it was harder to put a value on a soft outcome and that it was "a bit difficult to accept as a concept". Moreover, it highlighted the potential difficulty in presenting this information "in an accessible way" (Q 60). Similarly, the WEFO highlighted that there was no single "off the shelf" approach for measuring soft outcomes that would suit all projects because of the wide variety of activities and client groups involved and the difficulty in tying concepts such as "increased confidence" down (pp 188-9).

95.  Another disincentive associated with the use of soft outcomes was that organisations "are not paid a high premium for moving people along towards employment, unless they get an employment outcome" (Q 82). This point was echoed by Step Up, which highlighted that while hard outcomes are necessary for the release of funds, they did not reflect many of the soft outcomes, which it thought offered the greatest benefits to participants (Appendix 3).

96.  The TSEN felt that hard outcomes were easier to achieve with those closer to the labour market and that the 2007-13 programme focused on these participants as a result (pp 40-41). The Government Office for the North West also identified an emphasis on hard outcomes for the programmes, while the LSC was somewhat more understanding of this focus, recognising that the purpose of the ESF was to help people to return to employment and that this therefore had to be an in-built target in contracting arrangements for ESF programmes (QQ 17, 189). Lincolnshire County Council similarly identified "a clear need" for hard outcomes (Q 329).

97.  One of the advantages of using hard outcomes, as recognised by Higher Education European Funding Services Limited and the TSEN, was that they were easy to measure (QQ 60, 82). In fact, we heard that the Government have deliberately moved to more outcome-based measures of performance. Whereas in the past some programmes were funded for activity rather than achieving hard outcomes, the Minister argued that "if they were not getting them into work, then in the end we had to find someone who would" (Q 381). Furthermore, he explained that the Government were "holding prime contractors to account for outcomes in a way we have not done nearly so much before. We are not paying them unless and until they get them into work and keep them there to a much higher extent than before" (Q 389). The Commission was similarly intent on a greater emphasis on outcomes in future (Q 229).

98.  In this context, we heard from Tomorrow's People Trust that whereas under the previous programme providers received funds if they demonstrated that they were fulfilling the obligations under that project, there were now two elements to fees, with a management tranche and an output-related segment. The split between the two varies from project to project and we heard that there were instances where the Trust had decided not to bid for funding because the split entailed too much upfront expenditure and risk for the provider (Appendix 3).

Administrative requirements

99.  Many witnesses were critical of the auditing and administrative requirements for previous programmes. The LSC cited a high level of bureaucracy "where they had to go down to the level of providing bus tickets for learners". However, it suggested that "the co-financing arrangements have removed all of that" (Q 6). The Minister agreed that co-financing (see paragraphs 14-16) meant that organisations were under less onerous auditing obligations, having removed the burden for applicants of supplying their own match funding which was cited as a major source of audit problems in the past, a point echoed by the Commission (QQ 202, 390-1, p 210).

100.  Not all of our witnesses agreed that matters had improved under co-financing, however. Off the Streets and into Work (OSW) believed that its introduction had "significantly increased the bureaucratic burden on providers" (p 250) and the South West Regional Development Agency described the audit regime as being "on a scale where there is no comparison with the previous European programmes". It reported that it had attempted to "shield" some of the direct recipients and deliverers from that bureaucracy (QQ 141-2). The Chief Executive of Co-ordinating European Funding for the East Midlands Third Sector (CEFET) suggested that "the administrative burden on the delivery bodies has roughly quadrupled in terms of the staff time needed to fill in the returns and draw down the money" (Q 82, see also p 69). The WEFO and Community Service Volunteers were in agreement that bureaucracy was still an issue under co-financing (QQ 82, 303).

101.  OSW highlighted the risk that "the ever-increasing burden of paperwork endemic to ESF delivery" could exclude smaller, specialist providers (p 251). A similar view was expressed by others, including the Commission, the Scottish Government, the WEFO and Community Service Volunteers (QQ 234, 297, 303, p 47). Likewise, the TSEN thought that bureaucracy acted as a deterrent, as did the North West Universities Association which cited the retention period of 10 years for records as a disincentive for institutions to use ESF funding (Q 46, p 62, see also p 164). The Commission responded that "we cannot draw a line and say that there are audit requirements ... except for a range of organisations. That is not possible" (Q 234).

102.  Criticisms also included the variation in paperwork requirements between programmes, which Women Like Us thought could be a barrier for clients pursuing involvement in a programme (p 255).

103.  While many of our witnesses recognised that there was "an element of bureaucracy associated with the European funding" and the Commission acknowledged that it "would be lying if we said that we are not bureaucratic; of course we are", Higher Education European Funding Services Limited had "no problems" with the reporting requirements and "no real complaints" about the level of administration attached to the programme. Moreover, others thought it was worth accepting the additional auditing obligations and burdens in order to deliver a more detailed project (QQ 9, 46, 229, 308).

104.  As Higher Education European Funding Services Limited highlighted, "we would expect any programme of public funding to come with a degree of administration" (Q 46). The Mayor of London and London Councils believed that the audit regime was based on sound principles and ensured accountability of public funding (p 192). In addition, it was recognised that auditing requirements ensured that "money cannot go walkabout" (Q 12).

105.  The Commission also stressed the need for some bureaucracy and auditing requirements and highlighted the trade-off between spending money properly and reporting on what was achieved (Q 230). It described more recent innovations to try to reduce the bureaucracy, for example where "you do not account for each individual invoice in order to get it reimbursed from Brussels but you settle a standard cost for specific training, a specific intervention, and get it reimbursed once you demonstrate this activity has taken place" (Q 231, p 119).

106.  Moreover, we heard that the Commission adopted a lighter touch approach where a "contract of confidence" has been awarded (the English audit authority became the first to be awarded such a contract during the 2000-06 programming period). This contract formally recognises where the Commission has complete confidence in the audit arrangements and requires the Member State to provide certain assurances on the robustness of monitoring and audit systems (p 211).

Possible improvements

107.  The Mayor of London and London Councils wanted to see audits of co-financing organisations that included spot checks earlier in the programme to ensure that systems were compliant with ESF rules and regulations (p 193). The North West Universities Association, meanwhile, commented that the ESF seemed to have been designed for "an age when paper was predominant" and suggested that this had led to challenges for some institutions in transferring electronic records into hard copy for auditing purposes (Q 46).

108.  The Welsh Assembly Government called for a single audit body in the Commission with common audit standards, alongside proportionality of audit requirements, including the issue of document retention, which we heard from witnesses could be more of an obstacle for smaller organisations (Q 296, p 171). (See paragraph 101). This point was also addressed by the Government, who, in relation to the amending regulations going through the European Parliament and Council (see Box 9), are calling for EU rules that are proportionate to the nature and size of projects, and relaxation of the rules requiring even small projects to retain project documentation (p 211). The Mayor of London and London Councils were also keen to see a simpler regime (p 192).

Auditing problems

109.  Scotland failed to meet the audit requirements in the last programme. The Court of Auditors' report was described as "devastating" by the Commission, which considered that there were still some "major deficiencies" in the ESF audit system in Scotland. It intended to apply a financial correction and to ask the Scottish authorities to improve the management and auditing of the ESF programmes for the 2007-2013 period (QQ 236-7).

110.  For its part, the Scottish Government recognised that the audit trail was not perfect. It suggested that because it had moved from over 3,000 relatively small programmes under the old programme (which had difficulty in keeping the necessary audit trail) to larger projects under the current programme, this might prove easier with regard to audits, though it felt there was still an administrative burden (Q 296). Nevertheless, it did not think that the audit requirements would prevent it delivering programmes as it intended, though it suggested there was a risk that unless it could ensure that projects' audit trails were robust, applicants might "back off because the burden outweighs the benefits of drawing down the funds" (Q 307).

Conclusions and recommendations

111.  It is clear to us that a trusted and robust methodology for assessing effectiveness is key to the short- and long-term future of the European Social Fund. We conclude that there is substantial room for improvement.

112.  We welcome moves to strengthen the evaluation systems for the current programme, particularly the improved opportunities for mutual learning and sharing of best practice within and between Member States. However, we note that there are no set dates for evaluations under the new "ongoing" system. Data must be produced more quickly and regularly so that performance issues can be addressed mid-way through a programme, rather than after the seven years has run its course. Otherwise the value of evaluations could be lost and the time taken to conduct them wasted. We therefore recommend greater use of timely interim reporting to the Commission and the Government.

113.  Above all, it is clear to us that there is an excessive reliance on measuring hard outcomes almost to the exclusion of soft outcomes. We recognise the Minister's argument that job outcomes are the ultimate and legitimate aim of ESF intervention. However, intermediate work with the hard to reach is a necessary route towards the labour market, helping to improve people's employability. Although this may not lead to tangible outcomes during the lifetime of a programme, such action is nevertheless necessary and valid. We therefore recommend that the Government reconsider their rigid approach of increasingly withholding payment from providers unless they get people into work and keep them there, not least because a failure to secure a job does not in itself indicate a flaw in the activity.

114.  Moreover, we are not convinced that soft outcomes are sufficiently difficult to define to prevent their inclusion in assessments of the effectiveness of the ESF. We are concerned that the complete exclusion of soft outcomes would encourage providers to focus on those closer to the labour market at the expense of the harder to reach. We were particularly struck by the use of longitudinal cohort surveys and recommend that such surveys be made compulsory at a regional and co-financing organisation level. Not only do we consider that these would be of use in capturing soft outcomes, we believe they would also facilitate assessments of the sustainability of hard outcomes, such as job retention and progression.

115.  We note that the England ESF Operational Programme already includes soft outcome indicators and we recommend that the Government and the Commission explore what other measures could be used to record progress amongst those further from the labour market. Ultimately, programme providers should be required to collect data on soft and hard outcomes alike.

116.  As many of our witnesses pointed out, it is necessary to have an element of bureaucracy in any programme, particularly with the sums of money involved in the ESF and the sensitivity around the spending of EU money. Nevertheless, it is essential that this is proportionate to the size of the programme. We therefore recommend that the Government continue to press for the reduction of the 10-year records retention requirement for smaller organisations, whose valuable role in the ESF must not be hindered by disproportionate audit requirements.

117.  Given the differing views among our witnesses about the effect of co-financing on the bureaucratic burden faced by providers, we recommend that the Government carry out an assessment of this in order that any such issues can be identified and addressed systematically.

26   These include representatives of: Government; the European Commission; voluntary and social enterprise sectors; the social partners (TUC and CBI); further and higher education; local authorities; and regional bodies. Back

previous page contents next page

House of Lords home page Parliament home page House of Commons home page search page enquiries index

© Parliamentary copyright 2010