CHAPTER 3: Measuring the effectiveness
of the ESF|
How is effectiveness measured?
80. In England, the Department for Work and Pensions
(DWP) predicted that "the availability of individual level
data for the 2007-13 programme, alongside a longitudinal survey
of 10,000 participants ... will improve understanding of who takes
up ESF, performance and longer term outcomes." We heard too
that the ESF programme in England is regularly reviewed and monitored
by national and regional ESF committees
and at annual review meetings with the European Commission (pp 2,
81. The Scottish Evaluation Plan (2007-13)
is refreshed annually. The framework for enhanced monitoring and
evaluation systems, as set out by the Scottish Government, includes
political accountability and improved programme management and
performance. The Scottish Government warned that the methodology
for evaluations should not be prescriptive in order to ensure
that specific programme priority issues could be addressed and
efficiency and effectiveness maintained. It explained that for
the Scottish programme there would be evaluations which would
cover the overarching themes of the programme, in addition to
the issues targeted by each of the priorities (p 165).
82. We heard that in Wales, "evaluations
and monitoring reports for the 2007-13 programmes are being designed
to meet the information and analysis needs of WEFO [the Welsh
European Funding Office] and key stakeholders. WEFO will provide
the all-Wales Programme Monitoring Committee with the necessary
reports to enable them to assess the effectiveness and quality
of the implementation of the programmes." In addition, headline
targets were being used to track programmes' progress in meeting
their objectives and WEFO was commissioning annual longitudinal
surveys of ESF participants in order to supplement the programme
monitoring data to assess the effectiveness of the programmes
(Q 279, pp 171-2). (See paragraph 90 and Box 8 for more
about longitudinal surveys.)
83. The Commission reported that the evaluation
system for the current programming round had "been strengthened
in order to support the monitoring of ESF interventions in terms
of outputs, outcomes and impact". This included switching
to an ongoing evaluation system (p 118).
84. In addition to this, a database of Member
States' evaluations was set up by the Commission in November 2009
and an "evaluation expert network" will be launched
after the summer to synthesise the evaluations. The Commission
suggested that this would "improve mutual learning between
public authorities involved in evaluations" and "allow
for sharing information regarding the effectiveness of the ESF
across EU-27" (p 118).
How easy is it to assess effectiveness?
85. Several of our witnesses highlighted a difficulty
in assessing the effectiveness of the ESF. Criticism ranged from
an apparent paucity of data to suggestions that what data was
available was not the right kind or could not be used to assess
efficacy at a local, as opposed to a regional, level (QQ 55-6,
97, pp 46, 47, 91, 191, 253). The DWP took a similar view,
stating that "it has always been very difficult to estimate
the effectiveness of the ESF." In particular, they identified
a difficulty in isolating the impacts attributable to the ESF
compared with other provision (p 211).
86. While the Learning and Skills Council (LSC)
recognised that data had previously come "very late in the
day to be able to make any judgments on the performance of the
programme as a whole", it believed that data monitoring had
improved in the 2007-13 programme (Q 14). The Welsh Assembly
Government was also more positive about assessing the effectiveness
of the ESF during this programme, highlighting that figures on
qualification achievements were already beginning to filter through
(Q 279, pp 171-2).
87. The Commission was the most positive about
assessing the effectiveness of the ESF and considered that there
was "a great wealth of information available through monitoring
and evaluation studies that proves the effectiveness of the ESF"
(p 114). Like the LSC, it felt that evaluation arrangements
had improved in the 2007-13 programme (p 118).
Soft and hard outcomes
Soft and hard outcomes
Hard outcomes tend to be easy to measure and quantify,
for example, the number of people participating in ESF-funded
provision who enter into employment following that intervention;
or the number who obtain a qualification.
Soft outcomes are somewhat harder to measure and
to quantify and include acquiring a skill, developing participants'
confidence, improving someone's ability to work with others, or
securing an improvement in someone's attendance or timekeeping.
That is, improving a person's employability. They can be described
as the interim steps on someone's journey towards employment.
88. Our witnesses had differing views about the
use of soft and hard outcomes and their relative merit. Some stressed
the importance of an increased focus on soft outcomes in measuring
the effectiveness of the ESF, particularly in work undertaken
with those furthest from the labour market, the "hardest
to reach" (Q 346, pp 3, 142, 165). This was felt
to be particularly important because ESF intervention could help
people on their journey towards work, without that necessarily
leading to a hard outcome such as a job, at any rate during the
life of the programme. The Government Office for the North West
stressed that even so, it did not mean such action was not valuable,
or essential in helping participants to progress (Q 189).
89. We heard from a participant at Step Up (see
Box 3), which works with 14-18 year olds to help them gain new
skills and prepare for employment, that the emotional support
provided by the project was particularly welcome (Appendix 3).
Similarly, we heard from Tomorrow's People Trust (see Box 3)
that a large part of their work was motivational in nature and
that the objective of the programme was not only to secure a job
for people but to make them more employable (Appendix 3).
90. Longitudinal cohort surveys (see Box 8) measure
the progress of participants over time and were cited as a method
of capturing and examining soft outcomes: sustainability and retention
of employment; whether participants who have improved their qualifications
and skills progress to higher level and higher paid work; and
participants' views of the support they receive (p 221).
Many witnesses, including the Government, recognised the value
of longitudinal surveys in informing assessments of the ESF's
effectiveness (Q 280, pp 172, 212). The Government Office
for the North West in particular suggested that longitudinal research
was the most effective way of measuring the impact of the ESF,
though it regretted that the necessary measures were not currently
in place to "measure the steps as opposed to the whole journey"
(QQ 186, 189).
Longitudinal Cohort Surveys
A Longitudinal Cohort Survey is the name given
to a survey used to record and examine the progress of a group
of participants (cohort) over time. This progression can include
interim steps on a participant's journey towards employment, such
as beginning to actively seek employment or undertaking a work
placement, and can follow participants for the lifetime of the
programme (QQ 189, 280).
91. Nevertheless, we heard that many providers
already used soft outcomes in measuring effectiveness, such as
the Scottish Government and the Welsh Assembly Government. Lincolnshire
County Council had an index of well being and stressed the importance
of an effectiveness assessment that looked at "how a community
has changed and how a sector has changed just as much as about
the individual" (QQ 276-7, 278, 329).
92. The Commission supported the current soft
outcome indicators in the Welsh and English programmes, which
include showing the background of those on the programmes and
results indicators used to identify soft outcomes such as the
number of people engaged in job search activity (p 142).
The Commission welcomed further debate and decisions in the UK
about whether there could be an evaluation of distance travelled
towards employment in future programmes (Q 241).
93. However, the Minister and the Commission
both thought that the application of soft outcomes was limited
(Q 360, p 142). The Minister in particular was quite
critical about their use, stating that they were "all very
but in the end getting close to the labour market
does not mean much until you have a job and that is what we want
to get people into" (Q 360). He highlighted that the
Government were currently "focusing on what the outcome is
right at the end" (Q 361).
94. Other witnesses also had concerns about the
use of soft outcomes to assess the effectiveness of the ESF. The
LSC agreed that they were difficult to define and hard to capture
(p 15). Higher Education European Funding Services Limited
felt that it was harder to put a value on a soft outcome and that
it was "a bit difficult to accept as a concept". Moreover,
it highlighted the potential difficulty in presenting this information
"in an accessible way" (Q 60). Similarly, the WEFO
highlighted that there was no single "off the shelf"
approach for measuring soft outcomes that would suit all projects
because of the wide variety of activities and client groups involved
and the difficulty in tying concepts such as "increased confidence"
down (pp 188-9).
95. Another disincentive associated with the
use of soft outcomes was that organisations "are not paid
a high premium for moving people along towards employment, unless
they get an employment outcome" (Q 82). This point was
echoed by Step Up, which highlighted that while hard outcomes
are necessary for the release of funds, they did not reflect many
of the soft outcomes, which it thought offered the greatest benefits
to participants (Appendix 3).
96. The TSEN felt that hard outcomes were easier
to achieve with those closer to the labour market and that the
2007-13 programme focused on these participants as a result (pp 40-41).
The Government Office for the North West also identified an emphasis
on hard outcomes for the programmes, while the LSC was somewhat
more understanding of this focus, recognising that the purpose
of the ESF was to help people to return to employment and that
this therefore had to be an in-built target in contracting arrangements
for ESF programmes (QQ 17, 189). Lincolnshire County Council
similarly identified "a clear need" for hard outcomes
97. One of the advantages of using hard outcomes,
as recognised by Higher Education European Funding Services Limited
and the TSEN, was that they were easy to measure (QQ 60, 82).
In fact, we heard that the Government have deliberately moved
to more outcome-based measures of performance. Whereas in the
past some programmes were funded for activity rather than achieving
hard outcomes, the Minister argued that "if they were not
getting them into work, then in the end we had to find someone
who would" (Q 381). Furthermore, he explained that the
Government were "holding prime contractors to account for
outcomes in a way we have not done nearly so much before. We are
not paying them unless and until they get them into work and keep
them there to a much higher extent than before" (Q 389).
The Commission was similarly intent on a greater emphasis on outcomes
in future (Q 229).
98. In this context, we heard from Tomorrow's
People Trust that whereas under the previous programme providers
received funds if they demonstrated that they were fulfilling
the obligations under that project, there were now two elements
to fees, with a management tranche and an output-related segment.
The split between the two varies from project to project and we
heard that there were instances where the Trust had decided not
to bid for funding because the split entailed too much upfront
expenditure and risk for the provider (Appendix 3).
99. Many witnesses were critical of the auditing
and administrative requirements for previous programmes. The LSC
cited a high level of bureaucracy "where they had to go down
to the level of providing bus tickets for learners". However,
it suggested that "the co-financing arrangements have removed
all of that" (Q 6). The Minister agreed that co-financing
(see paragraphs 14-16) meant that organisations were under less
onerous auditing obligations, having removed the burden for applicants
of supplying their own match funding which was cited as a major
source of audit problems in the past, a point echoed by the Commission
(QQ 202, 390-1, p 210).
100. Not all of our witnesses agreed that matters
had improved under co-financing, however. Off the Streets and
into Work (OSW) believed that its introduction had "significantly
increased the bureaucratic burden on providers" (p 250)
and the South West Regional Development Agency described the audit
regime as being "on a scale where there is no comparison
with the previous European programmes". It reported that
it had attempted to "shield" some of the direct recipients
and deliverers from that bureaucracy (QQ 141-2). The Chief
Executive of Co-ordinating European Funding for the East Midlands
Third Sector (CEFET) suggested that "the administrative burden
on the delivery bodies has roughly quadrupled in terms of the
staff time needed to fill in the returns and draw down the money"
(Q 82, see also p 69). The WEFO and Community Service
Volunteers were in agreement that bureaucracy was still an issue
under co-financing (QQ 82, 303).
101. OSW highlighted the risk that "the
ever-increasing burden of paperwork endemic to ESF delivery"
could exclude smaller, specialist providers (p 251). A similar
view was expressed by others, including the Commission, the Scottish
Government, the WEFO and Community Service Volunteers (QQ 234,
297, 303, p 47). Likewise, the TSEN thought that bureaucracy
acted as a deterrent, as did the North West Universities Association
which cited the retention period of 10 years for records as a
disincentive for institutions to use ESF funding (Q 46, p 62,
see also p 164). The Commission responded that "we cannot
draw a line and say that there are audit requirements ... except
for a range of organisations. That is not possible" (Q 234).
102. Criticisms also included the variation in
paperwork requirements between programmes, which Women Like Us
thought could be a barrier for clients pursuing involvement in
a programme (p 255).
103. While many of our witnesses recognised that
there was "an element of bureaucracy associated with the
European funding" and the Commission acknowledged that it
"would be lying if we said that we are not bureaucratic;
of course we are", Higher Education European Funding Services
Limited had "no problems" with the reporting requirements
and "no real complaints" about the level of administration
attached to the programme. Moreover, others thought it was worth
accepting the additional auditing obligations and burdens in order
to deliver a more detailed project (QQ 9, 46, 229, 308).
104. As Higher Education European Funding Services
Limited highlighted, "we would expect any programme of public
funding to come with a degree of administration" (Q 46).
The Mayor of London and London Councils believed that the audit
regime was based on sound principles and ensured accountability
of public funding (p 192). In addition, it was recognised
that auditing requirements ensured that "money cannot go
walkabout" (Q 12).
105. The Commission also stressed the need for
some bureaucracy and auditing requirements and highlighted the
trade-off between spending money properly and reporting on what
was achieved (Q 230). It described more recent innovations
to try to reduce the bureaucracy, for example where "you
do not account for each individual invoice in order to get it
reimbursed from Brussels but you settle a standard cost for specific
training, a specific intervention, and get it reimbursed once
you demonstrate this activity has taken place" (Q 231,
106. Moreover, we heard that the Commission adopted
a lighter touch approach where a "contract of confidence"
has been awarded (the English audit authority became the first
to be awarded such a contract during the 2000-06 programming period).
This contract formally recognises where the Commission has complete
confidence in the audit arrangements and requires the Member State
to provide certain assurances on the robustness of monitoring
and audit systems (p 211).
107. The Mayor of London and London Councils
wanted to see audits of co-financing organisations that included
spot checks earlier in the programme to ensure that systems were
compliant with ESF rules and regulations (p 193). The North
West Universities Association, meanwhile, commented that the ESF
seemed to have been designed for "an age when paper was predominant"
and suggested that this had led to challenges for some institutions
in transferring electronic records into hard copy for auditing
purposes (Q 46).
108. The Welsh Assembly Government called for
a single audit body in the Commission with common audit standards,
alongside proportionality of audit requirements, including the
issue of document retention, which we heard from witnesses could
be more of an obstacle for smaller organisations (Q 296,
p 171). (See paragraph 101). This point was also addressed
by the Government, who, in relation to the amending regulations
going through the European Parliament and Council (see Box 9),
are calling for EU rules that are proportionate to the nature
and size of projects, and relaxation of the rules requiring even
small projects to retain project documentation (p 211). The
Mayor of London and London Councils were also keen to see a simpler
regime (p 192).
109. Scotland failed to meet the audit requirements
in the last programme. The Court of Auditors' report was described
as "devastating" by the Commission, which considered
that there were still some "major deficiencies" in the
ESF audit system in Scotland. It intended to apply a financial
correction and to ask the Scottish authorities to improve the
management and auditing of the ESF programmes for the 2007-2013
period (QQ 236-7).
110. For its part, the Scottish Government recognised
that the audit trail was not perfect. It suggested that because
it had moved from over 3,000 relatively small programmes under
the old programme (which had difficulty in keeping the necessary
audit trail) to larger projects under the current programme, this
might prove easier with regard to audits, though it felt there
was still an administrative burden (Q 296). Nevertheless,
it did not think that the audit requirements would prevent it
delivering programmes as it intended, though it suggested there
was a risk that unless it could ensure that projects' audit trails
were robust, applicants might "back off because the burden
outweighs the benefits of drawing down the funds" (Q 307).
Conclusions and recommendations
111. It is clear to us that a trusted and
robust methodology for assessing effectiveness is key to the short-
and long-term future of the European Social Fund. We conclude
that there is substantial room for improvement.
112. We welcome moves to strengthen the evaluation
systems for the current programme, particularly the improved opportunities
for mutual learning and sharing of best practice within and between
Member States. However, we note that there are no set dates for
evaluations under the new "ongoing" system. Data must
be produced more quickly and regularly so that performance issues
can be addressed mid-way through a programme, rather than after
the seven years has run its course. Otherwise the value of evaluations
could be lost and the time taken to conduct them wasted. We
therefore recommend greater use of timely interim reporting to
the Commission and the Government.
113. Above all, it is clear to us that there
is an excessive reliance on measuring hard outcomes almost to
the exclusion of soft outcomes. We recognise the Minister's argument
that job outcomes are the ultimate and legitimate aim of ESF intervention.
However, intermediate work with the hard to reach is a necessary
route towards the labour market, helping to improve people's employability.
Although this may not lead to tangible outcomes during the lifetime
of a programme, such action is nevertheless necessary and valid.
We therefore recommend that the Government reconsider their
rigid approach of increasingly withholding payment from providers
unless they get people into work and keep them there, not least
because a failure to secure a job does not in itself indicate
a flaw in the activity.
114. Moreover, we are not convinced that soft
outcomes are sufficiently difficult to define to prevent their
inclusion in assessments of the effectiveness of the ESF. We are
concerned that the complete exclusion of soft outcomes would encourage
providers to focus on those closer to the labour market at the
expense of the harder to reach. We were particularly struck
by the use of longitudinal cohort surveys and recommend that such
surveys be made compulsory at a regional and co-financing organisation
level. Not only do we consider that these would be of use
in capturing soft outcomes, we believe they would also facilitate
assessments of the sustainability of hard outcomes, such as job
retention and progression.
115. We note that the England ESF Operational
Programme already includes soft outcome indicators and we recommend
that the Government and the Commission explore what other measures
could be used to record progress amongst those further from the
labour market. Ultimately, programme providers should be required
to collect data on soft and hard outcomes alike.
116. As many of our witnesses pointed out, it
is necessary to have an element of bureaucracy in any programme,
particularly with the sums of money involved in the ESF and the
sensitivity around the spending of EU money. Nevertheless, it
is essential that this is proportionate to the size of the programme.
We therefore recommend that the Government continue to press
for the reduction of the 10-year records retention requirement
for smaller organisations, whose valuable role in the ESF must
not be hindered by disproportionate audit requirements.
117. Given the differing views among our witnesses
about the effect of co-financing on the bureaucratic burden faced
by providers, we recommend that the Government carry out an assessment
of this in order that any such issues can be identified and addressed
26 These include representatives of: Government; the
European Commission; voluntary and social enterprise sectors;
the social partners (TUC and CBI); further and higher education;
local authorities; and regional bodies. Back