Monitoring outcomes and Inspections
92. The European Commission requires that all member
states evaluate how well ESF is working in their country. For
ensuring that the member states own systems are satisfactory,
the Commission has "a rather complex monitoring and evaluation
system" in place. [183]
The Commission focuses on how the monitoring systems work, leaving
it up to member states to set up the particular way in which they
monitor the actual beneficiaries.[184]
The Commission undertakes regular check-ups and issues recommendations
and comments to individual member states on their monitoring systems.[185]
In the case of the objective 3 programme, the Commission told
us
"it was rather positive picture because
the UK has set a rather elaborate set of indicators which are
partly indicators on programme efficiency, partly how the control
systems work, the management system, and how the funds are being
used. If you look at the beneficiaries.[.].50 per cent of the
beneficiaries of social fund programmes are in work after leaving
the programme. That is a relatively good figure when you think
that we are focussing on those at a disadvantage in the labour
market. We have 88 per cent of the beneficiaries who are given
training or some form of course who are actually completing their
courses. About 40 per cent have increased their qualifications
through the structural funds. Then we have a set of surveys where
we ask about satisfaction. In 2001, 80 per cent said that ESF
funded courses met their needs, so they are relatively happy with
this; 55 per cent of participants felt that they improved their
computing skills or their IT skills which is a key area. This
should be looked at with the background that 35 per cent of participants
in the ESF funded courses had no prior qualifications whatsoever.
We feel that it is effective.[186]
93. The Government told us that there was some encouraging
evidence that the performance of the England Objective 3 was good.
[187] The Minister
told us:
"16 per cent of participants on that programme
are long-term unemployed; 35 per cent have no prior qualifications;
13 per cent have disabilities and 15 per cent are from black and
ethnic minorities. In the period 2000-02, 360,000 people in England
benefited from ESF. The final reports of projects tell us that
41 per cent of those gained a qualification, many more gained
or improved IT and computing skills and over three quarters76
per centof those who completed ESF courses were in jobs
or had started further education."
The Government also said that the programme exceeded
its expenditure target of £179 million for 2002 and so avoided
decommitment of any funding by the European Commission and was
on course to achieve its expenditure target for 2003.[188]
The Minister also told us that the UK was on course to achieve
the £100 million (performance reserve) for the Department's
Objective 3 (England) programme.[189]
94. More information will be available from the mid-term
evaluation. For example, the evaluation of England Objective 3
includes the issues of employment, skill development, and the
employment of women and information and communications technologies
(ICT).[190]
There is a separate evaluation of Equal, which is testing new
ways of combating discrimination and inequality in the labour
market. The Scottish Executive and the Welsh Assembly Government
are undertaking similar evaluations of Objective 3 in Scotland
and Wales respectively. The DWP will bring together the England,
Scotland and Wales Objective 3 evaluations, in an evaluation of
the overarching Great Britain Community Support Framework.
95. We are not in a position to comment on the effectiveness
of the ESF programme as a whole. However, we heard some indications
that the GB's Objective 3 programme was performing well. For example,
we were pleased to hear the European Commission describe the GB
Objective 3 programme as an exemplary model in terms of linking
outcomes with National Action Plan on employment (NAP) and that
it was used in the Commission's training.[191]
According to the Minister, the DWP was on track to secure the
performance reserve, which is an additional 4% of ESF expenditure
that is performance related and that the spending target for the
year had been achieved. [192]
96. We also note that in a written answer, the Minister
said:
"[Objective 3 and Equal]. are part of a system
of active labour market policies that, together with steady economic
growth, has contributed to 1.5 million more people into work since
1997, of which 850,000 are additional women in employment."[193]
97. As the DTI's consultation identified, while funding
was thought to be a useful source for skills-related projects,
there were concerns that Objective 3 funds may be spread too thinly
to all parts of the UK and that they may be poorly co-ordinated
with expenditure on Objective 2. Some respondents expressed the
view that targeting funding to areas of need would be better though
others disagreed and wanted continued wide coverage. We recommend
that Objective 3 funding should continue to be widely available
.[194]
98. The need to demonstrate "added value"
has meant that projects are subject to detailed monitoring. A
recurring criticism of ESF funding is the difficulty of quantifying
the so-called "soft outcomes" associated with some training
projects. For example, it is widely recognised that concepts such
as "employability" are exceedingly difficult to measure
accurately. One obviously narrow way of quantifying some aspects
of employability would be simply to count the number of beneficiaries
that find jobs at the end of the training programme. Although
relatively easy to quantify, such a measure, however, would be
too narrow to do justice to the full range of training activities
undertaken or the progress made by individual clients towards
their training objectives. For example, some final beneficiaries
on ESF projects may have made great strides towards work, without
necessarily obtaining employment. This progress should be recognised
in any measure of outcomes as a worthwhile activity. This progress
is especially important for those final beneficiaries with mental
or physical problems or those who lack basic skills. For some
people, the transition to work may be a long journey and their
acquisition of key skills or progress towards future employment,
however slight, should be recognised.[195]
For example, an individual's self-confidence may be improved
or their mental and physical condition stabilised as a result
of participating on a project. Projects that encourage this should
not be penalised. We raised this issue with the Minister and he
told us:
"[NEWTEC ]..used the very powerful example
of a lone parent with very few English language skills who is
making the journeyfrom being totally excludedtowards
the labour market but has not got there by the end of the project,
but nevertheless there has been considerable value added for her
as well as for the economy as a whole. That sort of thing is
very difficult to measure at a national evaluation level and certainly
in a European evaluation level of a programme like Equal, but
you can get some qualitative notion out of the reports at the
end of each phase of these projects. I think that is something
from which, particularly with a programme like Equal, you can
make some assessment about how valuable it is comparing with what
is happening in other parts of the European Union."[196]
99. While we recognise that in a world of targets
and quantification, it is inevitable that some monitoring of outcomes
is needed, we believe that it is equally important to devise meaningful
and valid measures that do not make excessive demands on service
providers for data. Under the current programming period, the
responsibility for monitoring projects and programmes has shifted
from the EC to the Government Offices and Monitoring Committees.
This delegation of responsibility allows different regions some
scope to apply different monitoring standards. According to a
study into the implementation of structural funds by Strathclyde
University:
"The process of developing a monitoring
and evaluation framework has been largely
directed from the 'top down',
driven by central
government departments, but with scope for some regions to make
their own decisions on indicators, monitoring procedures etc.
"While the national authorities have had
a key role in setting the framework and providing guidance, it
has been the regional Government Offices (in England) and the
Programme Management Executives (Scotland, Wales) that have given
practical effect to the guidance and the increasing commitment
of resources to evaluation. As in other areas of Structural Funds
implementation, the approach to monitoring differs between the
constituent parts of the UK. In England, the emphasis is on ensuring
that project monitoring is carried out efficiently and effectively,
so that national govermnent can ensure compliance with the financial
management, monitoring and control requirements of the EU. An
integrated system for reporting physical outputs is currently
being developed, with a common structure for collating core output
information across the English regions, and guidance on standard
monitoring procedures is being issued to regions as part of a
Structural Funds Manual. The English regions vary greatly in the
sophistication of their monitoring arrangements and the range
(and definition) of monitoring information collected. In Wales,
the emphasis also appears to be on regulatory compliance with
respect to financial and physical monitoring. By comparison,
in Scotland the greater transparency required as a result of devolution
has led to significant investment in new frameworks and systems
for monitoring, led by the Scottish Executive, to collate standardised
information across all Structural Fund programmes with common
definitions. These efforts are not only to ensure compliance
with the EU regulations but also to provide accountability to
the Scottish Parliament and to create an effective programming
tool"[197]
100. This difference in the degree of monitoring
in different parts of England and the UK generally was indicated
to us during our visit to Scotland. We heard that the rate of
returned forms were dramatically reduced when claim forms were
being checked by a different PME. The DWP told us:
"The Commission has adopted regulations
which lay down detailed rules on the eligibility of expenditure
and on management and control systems.[198]
These specify the minimum controls that Member States must apply
to ensure Structural Fund money is used correctly and enable the
Commission to verify the validity of Community expenditure. Commission
auditors visit the UK about four or five times a year to check
compliance with the regulations. The Commission is subject to
independent scrutiny by the European Court of Auditors. Auditors
from the Court visit the UK about once or twice a year."[199]
101. As we commented above, there is anecdotal evidence
that suggests that different GOs and Programme Management Executive
Committees may have different perceptions of the risk that the
Commission or the EUROPEAN COURT OF AUDITORS will intervene. As
a result, the degree of monitoring and financial control that
is imposed on service providers may vary considerably between
regions. We recommend that the DWP provides a comparative analysis
of any variation in monitoring standards across the UK and reassures
us that monitoring of service providers is proportionate to their
assessed risk and broadly comparable across the UK and the EU.
Feedback and accountability
102. We heard some criticism of the poor standard
of feedback provided to applicants whose application for ESF funding
had been rejected, specifically that the reasons for rejection
were not always given.[200]
In a note, the Government describes the feedback that is given
to unsuccessful applicants, including the fact that feedback letters
are quality assured by the relevant Regional Jobcentre Plus ESF
manager and at a national level by Jobcentre Plus Internal Audit
during any audit of ESF-related contracting activities. The Government
says that
"Following the appraisal of tenders for
ESF activity, the Jobcentre Plus Regional Office issues letters
to all unsuccessful applicants explaining that their application
cannot be supported. The letters either provide written feedback
or inform applicants that written feedback is available to them
on request.
"Written feedback is provided in the form
of a proforma indicating how the bid fared against the selection
criteria or by letter giving reasons why the application was unsuccessful
and providing advice on how any future bids might be improved.
Additionally, Jobcentre Plus Regional Offices arrange face-to-face
meetings, if unsuccessful applicants require more detailed feedback
on their application."[201]
103. We are pleased to hear that applicants are able
to receive written feedback, although in some cases this is clearly
considered insufficient. For example, in the case of Jobcentre
Plus the standard pro forma which uses ticked boxes is unlikely
to provide the necessary information to applicants to help them
understand the reasons for their rejection and to improve their
subsequent applications.[202]
In our view applicants need a reasonably detailed assessment
of the reasons why their application was unsuccessful. We
do not wish to see any further delay in awarding contracts, but
we consider that there is a case for more support and feedback
being given to applicants. Given the time that tender applications
and bids take to produce and the scarce resources that a number
of organisations invest in their applications, we recommend
that the Government considers ways in which it can provide more
support to potential applicants, before and after formal applications
are made. We believe that all rejected applicants should be provided
with reasonably full written feedback that identifies clearly
the areas in which the tender could be improved. We urge the Government
to review the use of the standard pro formas used by Jobcentre
Plus when informing applicants that they have been unsuccessful.
62 Ev 92 Back
63
Ev 92, para 5.7. See also Ev 135, para 3.7, Q 178. Back
64
Fraser report http://www.esf.gov.uk/evaluation/documents/Final%20Report%20vf.doc Back
65
Fraser report http://www.esf.gov.uk/evaluation/documents/Final%20Report%20vf.doc Back
66
Q178 Back
67
Q42 Back
68
Q178 Back
69
QQ203 and 205 Back
70
Q42 Back
71
Q42 Back
72
Q42 Back
73
Ev 94, para 6.4 Back
74
Ibid Back
75
Q89 Back
76
Q89 Back
77
Q90. Back
78
Q100 Back
79
Q100 Back
80
Ev 140 Back
81
Ev 61, The Co-financing consultation document was published in
October 2000. The consultation report and the Government's plans
for co-financing were published in February 2001. The Operational
Framework was published in March 2001. Back
82
Ev 128 Back
83
Ev 129 Back
84
Ev 138 Back
85
Q146 and see Q147 Back
86
Ev 130, para 36 Back
87
Ev 129, para 34 Back
88
Ev 150, para 3.10 Back
89
Ev 70, para 5.1 Back
90
Ev 42, para 8.4 and Ev 138 para 6(a) Back
91
Q19 Back
92
Ev 136, para 3.9 Back
93
Ev 135, para 3.3 Back
94
For example, see Ev 150, paras 3.8 and 3.10 Back
95
Ev 148 para 2.3 Back
96
For example, see Ev 141 para 25, Ev 126, para 12 Back
97
Q142 Back
98
Q142 Back
99
Ev 151 Back
100
Ev 71 Back
101
Ev 71 Back
102
Q198 Back
103
Evaluation was conducted by Fraser Associates, published 24 July
2002. Back
104
Evaluation of the Initial Implementation of European Social Fund
Co-Financing in England, DWP, 31st July 2002. Back
105
Q154 Back
106
Ev 125 Back
107
Q130 Back
108
Q130 and see Ev 126 Back
109
Q130 Back
110
Ev 125 Back
111
Ev 126 Back
112
Appendix 15 Back
113
Ev 141 Back
114
Ev 140 Back
115
Ev 141 Back
116
Capacity building helps communities develop their response to
their own identified problems. Q148 Back
117
Ev 141 Back
118
Ev 118 Back
119
Ev 118 Back
120
Q45 Back
121
Q22 Back
122
Q52 Back
123
Q178 Back
124
Q108 Back
125
Evaluation of the Initial Implementation of ESF cofinancing
in England, Fraser Associates, July 2002 Back
126
Q106 Back
127
Q46 Back
128
Ev 118 Back
129
For example Q111 Back
130
See paras 5.7 and 5.8 of the DWP Memorandum Back
131
We were told that in Scotland it took about three months from
the end of June when there were about 450 applications before
approval by the Minister was given. Back
132
Q56 Back
133
Ev 34 Back
134
The previous (1994-99) programmes were extended to June 2000 to
cover the gap. Back
135
For example, see Q13, Q177 Back
136
We were told that given labour market changes, it would be a little
unwise to give core funding permanently to organisations when
the need might not always be there. See Q189 Back
137
A social firm such as Rosie's café is required to generate
50% of its income through sales of goods or services and has social
aims. Back
138
ESF can also support training in the more traditional skills such
as building skills, especially plumbing and plastering. We welcome
such provision, which we view as being especially helpful for
women and people from ethnic minority groups. Back
139
Q138 Back
140
Ev 65 Back
141
Q130 Back
142
Q133 Back
143
Q149 Back
144
Ev 66 Back
145
Ev 135 Back
146
Ev 135 Back
147
Q185, see also written answer HC Deb, 19 June 2003, cc 4068W Back
148
Q186 Back
149
Q177 Back
150
Q138 Back
151
Q137 Back
152
Q138 and 142 Back
153
Although there are statistics that show which projects have been
successful and unsuccessful during each round in each region,
there is no consolidated information to show what has happened
to the same application in those regions over successive rounds
or why, in detail, a bid may have been rejected. Back
154
Global grants are made to intermediary bodies which are required
to provide the match-funding. The intermediary bodies then make
grants of up to £10,000 to small organisations. Back
155
Q188 Back
156
Q199 Back
157
http://www.esf.gov.uk/evaluation/documents/Final%20Report%20vf.doc Back
158
Q145 Back
159
We published our report into Childcare for Working Parents
on, 18 June 2003, HC564-I, Fifth Report 200203 Back
160
Ev 116 Back
161
Ev 116 Back
162
Although relatively small, the number of irregularities detected
or reported to the Commission is increasing. It is now 0.68 per
cent of the total funds. The UK has the fourth highest number
of reported irregularities. Q20 Back
163
All cases of non-compliance with the regulations must be communicated
to the Commission. The usual sanction imposed by the Commission
for non-compliance is repayment of funding, non-payment of outstanding
claims for funds, or decommitment of the programme, where the
entire funding for a specific programme is deemed ineligible and
has to be repaid. Back
164
The percentages are laid down in the EC regulations Back
165
Upfront funding is available from CFOs , but it needs to be requested
and justified by the provider. Q145 Back
166
European Social Fund, co-financing guidance, April 2003 available
from www.esf.gov.uk Back
167
Ev 156 Back
168
According to the regulations, original documents must be kept
until 31 December 2011. See ESF in GB, page 48. Back
169
These are companies limited by guarantee. In Scotland there are
5 PMEs for each of the programmes. Back
170
Q129 Back
171
We were told that the Commission accepts that where open and competitive
tendering is used to allocate bids, the audit trail will, in the
vast majority of cases, stop with CFOs. Back
172
Q63 Back
173
See http://www.esf.gov.uk/evaluation/documents/Final%20Report%20vf.doc Back
174
We were told that the IT system would allow claims forms to be
submitted online with certain parts of the form that require re-submission
of information presented with prepopulated entries. The
electronic form would be subject to detailed electronic checking
at the point of submission, which it is hoped would help correct
errors. Back
175
Other recommendations include abolition of management fees, better
information and support for Project Management System, more consultation
with beneficiaries Back
176
Ev 143 Back
177
For example, the Third Sector European Network (TSEN), which brings
together the v&c sector support agencies in all nine English
regions and their sister organisations in Scotland, including
Social Firms Scotland and Grampian ESF Network Back
178
Ev 69 Back
179
Q92 Back
180
Q92 Back
181
Q92 Back
182
Ev 136 Back
183
Q16 Back
184
Q18 Back
185
In the UK, evaluations of ESF programmes are conducted by the
Evaluation Team, which is based in the Analytical Services Directorate
of DWP in Sheffield. Data is collected from the applications forms
and the project closure forms. ESF service providers submit information,
usually quarterly, on their performance against their planned
expenditure and the delivery of their training outputs. These
quarterly reports form the basis for payments. In addition to
project monitoring, the DWP organises financial inspections covering
at least 5% of all ESF expenditure each year. Back
186
Q16 Back
187
Ev 93 Back
188
Ev 93 para 5.12. As regards Objective 1 programmes, these met
their ESF expenditure targets in 2002. Back
189
The performance reserve is funding that is withheld by the European
Commission until the performance of the country's programme is
assessed. Back
190
Much of the evidence is taken from monitoring data and surveys.
Sometimes, independent research is also commissioned. One of the
main research studies being conducted is the Leavers Survey that
involves a six-month follow-up survey of a sample of ESF beneficiaries.
The Monitoring Committee will consider the findings of the mid-term
evaluation and any proposals to amend the programme on the basis
of those findings. See written answer The Minister Back
191
Q3 Back
192
The European Commission is required by European Union legislation
to allocate by 31 March 2004 the 4 per cent. performance reserve
for Structural Fund Objectives 1, 2 and 3, which in the case of
England Objective 3 programme has a performance reserve of £105
million. The performance reserve will be released if the programme
is judged to be successful on the basis of indicators that will
be assessed as part of its mid-term evaluation. The Department
will submit the mid-term evaluation report to the Commission at
the end of 2003. The Department will agree the timing of the distribution
of the performance reserve with the Commission in early 2004. Back
193
HC Deb 7 July 2003, c606 Back
194
Extract from http://www.dti.gov.uk/europe/future.html and
http://www.scottish.parliament.uk/S1/official_report/cttee/europe-02/eur02-03-01.htm
Back
195
This problem of measurement is further compounded if unemployment
is on the rise in the area. Back
196
Q191 Back
197
http://www.eprc.strath.ac.uk/iqnet/iq-net/downloads/IQ-Net_Reports(Public)/8.2Monitoring.pdf.
Although this study concentrates on objective 2 areas, it's description
of the monitoring process applies equally to Objective 3. Back
198
Commission Regulation (EC) No 1685/2000 of 28 July 2000 laying
down detailed rules for the implementation of Council Regulation
(EC) No 1260/1999 as regards eligibility of expenditure of operations
co-financed by the Structural Funds; and Commission Regulation
(EC) No 438/2001 of 2 March 2001 laying down detailed rules for
the implementation of Council Regulation (EC) No 1260/1999 as
regards the management and control systems for assistance granted
under the Structural Funds. Back
199
Ev 88 Back
200
See appendix 12, Ev 159 Back
201
See Annex B, Ev 116 Back
202
Q201 Back