Project and Programme Evaluations
39. At present, there is a clear gap between the
specificity of project- and programme- level evaluations, and
DFID's overall objectives. It is this gap which must be filled
in order for DFID to be able to establish useful performance targets.
Sir John Vereker said in oral evidence that "we recognise
that there is potentially a gap between on the one hand our measurement
of the impact of our expenditure and on the other hand our achievement
of a policy outcome in a particular country".[35]
Sir John's answer in oral evidence to this "attribution gap"
was that DFID had invested significantly in policy analysis. There
is no doubt that there is a large amount of policy analysis information
now available, produced by DFID and by others, perhaps most notably
the World Bank. Whilst this is welcome, it is not sufficient for
DFID to learn from its own successes and failures, and to be held
properly to account for its expenditure. A significant investment
must also be made in expenditure analysis and evaluation procedures.
This is necessary in order to establish the effectiveness of individual
programmes and projects in producing the desired outcomes.
40. We must be able to assess how the levels of expenditure
on projects and programmes relate to each of DFID's objectives
and targets. This aspect of accountability is well catered for
in data recorded by the Policy Information Marker System, the
Poverty Aim Marker and the Policy Objective Marker.[36]
These markers provide an analysis of DFID's expenditure according
to key policy areas, its objectives, and the ways in which projects
are expected to contribute to poverty eradication.
41. In its 1998 and 1999 Departmental Reports,
DFID included in each section a table showing information on its
expenditure captured by Policy Information Marker System data,
which measures expenditure according to key policy areas. In its
2000 Departmental Report, DFID has discontinued this practice.
We recommend that in future Departmental Reports, in each section
of its report relating to one of its policy objectives, DFID include
a table showing relevant Policy Information Marker System, Policy
Objective Markers and Poverty Aim Marker data for the past three
years. This will provide a clear and readily-accessible picture
of DFID's expenditure profile in relation to its objectives.
42. There must also be information available on the
success or otherwise of individual projects and programmes in
meeting their objectives. All projects over £500,000 are
currently assessed on their completion by the project manager,
who must fill in a project completion report.[37]
The information provided in these reports is collated periodically
in synthesis reports, which provide an overview of the success
of projects in various sectors and countries in meeting their
objectives. Our analysis of the most recent synthesis study, which
covers PCRs completed during the period 1983 to 1998,[38]
is that the document is useful in that it provides an overview
of the areas of strength and weakness of DFID's programmes. Its
limitations lie in its lack of information about or analysis of
the reasons for the success or failure of projects and programmes
in meeting their objectives. This undermines the utility of the
study — lessons learned cannot be fed back into programmes
if they are not disseminated in studies such as this.
43. DFID's project completion report forms have
been revised recently to capture more information about lessons
learned from successes and failures, and we look forward to this
information being available on an annual basis in future project
completion report synthesis studies. This will represent a significant
improvement in their value as tools for learning lessons and applying
them in future projects and programmes. We recommend that, in
each section of future Departmental Reports, tables be included
showing the success of projects in each policy area in reaching
their objectives (as reported in the most recent project completion
report synthesis study), along with a summary of the key lessons
which have been learned.
44. The Project Completion Reports represent a snapshot
of a programme or project at its completion. The PCRs are not
independent (they are completed by the project manager), and they
focus on the objectives of the project or programme set by the
project manager. They do not include significant scope for an
analysis of any wider or unexpected impact which may have been
made beyond that intended by the project. The Project Completion
Reports alone do not present a comprehensive picture of DFID's
performance. They must be supported by evaluation studies which
are independent, published, are conducted after the completion
of projects and programmes, and which measure the broader and
longer-term impact of projects and programmes, and their contribution
to poverty eradication.
45. We asked DFID what proportion of its bilateral
expenditure was independently evaluated. The response was, "only
a small fraction".[39]
When we questioned him in oral evidence about the level of independent
evaluation of DFID's work, Sir John Vereker responded, "I
am not sure that it is right to distinguish quite so violently
between independent evaluations and our own. I think the head
of our evaluations department ... would assert fairly vigorously
that evaluations, whether they are done in-house or contracted
to outsiders, are in every case done by professional evaluators
according to well-established criteria, and all our evaluations
are published".[40]
46. Sir John Vereker went on, however, to state that
"Of course ... you have got to view quite sceptically anything
which contains the element of self-assessment which is in [the
Project Completion Reports]... It is quite difficult, without
being massively bureaucratic and elaborate, to come up with something
that is more obviously independent".[41]
We disagree with Sir John's assertion that to achieve a more independent
level of evaluation would be difficult in the way he suggests.
The evaluation studies published by DFID demonstrate that this
is not the case.
47. DFID commissions ad-hoc evaluation studies of
some of its projects and programmes. These involve a full retrospective
impact evaluation, which is the individual responsibility of the
authors of the report (which often include experts from outside
DFID as well as representatives from its own evaluation department),
rather than DFID. These provide an important supplement to the
project completion reports. In our Report on DFID's 1998 Departmental
Report, we noted that evaluation studies were conducted on around
12 projects each year. Sir John Vereker told us in evidence to
this inquiry, however, that there has been a shift in focus recently
in DFID's evaluation studies: they now concentrate on sectors,
countries or themes, rather than on individual projects and programmes,
and there are now fewer studies, each looking at several projects
at a time. Sir John explained that this shift in emphasis would
lead to evaluation studies which provided more valuable insights
into lessons which could be drawn.[42]
48. It is impossible to check what evaluation studies
have been published recently, since there is no list of recent
evaluation studies available either on the Department's Website
(which only includes evaluations carried out up to 1998), in its
publications catalogue (published in 2000), or in the Departmental
Report. We recommend that DFID revise its publications catalogue
to include evaluation studies, and that it update its Internet
site to include all evaluation studies on a regular basis. This
will enable other organisations outside DFID to benefit from the
studies more easily.
49. In the absence of readily available information,
we asked DFID to provide us with a list of recent evaluation studies.[43]
This showed that, in 1999, DFID published only three evaluation
studies in addition to the 1999 Project Completion Report Synthesis
which we discussed above. These concerned: Health Planning in
Pakistan; Indonesia Police; and an Evaluation of HMG's Response
to the Montserrat Volcanic Emergency. So far this year, only one
evaluation study has been published; an Environment Evaluation
Synthesis Study.
50. We consider independent evaluation of a much
greater proportion DFID's projects and programmes would be a very
valuable investment. The lessons which can be learned from such
evaluations could prove essential to future planning. We are
sure that DFID's in-house Project Completion Reports are compiled
in good faith. Nevertheless we would interpret cautiously an evaluation
portfolio which consisted almost entirely of self-assessment,
without any supplementary independent evaluations. At the very
least, there is an undeniable value in obtaining a second expert
opinion on the strengths and weaknesses of a project or programme
and the nature and extent of its impact, and on the lessons which
may be drawn for future work. We recommend that DFID significantly
increase the proportion of its projects and programmes that are
subjected to evaluation studies. We recommend that DFID publish
an evaluation strategy and invite suggestions from its staff and
from the development community on projects and programmes which
might usefully be independently reviewed. The strategy should
include all aspects of its work, including budgetary support and
sector wide adjustment programmes (SWAPs).
51. In the 2000 Departmental Report, DFID stated
that "In the future we will support more in-depth evaluations
of our effectiveness, both at country level and internationally.
This will help assess our impact in key policy areas covered by
the International Development Targets and will take forward the
development of systems to improve understanding of our impact
and influence in the international arena. A Development Impact
and Resource Centre will be set up in early 2000. This will provide
staff with high quality advice on how to carry out impact assessment
studies and improve in-country capacity to maintain and evaluate
performance".[44]
We look forward to the establishment of the new Development
and Impact Resource Centre, and request that DFID keep us informed
of progress in its work.
52. DFID publishes a large range of policy analysis
documents, strategy papers, and evaluation studies. It has in
place good evaluation systems, and has been working to improve
them. All this we welcome. There is a gap, however, between these
qualitative assessments of DFID's role and performance, and the
need to provide summary quantitative information about the overall
performance of the Department against clear and meaningful targets.
We recommend that DFID produce a set of revised departmental targets,
based on the results of its Project Completion Reports and Evaluation
Studies, disaggregated according to its objectives and expenditure
profile.
32 
