First annual letter from the Independent
Advisory Committee on Development Impact (IACDI) to the Secretary
of State for International Development
INDEPENDENT ADVISORY
COMMITTEE ON
DEVELOPMENT IMPACT
(IACDI)
The committee's terms of reference require that
I write to you once a year, and this is my first such letter.
It reflects conclusions we have reached at our first four meetings.
The minutes of these meetings are available on our website. In
our first year we have given priority to identifying changes needed
to DFID policies and processes to strengthen the central Evaluation
Department (EVD), to enhance the culture of learning and decentralised
evaluation across the department, and to encourage use of lessons
learned from evaluation by DFID staff.
We are conscious that high quality, independent
evaluation is particularly important for the management and accountability
of spending programmes like DFID's that are, expanding rapidly
and where the links between departmental policies and expenditures
and their impact—notably on poverty and achievement of the
Millennium Development Goals—are usually indirect and often
complex. High quality evaluation is needed both to provide assurance
that DFID's money is being well spent, and so that lessons from
experience are learned and applied in future. The creation of
IACDI is a signal of the Government's recognition of this.
Our initial assessment is that the quality and
independence of evaluations carried out by and for EVD are comparable
with other leading bilateral aid agencies. But we think DFID can
and should aim for a higher standard, as is implicit in IACDI's
mandate. Our conclusions on this were reached after in depth discussion
at two consecutive meetings of the committee and informed in part
by recent written reviews including in particular a helpful assessment
of the state of evaluation independence at DFID carried out by
one of the committee's members (Evaluation independence at DFID),
an assessment that in turn drew on criteria endorsed by the international
Evaluation Cooperation Group taking account also of DFID's specific
institutional arrangements. We have made a number of proposals
that would help ensure DFID practices meet best international
practice, and allow us to give the unqualified assurances about
the independence, quality and impact of DFID evaluations envisaged
in our terms of reference. These proposals are set out in full
in Annex A.
We welcome the broadly positive reactions of
your officials to these proposals, the actions taken to implement
some of them, and other relevant changes already underway. More
generally, we are encouraged by the support given by senior officials
for the work of IACDI, and their commitment to establishing a
stronger and more prominent role for evaluation in the department.
In particular we are delighted that the Head
of EVD has agreed to produce an annual report drawing out some
general themes from evaluations carried out during the year, and
I am pleased to enclose a copy of this first annual report with
this letter. I would underline his emphasis on the importance
of implementing the Paris Declaration (and now the Accra Agenda
for Action) commitments on better international coordination of
evaluations, and focus on building partner countries' systems
for monitoring and evaluation, within the context of efforts to
strengthen ownership and giving partner countries a stronger role,
not least by reducing unnecessary burdens on them.
We also welcome your officials' agreement to
produce an annual report of follow up to past EVD evaluations
and their recommendations. While future evaluation reports need
to be more selective and specific in their recommendations, the
department needs to be more specific and systematic in its responses.
Of course the department may sometimes disagree with the conclusions
of an evaluation—that is to be expected, and the disagreements
should be documented. But where DFID undertakes to make changes
following an evaluation, there is now a process in place for tracking
and reporting on implementation. The first such report, on follow
up to evaluations carried out in 2006-07, was enclosed with this
letter and reveals a patchy implementation record. There are positive
examples where evaluations have fed into and influenced DFID's
approach, but also others where they have not. We hope the department
will take action so that next year's report on follow up to 2007-08
evaluations will reveal an improved performance.
We have begun an in-depth review of evaluation
quality and welcome steps now underway to improve quality assurance
arrangements. We are also concerned that the ability of evaluations
to draw robust conclusions, especially on development impact,
remains limited by weaknesses in objective setting and relevant
data and monitoring information about DFID's interventions. The
effort now being made by DFID to improve systems for measuring
outcomes and impact is welcome and timely as is the substantial
support being given to international cooperation on impact evaluations.
We have spent more time discussing how to strengthen
independence. Independence of the evaluation function should not
mean isolation from decision makers and there needs to be a balance
between independence and engagement. However our detailed assessment
is that the structure and processes associated with DFID evaluations
need further strengthening to meet intenational best practice,
especially as regards EVD's independence and influence. Perceptions
are important too. We are concemed at some sample survey evidence
(NAO Survey of Perceptions of DFID Evaluation) suggesting that
most inside and outside DFID believe that EVD is less than entirely
or very independent.
We are less well placed as yet to make a judgment
on evaluation carried out in DFID outside EVD, but our preliminary
discussions have pointed to weaknesses especially in relevant
monitoring information. We would like to see stronger arrangements
in place to assure its quality.
We are also concerned to ensure that the level
of administrative resources available for evaluation in DFID both
inside and outside EVD is adequate for the task.
Several of our recommendations can be implemented
as part of the new DFID evaluation policy which is to be agreed
by early next year. We look forward to contributing to the design
of this policy, and to helping guide a process of public consultation
over the next few months. At the same time IACDI will consult
on the topics for evaluations by EVD for the period 2009-12.
We recognise that some of the steps we propose
will need to be worked through in greater depth, and that some
can only be implemented over a period of years, such as the proposals
to raise the grade and change the accountability arrangements
of the head of EVD. Also, as we continue our discussions we may
identify additional steps needed in other areas. I look forward
to further discussion with you and your officials about how best
to implement our proposals, so that in future we can give our
assurance that the independence and eftectiveness of DFID's processes
for evaluating development impact have achieved or are clearly
on track to achieve the high standard of excellence that the Government
committed to in establishing IACDI and in setting our terms of
reference.
I am sending copies of this letter and attachments
to members of the International Development Committee of the House
of Commons.
David Peretz
Chair, Independent Advisory Committee on Development
Impact
Annex A
IACDI PROPOSALS TO STRENGTHEN THE INDEPENDENCE,
EFFECTIVENESS AND IMPACT OF EVALUATION IN DFlD
A. PROPOSALS
ALREADY BEING
IMPLEMENTED
1. DFlD will produce an annual report of follow
up action taken following the previous year's evaluations, and
nominate a Director responsible for follow up to each evaluation
report. IACDI may on occasion ask to meet with Directors responsible
to discuss progress. The first such report is appended to this
letter.
2. The Head of Evaluation Department (EVD) has
agreed to make an annual report drawing lessons and common themes
from reports completed during the year and to discuss it with
IACDI in draft form. The first such report was enclosed with this
letter.
3. DFID Management Board has agreed to hold
at least one annual discussion of evaluation work, and the first
such discussion is scheduled for November 2008.
4. There should be an agreed Departmental policy
on evaluation that meets internationally recognised criteria,
and is endorsed by IACDI. The Committee has discussed a first
draft of a new policy and will help guide a process of public
consultation before formal adoption of the policy early next year.
B. OTHER PROPOSALS
TO BE
IMPLEMENTED IN
THE IMMEDIATE
FUTURE
5. Evaluation in DFID undertaken outside EVD
needs to be strengthened. This will require the creation of the
right incentives. There is a case for EVD taking on appropriate
responsibility for oversight, quality assurance and guidance on
an evaluations carried out in DFID, as at present this function
seems fragmented and largely left to the judgment of line managers.
This would require extra resources for EVD and is an issue to
be resolved as part of the new evaluation policy.
6. A formal role for IACDI or its chair in agreeing
a written job description, protocols and in arrangements for performance
review will help buttress the independence of the head of EVD.
7. There should be clear written protocols for
unimpeded access by EVD and their consultants to information in
DFID: for rules of engagement with DFID staff in discussing draft
reports; for rules for avoiding staff conflicts of interest; and
a written policy on disclosure of reports.
C. PROPOSALS
TO BE
IMPLEMENTED OVER
A 2 TO
3 YEAR PERIOD
8. To match good practice the Head of EVD should
be accountable to the Permanent Secretary of DFID, and be able
to make reports without clearance from the management line. The
Committee has welcomed the recent decision to make the head of
EVD directly accountable to the DG Corporate Performance as a
step in the right direction, but a further step is needed.
9. The post should also have a more senior grade
and status to give greater visibility and clout.
10. There is a contradiction between the need
for more, high quality evaluation and a declining administrative
budget for evaluations, suggesting a need to explore ways to protect
the budget for evaluation. IACDI should have a central role in
deciding EVD's budget in future.
11. There should be changes in modalities for
(and control of the Head of EVD over) staffing, probably with
more staff recruited from outside DFID—over time EVD may
need to change the balance towards an increased role for EVD staff
and a lesser role for external consultants.
12. Future heads of evaluation should probably
be appointed on an understanding precluding employment elsewhere
in DFID, and the contract should be for a fixed term, possibly
renewable on the advice of IACDI.
IACDI would expect to see this last group of
proposals implemented after the more immediate changes have been
made, over the course of the next 2 to 3 years, with some changes
to be implemented, perhaps, when the current head of EVD is replaced.
1 December, 2008.
| 
