DFID Annual Report 2008 - International Development Committee Contents


First annual letter from the Independent Advisory Committee on Development Impact (IACDI) to the Secretary of State for International Development

INDEPENDENT ADVISORY COMMITTEE ON DEVELOPMENT IMPACT (IACDI)

  The committee's terms of reference require that I write to you once a year, and this is my first such letter. It reflects conclusions we have reached at our first four meetings. The minutes of these meetings are available on our website. In our first year we have given priority to identifying changes needed to DFID policies and processes to strengthen the central Evaluation Department (EVD), to enhance the culture of learning and decentralised evaluation across the department, and to encourage use of lessons learned from evaluation by DFID staff.

  We are conscious that high quality, independent evaluation is particularly important for the management and accountability of spending programmes like DFID's that are, expanding rapidly and where the links between departmental policies and expenditures and their impact—notably on poverty and achievement of the Millennium Development Goals—are usually indirect and often complex. High quality evaluation is needed both to provide assurance that DFID's money is being well spent, and so that lessons from experience are learned and applied in future. The creation of IACDI is a signal of the Government's recognition of this.

  Our initial assessment is that the quality and independence of evaluations carried out by and for EVD are comparable with other leading bilateral aid agencies. But we think DFID can and should aim for a higher standard, as is implicit in IACDI's mandate. Our conclusions on this were reached after in depth discussion at two consecutive meetings of the committee and informed in part by recent written reviews including in particular a helpful assessment of the state of evaluation independence at DFID carried out by one of the committee's members (Evaluation independence at DFID), an assessment that in turn drew on criteria endorsed by the international Evaluation Cooperation Group taking account also of DFID's specific institutional arrangements. We have made a number of proposals that would help ensure DFID practices meet best international practice, and allow us to give the unqualified assurances about the independence, quality and impact of DFID evaluations envisaged in our terms of reference. These proposals are set out in full in Annex A.

  We welcome the broadly positive reactions of your officials to these proposals, the actions taken to implement some of them, and other relevant changes already underway. More generally, we are encouraged by the support given by senior officials for the work of IACDI, and their commitment to establishing a stronger and more prominent role for evaluation in the department.

  In particular we are delighted that the Head of EVD has agreed to produce an annual report drawing out some general themes from evaluations carried out during the year, and I am pleased to enclose a copy of this first annual report with this letter. I would underline his emphasis on the importance of implementing the Paris Declaration (and now the Accra Agenda for Action) commitments on better international coordination of evaluations, and focus on building partner countries' systems for monitoring and evaluation, within the context of efforts to strengthen ownership and giving partner countries a stronger role, not least by reducing unnecessary burdens on them.

  We also welcome your officials' agreement to produce an annual report of follow up to past EVD evaluations and their recommendations. While future evaluation reports need to be more selective and specific in their recommendations, the department needs to be more specific and systematic in its responses. Of course the department may sometimes disagree with the conclusions of an evaluation—that is to be expected, and the disagreements should be documented. But where DFID undertakes to make changes following an evaluation, there is now a process in place for tracking and reporting on implementation. The first such report, on follow up to evaluations carried out in 2006-07, was enclosed with this letter and reveals a patchy implementation record. There are positive examples where evaluations have fed into and influenced DFID's approach, but also others where they have not. We hope the department will take action so that next year's report on follow up to 2007-08 evaluations will reveal an improved performance.

  We have begun an in-depth review of evaluation quality and welcome steps now underway to improve quality assurance arrangements. We are also concerned that the ability of evaluations to draw robust conclusions, especially on development impact, remains limited by weaknesses in objective setting and relevant data and monitoring information about DFID's interventions. The effort now being made by DFID to improve systems for measuring outcomes and impact is welcome and timely as is the substantial support being given to international cooperation on impact evaluations.

  We have spent more time discussing how to strengthen independence. Independence of the evaluation function should not mean isolation from decision makers and there needs to be a balance between independence and engagement. However our detailed assessment is that the structure and processes associated with DFID evaluations need further strengthening to meet intenational best practice, especially as regards EVD's independence and influence. Perceptions are important too. We are concemed at some sample survey evidence (NAO Survey of Perceptions of DFID Evaluation) suggesting that most inside and outside DFID believe that EVD is less than entirely or very independent.

  We are less well placed as yet to make a judgment on evaluation carried out in DFID outside EVD, but our preliminary discussions have pointed to weaknesses especially in relevant monitoring information. We would like to see stronger arrangements in place to assure its quality.

  We are also concerned to ensure that the level of administrative resources available for evaluation in DFID both inside and outside EVD is adequate for the task.

  Several of our recommendations can be implemented as part of the new DFID evaluation policy which is to be agreed by early next year. We look forward to contributing to the design of this policy, and to helping guide a process of public consultation over the next few months. At the same time IACDI will consult on the topics for evaluations by EVD for the period 2009-12.

  We recognise that some of the steps we propose will need to be worked through in greater depth, and that some can only be implemented over a period of years, such as the proposals to raise the grade and change the accountability arrangements of the head of EVD. Also, as we continue our discussions we may identify additional steps needed in other areas. I look forward to further discussion with you and your officials about how best to implement our proposals, so that in future we can give our assurance that the independence and eftectiveness of DFID's processes for evaluating development impact have achieved or are clearly on track to achieve the high standard of excellence that the Government committed to in establishing IACDI and in setting our terms of reference.

  I am sending copies of this letter and attachments to members of the International Development Committee of the House of Commons.

David Peretz

Chair, Independent Advisory Committee on Development Impact

Annex A

IACDI PROPOSALS TO STRENGTHEN THE INDEPENDENCE, EFFECTIVENESS AND IMPACT OF EVALUATION IN DFlD

A.  PROPOSALS ALREADY BEING IMPLEMENTED

  1. DFlD will produce an annual report of follow up action taken following the previous year's evaluations, and nominate a Director responsible for follow up to each evaluation report. IACDI may on occasion ask to meet with Directors responsible to discuss progress. The first such report is appended to this letter.

  2. The Head of Evaluation Department (EVD) has agreed to make an annual report drawing lessons and common themes from reports completed during the year and to discuss it with IACDI in draft form. The first such report was enclosed with this letter.

  3. DFID Management Board has agreed to hold at least one annual discussion of evaluation work, and the first such discussion is scheduled for November 2008.

  4. There should be an agreed Departmental policy on evaluation that meets internationally recognised criteria, and is endorsed by IACDI. The Committee has discussed a first draft of a new policy and will help guide a process of public consultation before formal adoption of the policy early next year.

B.  OTHER PROPOSALS TO BE IMPLEMENTED IN THE IMMEDIATE FUTURE

  5. Evaluation in DFID undertaken outside EVD needs to be strengthened. This will require the creation of the right incentives. There is a case for EVD taking on appropriate responsibility for oversight, quality assurance and guidance on an evaluations carried out in DFID, as at present this function seems fragmented and largely left to the judgment of line managers. This would require extra resources for EVD and is an issue to be resolved as part of the new evaluation policy.

  6. A formal role for IACDI or its chair in agreeing a written job description, protocols and in arrangements for performance review will help buttress the independence of the head of EVD.

  7. There should be clear written protocols for unimpeded access by EVD and their consultants to information in DFID: for rules of engagement with DFID staff in discussing draft reports; for rules for avoiding staff conflicts of interest; and a written policy on disclosure of reports.

C.  PROPOSALS TO BE IMPLEMENTED OVER A 2 TO 3 YEAR PERIOD

  8. To match good practice the Head of EVD should be accountable to the Permanent Secretary of DFID, and be able to make reports without clearance from the management line. The Committee has welcomed the recent decision to make the head of EVD directly accountable to the DG Corporate Performance as a step in the right direction, but a further step is needed.

  9. The post should also have a more senior grade and status to give greater visibility and clout.

  10. There is a contradiction between the need for more, high quality evaluation and a declining administrative budget for evaluations, suggesting a need to explore ways to protect the budget for evaluation. IACDI should have a central role in deciding EVD's budget in future.

  11. There should be changes in modalities for (and control of the Head of EVD over) staffing, probably with more staff recruited from outside DFID—over time EVD may need to change the balance towards an increased role for EVD staff and a lesser role for external consultants.

  12. Future heads of evaluation should probably be appointed on an understanding precluding employment elsewhere in DFID, and the contract should be for a fixed term, possibly renewable on the advice of IACDI.

  IACDI would expect to see this last group of proposals implemented after the more immediate changes have been made, over the course of the next 2 to 3 years, with some changes to be implemented, perhaps, when the current head of EVD is replaced.

1 December, 2008.





 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2009
Prepared 19 February 2009