The Work of the Independent Commission for Aid Impact - June 2012

Written evidence submitted by The UK Aid Network (UKAN) and BOND

Bond and UKAN with our members are grateful for the IDC to open this inquiry and to ask for input into the work and recent reports of the Independent Committee for Aid Impact (ICAI). In the following submission we focus on the way ICAI conducts their evaluations, on their value for money methodology and of the approaches they have taken to their reports published in 2012.

1. ICAI’s overall structure and ways of working

1.1 The UK NGO sector broadly welcomes ICAI’s role in adding an additional, independent layer of scrutiny for the impact and effectiveness of UK aid. However, based on our initial submission on the formation of ICAI [1] , some key concerns remain about the way ICAI is set up and operates.

1.2 We welcome the strong and active role that the Commissioners and Chief Commissioner Graham Ward play in the ICAI reports. Yet, we are still concerned that while the Commissioners represent a range of important areas of expertise, there is still no Commissioner with extensive international development expertise (i.e. from the academic, programme implementation or evaluation side of this field). We believe that an additional Commissioner with such a background would add great value to ICAI’s work. We would therefore be very interested to see this inquiry explore with ICAI whether their work has in any way suffered from not having such experience amongst its Commissioners and future steps to address such challenges if they are in fact relevant.

1.3 We are aware that ICAI has been trying to include the voice of aid recipients in its reports, consistent with its approach as outlined in their value for money methodology that the intended beneficiaries of aid must come first. However, it is not clear to us to which extent they have been successful in achieving this goal, especially given the limited time available for field visits in the process of producing their reports.

1.4 As part of its effort to include the voice of intended beneficiaries ICAI has been making efforts to consult Bond members during some of their studies. These efforts have been greatly appreciated, although in many cases there has been limited notice given for meetings, hindering efforts to gather insights from NGO field offices, and also in some cases such meetings have come quite late in the process leaving little space for influencing the findings. ICAI has now started to systematically publish Terms of Reference for its studies on their website before they are finalised which is helpful in encouraging input from Bond members. However, it is not clear how submissions our members have made are being used and what impact they might be having.

1.5 We are aware that ICAI reports do have a strong impact on DFID’s practise and that DFID is keen to implement suggestions as outlined in the reports and ICAI therefore represents a key evaluation function of DFID’s programmes. Yet, as mentioned above, we do have concerns about the limited timeline for each report and the ambitious ICAI 3 years plan and want to stress that time constraints should not undermine a thorough evaluation of DFID’s programmes and long term impact.

2. ICAI’s value for money methodology and traffic light system

2.1 We welcome ICAI’s value for money methodology which outlines 4 criteria – objectives, delivery, impact and learning – to assess DFID’s performance and do deliver a comprehensive view of DFID’s efforts. The used traffic light system to present DFID’s performance in an accessible way is useful to see shortfalls of programmes and approaches and get an idea of overall performance.

Yet in a media context this traffic light system can be too simplistic as UK media has mainly picked up on the negative headlines from each report over the past months. There should be a discussion on how ICAI interacts with the media given that the nature of evaluations is to highlight challenges and shortcomings and give advice on how to improve work. We believe ICAI should be less focusing on active media work and more on working closely with DFID to improve programmes, procedures and policies.

2.2 We welcome the addition of equity as a 4th ‘E’ alongside economy, efficiency and effectiveness in the methodology. This speaks to the goal of NGOs to ensure that their work benefits the poorest and most marginalised

2.3 We welcome the focus the need to demonstrate value for money across objectives, delivery, impact, and learning, and the recognition that strong programme design, monitoring, evaluation and learning is essential making a strong case for value for money. Whilst econometric value for money methodologies and statistical methods such as randomised control trials can produce some interesting data and results, we agree with ICAI that these methodologies are not essential to a robust value for money assessment.

2.4 We agree with the ICAI’s statement that it is  "important not to shy away from the difficult to measure" (3.4) and the recognition that a combination of quantitative and qualitative methods are needed to measure performance. Concerns have been raised that the value for money agenda will drive organisations towards implementing more easy-to-measure interventions, and this is a welcome commitment by ICAI ensuring that difficult-to-measure changes such as improvements in empowerment and accountability will not be disadvantaged. [2]

2.5 We welcome ICAI’s commitment to taking a high quality and rigorous approach to assessing effectiveness and value for money that states that "the intended beneficiaries come first" (4.3.i) and that in most cases  "direct engagement with individuals such as intended beneficiaries, programme staff, delivery chain partners and independent observers" (4.15) will form part of the assessment. Beneficiaries should take the lead in defining ‘value’ in a value for money assessment and their views on programme effectiveness and value for money should have a high profile place in ICAI’s future assessments.

Yet, as stressed before, it is not clear to us how much this is translated into practice for each report given very limited and short time frames and less focus on consultations with civil society. The key challenge, we see for ICAI and the organisations who deliver the evaluations, is to collect sufficient and the right data in the short time frame they have.

2.6 Organisations working in international development acknowledge that aid delivery also has a strong political component. ICAI’s methodology doesn’t really cover nor does it mention concepts of long term social change, rights based approach and empowerment.

3. ICAI’s approaches to their reports

3.1 ICAI’s reports have addressed a wide range of important challenges that DFID faces in implementing its programmes effectively and this scrutiny and the recommendations ICAI has made will certainly help to improve the effectiveness of its programmes. Having said this we do want to raise some questions about some elements that may be under-emphasised in ICAI’s reports and therefore neglected in its scrutiny efforts.

3.2 Firstly, ICAI’s reports to date have predominantly focused on the management, compliance and oversight side of DFID’s programmes and there has been limited focus to date on the actual impact of its work and the challenges of day to day programme implementation, a focus which is critical for DFID to learn from and improve its practice. The Afghanistan and Budget Support studies are good examples of how this focus has predominated, and even in the India education and health study, which aimed to explore the performance of DFID’s work in India it highlighted the need for DFID to better track how its work is catalytic and influences Government policy but made little effort to explore some of these impacts itself.

3.3 Secondly, ICAI’s reports only very weakly address questions about how DFID should work in a multi-donor environment and strategically work with and compliment the activities of other donors, despite the fact it is working alongside many donors in every country in which it is operating and donor coordination is one of the biggest challenges facing the aid community. Such analysis is especially important when it comes to issues such as the mix of aid modalities DFID should use and steps it should take to undertake oversight of programmes, as DFID cannot do everything on its own and needs to coordinate efforts with others. As an example, ICAI’s India and Budget Support studies recommend DFID moves more towards technical assistance without any detailed assessment of what other donors are doing in these areas and how it can compliment this work.

June 2012


[2] Also see Bond’s Value for Money Paper:

Prepared 9th July 2012