The Independent Commission for Aid Impact's Annual Report 2012-13 - International Development Committee Contents

3  Operational issues for ICAI

20. Whilst overall ICAI's work was well-received by witnesses, several challenges relating to how the Commission operates were highlighted, namely: ICAI's use of its contractor; ICAI's work alongside other evaluators; its traffic light system; the scope of its evaluations; and its use of the evidence base.

ICAI's contractor

21. ICAI undertook a formal review of its contractor's performance in May 2013.[24] ICAI said that it remained "content" that the contractor was meeting the requirements of its contract "in both the spirit and the letter" and was delivering good value for money for the taxpayer."[25]

22. The contractor designs and participates in every country visit that ICAI undertakes.[26] DFID is responsible for helping induct and orient ICAI contractors when they arrive in-country for an evaluation. In April, DFID's Permanent Secretary, Mark Lowcock, told the sub-Committee that the induction process was not as efficient as it could be:

    One of the issues that we and [the ICAI Commissioners] have kept talking about is how we ensure that the consultants doing a particular study—which may be their first interaction with us or ICAI in many cases—are brought up to speed as fast as possible, and in a way that minimises the amount of time they need to spend being tutored and briefed and having things explained to them by the staff of the Department.[27]

ICAI said there had been progress this year in its contractor deploying an increased pool of staff with specific expertise for the report in question, which—it was hoped—would speed up the orientation process.[28] However, the Commissioners said that they were keen to balance the deployment of specialised staff with continuity between contractor teams, and emphasised that the contractor had a "consistent core" to "join the dots" between evaluations.[29]

23. ICAI admitted that its contractor could do more to ensure learning is shared between staff, and "from one review [...] to others."[30] However, this is just one aspect of lesson-sharing. Another is disseminating learning to DFID itself—and, indeed, to taxpayers. In written evidence, consultant David Peretz suggested ICAI should take steps to improve lesson-sharing, not least because DFID's management responses to reports were limited to a "formal exchange".[31] He suggested that ICAI organise seminars and outreach events after each evaluation, both for DFID officials and wider audiences.[32]

24. We were concerned to hear that orienting ICAI's contractor in-country was proving to be time-consuming for DFID staff. We were pleased to hear that the contractor was now sending more specialised staff in-country. We hope this will help reduce the time spent by DFID country staff inducting and orienting ICAI evaluators.

25. We agree with ICAI that continuity of staff across evaluations is important, and believe that paying attention to continuity will also help promote lesson-learning amongst staff across evaluations. It is also important that lessons are shared with DFID, the development community and wider audiences. We recommend that ICAI take further steps to improve lesson-sharing from ICAI's work, for example by holding seminars and outreach events following each evaluation. This would help improve knowledge dissemination, both to DFID and to wider audiences.

ICAI's work within the wider evaluation field

26. As well as ourselves, there are three further bodies scrutinising UK Government expenditure on development: ICAI; the National Audit Office; and DFID's own evaluation department, which has recently been reinvigorated via a new Evaluation Policy (May 2013). The Preface to DFID's new Evaluation Policy states that the creation of ICAI "has been matched by the embedding of evaluation across DFID to ensure that lessons are learned during the development process."[33]

27. The build-up of DFID's own evaluation function was welcome by witnesses such as David Peretz, who said he would like to see ICAI making more use of DFID's own evaluations.[34] In August 2012 DFID awarded the Global Evaluation Framework Agreement (GEFA) which has put out to tender contracts to evaluate work across all DFID's programmes and projects.[35] We were surprised to see that the website advertising the original GEFA tender put the "lowest and highest offer" for the total final value of contract(s) at £150 million, a huge amount compared to other evaluation efforts (ICAI's current budget, for example, is £3.26 million).[36]

28. However, when we questioned DFID's Permanent Secretary about the GEFA, he said that the projected spend was far lower than this: as at 1 July 2013, 11 contracts had been awarded through GEFA at a total value of £8.2 million. He said there were a further 16 live competitions underway with a value around £15.9 million.[37] He told us that services to the International Climate Fund (ICF)—which has a substantial budget of £3.87 billion between 2011 and 2015—were included in the GEFA to "provide a more joined-up approach" across departments.[38]

29. We agree that working with other evaluators is a positive step for ICAI- as long as duplication is avoided. We have been alert to possible duplication between ICAI and either ourselves or other scrutiny bodies over the past year. ICAI's official Government-appointed role in providing independent scrutiny makes it important that its own role and relationship to other official scrutiny mechanisms are as clear as possible. Our sub-Committee on ICAI undertakes scrutiny of ICAI's future workplan towards this end. We are aware that both our own Committee and ICAI undertook evaluations of DFID's Pakistan programme in 2013. There were some useful aspects of this dual focus for us. For example, ICAI's evaluation (which took place before our own) included a detailed look at the health sector, which we were able to draw upon in our own inquiry. Equally, in Year 3, it has been useful to read ICAI's comprehensive evaluation of DFID's Health Programmes in Burma (July 2013) before we begin our own inquiry into 'Democracy and Development in Burma' in the autumn of 2013.

30. We asked DFID's Permanent Secretary whether it concerned him when two scrutiny bodies looked at the same issue. He told us

    I am much less worried about the potential overlap between the Committee's work and ICAI's work than I am about the more similar work that the NAO and ICAI do. [...] The approach is similar. It is an assurance based approach, it is quite process-heavy and it is a bit less on development policy and exploration of development insight, which is, if I may say so, a much heavier focus of the Committee's work. I worry a bit more about those kinds of overlaps and duplications.[39]

ICAI told us that it is in regular contact with the NAO to ensure "complementarity of reports", as are we ourselves.[40] We noted that, although both the NAO and ICAI published reports on the cross-Whitehall Conflict Pool in 2012, they had very different conclusions, with the NAO's findings being far more positive than ICAI's.[41]

31. Following the Cabinet Office Review, we intend to expand the role of the sub-Committee in order to strengthen ICAI's relationship with IDC, so that we can both be more strategic about developing our respective workplans and reports. We are aware of the similarities to the relationship between the NAO and the Public Accounts Committee (PAC).[42] Although, clearly, ICAI is not the NAO, and IDC is not the PAC, we intend to learn from the relationship between those bodies in our future work.

32. We are concerned about the importance of avoiding overlap between the four main bodies scrutinising DFID: ICAI, IDC, the NAO and DFID's own evaluation department. We await with interest the publication of the ICAI's Triennial Review, due out in the autumn of 2013, and any conclusions it may make on the specific types of evaluation it seeks from ICAI. The establishment of our IDC sub-Committee has helped us establish a closer, more mutually beneficial working arrangement that includes the sharing of ideas between inquires and evaluations. We intend to further strengthen the relationship following the forthcoming Cabinet Office Triennial Review of ICAI, so that our workplans and reports adopt a more strategic approach to development. Our impression is that duplication is not currently a problem, but we will await to see if this view is reflected in the Triennial Review.

The traffic light system

33. In order to offer taxpayers a simple assessment of DFID performance, ICAI uses a traffic light system to award overall ratings within evaluations. The system has received a mixed reception since ICAI's establishment in 2011. In written evidence, Professor Robert Picciotto said of ICAI's ratings system, "The link between the ratings and the evidence is often tenuous and the consistency between the overall rating and the detailed ratings is not always self-evident."[43]

34. ICAI's recent reports on i) DFID and Programme Partnership Arrangements[44] and ii) DFID's Use of Contractors awarded both evaluations 'green-amber' overall. Yet these reports contain a number of serious criticisms of DFID, notably that:

  • The Department does not make strategic use of contractors and NGOs
  • A 'master-servant' relationship, rather than proper partnership, is operated by DFID;
  • Monitoring and evaluation should improve; and
  • Learning from contractors needs to be more effectively captured.

35. During an oral evidence session in April 2013, we asked Mark Lowcock, DFID's Permanent Secretary, for his view on the traffic light system. He told us that the scores were "not the first thing I look at". He said, rather, that he was "interested in the lessons and the content recommendations".[45]

36. Evidence received in our inquiry raised concerns about the traffic light system, saying the link between ratings and evidence is often tenuous. We were struck by the contrast between the positive 'green-amber' rating awarded to ICAI's recent reports on 'DFID's Support for Civil Society Organisations through Programme Partnership Arrangements' and 'DFID's Use of Contractors to Deliver Aid Programmes', and the relatively critical content of the reports. We understand that the traffic light system is designed to offer taxpayers a simple assessment of DFID performance. However, we wonder whether a more nuanced process might be more appropriate that the current red, amber or green ratings. We recommend ICAI keep its traffic light system under review.

ICAI evaluations: scope and use of evidence


37. ICAI selects the topics in its workplan using four criteria: coverage, materiality, interest, and risk. This process has led to an overall ICAI portfolio focusing on 'snapshots' of DFID's work rather than cross-cutting issues. Witnesses criticised this approach.[46] ITAD, an organisation specialising in evaluation, stated

Evaluations of particular sectors of operation in particular countries may improve the way in which a handful of programmes are implemented, but does not necessarily inform whether these are strategically the best lines of investment in the first place. The choice of evaluations could be more strategic.[47]

David Peretz agreed that ICAI should "focus more on broader and cross cutting evaluations" and "less on the evaluation of individual projects."[48]

38. We put this point to ICAI's Chief Commissioner, Graham Ward. He said that he understood the concern, and that in Years 3 and 4, ICAI would undertake a number of more thematic reports, for example: one on learning; one on the private sector; and one on climate finance.[49] ICAI's approach of choosing individual projects or single-issue 'snapshots' of DFID's work has resulted in an impressive portfolio that covers a large portion of DFID's programmes. However, we agree with witnesses that including some thematic, cross-cutting evaluations will help create a more complete picture of the UK's aid programme. We were pleased to hear that ICAI will undertake a number of more thematic evaluations in Years 3 and 4.


39. Several written submissions expressed concern about ICAI's evaluation techniques. Professor Picciotto said of ICAI's reports

    They tend to rely on hearsay rather than concrete and rigorous analysis. Some anecdotal information is provided instead of focused evidence regarding the key assumptions that link objectives to impact.[50]

He went on to say, "In a nutshell, ICAI delivers competent comprehensive audit reports. But auditing is not evaluation." He recommended that ICAI upgrade and/or update the evaluation skills of its contractors, so that they make more use of theory-based evaluation approaches or participatory evaluation techniques.[51]

40. David Peretz said that, whilst recent reports suggested "some improvement" in ICAI's use of the evidence base, more could be done by ICAI to "collect and develop its own evidence." Specifically, Mr Peretz said that more studies would have benefited from properly structured surveys of opinions, and that the evaluation of DFID's Oversight of the EU's Aid to Low-Income Countries would have benefited from an assessment of other major EU donors' aid programmes.[52]

41. Linking evidence to impact is a timely issue for ICAI, given that it is shortly to evaluate the concept of impact. In its Annual Report, ICAI said it will carry out a Year 4 report looking specifically at "how best to deliver and monitor impact". Witnesses felt such a study was overdue.[53] ICAI said its evidence base would consist of its "completed reports, stakeholder engagement and some new work looking at how we can provide the taxpayer with assurance that impact has indeed been delivered."[54]

42. We were concerned to hear that witnesses had reservations about ICAI's evaluation techniques. We feel it is not our role to make technical recommendations on using evidence and/or specific evaluation techniques. We recognise that ICAI needs to try different approaches until it finds the right ones. However, it should now look to establish a set of approaches which it knows will be completely reliable. We anticipate hearing in DFID's response to our report its own view of how ICAI uses evidence, and what adjustments (if any) are needed to improve ICAI's evaluation methods.

24   The review was in line with ICAI's Framework Agreement with DFID. Back

25   ICAI Annual Report 2012-13, paras 7-8 Back

26   Alongside Commissioners and staff, as appropriate. Back

27   Q 25 Back

28   ICAI Annual Report 2012-13, para 9 Back

29   Qq 9-10 Back

30   ICAI Annual Report 2012-13, para 10  Back

31   DFID has three weeks to issue a 'management response' to each ICAI report. Back

32   Ev w4 Back

33   DFID, Evaluation Policy (2013) and Ev w7 Back

34   Ev w4 Back

35  The Global Evaluation Framework Agreement got underway in August 2012 and will operate for an initial 2 year period with the option for a further one-year extension. Back

36   Original advert:

Final, full award notice (lists all (approx 30) suppliers): Back

37  Ev 16 Back

38  Government funding to the ICF for the period 2011-15 is divided across three departments as follows: DFID will give £2.13 billion to the Fund; Department of Energy and Climate Change (DECC) will give £1.6 billion; and Department for Environment, Food and Rural Affairs (DEFRA) will give £140 million.


39   Sub-Committee on ICAI, evidence session, 16 April 2013, Q6 Back

40   Ev 18 Back

41   NAO, 'Review of the Conflict Pool' (March 2012) Back

42   The NAO carries out audits of Government expenditure. It reports these to the Comptroller and Auditor General, who is an officer of the House of Commons and in turn reports the audits to the Public Accounts Committee. The PAC appoints NAO's external auditors and scrutinises its performance. Back

43   Ev w6 Back

44   Programme Partnership Arrangements (PPAs) are one of DFID's principal mechanisms for funding civil society organisations. Back

45   International Development Committee,16 April 2013 evidence session, Q7 Back

46   Ev w4 Back

47   Ev w5 Back

48   Ev w4 Back

49   Q 3 Back

50   Ev w6 Back

51  Ev w6. Theory-based evaluation approaches refer to any evaluation approach that examines the underlying assumptions to the evaluated intervention. Participatory evaluation is a partnership approach to evaluation in which stakeholders actively engage in developing the evaluation and all phases of its implementation. Back

52  Ev w4 Back

53   Ev w8 Back

54   ICAI Annual Report 2012-13, p.17 Back

previous page contents next page

© Parliamentary copyright 2013
Prepared 22 October 2013