3 Operational issues for ICAI |
20. Whilst overall ICAI's work was well-received
by witnesses, several challenges relating to how the Commission
operates were highlighted, namely: ICAI's use of its contractor;
ICAI's work alongside other evaluators; its traffic light system;
the scope of its evaluations; and its use of the evidence base.
21. ICAI undertook a formal review of its contractor's
performance in May 2013.
ICAI said that it remained "content" that the contractor
was meeting the requirements of its contract "in both the
spirit and the letter" and was delivering good value for
money for the taxpayer."
22. The contractor designs and participates in every
country visit that ICAI undertakes.
DFID is responsible for helping induct and orient ICAI contractors
when they arrive in-country for an evaluation. In April, DFID's
Permanent Secretary, Mark Lowcock, told the sub-Committee that
the induction process was not as efficient as it could be:
One of the issues that we and [the ICAI Commissioners]
have kept talking about is how we ensure that the consultants
doing a particular studywhich may be their first interaction
with us or ICAI in many casesare brought up to speed as
fast as possible, and in a way that minimises the amount of time
they need to spend being tutored and briefed and having things
explained to them by the staff of the Department.
ICAI said there had been progress this year in its
contractor deploying an increased pool of staff with specific
expertise for the report in question, whichit was hopedwould
speed up the orientation process.
However, the Commissioners said that they were keen to balance
the deployment of specialised staff with continuity between contractor
teams, and emphasised that the contractor had a "consistent
core" to "join the dots" between evaluations.
23. ICAI admitted that its contractor could do more
to ensure learning is shared between staff, and "from one
review [...] to others."
However, this is just one aspect of lesson-sharing. Another
is disseminating learning to DFID itselfand, indeed, to
taxpayers. In written evidence, consultant David Peretz suggested
ICAI should take steps to improve lesson-sharing, not least because
DFID's management responses to reports were limited to a "formal
suggested that ICAI organise seminars and outreach events after
each evaluation, both for DFID officials and wider audiences.
24. We were concerned to hear that orienting ICAI's
contractor in-country was proving to be time-consuming for DFID
staff. We were pleased to hear that the contractor was now sending
more specialised staff in-country. We hope this will help reduce
the time spent by DFID country staff inducting and orienting ICAI
25. We agree with ICAI that continuity of staff
across evaluations is important, and believe that paying attention
to continuity will also help promote lesson-learning amongst staff
across evaluations. It is also important that lessons are shared
with DFID, the development community and wider audiences. We
recommend that ICAI take further steps to improve lesson-sharing
from ICAI's work, for example by holding seminars and outreach
events following each evaluation. This would help improve knowledge
dissemination, both to DFID and to wider audiences.
ICAI's work within the wider evaluation
26. As well as ourselves, there are three further
bodies scrutinising UK Government expenditure on development:
ICAI; the National Audit Office; and DFID's own evaluation department,
which has recently been reinvigorated via a new Evaluation Policy
(May 2013). The Preface to DFID's new Evaluation Policy states
that the creation of ICAI "has been matched by the embedding
of evaluation across DFID to ensure that lessons are learned during
the development process."
27. The build-up of DFID's own evaluation function
was welcome by witnesses such as David Peretz, who said he would
like to see ICAI making more use of DFID's own evaluations.
In August 2012 DFID awarded the Global Evaluation Framework Agreement
(GEFA) which has put out to tender contracts to evaluate work
across all DFID's programmes and projects.
We were surprised to see that the website advertising the original
GEFA tender put the "lowest and highest offer" for the
total final value of contract(s) at £150 million, a huge
amount compared to other evaluation efforts (ICAI's current budget,
for example, is £3.26 million).
28. However, when we questioned DFID's Permanent
Secretary about the GEFA, he said that the projected spend was
far lower than this: as at 1 July 2013, 11 contracts had been
awarded through GEFA at a total value of £8.2 million. He
said there were a further 16 live competitions underway with a
value around £15.9 million.
He told us that services to the International Climate Fund (ICF)which
has a substantial budget of £3.87 billion between 2011 and
2015were included in the GEFA to "provide a more joined-up
approach" across departments.
29. We agree that working with other evaluators
is a positive step for ICAI- as long as duplication is avoided.
We have been alert to possible duplication between ICAI and either
ourselves or other scrutiny bodies over the past year. ICAI's
official Government-appointed role in providing independent scrutiny
makes it important that its own role and relationship to other
official scrutiny mechanisms are as clear as possible. Our sub-Committee
on ICAI undertakes scrutiny of ICAI's future workplan towards
this end. We are aware that both our own Committee and ICAI undertook
evaluations of DFID's Pakistan programme in 2013. There were some
useful aspects of this dual focus for us. For example, ICAI's
evaluation (which took place before our own) included a detailed
look at the health sector, which we were able to draw upon in
our own inquiry. Equally, in Year 3, it has been useful to read
ICAI's comprehensive evaluation of DFID's Health Programmes
in Burma (July 2013) before we begin our own inquiry into
'Democracy and Development in Burma' in the autumn of 2013.
30. We asked DFID's Permanent Secretary whether it
concerned him when two scrutiny bodies looked at the same issue.
He told us
I am much less worried about the potential overlap
between the Committee's work and ICAI's work than I am about the
more similar work that the NAO and ICAI do. [...] The approach
is similar. It is an assurance based approach, it is quite process-heavy
and it is a bit less on development policy and exploration of
development insight, which is, if I may say so, a much heavier
focus of the Committee's work. I worry a bit more about those
kinds of overlaps and duplications.
ICAI told us that it is in regular contact with the
NAO to ensure "complementarity of reports", as are we
ourselves. We noted
that, although both the NAO and ICAI published reports on the
cross-Whitehall Conflict Pool in 2012, they had very different
conclusions, with the NAO's findings being far more positive than
31. Following the Cabinet Office Review, we intend
to expand the role of the sub-Committee in order to strengthen
ICAI's relationship with IDC, so that we can both be more strategic
about developing our respective workplans and reports. We are
aware of the similarities to the relationship between the NAO
and the Public Accounts Committee (PAC).
Although, clearly, ICAI is not the NAO, and IDC is not the PAC,
we intend to learn from the relationship between those bodies
in our future work.
32. We are concerned about the importance of avoiding
overlap between the four main bodies scrutinising DFID: ICAI,
IDC, the NAO and DFID's own evaluation department. We await with
interest the publication of the ICAI's Triennial Review, due out
in the autumn of 2013, and any conclusions it may make on the
specific types of evaluation it seeks from ICAI. The establishment
of our IDC sub-Committee has helped us establish a closer, more
mutually beneficial working arrangement that includes the sharing
of ideas between inquires and evaluations. We intend to further
strengthen the relationship following the forthcoming Cabinet
Office Triennial Review of ICAI, so that our workplans and reports
adopt a more strategic approach to development. Our impression
is that duplication is not currently a problem, but we will await
to see if this view is reflected in the Triennial Review.
The traffic light system
33. In order to offer taxpayers a simple assessment
of DFID performance, ICAI uses a traffic light system to award
overall ratings within evaluations. The system has received a
mixed reception since ICAI's establishment in 2011. In written
evidence, Professor Robert Picciotto said of ICAI's ratings system,
"The link between the ratings and the evidence is often tenuous
and the consistency between the overall rating and the detailed
ratings is not always self-evident."
34. ICAI's recent reports on i) DFID and Programme
and ii) DFID's Use of Contractors awarded both evaluations
'green-amber' overall. Yet these reports contain a number of serious
criticisms of DFID, notably that:
- The Department does not make strategic use of
contractors and NGOs
- A 'master-servant' relationship, rather than
proper partnership, is operated by DFID;
- Monitoring and evaluation should improve; and
- Learning from contractors needs to be more effectively
35. During an oral evidence session in April 2013,
we asked Mark Lowcock, DFID's Permanent Secretary, for his view
on the traffic light system. He told us that the scores were "not
the first thing I look at". He said, rather, that he was
"interested in the lessons and the content recommendations".
36. Evidence received in our inquiry raised concerns
about the traffic light system, saying the link between ratings
and evidence is often tenuous. We were struck by the contrast
between the positive 'green-amber' rating awarded to ICAI's recent
reports on 'DFID's Support for Civil Society Organisations through
Programme Partnership Arrangements' and 'DFID's Use of Contractors
to Deliver Aid Programmes', and the relatively critical content
of the reports. We understand that the traffic light system is
designed to offer taxpayers a simple assessment of DFID performance.
However, we wonder whether a more nuanced process might be more
appropriate that the current red, amber or green ratings. We
recommend ICAI keep its traffic light system under review.
ICAI evaluations: scope and use
37. ICAI selects the topics in its workplan using
four criteria: coverage, materiality, interest, and risk. This
process has led to an overall ICAI portfolio focusing on 'snapshots'
of DFID's work rather than cross-cutting issues. Witnesses criticised
this approach. ITAD,
an organisation specialising in evaluation, stated
Evaluations of particular sectors
of operation in particular countries may improve the way in which
a handful of programmes are implemented, but does not necessarily
inform whether these are strategically the best lines of investment
in the first place. The choice of evaluations could be more strategic.
David Peretz agreed that ICAI should "focus
more on broader and cross cutting evaluations" and "less
on the evaluation of individual projects."
38. We put this point to ICAI's Chief Commissioner,
Graham Ward. He said that he understood the concern, and that
in Years 3 and 4, ICAI would undertake a number of more thematic
reports, for example: one on learning; one on the private sector;
and one on climate finance.
ICAI's approach of choosing individual projects or single-issue
'snapshots' of DFID's work has resulted in an impressive portfolio
that covers a large portion of DFID's programmes. However, we
agree with witnesses that including some thematic, cross-cutting
evaluations will help create a more complete picture of the UK's
aid programme. We were pleased to hear that ICAI will undertake
a number of more thematic evaluations in Years 3 and 4.
USE OF EVIDENCE BASE
39. Several written submissions expressed concern
about ICAI's evaluation techniques. Professor Picciotto said of
They tend to rely on hearsay rather than concrete
and rigorous analysis. Some anecdotal information is provided
instead of focused evidence regarding the key assumptions that
link objectives to impact.
He went on to say, "In a nutshell, ICAI delivers
competent comprehensive audit reports. But auditing is not evaluation."
He recommended that ICAI upgrade and/or update the evaluation
skills of its contractors, so that they make more use of theory-based
evaluation approaches or participatory evaluation techniques.
40. David Peretz said that, whilst recent reports
suggested "some improvement" in ICAI's use of the evidence
base, more could be done by ICAI to "collect and develop
its own evidence." Specifically, Mr Peretz said that more
studies would have benefited from properly structured surveys
of opinions, and that the evaluation of DFID's Oversight of
the EU's Aid to Low-Income Countries would have benefited
from an assessment of other major EU donors' aid programmes.
41. Linking evidence to impact is a timely issue
for ICAI, given that it is shortly to evaluate the concept of
impact. In its Annual Report, ICAI said it will carry out a Year
4 report looking specifically at "how best to deliver and
monitor impact". Witnesses felt such
a study was overdue.
ICAI said its evidence base would consist of its "completed
reports, stakeholder engagement and some new work looking at how
we can provide the taxpayer with assurance
that impact has indeed been delivered."
42. We were concerned to hear that witnesses had
reservations about ICAI's evaluation techniques. We feel it is
not our role to make technical recommendations on using evidence
and/or specific evaluation techniques. We recognise that ICAI
needs to try different approaches until it finds the right ones.
However, it should now look to establish a set of approaches which
it knows will be completely reliable. We anticipate hearing in
DFID's response to our report its own view of how ICAI uses evidence,
and what adjustments (if any) are needed to improve ICAI's evaluation
24 The review was in line with ICAI's Framework Agreement
with DFID. Back
ICAI Annual Report 2012-13, paras 7-8 Back
Alongside Commissioners and staff, as appropriate. Back
Q 25 Back
ICAI Annual Report 2012-13, para 9 Back
Qq 9-10 Back
ICAI Annual Report 2012-13, para 10 Back
DFID has three weeks to issue a 'management response' to each
ICAI report. Back
Ev w4 Back
DFID, Evaluation Policy (2013) and Ev w7 Back
Ev w4 Back
Global Evaluation Framework Agreement got underway in August 2012
and will operate for an initial 2 year period with the option
for a further one-year extension. Back
Original advert: http://ted.europa.eu/udl?uri=TED:NOTICE:106441-2012:TEXT:EN:HTML&src=0
Final, full award notice (lists all
(approx 30) suppliers): http://ted.europa.eu/udl?uri=TED:NOTICE:279664-2012:TEXT:EN:HTML&src=0 Back
funding to the ICF for the period 2011-15 is divided across three
departments as follows: DFID will give £2.13 billion to the
Fund; Department of Energy and Climate Change (DECC) will give
£1.6 billion; and Department for Environment, Food and Rural
Affairs (DEFRA) will give £140 million.
Sub-Committee on ICAI, evidence session, 16 April 2013, Q6 Back
Ev 18 Back
NAO, 'Review of the Conflict Pool' (March 2012) Back
The NAO carries out audits of Government expenditure. It reports
these to the Comptroller and Auditor General, who is an officer
of the House of Commons and in turn reports the audits to the
Public Accounts Committee. The PAC appoints NAO's external auditors
and scrutinises its performance. Back
Ev w6 Back
Programme Partnership Arrangements (PPAs) are one of DFID's principal
mechanisms for funding civil society organisations. Back
International Development Committee,16 April 2013 evidence session,
Ev w4 Back
Ev w5 Back
Ev w4 Back
Q 3 Back
Ev w6 Back
w6. Theory-based evaluation approaches refer to any evaluation
approach that examines the underlying assumptions to the evaluated
intervention. Participatory evaluation is a partnership approach
to evaluation in which stakeholders actively engage in developing
the evaluation and all phases of its implementation. Back
Ev w8 Back
ICAI Annual Report 2012-13, p.17 Back