Examination of Witnesses (Quesitons 40-45)
MR DAVID
PERETZ, MR
ANTHONY KILLICK,
MR ROBERT
PICCIOTTO AND
MS ALISON
GIRDWOOD
9 JULY 2008
Q40 Mr Crabb: Have you seen any evidence
yourselves of that concern being affected in practice? Have you
seen evidence of, for example, a positive report about a particular
aspect of aid by a consultant where the suspicion is that actually
they have been a bit weak, because of the issue we have just talked
about?
Mr Picciotto: I am talking now
as an evaluator and not for the committee, because we have not
finished our examination of the quality of reports; but one of
the criterion of independence is behavioural independence, which
means producing on a regular basis uncompromising reports which
go under the skin of DFID. My review of the reports suggests that
it is a mixed bag. Some reports do get under the skin; others
do not. By strengthening the independence of the whole system
and by changing staff incentives, one will gradually see much
more behavioural independence than can be observed now by looking
at the reports. That said, DFID evaluation reports, as compared
to those produced in the bilateral development aid community,
are well above average. It is not that the reports are bad; it
is just that they could be better and tougher in some cases.
Q41 Mr Crabb: Very roughly, what
estimate would you make of the number of consultants working in
the area of evaluation who are former employees of the Department?
Mr Picciotto: I do not know.
Mr Killick: I would not identify
that as a major issue. I think that most of the consultants are
academics or ex-academics. I do not think it is an issue so much
of buying in to people who have previously been working in the
Department. That is not my impression.
Mr Peretz: In fact, previous departmental
members may be the most critical. The point about conflicts of
interest applies, clearly. You would not want an ex-member of
staff to be evaluating something he had been involved in as a
member of staff. At our last meeting we had our first discussion
about the quality of evaluations. We reached a conclusion rather
similar to somebody else who gave evidence to you that, in comparison
to other bilateral agencies' evaluations, DFID scores pretty well.
I think we said that it is well up, if not the best, of the bilaterals—but
that is not a particularly high standard and we, as a committee,
will be aiming for something much better than that.
Q42 Jim Sheridan: I still remain
unclear about just how your advisory committee would ensure that
the self-evaluation of DFID will become independent. Would there
be some other alternatives, other than what you have suggested
today, to ensure that DFID has that independence? What alternatives
could there have been?
Mr Peretz: I think that you are
asking a question about evaluation, not in the central Evaluation
Department but elsewhere in the Department, which we said we were
doubtful about. To be honest, we need to find ways to get to grips
with that. We suggested one, which is that the Evaluation Department
itself should have a role, and may have quite a key role, in assuring
quality. It is not just independence that you are assuring; it
is whether it is being done well and, indeed, are the lessons
being learned as you do it yourself. That would be my answer to
that bit of the question, but there was another half of the question
as well.
Q43 Jim Sheridan: You could not identify
any other alternative other than what is on the table today?
Mr Picciotto: I would say that
if these 11 recommendations are implemented, DFID evaluation would
move to the "top of the class"; but they are not yet
implemented and it will not happen overnight. It takes time to
develop the skills, to develop the organisation, to change the
culture; but it should not take too much time either. In two or
three-years we should see DFID become a centre of excellence for
evaluation. This is not the case now.
Q44 Chairman: In Mr Sheridan's earlier
intervention he was expressing concern about money being spent
on evaluation that could have been spent on aid. The counter to
that is that what people will want to know is that the money spent
on evaluation actually leads to changes in policy and implementation
that will improve the delivery of aid. First of all, do you get
the impression that that is currently what is happening with evaluation
within DFID? You shake your head. Is it your intention to ensure
that in future it does?
Mr Peretz: We have got DFID to
agree to institute this process of producing an annual report
of follow-up to previous evaluations. They have never done it
before. It will be published; you will see it. It is quite a patchy
document, because it covers a period when nobody knew they were
going to have to produce this, and in future they will. The departmental
response to evaluations is sometimes lacking. Very often it does
not cover all the recommendations; so you do not know what they
are going to do and what they are not going to do at the beginning.
The follow-up is then patchy, I would say. However, I think that
we have set in train a process and we have also said, and got
it agreed, that for each evaluation report done by the Evaluation
Department a director in DFID will be responsible for managing
the response and the follow-up; and our committee may call directors
to meet with us and explain what they have been doing. We have
now created some incentive, therefore, for lesson-learning and
for follow-up which was not there before—and that is a bit
of a success.
Q45 Chairman: It has been a very
interesting and useful exchange, because it is quite clear that
you have set in process a whole range of actions which could clearly
make a material difference, it seems to me at a reasonably affordable
cost, in terms of what is involved. Obviously we will look forward
to being copied in on your letter to the Secretary of State and,
to take up your own remarks, I think that we may well want to
meet again. Perhaps there may be value in having an informal meeting
as well and not necessarily always to do this in a formal evidence
session. Clearly, what we are interested in is the extent to which
you will be able to improve the culture, which you indicate was
lacking in DFID, and to make the point of evaluation to deliver
better aid and development funding, rather than simply to prove
that DFID is right to be pleased with itself—which I guess
is not really a justifiable use of public money. Can I thank you
for that first initial engagement. To pick up on Mr Kawczynski's
point, the time may come when confirmation hearings for people
in your position will be part of the process. We are working on
it.
Mr Peretz: Can I thank you and
say that, as to the idea of an informal meeting, over the winter
we will be consulting on the future work plan and also DFID's
evaluation policy. It will be very helpful to get this Committee's
informal input into that process in some way.
Chairman: I think that the Committee
would appreciate that opportunity. It would be mutually beneficial.
We will try to fit it into the diary at an appropriate time.
| 
