Examination of Witnesses (Questions 140-155)
MR JOHN
MITCHELL AND
MR NICHOLAS
STOCKTON
20 JUNE 2006
Q140 Joan Ruddock: Are there better
other principles, other criteria that might be used?
Mr Mitchell: It depends exactly
what you want to evaluate. The Disasters Emergency Committee used
the Red Cross, Red Crescent code of conduct in a similar way.
There were 10 principles. They will use each of those principles
as a benchmark for the evaluation and one of those principles
of course is accountability to beneficiaries. The DEC evaluations
up until now have favoured that approach, but you could use GHD.
You could use Sphere. I think there is a number of codes and standards
that are out there, that have been put in place, as Nick says,
after the Rwanda evaluation, that give us the basis for talking
about these issues and developing methodologies and trying to
assess our performance. Both of us are saying, I suppose, that
these structures are in place but better performance is a rather
elusive goal.
Q141 Joan Ruddock: It was rather
disturbing to hear in your introductory remarks that the situation
that prevailed in Rwanda probably does today to a degree. There
is nothing being learned from these evaluations. There is nothing
being applied either to future evaluations or indeed to outcomes.
Mr Mitchell: That is the question
that all of us are asking at the moment. I think since Rwanda
we now have mechanisms for monitoring what happens through the
ALNAP review of humanitarian action, evaluation synthesis and
other endeavours that are out there, so we are much more in touch
with what we are doing. "Lessons learned" papers are
being compiled for the ALNAP reports database and put out into
the public domain. For example, after the tsunami we did one on
responses to earthquakes and floods. After the Pakistan earthquake
we went through our evaluation database and compiled all the similar
kinds of evaluations and wrote a "lessons learned" paper.
That went out before teams went out to the Pakistan earthquake.
That information that was not there before is definitely there
now. I am not sure it is making that much difference, I suppose.
On the question you ask: Are these lessons being put into practice?
I think people are trying, but I cannot put my hand on my heart
and say there has been a significant improvement. Every year we
do an evaluation synthesis of ALNAP and the same lessons are coming
round time and time again now. There are about eight or 10 key
areas that we see keep recurring. Progress is a bit like the First
World War—you know, it is very, very slow. Very slow.
Q142 Joan Ruddock: Does your membership
continue to have faith in the organisation, if that is the case?
Do you have a sense that there is a purpose in going on?
Mr Mitchell: Yes. I do not think
our membership thinks that the performance of the humanitarian
sector is dependent on one organisation. The sense that we have
a forum now for sharing experience right across the sector, donors,
NGOs, the Red Cross Movement and so on, is definitely healthy.
We all feel as though we know more about the problems. We all
feel as though we understand the challenges more. I think there
is more chance that we are working collectively to deal with it;
it is just difficult.
Q143 Richard Burden: I think I see
that last set of answers as sort of optimistic, although the comparison
with the First World War made me wonder if you feel progress is
being made. If I read what you are saying correctly, it is that,
given the fact that when you got to the kind of questions TEC
was starting to ask there was this depressing feeling as to what
had been learned since Rwanda, you have put mechanisms in place
to do some of those evaluations a bit more but you are uncertain
about how that translates on the ground. I guess my question may
be a bit difficult to answer, given the final synthesis report
on the TEC is not yet out, but what further lessons do you think
have come out from the process of the TEC about approaches to
evaluation rather than simply on the question of action?
Mr Mitchell: On the approaches
to evaluation, TEC is a joint evaluation; in other words, up to
70 different organisations have been involved in this evaluation
and the premise behind joint evaluations is that you can get a
system-wide view. The theory is that if you get a system-wide
analysis you are more able to have a positive influence on policy
and practice. Single agency evaluations are focusing on one agency
doing their own thing, so they do not provide a big picture. That
is the theory. But there is an opportunity cost, because the TEC
is a very complicated process. There are different bits to it.
There is a core management group with 15 different agencies. There
have been five thematic evaluations which have been conducted
by steering committees of four or five different agencies. There
is the ALNAP secretariat, which is a focal point of this. I have
had to employ five or six staff over the past 18 months to write
the synthesis report. There is a peer review group, and other
actors are becoming involved: the Office of the Special Envoy
in the UN, Clinton's office, and others. It is very, very difficult
to set up and organise and it is complicated. There is an opportunity
cost. Whether we actually do get this higher level analysis that
is really going to change policy and practice, remains to be seen.
In fact, I have just come from a meeting of the ODI where we are
reviewing the practice of joint evaluations right now, as a first
step of trying to see whether it is useful or not. In a sense,
I think, the answer to your question is: The jury is out. We felt
this was very important to do, to improve accountability. Because
the tsunami response was such an extraordinary event, the evaluation
community wanted to have a rather extraordinary response, so now
we have the biggest evaluation since the Rwanda evaluation and
we are hoping that something really useful will come out of that.
Q144 Richard Burden: Has anybody
put a figure on the cost?
Mr Mitchell: Yes. Maybe to you
it does not sound a lot. To us it sounds a lot. The whole evaluation
cost $3 million.
Q145 John Battle: There is a sense
in which "small is beautiful": little projects doing
well and Nick's thesis of the moral imperative to help, but sometimes
people's motives are mixed. Sometimes, on a small scale in the
tsunami, a well-intentioned individual may well say, "I am
going to help. I am a nobody, but I am going to try to get television
coverage for it," set out, raise funds, go out there and
then decide that the real thing that is needed is a hotel, which
they run really. Then they come back, and everybody is saying
what a waste of money, and the media present that as if the whole
aid project is a waste of money, when it has been the supporting
of the little person with mixed motives. It is not wrong to build
hotels but the funding for it perhaps ought not to have come from
popular support but from something else. How do we disentangle,
then, small is beautiful versus the coordinated aid giving? With
that one image, they were blasting across the media that all the
tsunami aid was corrupt and useless, which was not the case. Did
you take those small, half well-intentioned accounts into play
when you were doing the analysis?
Mr Stockton: I have been partly
involved and certainly have been following the production of all
the drafts of the Tsunami Evaluation Coalition and there are anecdotal
references to precisely those sorts of concerns. That then flows
into yet again another recommendation for yet more coordination.
I would ask you: When are you going to stop recommending more
coordination? How much does it already cost? The worry for me
is that this seems to be absolutely out of all proportion to the
problem.
Q146 John Battle: We do not need
700 people to check.
Mr Stockton: Exactly. It is a
really important thing to say it creates a problem in the media,
for sure, but, then again, organisations like Oxfam, Save the
Children Fund, the Red Cross Movement and so on are pretty good
at looking at themselves, are they not, on these things?—so
I would not worry too much about them It does create some bad
press, if you like, but the important question for DFID and yourselves,
and generally for British citizens, is: How much of a problem
is this really? How much real damage does it do? I think it is
very difficult to find any evidence of any damage being done,
to be honest. I think there are a lot of scares about humanitarian
assistance, for example, fuelling war. If you ask, "Give
me the evidence for that," you will find when you try to
track it down that there is almost none. There is no hard evidence
at all of it fuelling warfare, which is a typical folk panic,
if you like, which is almost evidence-free as a claim if you go
back to looking carefully at the way resources are trying to get
misdirected. The trouble is that those sorts of claims, and the
kind of evidence-less claims of the terrible problem of NGO proliferation,
consistently lead to this growing monster of coordination, and
it just does not deliver value for money in my view—it genuinely
does not. I am not anti-UN, I would stress—quite the opposite.
Q147 Chairman: You are quite clear
about that. In your article, you used the analysis of Toyota and
customer orientation.
Mr Stockton: Yes.
Q148 Chairman: When you are talking
about the Red Cross and a number of other agencies, their customers
are the donors, whether it is governments or individuals, and
yet the services are being provided to recipients. Hopefully,
those recipients are only subject to one humanitarian disaster
in their life in the one or two places that they occur, and becoming
more expert surely should not happen, but is there not a slight
problem there, in that there are two different customers, one
of whom has some response to the market and one who does not.
If we could draw the thing to a close, we are interested in any
suggestions you have as to how well DFID evaluates what it does—because
it has been criticised in the past for not having done it very
well—and what more it could do, taking your point, Mr Stockton,
that you could waste an awful lot of time and effort doing too
much.
Mr Stockton: Very briefly, we
desperately need, as all industries need, a quality assurance
mechanism. We desperately need that in the humanitarian system
and we do not frankly have it.
Q149 Chairman: That is what you are
trying to provide?
Mr Stockton: In a particular way
and with a particular approach to it. It is small, it has been
a struggle to get to where we have got to. We appreciate the DFID
backing and that of now seven other major donors, but it is political
support for this which is really absent. It is fine to chuck a
bit of money at the Humanitarian International Partnership or
ALNAP; it is a serious political fact to apply the Good Humanitarian
Donor principle seriously, which would make the difference, in
my view, and working that through around a quality assurance model
is what really needs to be done.
Q150 Chairman: Mr Mitchell, the National
Audit Office said in 2003 that "DFID's partners noted the
limited extent to which DFID was prepared to fund evaluation of
their projects aside from the monitoring report it required them
to submit." You are in the same vulnerable position as anybody
else. You need funding to do what you do. Have things changed
in DFID? Are they doing it and are they doing it to proper extent?
Mr Mitchell: To the best of our
knowledge DFID are doing evaluations. They submit their evaluations
to our evaluations report database every year—we think more
from other agencies, but certainly many of the agencies that DFID
have funded, which have been evaluated as part of the requirements,
submit their evaluations too. We certainly have evidence of many
DFID-funded evaluations in our evaluations report database. I
asked one of my staff to have a little check to see what the quality
of these evaluation reports were and they are certainly as good
as anybody else's. I am not talking about a huge sample here,
but they have scored high in this thing we call our quality pro
forma, which indicates that the evaluations that DFID have submitted
to us are quite good. However, I would say one thing, which complicates
matters a little bit, I suppose. We have found from the analysis
of our evaluations that there is not the correlation between the
good evaluation and the good impact. I think the important thing
to recognise is not so much the quality of the report; it is how
the evaluation is seen within the organisation. Currently I think
that is a political question.
Q151 Chairman: In other words: Did
we do a good job? Or: How do we do a better job?
Mr Mitchell: Yes. Also, transparency,
because it is very important, I think, for accountability that
organisations commission independent evaluation reports and put
them in the public domain. Certainly, we have seen, after the
tsunami, that certain UK chief executive officers from UK NGOs
are rather concerned about putting evaluation reports in the public
domain.
Q152 Chairman: We have had a discussion
about that—for which the argument was that it was picked
on by the media to rubbish the entire event and therefore one
tried to find ways of getting a better balance. That was the response.
Mr Mitchell: I would agree with
that. It is easy to do that.
Q153 Chairman: Let us assume that
Hilary Benn came to you and said, "How does DFID and the
UK Government cost-effectively evaluate what we do?" what
would you suggest they did that they are not doing now?
Mr Stockton: For me, I think you
have answered your own question: you have talked about cost-effectiveness.
The thing which most of all is absent in the current standards,
in the current paradigm of evaluation, is cost-effectiveness.
It is simply not in the lexicon of evaluations, I am afraid. It
just is not there. In my view, there are enormous inefficiencies
within the system which simply never come to light within the
current mechanisms associated with evaluation. If the question
about evaluation, then, I would say, focus on questions of cost-effectiveness—
Q154 Chairman: The cost-effectiveness
of the evaluation or of the response?
Mr Stockton: If you want to use
evaluation as the tool of learning, which I do not myself think
is the best thing to do, there are different and more efficient
ways of learning associated with, if you like, a quality management
model—which, to go back to the Toyota case, is why they
are so successful at what they do. They learn on the job, rather
than waiting for men in grey coats to come along and learn after
the event—which is a fairly silly thing to do, if you think
about it. Learning, as it were, in the process of managing what
you are doing is the goal, I would say, that we really, really
need to try to achieve within the humanitarian aid system. It
is something that every other business pretty well has recognised.
Whether it is concerned with public service delivery or whether
it is selling products, it is recognised that that is the key
to getting real efficiency into the system, and that we do not
currently have.
Mr Mitchell: I think I would say
to Hilary Benn that DFID should focus on an impact evaluation
of humanitarian aid but as part of a wider joint response—so
not just DFID but system wide. I tend to agree with Nick that
evaluations are very, very useful tools, but for learning perhaps
they are not the best, and so I would develop a methodology that
was over and above an evaluation and more of a research methodology
to take a very, very serious look at the impact of humanitarian
aid and how much it costs.
Q155 Chairman: Thank you both very
much indeed. That is extremely helpful. Thank you for taking the
time and trouble to come along. We will ask questions in Pakistan
and report what we have been told as an example.
Mr Stockton: I hope you have a
very good visit there.
Mr Mitchell: Thank you.
|
