Examination of witnesses (Questions 20
- 39)
TUESDAY 28 APRIL 1998
MR J VEREKER,
MR R MANNING
and MR G STEGMANN
20. I remember people who acted for you
as consultants telling me that they were extremely embarrassed
to present a report which was sometimes very critical of projects
and then find that the published report had most of the criticisms
taken out. They were unable, obviously because they worked for
you on a consultancy basis, to make their own distress at that
situation known.
(Mr Manning) I can certainly assure the Committee
that in the last few months we have taken steps to make clear
to all those producing the evaluations that the evaluations will
issue on their own responsibility. While each evaluation is seen
by the Department in draft and is subject of discussion with the
Department, which is only right, we have made it very clear in
the handling of these that the report is to be the responsibility
of the people producing it and that it is up to them what they
want to put in the public domain. I can certainly give that assurance.
21. But they might not work for you in future
if they are unduly critical of some of the projects. That was
the view put to me.
(Mr Manning) I would be very surprised if you
could find a consultant who would say that under the terms of
the guidance we have now issued.
Mr Rowe
22. I have had brought to my attention just
last week a disgraceful case where a consultant produced a report
which was clearly unacceptable to the Americans who were the biggest
contributor to this particular multinational organisation. She
now knows, because one of her friends works in one of the multinational
agencies of the United Nations, that she has been listed in that
organisation as unacceptable for future work and yet she is one
of the most experienced international experts in this field and
that is the only reason why she has been blacklisted. I am taking
that up and expect to find that some action will be taken but
it is not the first time I have heard that kind of criticism.
I do think it is an issue we need to take very seriously.
(Mr Vereker) Perhaps I could say something about
this. It is important that an organisation's culture is one which
recognises the validity of outside criticism and I should like
to persuade the Committee that this Department and its predecessor
Department have been in the forefront of setting up objective
independent and published evaluations all along. If there were
a serious flaw in the way in which we are going about something,
all our instincts as a department would be to want to know about
it. Our instincts would not be to want to brush it under the carpet.
It is very important to us that we preserve our reputation as
being one of the world's leading development agencies. The last
thing we would want would be to suppress criticism. We would be
deeply distressed if we thought there were consultants out there
who felt they had been disadvantaged because they had been critical.
There are consultants out there who are disadvantaged because
they have not been very good; that is different. We are allowed
to take that view. We would not want to find ourselves in the
position of being defensive. By tradition and culture all along
we have been, I believe, extraordinarily open and willing to learn
and we remain that way.
Mrs Kingham
23. Do you have a list? When these evaluation
reports are produced obviously it is not appropriate to circulate
all of them to the Committee, but is it possible to produce a
list of which evaluations are being carried out by the Department
so that if we have a particular interest in a particular country
or area or sectoral approach we could as individuals on the Committee
perhaps request to see specific reports?
(Mr Vereker) Yes. It would be very straightforward
for us to produce in each Departmental Report year on year lists
of project completion reports and of evaluations which have been
completed and individual Committee members, or members of the
public can ask to see them.
Ann Clwyd
24. You keep saying there were not reports
under the previous administration. I can produce those reports:
one was internal, the other was public and most of the criticism
was taken out. They actually looked identical but they had different
covers. Either somebody has a short memory or you were not there
at the time but it is not right to say that there were not two
separate reports on certain projects.
(Mr Manning) It would be very helpful if you could
send us a copy and we will respond formally. This allegation keeps
being made and I am anxious to nail it down.[5]
(Mr Vereker) Are these separate reports
and not successive drafts?
25. Reports on the same subject.
(Mr Vereker) Not successive drafts.
26. I do not know what you mean by successive
drafts. Apart from colour they are two identical reports. They
looked like this and they are on the same set of projects. One
was internal; it was sent to me by mistake. The other was for
public consumption. They look identical and are called evaluation
reports.
(Mr Vereker) Any document goes through a process
of drafting, iteration and then final product. That is the way
in which any large document is produced.
Mrs Kingham
27. The question is whether during that
drafting process the criticisms get removed and it gets adapted.
(Mr Vereker) I am happy to look at that.
Ann Clwyd
28. How effective is the monitoring of performance
by the multilateral institutions to which DFID contributes? Do
you think that monitoring is consistent with that of DFID?
(Mr Vereker) The growing international recognition
of the primacy of the international development targets makes
this task easier. In the past it has been harder because it has
not been so clear against what criteria the performance of an
institution is being measured. The whole trick about performance
measurement is to establish at the beginning against what criteria
you are going to be measured rather than looking back on the year
and not being quite sure what you were trying to achieve. The
development of the international development goals and the 21
subordinate indicators which are going to be measured will enable
people to look in a much more structured way at the impact of
individual institutions. In addition to that, virtually all international
institutions working in the development system have developed
or are developing the same kind of internal processes which we
have described to you for evaluation of their individual interventions.
I am not sure they are developing output and performance analysis
in quite the same way as we are; that may come in some areas.
Certainly the World Bank has an evaluation department very similar
to ours, the European Bank for Reconstruction and Development
has and the European Commission has in most of the areas with
which we deal. Yes, we are satisfied that in general these are
delivering reasonably useful information.
Mr Rowe
29. What sanctions does the international
aid community either have or is it moving towards? I remember
being told of a scandalous case by the EU in Delhi about the poultry
project in Bihar which cost millions of pounds and resulted in
not a single chicken appearing in Bihar. The scale of the corruption
in that particular state is just horrendous anyway. I was told
that new auditing procedures had been introduced, that for example
from now on the EU will insist on having its own auditors rather
than auditors appointed by the government. In the end the only
effective sanction would appear to be to say to the Indian Government
or the state government certainly that unless this kind of behaviour
never happens again we are not actually ever going to put any
more international aid money into their state. Are there sanctions
and if so do you use them?
(Mr Vereker) It depends what the perceived problem
is. If the perceived problem is that the Indian partner is not
performing properly then there is a choice between endeavouring
to build up the capacity of the partner so that it can operate
properly or walking away. If you cannot build a partnership upon
which you can rely then you should walk away. If the perceived
problem is that the development agency, in your example the European
Commission, is getting its procedures wrong, then I would look
to the Court of Auditors to say so, I would look to the management
committee responsible for whatever the intervention was, the Asia
and Latin America programme in this case, in Brussels to take
the matter up.
(Mr Manning) We do take this very seriously and
there are perhaps two things I would mention that are relevant.
One is that the European Court of Auditors is just embarking on
what will be a very major study of the development programmes
of the Community. There is going to be a lot of work coming out
of that. The second, which I greatly welcome, is an increasing
trend among donors to look at joint evaluation so that we all
look together at some activity. The Committee will probably be
familiar with the very useful evaluation the Danes led and to
which we contributed along with others on Rwanda. Within the European
Union there are joint reviews going on at the moment of a number
of European Community programmes which are being done jointly
by the Commission and by the member states. This is quite a useful
technique for bringing pressure to bear in areas which need a
really serious look.
Ann Clwyd
30. On page 51 of your report you say, "In
the African Development Bank, problems of poor management and
inappropriate policies led to ineffective operations and serious
financial difficulties. In concert with other donors, we are supporting
a programme of institutional reforms being implemented under the
Bank's new President". Could you tell us what information
was provided to Parliament at the time on the poor performance
of the bank? What plans are there to improve the reporting to
Parliament of the performance of multilateral institutions to
which we contribute?
(Mr Vereker) I am not sure I can answer the first
part of your question; we may have to do a bit of research to
check back what information, largely under the previous administration,
we may have placed before Parliament.[6]
On the second part, we are very much in your hands. If the Committee
would like us to tell you more about the multilateral development
banking system we are happy to do so.
31. I should like to question you about
reporting to Parliament overall. You have just produced the human
rights report jointly with the FCO. There has been no statement
in Parliament, there has been no offer of a debate, yet you place
great emphasis in this report on the importance of human rights,
central to the whole operation of the Department and so on, yet
no-one saw it as appropriate to make a statement to Parliament
on its first year's report.
(Mr Vereker) That is a matter I can only draw
to the attention of our Ministers.
32. Would you agree that it is not very
satisfactory as far as parliamentarians are concerned to find
reports talked about on the Today programme before we even know?
This Committee did not know of its existence and neither did anybody
else here as far as I know.
(Mr Vereker) It is not for me to make judgements
on whether this is a satisfactory way of doing it.
Mr Canavan: We should
be grateful if you would report that back to the Secretary of
State and other members of your staff because I understand that
the human rights report was not even laid before Parliament. There
is a general concern in Parliament, not just in this Committee
but other Select Committees, that certain government departments
are not being as open as they should in terms of accountability
to Parliament and reporting such important matters to Parliament.
I should be grateful if you would convey that message to the Secretary
of State.
Mrs Kingham
33. I fully endorse what you have said there
about ensuring that is fed back into the Department, in particular
from this Committee, that we are given equal status with other
announcements which are made on the floor such as on drugs and
young people or health announcements. The human rights approach
around the world is equally important so I would appreciate it
if that were fed back.
(Mr Vereker) I shall certainly draw that to the
attention of my Secretary of State. Of course this was a report
jointly published by the Foreign Secretary and my Secretary of
State; it is not simply a matter for this Department.
34. You mentioned PIMS, the Policy Information
Marker System. You did suggest that perhaps the Committee were
already familiar with it but I have to confess that I am not and
maybe some others also from the looks going round. Could you give
us a very brief overview of how it operates and whether you feel
it is being useful to DFID?
(Mr Vereker) The Policy Information Marker System
was established four or five years ago now and when we first explained
it to Parliament I explained that it was something which would
adapt. It was an infant programme and we wanted to see whether
it was useful. The essence of it was that we would attach a marker,
a flag as it were, to each one of our interventions to say what
category of objective it was designed to achieve. There was a
sophisticated part to this which was that we distinguished between
a principal objective of the intervention and a supporting objective.
If it was principally determined to do something it got a big
marker and if it just contributed towards it it got a smaller
marker.
35. Can you give us an example?
(Mr Vereker) Examples of the PIMS marker? Women
in development, if it was designed to promote the position of
women in developing countries. At the time there were seven of
these objectives against which we were marking. That generated
some quite useful information but it also contained some drawbacks,
one of which was that you could attach more than one marker to
a project because it could achieve more than one objective and
the best ones did, as a result of which it summed to more than
100 per cent which created horrendous conceptual difficulties,
for those trying to read the data. They are still quite useful
and still there. What has now happened is that, first of all,
the Department has redefined its overarching aims and objectives
and you will be familiar from the White Paper and repeated in
the Departmental Report with the overarching aim and the three
broad objectives. The marking system needs to be adapted to that
a bit. We were also particularly dissatisfied with the way in
which the policy information marker handled poverty, because poverty
in the old system was simply one of these seven or eight objectives
and had to be ticked only when a project was principally directed
to doing something about poverty. It got a rather oddly small
number of ticks because although of course in a very real sense
everything we are concerned with is to do with poverty, quite
a lot of them were principally concerned with something else and
were only in a supporting role doing something about poverty.
We decided it would be useful actually to have two supporting
systems. More acronyms coming, I am afraid; I know this Committee
hates acronyms. We have PIMS, but we also now have PAM and POM.
PAM is a poverty aim marker and it is attached to each of our
interventions to decide in what sort of way we are attacking poverty.
It sums to 100 per cent because we recognise that all our activities
under our new Government and our new mission statement are directed
towards poverty. We will start generating a flow of information
which says how in each case we are going to do something about
poverty. And then we have POM, which is a policy objective marker,
which quite simply marks it against the policy objectives in our
mission statement. All this will enable us to generate a time
series of information which will show how we are using the resources
we are given in relation to our objectives, but—and it is
a big but—it is what I would describe as input information.
It describes what we are committing and what we are spending:
it does not say much about the impact. For the impact you have
to go to the evaluation system. I have two experts on either side
who can talk at even greater length on the subject if you would
like.
Mrs Kingham: That
was actually quite clear; no, thank you.
Mr Canavan
36. Did you say that PAM is policy or poverty
aim marker?
(Mr Vereker) PAM is a poverty aim marker. It looks
at how we are going about poverty.
Mrs Kingham
37. I am quite interested to get more detail
about how DFID sets objectives and targets. Are objectives and
targets set on a country basis and with the agreement of recipient
country as a process? Are they set on a sectoral basis? Are there
targets for specific countries? How does the actual system operate?
Can I give one example of something which came out of our visit
to East Africa? This is brought up later in terms of gender impact
work. Targets were obviously set for a particular project we saw
in terms of gender impact, how it impacted on women in terms of
the recipients of the project but no targets seem to have been
set, and I discussed this with DFID staff at the time, to ensure
that it was not just beneficiaries of projects, women who benefited
from the projects, but that actually women were incorporated into
management structures and actually organising the project as it
grew. It was that women were beneficiaries but that the actual
project which benefited women 100 per cent was actually organised
and run by a totally male committee. There seemed to be no aims
and objectives set to ensure that women were actually incorporated
as part of that project into developing within the project structures.
It seemed to me that the aims of the objective setting had gone
a bit out of sync there and maybe it relates back to your PIMS
programme.
(Mr Vereker) You give a very interesting example.
Let me try to answer the question about how we do it. I cannot
comment on that particular project without seeing the detail.
The starting point is the country and institutional strategy paper
system and the broad architecture is that we secure the resources
from Parliament in the course of the public spending discussions
that Ministers have. We allocate these resources by broad purpose,
region, country, office, whatever. Then we invite our country
programme managers and the managers of our institutional programmes
to write strategy papers which will set out the objectives they
are trying to achieve over the next period, typically three years
ahead, with the resources they have been given. We are refining
our system of strategy papers in a number of ways and perhaps
I should mention the two most important. One is that increasingly
they are going to be written consultatively with a wide array
of people but importantly including the recipient governments
and the recipient administrations. I was talking this morning
to our office in Delhi who were telling me that they have been
having a dialogue over the last couple of days with the Indians,
in the course of which much of the drafting is actually being
done by the Government of India. That is the ideal model in a
sense, that the ownership of the strategy is transferred somewhat.
The other important feature is that all these documents—and
there will be a very large number of them because we are going
to try to do them all in the course of this year at the beginning
of the new cycle—will be placed in the public domain so
that they will be entirely open to Parliament and anyone else
to interrogate us on. There will not be a secret version left
behind with us. Those strategy papers, if you take a country,
will typically set out sector by sector the sort of areas in which
we want to work and they will set some objectives but they will
be rather high level objectives like, do more in this area, or
try to bring about some impact in that area. The next stage is
designing the particular project intervention and in what I described
as an old style project, where you can see a project and take
a photograph of it, which is a diminishing part of what we do
now, we would set out a whole series of objectives, impact measures,
intermediate objectives, process objectives against which progress
would be monitored. In the more complicated process projects,
institutional building, participatory projects where we are not
only trying to make an investment in some hardware but also build
up sector management, build up local capacity, we will try to
do the same thing, but in the nature of a process project we may
adapt the particular targets as we go along in the light of experience.
The important thing is that in each case the manager responsible
for the resources has got in front of him or her a document which
sets out very specific indicators of success in terms of time
bound indicators of process, or intermediate output and of impact.
38. Are objectives actually set for each
stage of the process: the planning, the implementation, the monitoring
of the project? Are there set stages for a project, at which point
it would be monitored, evaluated, stopped if necessary or targets
and objectives moved?
(Mr Vereker) In general, yes. It will vary from
intervention to intervention but in general a target will be set
for the frequency of monitoring and there will be a target date
for the conclusion of the expenditure and a target date for the
completion of the project completion report.
(Mr Manning) Yes and we also will usually want
to have at least once during the project life, if not more regularly,
a review which looks at whether the outputs of this project are
achieving the purposes of this project. In other words, we may
be building the school but is it actually helping children to
get better education? We do try to focus at least once in the
life of a project on how far the expenditure is not only producing
a concrete result but is actually achieving the objective which
the project is supposed to be producing.
Dr Tonge
39. May I ask a question relating to page
68 of the report on running costs? I am very anxious to establish,
coming from the health service as I do, that not too much money
is being spent on administration. I note that the overall budget
for DFID is not increasing next year. Can you tell us something
about the running costs? In your view are they reasonable? We
do not get any breakdown of them at all. They are all lumped together
in one figure of £59 million. I am a little bit anxious that
you should be looking at your own departmental costs and making
sure that they are not excessive.
(Mr Vereker) The first thing I should say to the
Committee is with apologies that there is an error in the line
on aid administration running costs on page 12 which I think I
should draw to your attention at this point because it is germane.
Table 3 on page 12, three lines from the bottom there is a line
saying "Aid Administration Running Costs". You will
see the line goes 56, 65, 70, 64. That figure of 70, which is
obviously an outlier, I am terribly sorry, is wrong and should
have been 64. I am told we double counted capital. It is just
one of those things. I do apologise.
5 Annex 2, Table 1 "Bilateral Aid by Country 1996-97"
Back
6
See Evidence, p. 4. Back
|
