You are here: Parliament home page > Parliamentary business > Publications and Records > Committee Publications > All Select Committee Publications > Commons Select Committees > International Development > International Development

CORRECTED TRANSCRIPT OF ORAL EVIDENCE To be published as HC 945-i

House of COMMONS

MINUTES OF EVIDENCE

TAKEN BEFORE

INTERNATIONAL DEVELOPMENT COMMITTEE

 

 

DFID ANNUAL REPORT 2008

 

 

Wednesday 9 July 2008

MR DAVID PERETZ, MR ANTHONY KILLICK, MR ROBERT PICCIOTTO

and MS ALISON GIRDWOOD

Evidence heard in Public Questions 1 - 45

 

USE OF THE TRANSCRIPT

1.

This is a corrected transcript of evidence taken in public and reported to the House. The transcript has been placed on the internet on the authority of the Committee, and copies have been made available by the Vote Office for the use of Members and others.

 

2.

The transcript is an approved formal record of these proceedings. It will be printed in due course.

 


Oral Evidence

Taken before the International Development Committee

on Wednesday 9 July 2008

Members present

Malcolm Bruce, in the Chair

Richard Burden

Mr Stephen Crabb

Daniel Kawczynski

Jim Sheridan

Sir Robert Smith

________________

Witnesses: Mr David Peretz, Chairman, Mr Anthony Killick, Committee Member, and Mr Robert Picciotto, Committee Member, Independent Advisory Committee, and Ms Alison Girdwood, DFID Evaluation Department, Department for International Development, gave evidence.

Q1 Chairman: Good morning, Mr Peretz, and your colleagues. Thank you for coming to give evidence to us. We had said at the time when Hilary Benn announced the establishment of your committee that we would, in due course, take evidence from you. I think it has taken slightly longer than we intended, but it also fits into the report we will be doing on the department's Annual Report. Clearly, evaluation is an important part of it. I wonder if you could introduce your colleagues. I guess Alison should be introducing herself as she has a slightly oblique role in this.

Mr Peretz: I am David Peretz, and I am Chair of the Independent Advisory Committee on Development Impact. Can I call it IACDI for short? Robert Picciotto is on my left.

Mr Picciotto: I am a professor at King's College, London, I suppose that I am on the committee because I used to head evaluation at the World Bank and because I sit on the Boards of the UK Evaluation Society and the European Evaluation Society.

Mr Peretz: Anthony Killick, who is another member of the committee.

Mr Killick: I am a development economist who has specialised in Africa and I am a Senior Associate of the Overseas Development Institute (ODI) in London.

Ms Girdwood: I am Alison Girdwood, representing the Evaluation Department at DFID.

Q2 Chairman: Which the committee did visit when we were in East Kilbride last year.

Mr Peretz: The Evaluation Department provides the secretariat and secretarial support for our committee.

Q3 Chairman: Which, I suppose, is a bit of an issue we might want to explore too, but perhaps we can get into the meat of it. Clearly we are in a situation where we have a rising aid budget and we need to know whether that budget has been effectively spent, and a whole raft of questions, obviously, arise out of that. This committee frequently asks the question: what works and how good is DFID at delivering what works? I wonder whether you could give an indication of how your committee goes about its work in answering those kinds of questions. You were part of that discussion at the ODI, but do you accept that that is a good starting point for both DFID and for your work as an independent committee?

Mr Peretz: I think it is a fine starting point. It is a question of what works and do you learn lessons from things which do not work? Could I make one point at the beginning? I am very glad to have this opportunity to meet with the committee, and I hope we might have a continuing relationship. I am required, as chairman, to write an annual letter to the Secretary of State and copy it to members of this committee, which suggests that you might have quite a role, if you agree with our recommendations, both in helping to make sure they get implemented but perhaps also, and perhaps more immediately, we on the committee would be very interested that your concerns, either at this meeting or in other ways, feed into us. One of the things we are going to have to do quite shortly is decide on, or begin to discuss, a work programme of evaluations for the next three years, and the question of what is to be evaluated is the sort of question on which I would have thought this committee might have views. Shall I just give a brief sketch of what we have done?

Q4 Chairman: If you could, briefly. I should say, we are aiming for this session to be around about an hour.

Mr Peretz: I will try and be very quick, partly because we are being very transparent and publishing the minutes of our meetings, which I think you have got, so you will have seen those. One point to make at the beginning is we have seen our role as covering both evaluations carried out centrally in the Evaluation Department of DFID but also evaluations, some people call them self‑evaluations, carried out across the department. It is much easier to get a handle on what the Evaluation Department is doing than on what is being done at the various places throughout DFID, but we are trying to look at both, and I think, on the latter, what I would say at this point is my impression is that there is not a very strong culture of self‑evaluation in DFID, there needs to be something of a cultural change, and we are thinking about ways you could do that. One of our recommendations is that the central Evaluation Department should have some role in quality assurance and control over self‑evaluations. So far we have had three meetings. We have given priority to trying to get the structures and processes right in order to be able, in due course, to give the assurances that our terms of reference require us to do about independence, impact, effectiveness and evaluation. I am going to ask Bob in a minute to say something. We have been looking at how to assure independence and bring it up to international standards. You will have seen in our most recent minutes we have a set of recommendations about that.

Q5 Chairman: We have, obviously, a number of questions to ask you which will draw out some of these points.

Mr Peretz: Let me just say that, looking forward for our work plan, at our next meeting we are going to discuss a new evaluation policy for DFID, which I think could be quite important, and begin a process of consultation about a three‑year work plan, which I mentioned, for the central Evaluation Department. Do you want to go on with questions?

Q6 Chairman: We have a number of questions, so I think we can draw things out in the questions. As a supplementary to the more general question, the discussion we have had as a committee (and we have had some advice from the Scrutiny Unit in the House of Commons to try and explore this a bit more) is that DFID in its Annual Report is using its traffic‑light measures against the Millennium Development Goals. The Millennium Development Goals are big international objectives and it is pretty difficult to relate back what DFID is doing to achieve those outcomes. DFID, I think, will acknowledge that, but do you feel your approach might help to fill in the space? The sort of thing we have been looking for is that DFID should be saying, if fulfilling the MDGs, either generally or by country that we are engaged in, in part of our objective, how do we identify the specifics that DFID is doing that directly relate to that achievement, as opposed to taking collective credit or blame for something which may have no causal link at all? Is what you are doing trying to help us answer that question? Mr Picciotto is nodding.

Mr Picciotto: Yes. What you are asking for is at the core of the evaluation function. Namely, you are asking for accountability and for learning about what works and what does not work. This is what an independent evaluation department ought to be delivering. We are focusing on accountability, but one has to be accountable for learning as well: what are the policy principles and the agreed programme goals which have been approved by the governance of DFID? Evaluation checks whether these objectives are relevant, whether the policies are relevant, whether they are being achieved in an efficient way. Next there is the question of impact-what works, what does not work. Impact goes beyond outcomes, and for evaluating impact one needs very sophisticated methods, which are very close to social research. So it is in these two areas that the committee is going to focus. In order to generate the kind of knowledge needed for this, you need two things: independence and quality. Independence is crucial for credibility. As in auditing of accounts, you need an independent function, and that is what the committee focused on at the very beginning and we hope to produce a detailed report on this aspect. We used good practice standards accepted by the international community and based on the experience of governments and auditing organisations, and we worked very systematically through a template designed to assess how independent is the evaluation function in DFID. We started reviewing the quality of the reports. How are they produced? Are the methods right? Are the skills appropriate? 

Q7 Chairman: It will be interesting to see how that develops.

Mr Peretz: Can I try and address this? I think the question you were asking, Chairman, was about the attribution of what DFID do. There is a big problem of attribution in the aid business, and I should say, these are not issues we have discussed in the committee, I am just talking as an individual, but successful development, it certainly seems to me, is usually the result of something of a team effort between the government of the country crucially, having the right sorts of policies, and a whole set of donors supporting that and other agencies. I have an analogy: it is a bit like trying to assess the performance of an individual member of a football team when you are asking, "What is DFID doing and what is the impact of what it is doing?" An important part of evaluation is to make sure that the whole process is working, that the combination of government policies, support from any donors, is actually reducing poverty, making progress towards the MDGs. Then you have to ask a separate question, and it is like asking what contribution does an individual member of a football team make to the fact that it is a successful team or is at the top of the league table? What is the full‑back contributing? You have to ask some rather more subtle questions. It can be done, but it is not a question of being able to say that this money from DFID translates directly into so many children out of poverty or so many people getting HIV/AIDS treatment. It is more a question of asking questions like, as with the football player, "Is the ball being passed to other players at the right time?", and at DFID you would say, "Are they co‑operating with others? Is the advice they are giving to government good advice or bad advice? Are they helping to strengthen the national systems of governance?", and so on. These are things that you can evaluate, but it is not a simple thing. I sense that some people are looking for a very simple answer to the question, "How much does each pound a day do in terms of the MDGs?" Tony may want to add something.

Q8 Mr Crabb: I take the point, Mr Peretz, but DFID does claim that they lift three million people out of poverty every year. Are you saying that that is an impossible claim for the department to make?

Mr Peretz: Tony may be able to answer. As I understand it, this is based on the Collier/Dollar research, which is a cross‑country macro analysis of what you would expect aid to do in good circumstances. It is perfectly reputable research, but it is not looking at‑‑‑. Tony you probably know more about the research than I do.

Mr Killick: As David has said, claims of that kind can only be derived from very macro level research. I guess one of the underlying facts on which that claim is based is the fact that, on the whole, British aid is rather well distributed in favour of poor countries by comparison with various other donor organisations. If one takes the view that aid helps reduce poverty, then certainly the orientation of British aid is of a nature that helps in that direction, but evaluation, which is the concern of the committee, operates at a rather different level. It does not operate at that meta‑level of trying to estimate numbers of millions of people pulled out of poverty from the programme as a whole. The evaluation is looking at specific interventions and what the effects of those interventions are and the cost‑effectiveness thereof, which is a rather more micro-level of examination.

Chairman: Perhaps we could move on to some other aspects of that. Richard Burden.

Q9 Richard Burden: I would like, if I may, to ask you a little bit about an area which, I suppose, brings out this problem in very large quantities, which is how you assess the effectiveness of budget supports. 20% of DFID's bilateral aid programme goes through budget supports, very serious amounts of money, and the Public Accounts Committee recently had a look at that and, on the one hand, they seemed to be saying that this did seem to be effective in terms of increasing services to the benefit of the poor, expanding access to free health and education, and so on, and it said that in 6 out of 9 countries they assessed that that was the case, but then it also looked at areas where budget support was not necessarily used and found fairly similar results. How do you think we can assess whether budget support is working or not? What are the kinds of indicators we can use on that?

Mr Peretz: This is not an issue which my committee has discussed yet, but we will, and I will ask Tony to say something in a minute about it. There was, of course, a big multi‑national multi‑country evaluation of budget support, which I think this committee has seen and had presented to you, which was pretty positive. My own view, I am speaking personally, but I have seen budget support working in a number of countries and, in the right circumstances, it works very well and it avoids doing what traditional methods of aid often do: it avoids undermining or sidelining the national systems of setting budget priorities and financial management.

Q10 Richard Burden: It is difficult to know whether the Public Accounts Committee looked at the effect on developing long‑term capacity.

Mr Peretz: That is a big issue. I am personally quite a fan of budget support, but there are circumstances where it is clearly not the right thing to do and there are circumstances where it is. As we talk about a three‑year programme of evaluations to be done by DFID Evaluation Department, the question of what evaluation or evaluations should, or might, be done in this area is one that we will address. Tony, you might want to add to this. I know that you have looked at the PAC report.

Mr Killick: Yes. I share David's view that there is a strong prima facie case in favour of budget support, by comparison with the traditional sort of project‑based approach which leads to very fragmented and incoherent and highly costly types of intervention, but budget support is not universally applicable. One is, after all, putting money into fiscal systems which are often quite weak, and so there is an element of risk involved there, sometimes a substantial element of risk. I think risk is perfectly justified, but it does have to be justified in terms of a careful appraisal of the situation before one goes into it. I had a look at the Public Accounts Committee report and one of the things I took from that was its conclusion that DFID, as an institution, is not collecting the information and does not have the systems in place to enable it to reach firm judgments about cost‑effectiveness, and I think that reinforces what David said about the need to look at that rather carefully. There has been this multi‑donor, multi‑country evaluation back in 2006, which arrived at really quite positive conclusions overall, but it was not addressing issues like are there systems in place within DFID to take sensible decisions about cost‑effectiveness, and maybe some more process‑oriented evaluations of that kind could be a very appropriate response to the PAC report.

Q11 Richard Burden: It would seem to me that there is likely to be greater focus on this and, therefore, the need to develop robust evaluation mechanisms on that. It is not only necessary, on which I think we are all agreed, but there are also probably quite urgent ones that build in. The point that Malcolm said about issues of building capacity as well as more tangible results. You said this is going to be an area you are going to be looking at. Is there any likely timescale on that? I am looking for developing mechanisms rather than looking at DFID.

Mr Peretz: What I said was it will be part of our discussion. One of the few things that this committee actually has the power to do, we have the power to agree the future work plan of the Evaluation Department. What we have decided is that we will consult on a three-year work plan, which will start next March, and we will consult over the winter about that with quite a long list. One of the elements of that list will be an evaluation, or several evaluations, of budget support. I should say, though, there are things that certainly could be done in the meantime. The Evaluation Department already do about 5 or 6 country programme evaluations a year looking at individual countries, many of which have budget support as one of the instruments, and one of the things we have agreed as a committee, and I am very glad that the head of the Evaluation Department has volunteered to do this, he is going to write, from now on, an annual report which will look across the evaluations that have been done over the year and try and draw out themes. One of the themes might well be the circumstances where budget support works and where it does not, just looking across those evaluations which have already been done.

Chairman: Sir Robert Smith has some follow‑on questions relating to that. It might be appropriate for him to come in, if that is all right with you.

Q12 Sir Robert Smith: On the methodology of evaluation and how rigorous it is, it has been put to us that you can have an informal evaluation that says the aim of the intervention by DFID was to achieve the goal of, say, 20,000 odd people getting immunised, or something like that, and then the evaluation comes along and says, "We put the money in and we ticked the box because 20,000 people got immunised". Or is there the more rigorous evaluation which says, "If we had not made the intervention, was that going to happen anyway? Was there a causal link between the intervention and the outcome?" Where on the spectrum does the evaluation tend to take place: more towards the informal or the more rigorous?

Mr Peretz: This is what I was trying to say earlier. Tracking through money to effects is never that easy. If you say, if we take your example, money was put in for inoculations but, probably, for that to work, it depended not just on the DFID money but on things the government was doing in terms of having health centres and transforming the health service and even things like transport arrangements that get people to the clinics, all these things have several players, and that is the problem with attribution. On actual evaluation methods, I turn to Bob, who is the expert on this.

Mr Picciotto: There is a need to focus on impact evaluation of the kind that you are suggesting. In fact DFID is chairing a group which is focusing on how the development community should approach impact evaluation. There is a lot of momentum behind randomised control trials where, indeed, you distinguish between two groups: the experimental group and the control group, and you select a group of individuals affected, (let us say by a vaccination programme) on a random basis. This kind of approach removes the bias that exists as you are implying, if one simply looks at what happens before and after the interventions since it may be the result of other factors than the intervention. These are very powerful methods, but on the other hand they have their own limitations. You have something called the Hawthorne effect, where the simple fact of doing the intervention modifies the behaviour of participants; you have the John Henry effect, where people who are not in the control group are also affected; you have a selection bias, and so forth. This is why the evaluation profession is arguing for a mix of methods: randomised control trials, at other methods of matching through propensity scores, interrupted time‑series, and other techniques which are very close to research. That is why one of the issues in an evaluation policy is how the evaluation department connects with the research department and the chief economist, and so forth. At the World Bank both the research and the evaluation departments are working together on developing these techniques, but one has to be very careful to triangulate between these techniques and more traditional techniques which go beyond whether the intervention works or not to figure out why does it work, how does it work, who makes it work? One of the problems with impact evaluation of the highly abstract type of the kind I described is that it is weak on accountability, because all it does is answer the question, "does it work or does it not work?" It does not tell who is supposed to make it work, which is the point the Chairman was raising earlier. So we are recommending a mix of methods, and I think it is the choice of methods and their mix which the Evaluation Department should be overseeing.

Q13 Sir Robert Smith: You mentioned earlier how development and aid is a team effort between the recipient and the donor and many other partners involved, but then, obviously, the partner countries do not necessarily have the resources, the capacity, to gather the data. What is DFID doing to overcome these challenges to make sure the recipients have the resources to take part in the evaluation?

Mr Peretz: I think it is a very important point. This is the first part of my analogy. How do you judge the performance of the team? Is poverty being reduced? Are more children going to school? Are education and health outcomes better? Are more people being immunised? Collecting statistics and the national capacity to do this is absolutely essential. This is something Alison might want to add to, but my impression is that DFID are doing quite a lot to try and strengthen national systems of statistics and monitoring arrangements of this kind, and it is critical.

Ms Girdwood: The main areas of work are supporting the Marrakech Action Plan for Statistics, which is mainstreaming strategic planning of statistical systems and assisting capacity to develop national strategies for statistics, supporting internationally the 2010 Census around the Household Survey Network and general statistical capacity building, largely through the Paris 21 Consortium. We are supporting that to quite a large extent.

Mr Peretz: This is a consortium of donor countries.

Ms Girdwood: Yes. We are also doing an evaluation through the Joint Paris Declaration, Evaluation of National Statistics Capacity Building, and internationally how best to support it.

Q14 Sir Robert Smith: So there is proper co-operation between donors as well. Joint evaluations, presumably, can pool resources?

Mr Peretz: Yes; absolutely. Certainly our committee are very much in favour of joint evaluations as part of the mix, and when you are looking at country programmes certainly the first half of my question ‑ how is the whole thing working-would be much better done that way. On the other hand, there are great problems in actually organising and getting people together to agree to do joint evaluations. It is one of the things where there is quite a lot of money in the Evaluation Department's budget which will be available for this which they are having difficulty spending, but we are going to get on to budgetary issues later.

Q15 Chairman: I am not sure you are there to help them spend; you are there to help them to get results.

Mr Peretz: They are having difficulties spending effectively, I should say.

Q16 Sir Robert Smith: You see a big benefit in joint working, but the barrier is the different methodologies and, presumably, different cultures within the different donors?

Mr Killick: Yes. There are different methods, different approaches, but also different objectives. Different donors have different objectives and want to get different things out of the system. The point I would be particularly keen to make, in the context of your question, is that we should not have an unrealistic expectation of what aid and what aid donors can do by way of strengthening institutions. What we have learned about the institutional factors, and how institutions change and improve over time is that the key factors are domestic, not external and, of course, donors such as DFID can provide technical assistance, can provide training, other types of resources, but in the end the leadership and the motivation for this has to come from within the countries, which is one of the reasons why one needs to be really quite careful about choosing the countries or the governments that one is assisting in these ways: because in some situations it is so easy for a donor to design a fancy programme for strengthening this institution and that ,but without any real local buy‑in that is most unlikely to be effective.

Mr Picciotto: Quickly, on this particular point, I would say three things. First of all, the Paris Declaration asks for harmonisation across donors for all aid practices, and this applies to evaluation as well. Secondly, harmonisation of evaluation methods is under way both in the multilateral system and in the bilateral system under the OECD Development Assistance Committee, where Nick York who heads the DFID evaluation department is taking over the evaluation working party. Thus the harmonisation evaluation method is very much on the agenda of the evaluation community. Thirdly, there is the issue of evaluation capacity development in poor countries, connected, for example, to the PRSC System (the Poverty Reduction Strategy Paper System) involving selecting the right households and getting the statistical system connected to the evaluation systems. Building evaluation capacity in developing countries is important. It is one of the things we will take a look at in terms of how the Evaluation Department of DFID is contributing to this priority.

Q17 Jim Sheridan: I think evaluation of how taxpayers' money is used in terms of development is absolutely crucial, and I do not think anyone would deny that, but there is a growing number of evaluators now coming into this field and we have evaluators evaluating the evaluators, which is somewhat concerning. I am just a bit concerned that, if we are going to be focusing on evaluation of evaluation, we are diverting resources away from where it should be going, and that is to the people that actually need it. The question I really want to ask is: what so is unique about your organisation that other organisations have not brought to the table so far? Is there a possibility of duplication, but, most importantly, how can you assist DFID in its overall objective of getting development to the people that need it?

Mr Peretz: I think we were set up as a committee with terms of reference to assure the independence of evaluation in DFID, to try to make sure that lessons from evaluations are learnt and, where recommendations are made and accepted, that there is follow‑up, and also to improve the clout of the evaluation, to get it listened to in the department: because a lot of work is done and there are lessons to be learned. Evaluation is partly about accountability, it is about saying whether a thing is well done or not, but also at least half is about lesson‑learning. I think we have begun to make some impact. You will see that one of the things we have got now, we are having an annual report from the department on follow‑up to past evaluations so that these are not just books which are put on the shelf, the department is going to account for what it is doing as a result of recommendations. Obviously, some of the recommendations will be rejected, but where they are accepted, there should be follow‑through, and one of the things we are doing as a committee is making sure that happens. I think we are taking steps to improve the independence of the central Evaluation Department in DFID, and you will see we have 11 proposals in our latest set of minutes which we are going to take forward, and I will attach them to my annual letter to the department.

Q18 Jim Sheridan: I would hazard a guess that similar organisations would say exactly the same. Can I ask Alison what is unique?

Mr Peretz: Can I say, there is one other country which has a committee similar to ours, which is Ireland. We have now some inquiries from other countries. I think the Dutch Head of Evaluation wants to come to our next meeting. It is early days yet, but I think it could have quite a big impact.

Q19 Jim Sheridan: The point I am trying to make is that there is a finite amount of resources. I do not want resources spent searching for statistics or evaluations. I am interested, Alison, from a DFID perspective, what is unique about this advisory committee?

Ms Girdwood: Soon it will not be unique. It is actually being copied by a number of our partner evaluation departments. I think the Netherlands are adopting the same model. So I think there is a general feeling across the other agencies that they need something of this sort to strengthen the function internationally.

Mr Picciotto: You are making two points. First, what is unique about this committee? Your Secondly, who evaluate the evaluators? Our committee does not do evaluation. Is too much going into evaluation whether in DFID or in this committee? The committee has only met three times. My own view is that the resources assigned to the committee are tight, but probably appropriately tight, to make sure there is subsidiarity and we do not start doing other people's work. And if you look at how much resource goes into evaluation in DFID today, it is 0.1 % of the total programme budget, and as a percentage of the evaluation budget it is roughly 2 %, which is pretty much in line with other aid organisations. The point that the Chairman was making is that, as the programme budget of DFID goes up, for quality purposes, and for accountability purposes, the evaluation budget should also increase in a reasonable way in line with good practice. This committee is filling a gap: since before the committee was established there were questions about how independent is the Evaluation Department in DFID. This is the key value‑added by this committee.

Mr Peretz: The exact figure is 0.09 % of the total-that is for the current year-but, as I indicated, some part of that will not be spent because it is actually allocated only for international co‑operative evaluations which, as we have discussed, are quite difficult to mount, and that is more than half of it. If you look at the administrative budget, which is what DFID spend themselves and use to spend on consultants for evaluating DFID programmes, it is 0.04 % of the total spend this year. I think what has concerned the committee in our discussions so far is not so much the absolute size of these figures, which we have not really discussed, but the fact that this is a declining figure rather than a rising figure at a time when the total programme budget is rising quite fast.

Daniel Kawczynski: I think my colleague, Mr Sheridan, has touched upon a very important point. We obviously want co‑ordination of evaluation, and Mr Picciotto said that there had been three meetings, is that right, that you had had three meetings, and that the budget was how much: 0.1 %?

Q20 Chairman: I think you were talking about DFID's budget.

Mr Peretz: DFID's Evaluation Department budget is 0.09 % of the total budget. Our committee is a tiny proportion of that.

Daniel Kawczynski: We too also here are meant to be scrutinising DFID and evaluating taxpayers' money. My only concern is who actually appointed you-that is one question- and how long is your term of office? I have a concern that it really should be Parliament who is setting up these organisations and not government. I am a little bit concerned that there is actually a member of the department here, because I would have really liked to have seen you without anybody from DFID?

Q21 Chairman: It is a public meeting.

Mr Peretz: The factual answer is that both I and all members of the committee were appointed by the Secretary of State, who appoints us. Personally, I have a three‑year contract, which is renewable for another three years, which will take me to over 70, which is probably the time I should give up anyway.

Q22 Chairman: Mr Kawczynski is making a point which Parliament quite often is seeking, which is that there should be more of these kinds of public appointments which are, at the very least, to some extent ratified or endorsed by parliamentarians. This is something that committees are trying to achieve. The Government, to be fair, have acknowledged in some quarters that they are willing to do this. You have not fallen into this category yet.

Mr Peretz: I do not fall into it. I am not sure it is an issue on which I have a view. As people say, it is a bit above my pay grade. I think it is a question you might want to ask of the Secretary of State. I guess he is the person who would reply. What I would say is it is a very high quality committee. We have a lot of experience and expertise round the table. I do not think there is anybody round the table who feels what they say is in any way affected by the fact they were appointed by the Secretary of State. We are appointed as individuals to give our independent advice, and that is what we will do. The fact that Alison is here is partly because, as chairman, I invited her. As I said earlier, the Evaluation Department provide secretarial services to the committee, and I thought it might be helpful to have somebody from the Evaluation Department here in case there were questions, as there have been, about what DFID is doing, rather than what the committee is doing, which she could helpfully answer.

Mr Picciotto: It is an international practice. You have two ways: one is reporting to the head or the deputy head of the organisation, which is what our committee is recommending, that the Evaluation Department should eventually report to the Permanent Secretary, which is quite an acceptable way of defining independence. The other way, for example, in the United States, the General Accounting Office reports to Parliament. This is also quite a legitimate way of doing it. All I am saying is that it really depends on the government of the countries concerned and it is quite respectable to have it one way or other. So, it is really a matter essentially for the UK authorities to decide how they handle it. All I am saying is that this structure does meet good practice internationally. In other words, to have a unit which reports to the head or the deputy head of the department would meet good practice. At the moment it does not, the Evaluation Department does not yet report to the Permanent Secretary, but when it does it will meet good practice internationally. The alternative would be for your committee to set up an evaluation unit.

Q23 Daniel Kawczynski: I have no doubt that you are extremely eminent and competent people, and I applaud the work you do. My only concern is that as a parliamentarian, as a democrat, I believe that you should be appointed by Parliament itself and be accountable to Parliament directly, rather than DFID appointing you and having you appoint them. I have great reservations about that. Coming on to my official question, what do you think are the organisational and institutional challenges to effective impact evaluation which DFID faces? What I am trying to get it is: we obviously have various difficulties in terms of probing DFID through their structure and organisation. Could you perhaps enlighten us on any negatives that you are experiencing in terms of being able to get under the skin of DFID?

Mr Peretz: It is early days yet. We have had three meetings, but I would say our experience so far has been quite good in that proposals we have made have been accepted. I am not sure "getting under the skin" is the phrase I would use, but I think there is an issue, which I alluded to earlier, not so much about the work of the Evaluation Department, which we see and we see a lot of, but the evaluations done across the department as part of their regular business: self‑evaluation. I think I said at the beginning, our feeling as a committee is that there is not yet a proper culture of self‑evaluation in the department of looking at how you are going along with a project or a programme and regularly monitoring, evaluating and reporting back, and that is an issue which we want to get more involved in. We are going to discuss it more at our next meeting when we come to talk about the evaluation policy for the department, but there are limits as to what an advisory committee like us can do except apply pressure where we think there are issues and ask for evaluations. The one thing we have suggested is that the central Evaluation Department itself should play a role in trying to raise standards of evaluation across the department. You asked me where I would see problems. That is certainly one of the areas where we do see a problem.

Q24 Mr Crabb: Can I come back to the issue of money allocated by the department for evaluation. You talked of 0.09 %. The cash amount was £2 million pounds for the current financial year. That is the 0.09 % figure that Mr Peretz referred to?

Mr Peretz: £2.3 million for the administrative budget.

Q25 Mr Crabb: £2.3 million?

Mr Peretz: To get the total budget you need to add together two figures. I have got a table here I can give you. But these are the figures for the Evaluation Department, not for other evaluations carried out by DFID.

Q26 Mr Crabb: So the total sum being spent by DFID for all of its aid evaluation may be considerably‑‑‑

Mr Peretz: It may well be bigger than this, yes, quite a lot bigger.

Q27 Mr Crabb: One of the questions I was going to ask, and you have probably answered it already, is whether you believe that the amount of money DFID is allocating for this activity is appropriate and commensurate with the significantly increasing size of its overall departmental budget?

Mr Peretz: This is an issue we have only partly discussed as a committee. One of our recommendations is if you are going to have an independent evaluation department, then our committee should have some control over the budget. What we have said is we should have "a central role" in setting the budget. What the budget should be we have not really discussed. As Bob says, if you look at it one way, it is a little under 2 % of the administrative spend of the department, which is probably in line with international practice. If you look at it another way, the way I was saying, it is at present less than 0.1 % of the total spend of the department, and actually it is the administrative budget which is the most usable part of what they have, and that is less than half of the Evaluation Department's budget. The figure there is 0.4 %. What actually concerns us all, as much as the overall figures, is the fact that these are falling figures. If you look at the projections for future years it is due to come down.

Q28 Mr Crabb: As a percentage, not overall cash?

Mr Peretz: No, for the administrative budget in absolute terms.

Q29 Chairman: Can I pick you up on that.

Mr Peretz: Both as a percentage and in absolute terms for the administrative budget.

Q30 Chairman: This committee has just done two reports on DFID's funding contribution to both the World Bank and the African Development Bank, because they are very substantial donations, and we took the view that we should evaluate that. Have you done that, or would you be doing that: because I think what concerned us is, if we had not done so, an awful lot of money would have been devoted to the World Bank without any external check at all?

Mr Peretz: I can say, we will discuss this when we discuss the future work programme. It is quite clear that evaluations of contributions to multilateral organisations will be an important item on the programme, simply because it is such a fast-rising proportion of the total, and you can do it at two levels. You can evaluate the performance of the multilaterals themselves, and they have their own evaluation departments, but the other thing you need to do is to evaluate how DFID are influencing those organisations.

Q31 Chairman: Obviously, that is what our report did.

Mr Peretz: That is another thing to be added to the list, but, going back to the budget, as I was trying to say, our concern is that it is a declining figure both in percentage and absolute terms at a time-I speak now as an ex‑Treasury official, if you like, but I know very well that when departments have its expenditure rising at a very rapid rate, it is the very time you need to be doing the most evaluation.

Q32 Mr Crabb: Given that we are in a period of a very aggressive ramp up of spending within the department and given that there is, I think it is fair to say, a growing demand within Parliament, and certainly maybe out in the public as well, for much better knowledge and awareness of what actual impacts we are getting for the money that we are spending, what kind of size of evaluation budget should the department be deploying? Are you in a position to give us any idea of that?

Mr Peretz: We have not really discussed this yet. One thing we have said is that if the central department is to take on the extra role which we want it to of assuring quality of evaluations across the whole of DFID, then you will need extra resources, so you would need a bigger budget for that. We have also said over the next two to three years that the head of the Evaluation Department should be of a more senior grade, which would cost a bit more money, and should probably be a person who has said, "I will not, after this work, stay in DFID."

Q33 Chairman: An end of career post.

Mr Peretz: Probably an end of career post, yes, but not necessarily. It might be an expert in evaluation who sees his career in evaluation somewhere else.

Q34 Mr Crabb: Just on a human resources point, do you think there are enough people within DFID actually working on evaluation? Do you think that the head-count of evaluators within the department is right, or does that need to be increased too?

Mr Picciotto: It is about the same as other bilaterals, but we need to analyse this more systematically. The key policy for evaluation is the extent to which evaluation is done by consultants or whether it is done by DFID staff. When I used to run the department at the World Bank we essentially did most of the evaluations, using consultants to a very limited extent. Here it is the other way round. So, looking at the interface between the Evaluation Department and its consultants is one area we are going to look at.

Q35 Mr Crabb: I think a colleague of mine is going to come on to the consultants question. Finally, the comparative study last December by ODI reached a number of conclusions, one of which was that there is a disconnect between the rhetoric about the importance of evaluation and the practice within agencies. It also characterised impact evaluation as "a function in search of an identity". Do you think either of those conclusions are fair and correct for DFID?

Mr Peretz: It was an interesting report, and it also had something to say about how the head of evaluation department in DFID fitted into the structure; and there has been a change since that report in the right direction, from our point of view, though I think what we are now recommending is that he or she should report directly to the Permanent Secretary. We have talked about impact evaluation. I do not know if Bob‑‑‑

Mr Picciotto: Impact evaluation is obviously eliciting a great amount of debate at the moment, and DFID is at the centre of this debate. No, I do not agree with the conclusion that it is a function looking for an identity, because there is not yet a full consensus on the precise role of impact evaluation. To do it well is very expensive and it does not always answer the accountability questions. On the other hand it can have a huge impact if applied in the right circumstances at the right time. So its role is part of the evaluation policy which we hope will be developed by DFID.

Mr Killick: Can I add one thing? It would be possible to interpret the Minister's decision to create us, this committee, as an attempt to narrow the gap between the rhetoric and the reality to give him greater assurance about the quality and independence of the evaluation function and the lesson learning which goes on as a result of the evaluation. I am sure there is still a rhetoric reality gap, but I think we are a practical way of trying to narrow that gap.

Q36 Richard Burden: A few moments ago you mentioned the possibility of the head of the Evaluation Department having some greater autonomy. I think the comment was made, an end of career post. We were talking about this before. If you create such a post it will be known in advance that it is an end of career post rather than arising out of what you did in that post! Do I take it from that, though, that you think, as things stand at present, the Evaluation Department really does not have the clout that it needs for access at the level it needs in the department, the access to the documents it needs, and so on?

Mr Peretz: I think there is a set of questions there. Yes, and if you read the minutes of our last meeting we have got an 11‑ point plan, but, yes, we say the head of the department should be accountable directly to the Permanent Secretary, should be a more senior grade and status, and the budget probably should increase. Certainly there should be a role for IACDI in deciding the budget. We also say there should be rules about access to information. I do not think we really have any indication that access to information is denied. There may be occasions where it happens, but in general it is quite good. But these are the sorts of things which there should be written rules about which everybody knows and understands. I suspect more of a problem, particularly in current times when staffing is very tight across the department, is each evaluation actually takes up the time of departmental staff as well. They have to devote time to helping the evaluators and when they have other things to do it may not be top priority. This is part of what I was talking about earlier about changing the culture. I think that needs to be done, and there need to be rules about them giving it priority too. Alison, you have had experience of evaluations. Do you have any problems you want to point to?

Ms Girdwood: There has been a change recently where some evaluations are now mandatory rather than discretionary-beforehand it was a process of negotiation-which has been helpful in terms of ensuring that we can carry out the programme we are expected to carry out.

Mr Peretz: If you have a look at our list of propsals, we said there should be clear written protocols for unimpeded access by DFID staff and their consultants to information in DFID; there should be clear rules of engagement with DFID staff when discussing draft reports, because that is where, if you are not careful, you can get into a negotiation process, toning down what reports say, and they become less than independent and rigorous; there should be rules with DFID for avoiding staff conflicts of interest, and if staff are evaluating something they have previously worked on, they should not do it; there should be a written policy on publication of all reports, in fact I think they are all published at the moment, but there is no policy. So there is a set of things that we have recommended.

Q37 Richard Burden: Are you expecting, if you like, a composite response to that?

Mr Peretz: Yes.

Q38 Richard Burden: If so, roughly when and are you hopeful?

Mr Peretz: If you look through these 11 points, most of them we have had, I would say, some indication that they would be accepted and a lot of them do not cause any problem, they are just putting down in written form what has been practised. But, no, I have not had a response yet. What we said in the minutes of our meeting, which are published, is that I will attach these-I am going to write an annual letter, my annual letter, in October to the Secretary of State, and I will attach these, and hopefully some of them will be acted on by then, but we will see.

Q39 Mr Crabb: I was the colleague that was going to raise consultants. Coming back to the issue of consultants, the ODI report, rightly in my view, raised the question of how truly independent a consultant can be if they are relying on commissions from the department for their pay and then they have to basically evaluate the people who they are relying on for their livelihoods. Is that a concern that you share, and what other concerns would you have about these consultants in evaluating aid?

Mr Peretz: As Bob indicated earlier, I think the committee does share that concern. I think traditionally it was thought that using consultants guaranteed independence because they are were not departmental staff, and I think what the ODI report had to say is absolutely true, that consultants are no more or less independent than anybody else. What you need to do is to have the structures in place to make sure that the whole of the Evaluation Department is independent, particularly the head, which is what we are trying to do. As a committee, one of our 11 points is that we think, over time, there should be a move to doing more of the work in-house and using consultants less. There are other problems with consultants, I understand, just ensuring the quality of what they do, which varies. Some are very good and some are not so good. There is just the whole process of managing them.

Mr Picciotto: Fee dependence is what you are concerned about, which is right. The consultants are dependent on fees and they may not be as independent as you may think they are. What we are looking for is a golden thread connecting an independent head of the evaluation function who has the authority for setting incentives for the evaluation staff, who have clear accountability over the work of the consultants. The other aspect is clear methods, guidelines and standards that consultants should meet. All of these things take time to instil, and an action plan to develop those things is needed. Given the need for flexibility there is nothing wrong in using consultants, and if the independence recommendations are implemented and if we work on improving evaluation methods, we will go a long way in dealing with the ODI concerns.

Q40 Mr Crabb: Have you seen any evidence yourselves of that concern being affected in practice? Have you seen evidence of, for example, a positive report about a particular aspect of aid by a consultant where the suspicion is that actually they have been a bit weak, because of the issue we have just talked about?

Mr Picciotto: I am talking now as an evaluator and not for the committee, because we have not finished our examination of the quality of reports; but one of the criterion of independence is behavioural independence, which means producing on a regular basis uncompromising reports which go under the skin of DFID. My review of the reports suggests that it is a mixed bag. Some reports do get under the skin; others do not. By strengthening the independence of the whole system and by changing staff incentives, one will gradually see much more behavioural independence than can be observed now by looking at the reports. That said, DFID evaluation reports, as compared to those produced in the bilateral development aid community, are well above average. It is not that the reports are bad; it is just that they could be better and tougher in some cases.

Q41 Mr Crabb: Very roughly, what estimate would you make of the number of consultants working in the area of evaluation who are former employees of the Department?

Mr Picciotto: I do not know.

Mr Killick: I would not identify that as a major issue. I think that most of the consultants are academics or ex-academics. I do not think it is an issue so much of buying in to people who have previously been working in the Department. That is not my impression.

Mr Peretz: In fact, previous departmental members may be the most critical. The point about conflicts of interest applies, clearly. You would not want an ex-member of staff to be evaluating something he had been involved in as a member of staff. At our last meeting we had our first discussion about the quality of evaluations. We reached a conclusion rather similar to somebody else who gave evidence to you that, in comparison to other bilateral agencies' evaluations, DFID scores pretty well. I think we said that it is well up, if not the best, of the bilaterals-but that is not a particularly high standard and we, as a committee, will be aiming for something much better than that.

Q42 Jim Sheridan: I still remain unclear about just how your advisory committee would ensure that the self-evaluation of DFID will become independent. Would there be some other alternatives, other than what you have suggested today, to ensure that DFID has that independence? What alternatives could there have been?

Mr Peretz: I think that you are asking a question about evaluation, not in the central Evaluation Department but elsewhere in the Department, which we said we were doubtful about. To be honest, we need to find ways to get to grips with that. We suggested one, which is that the Evaluation Department itself should have a role, and may have quite a key role, in assuring quality. It is not just independence that you are assuring; it is whether it is being done well and, indeed, are the lessons being learned as you do it yourself. That would be my answer to that bit of the question, but there was another half of the question as well.

Q43 Jim Sheridan: You could not identify any other alternative other than what is on the table today?

Mr Picciotto: I would say that if these 11 recommendations are implemented, DFID evaluation would move to the "top of the class"; but they are not yet implemented and it will not happen overnight. It takes time to develop the skills, to develop the organisation, to change the culture; but it should not take too much time either. In two or three-years we should see DFID become a centre of excellence for evaluation. This is not the case now.

Q44 Chairman: In Mr Sheridan's earlier intervention he was expressing concern about money being spent on evaluation that could have been spent on aid. The counter to that is that what people will want to know is that the money spent on evaluation actually leads to changes in policy and implementation that will improve the delivery of aid. First of all, do you get the impression that that is currently what is happening with evaluation within DFID? You shake your head. Is it your intention to ensure that in future it does?

Mr Peretz: We have got DFID to agree to institute this process of producing an annual report of follow-up to previous evaluations. They have never done it before. It will be published; you will see it. It is quite a patchy document, because it covers a period when nobody knew they were going to have to produce this, and in future they will. The departmental response to evaluations is sometimes lacking. Very often it does not cover all the recommendations; so you do not know what they are going to do and what they are not going to do at the beginning. The follow-up is then patchy, I would say. However, I think that we have set in train a process and we have also said, and got it agreed, that for each evaluation report done by the Evaluation Department a director in DFID will be responsible for managing the response and the follow‑up; and our committee may call directors to meet with us and explain what they have been doing. We have now created some incentive, therefore, for lesson-learning and for follow-up which was not there before-and that is a bit of a success.

Q45 Chairman: It has been a very interesting and useful exchange, because it is quite clear that you have set in process a whole range of actions which could clearly make a material difference, it seems to me at a reasonably affordable cost, in terms of what is involved. Obviously we will look forward to being copied in on your letter to the Secretary of State and, to take up your own remarks, I think that we may well want to meet again. Perhaps there may be value in having an informal meeting as well and not necessarily always to do this in a formal evidence session. Clearly, what we are interested in is the extent to which you will be able to improve the culture, which you indicate was lacking in DFID, and to make the point of evaluation to deliver better aid and development funding, rather than simply to prove that DFID is right to be pleased with itself-which I guess is not really a justifiable use of public money. Can I thank you for that first initial engagement. To pick up on Mr Kawczynski's point, the time may come when confirmation hearings for people in your position will be part of the process. We are working on it.

Mr Peretz: Can I thank you and say that, as to the idea of an informal meeting, over the winter we will be consulting on the future work plan and also DFID's evaluation policy. It will be very helpful to get this Committee's informal input into that process in some way.

Chairman: I think that the Committee would appreciate that opportunity. It would be mutually beneficial. We will try to fit it into the diary at an appropriate time.