To be published as HC 1743-ii

House of COMMONS





Wednesday 8 FEBRUARY 2012

professor anthony costello, dr john kirkland,

professor melissa leach and professor andrew westby

Evidence heard in Public Questions 24 – 55



This is an uncorrected transcript of evidence taken in public and reported to the House. The transcript has been placed on the internet on the authority of the Committee, and copies have been made available by the Vote Office for the use of Members and others.


Any public use of, or reference to, the contents should make clear that neither witnesses nor Members have had the opportunity to correct the record. The transcript is not yet an approved formal record of these proceedings.


Members who receive this for the purpose of correcting questions addressed by them to witnesses are asked to send corrections to the Committee Assistant.


Prospective witnesses may receive this in preparation for any written or oral evidence they may in due course give to the Committee.

Oral Evidence

Taken before the Science and Technology Committee

on Wednesday 8 February 2012

Members present:

Andrew Miller (Chair)

Stephen Metcalfe

Stephen Mosley

Pamela Nash

Graham Stringer


Examination of Witnesses

Witnesses: Professor Anthony Costello, Professor of International Child Health and Director, UCL Institute for Global Health, Dr John Kirkland, Deputy Secretary General, Association of Commonwealth Universities, Professor Melissa Leach, Director, STEPS Centre, and Professor Andrew Westby, Director, Natural Resources Institute, University of Greenwich, gave evidence.

Q24 Chair: I welcome you to this session and thank you for attending this morning. It would be helpful for the record if you could all introduce yourselves.

Professor Costello: I am Anthony Costello, director of the UCL Institute for Global Health.

Dr Kirkland: I am John Kirkland, deputy secretary general of the Association of Commonwealth Universities.

Professor Leach: I am Melissa Leach, director of the SLC STEPS Centre at the Institute of Development Studies in Sussex.

Professor Westby: I am Andrew Westby, director of the Natural Resources Institute at the University of Greenwich and also president of Agrinatura, which is a European association of universities working on agricultural research and development.

Q25 Chair: Thank you very much. If we may start with definitions, DFID’s definition of capacity building is "enhancing the abilities of individuals, organisations and systems to undertake and disseminate high quality research efficiently and effectively". Is that a fit-for-purpose definition on which we can work, or would you encourage us to look at something slightly different?

Professor Costello: It is more or less fit for purpose. It goes on later to talk in more detail about social capital, trust, knowledge, skills and attitudes; it puts a little more flesh on the bones, but it is a reasonable definition from my perspective.

Dr Kirkland: I agree with that. The three levels-individual, organisational and what we call system or environmental-are a particularly useful way of hanging on later analysis.

Professor Leach: I would agree. Beyond that broad catch-all we need to ask some harder questions about the capacity of whom to do what precisely, but that debate follows on from that broad definition. I would also emphasise that the systemic aspect, thinking about how individuals relate to organisations and broader networks, is critical. I am sure it is something that will come out of our later discussions.

Professor Westby: I am also broadly in agreement. For me, the key issue is capacity building to do something rather than capacity building in its own right. In my field of agriculture, capacity building to improve the livelihoods of poor people would be important for us. It is also important to distinguish between capacity building in science for doing the research-research capacity building-and capacity building for the extension of science to have impact on the ground. I would like to flag up that they are two quite different things.

Q26 Chair: Following that, how does building research capacity relate to building scientific capacity? Is the distinction between those terms understood particularly by Government?

Professor Westby: It is understood, but often when we talk about capacity building we mean research capacity building. You definitely need that. For example, in my own field we need researchers in developing countries to be able to deliver new varieties and so on, but we also need a trained cadre of scientifically aware extension workers-for example, in the public-private or NGO sector-who know about science and are able to deliver good scientific advice to farmers to increase productivity and things like that.

Dr Kirkland: I agree with that. I am sure this will be said at several points today and has been said in your previous discussion. It is amazing how many of these issues could equally be cited to some extent in the British higher education or knowledge transfer system. I strongly agree with the importance of extension work and also the quality of science graduates entering the system in those countries.

Q27 Graham Stringer: Following on from what Andrew said about building research capacity and other answers, is it better to invest in individuals or institutions?

Professor Costello: I would say the latter. For example, if you look at the Wellcome Trust or Medical Research Council-I come more from the health angle-traditionally, over the years they have invested huge amounts of money in largely British-run international research stations, and there are lots of other examples of that. About five years ago Sir Mark Walport went round Africa and India and realised something we had pointed out 20 years ago. It has short-term benefits for high-quality research but does not invest in the institutions that you want to strengthen in countries; indeed, it detracts from them because it tends to take people out of the system. He realised that they were training up lots of competent African postgraduates and postdoctoral students who had nowhere to go because the university system had collapsed.

The model they are now moving towards, which says, "We won’t take people to do PhDs in Oxford, UCL and Sussex; we want to try to develop the system through the local university but give them a similar experience," is one that we should be moving towards. Although we have been trying to do this for about four years as part of a partnership, we are just starting to see some of the benefits of it. You have to hold your nerve and work with institutions in order to try to develop it. I have just been in Malawi where we are finally getting a PhD programme running. We are linking with the local systems; we have enrolled a number of local PhD students of different levels. The short-term benefits of an NGO approach or internationally-run centres have long-term disadvantages, and I think we should be moving to a more institutional model, personally.

Professor Leach: Building on that, I would absolutely agree that training PhDs and thinking of new kinds of partnership models to make that work is crucial, but we also need to move beyond a simple focus on what our submission terms "centres of excellence". That is very much the discourse that DFID and others have used in promoting science capacity building, which is about generating capacity, as it were, to compete on an international world stage on the scientific issues and debates of the moment.

That is really important but does not answer the question raised by Andrew about how we build the capacity to deliver innovations and applied technologies that will improve the livelihoods and health of people on the ground. That requires a broader approach both to individuals and institutions that recognises there is knowledge, innovation and capacity among citizens, small businesses, farmers, health practitioners and extension workers. Part of the requirement is to get those networks and people talking to one another so that science and research can respond to the needs of users and, in turn, can get out better to the people who need to make use of it and allow their innovations to flourish.

There, I think there is a role for some new kinds of institutions that might be linked to universities but also move beyond them. For instance, in the climate change field, DFID itself has been pioneering new climate change innovation centres where it is working with a group called infoDev. It has piloted them in India and Kenya. These are hubs that bring together scientists and business entrepreneurs-people who know about funding and local conditions-who can work together to generate new kinds of science.

Another model in the field of agriculture would be the Future Agricultures Consortium, which has been a kind of policy engagement research hub. That is now pioneering some Africa hubs, which again will bring together researchers, field people and policymakers to try to generate the kinds of research that are really needed. These new institutional experiments, as it were, are valuable and need better recognition and support.

Professor Westby: You asked particularly about supporting organisations or individuals. There is a role for both. There are issues of retention of individuals, particularly if they come out of the system to go back into it. It is important to think through the issues in terms of how donors such as DFID create an environment that will encourage scientists to go back. From our own experience, there are three important things. Any capacity-building initiative should be locally owned, in the sense it is owned within the system so that there is an encouragement to return; secondly, it should be long-term; and thirdly, it should be carefully nuanced to address the real needs of whichever sector it is. In our case it is farmers or other people involved in an agricultural value chain. If you can test those three things, you can look at the balance between individual and institutional. I don’t think it is one or the other; it is a combination of them, but within a framework that encourages people to remain in the system and contribute towards development goals.

Dr Kirkland: The answer is that we need a balanced approach between individuals and institutions, but, following on from what colleagues have said, I agree that too little attention has been paid at the institutional level. The Committee may be aware, in the context of research, of a phenomenon that one of my colleagues at the University of Stellenbosch in South Africa has called the de-institutionalisation of research. The fact is that quite often research commissioned by donors is now undertaken by individuals, and the question is how far that expertise and capacity building penetrate down into the institution.

Another point I would make on institutional links, which follows on from something said by Professor Costello, is that institutional capacity building is not for the faint-hearted. For individuals, one can get them a degree or qualification, and that can count as a success; for institutions, it is much more difficult and long-term, and it requires a lot of understanding that we don’t have.

One example of that is in the field of access to journals, which is commonly cited as critical for scientific capacity building. Some research undertaken by my colleague Jonathan Harle at the ACU shows that most leading African universities now have electronic access to as many journals as the leading northern universities, yet the take-up is much lower and that cannot be explained simply by bandwidth and internet access. These are deeply cultural issues relating to incentives and relationships. I quote that only because institutional support needs a great deal of understanding and needs to be more long-term than individual.

Q28 Graham Stringer: Would you care to expand on that? It is easy to say these are cultural things. It is very odd that a research institute is not looking at the latest research. Can you give a more detailed analysis about that?

Dr Kirkland: I could certainly let the Committee have the entire report, but two particular points come to mind. First, we tend to assume in the UK that good academics want to do research and it is a priority for them. Frankly, in some developing countries the incentive structure runs quite against that: first, because of the balance of time and an overwhelming amount of teaching; secondly, there is no clear structure for buying out time; and thirdly, as described earlier, the de-institutionalisation of research, whereby personal consultancy drives an awful lot of people’s salaries and international exposure. Added to that are probably issues about the relationship between academics and librarians in this case; the administrators and managers may not be well enough respected, and so on. It is quite a complex mishmash of things. I don’t want to digress too much into that, but I think the understanding of how institutions work in some parts of the world is something we should not take too lightly.

Q29 Graham Stringer: That leads neatly into my next question. Who is responsible for achieving a balance between policy-driven, practical research and more rigorous pure basic research? Who and how is that determined in developing countries?

Professor Costello: First, I challenge the assumption that basic is more rigorous than policy.

Graham Stringer: Okay. We won’t go down that road.

Professor Costello: You can do very rigorous population-based research, which is often more rigorous than laboratory stuff-but it is a very good question. Remember, there is not a single African or south Asian university in the top 200, if you take out Cape Town. Most universities just teach; there is very little research culture there because they don’t have the time and resources. A lot of the time, people in senior positions feel that they lack the skills or are disempowered about it. As to policy-relevant stuff, obviously the donors who fund most of the research or the major funding institutions set those agendas. There are always lively debates between those who want to do more blue-sky or more directed research. In the last five years there has been a definite trend towards the major funders, including people like the Gates Foundation, DFID and others, to be very directive, to a degree that is probably not healthy. They are asking specific questions and almost subcontracting universities to do precise bits of work. That, I think, is not necessarily a good thing. You may need a diverse range where you want to look at, say, gender empowerment or outreach agriculture and come up with some ideas. That would be a better way of doing it. The Gates Foundation almost always ask you to do something specific, whereas more mature funders like the Wellcome Trust or the National Institutes of Health in the States don’t do that; they still want it to be largely research-led.

Professor Leach: To build on that, the dichotomy between basic and applied can be challenged, and a lot of the best problem-driven research is a hybrid; it is a mixture of both. It is crucial that the broader power relations of the setting and funding of research leave developing country institutions a little freer to define those problems in ways that make sense to them on the ground in their particular settings. As Professor Costello has emphasised, some of the funding modes we have seen through the Gates Foundation and others have been rather too directive, or at least they have framed the issues in ways that respond to international concerns-grand challenges in health as seen from a global point of view-and that is quite off-putting to researchers who don’t feel empowered to look at where they are in their countries, to respond to the issues that are emerging on the ground. Some of those more open funding mechanisms could really help.

Professor Westby: As I said in answer to the previous question, it is all about balance. We need strong policymakers, who are well informed and have a good background in the scientific issues. That then feeds through to funding, basic research and so on. When we talk about capacity building, it is most important to think about it across the spectrum. I hate to say it, but MPs need capacity building. Tony Worthington, who sat next to me, did a lot of work for AWEPA, but in developing countries MPs need to understand the issues in their constituencies so that they are better able to frame the right policies and laws. I think it is wrong to say we need more of this and that; if we are truthful, we need a lot more of everything.

Dr Kirkland: To make one point about policy uptake, I am not in this company trying to get involved in basic versus applied research, but it ought to be recognised that DFID has for some years perhaps been seen as a leader in this area. DFID was one of the pioneers of the idea that you build into a research grant a small element of the budget for transfer or dissemination activities. You can use all kinds of different words. It is now time to build on that. It was a great step forward, but it is increasingly recognised now that, however much you recognise it in the funding, many researchers and academics have neither the time nor inclination, or in many cases the ability, to be great communicators of their research. That is true in the UK as well as in many developing countries. At the moment the ACU is involved in a project whereby we are trying to move on from that to try to get institutional support on a more regular basis for those projects as they complete and come to fruition. It comes back to your earlier point about the individual or institution. There is a much greater prize if we can get institutions offering support for these kinds of activities, but it is not for the faint-hearted; it is a much longer-term strategy.

Q30 Graham Stringer: Professor Leach, are there any specific examples of projects where groups outside mainstream science are participating in capacity building?

Professor Leach: The climate innovation centres that DFID is piloting at the moment are an emerging example of this. We are talking here about an approach to capacity building that is about mutual learning more than the delivery of teaching or a PhD programme. It is about the kinds of capacities that emerge from perhaps quite diverse groups-citizens, farmer innovators, small businesses-who have been trying to do things at the innovation cutting edge and responding to day-to-day challenges in their lives, coming together and learning from one another about ways they might scale up what they are doing to other settings, and get funding to make an innovation fly and become something that can really operate. It is a rather different model of capacity building altogether; it is about an approach to mutual learning and mutual capacity to do things with knowledge, which moves somewhat beyond the conventional, slightly more top-down approach to teaching and training we have seen in the university sector. A group report called "The Capacity Collective" which I referred to in my submission brought together a number of international agencies and thinkers in this area a couple of years ago, and that gives a number of examples of this broader approach to capacity.

The other element that might be worth highlighting at this point is that, increasingly, the challenges faced by people in developing countries require approaches that go beyond the boundaries of specific disciplines. Many universities are still organised on the basis of departments dealing with agronomy, medical science and so on, or indeed have a division between the more technical disciplines and those coming from the social side. Yet, increasingly, many of the most important development challenges require knowledge from those different angles to be brought together, not in a way that creates a lowest common denominator squadge. If you are tackling a health problem, say, you need some really sharp social science to understand the aspects of behavioural change with the sharpest minds from medical science. Often universities are not quite set up to do that. We have seen important initiatives in some of the UK’s funding, including DFID’s, of cross-disciplinary research programmes. There have been some very good recent examples in the environment and development field, including the Ecosystems Services for Poverty Alleviation consortium approaches, which try to think of science as including the social aspects. In turn, that opens up room for people who are sometimes mobilising around the more social and political aspects of these issues to come on board in these networks and learn from one another.

Q31 Pamela Nash: I am interested in the evidence you gave earlier. A lot of that feeds into questions that I am going to ask. I would like to think about how we can make the capacity-building initiatives that are taking place more long-term and ensure their success. Are there any key parts of that that you think we should incorporate that are not already included, or are there any examples of best practice you could share with us today?

Professor Westby: I go back to the three things I mentioned before. You mentioned the long term. The first point is the locally-owned nature of it. You may have some questions about SCARDA, a project for capacity building in agricultural research in Africa that DFID is funding. The long-term nature of that starts with trying to fit capacity building within the agreed mechanisms that exist in Africa. Rather than parachuting something in, you start from where the Africans are starting themselves. They have a coordination mechanism, which is the Forum for Agricultural Research in Africa-FARA-against a generally agreed plan called CADAP. In our case that particular capacity-building programme was based very much on the needs identified in those countries and then working with sub-regional organisations and national organisations within that. The important thing about the long-term nature of it is to work within existing systems and try to build them up and make them more effective. Maybe I will leave that as a first point.

Professor Costello: I have two major concerns about the way a lot of stuff is done now through contracting agencies. A lot of DFID money goes through NGO contracting agencies at this end, which often work through NGOs at the other end. I run partnerships in India, Bangladesh, Nepal and Malawi, and I work through a mixture of institutions and NGOs locally, and both have advantages and disadvantages. If you invest through a university like UCL, Sussex or whatever, we are in it for the long term; we are not there just for the contract. All my relationships have continued. The longest is 22 years; the shortest is 10 years. Even when we go through a fallow period, we will still hunt for funding from other sources because we are interested in the long-term relationship and investment, whereas the NGO is not; it is interested only in the contract and its renewal, by and large.

But the second huge issue, which comes back to definitions, is what we mean by research and what we mean by the rules of evidence. One of our biggest concerns-both UCL and Imperial took DFID to task over this with the recent awards of research programme consortia-was the way it interpreted what research was. It awarded some of its consortia to international NGOs, which basically publish on websites and working papers. The World Bank does this; it does not subject it to the rules of evidence, which are peer-reviewed publications. We were extremely annoyed. I have run six major population-based trials. My interest is women’s groups and the impact on maternal and newborn mortality. We spend a decade investing in really high-quality stuff, and people who come along and write a case study for the Minister’s speech get the major research programmes. That is wrong and very dangerous, for reasons that can be seen if you look at climate change.

Every five years the Intergovernmental Panel on Climate Change reviews the entire world literature. Usually, more than 5,000 scientific papers are reviewed and put into their report. You will recall 18 months ago the stuff about Himalayan glaciers melting, which came from one NGO report that was not peer-reviewed. The whole edifice almost came down. Rajendra Pachauri almost had to resign as chairman of IPCC. Climate sceptics had a field day, saying this was not proper scientific evidence. Yet, if you look at an awful lot of the publications coming out of DFID-I looked at a whole review article last year on cash transfers, which is a big issue right now-its rules of evidence would fail an MSc at UCL. It did not have any methodology about searching. Of the 170 publications it reviewed, 160 were websites, and it came to the wrong conclusion because it did not look at some of the published literature. To be fair, the chief scientific adviser in DFID, for whom I have a lot of time, and the head of research are trying to bring about a culture change here, but it takes a long time because most people don’t abide by the rules of evidence by which universities have to abide.

Q32 Pamela Nash: That could be the subject of a whole other inquiry.

Professor Leach: It picks up a very direct capacity-building issue. Going back to African universities, researchers often don’t have the confidence or feel they have the networks or nous to publish in peer review journals that will give the evidence the credibility it needs for both scientific and policy issues. It is one of the key roles in long-term UK research relationships. Anthony has established many; I have relationships with people in west Africa going back 25 years. It is partly about working, often in quite personalised ways as well as through departmental links, to enable researchers to get excellent ideas out into fora where they can be subjected to that kind of peer review and become credible on a world stage. That is also a long process.

Q33 Pamela Nash: One thing I want to go into is the promotion of better career structures for researchers in developing countries. Do you think Government have the responsibility in promoting that, or is it just universities and NGOs that have these partnerships? How can we help to make sure there is a career structure and there is also confidence and capacity building to allow those researchers to have their work published?

Dr Kirkland: It is a very important issue and one that historically has not been thought through very well. Historically, the support given for developing the career of researchers and academics at the start is by way of scholarships-the UK and DFID have a good record on scholarships-followed perhaps by mid-career training or what we call typically fellowships. One of the problems is that the fellowship world tends to concentrate on people in their mid40s, if I may say so. When somebody returns home with a PhD, they are assumed to have had their share of training for the last few years and therefore should now get on with building a career. In 10 years’ time they may need some updating and so on. One tends to find from the point of view of the career of an academic that in their mid-40s, if they are doing well, suddenly they are inundated by fellowship providers who will give overseas fellowships. When they get to their late 40s or 50s they are too old to have support from outside, so it is a very disjointed attitude.

Q34 Pamela Nash: Is the structure you have outlined a UK one only?

Dr Kirkland: I think it is international.

Q35 Pamela Nash: In the developed world?

Dr Kirkland: No. I think it is the way in which the developed world tends to see the developing world, if I may say so. If I may declare an association with the Commonwealth Scholarship programme, which DFID supports in the UK, we have high numbers of students returning home. The brain drain in that sense has not been a problem, but some of the problems they face when they return home are quite unexpected. One can say there is a lack of equipment and facilities, but in some cases one problem they face is receiving promotion too early. To be made head of a department in an African university is a double-edged sword, because it means that critical years of research can be spent working out timetable rotas and things like that. Our students are very well-regarded-in fact, in some ways too well-regarded because they are promoted too quickly.

In answer to your question, there is a lot of scope for free thinking about how we support those people within the critical five years when they go back. Traditional postdoctorals of one year here and one year there are not the answer. One way is to keep a contact, say, with their home university in the UK in an innovative way while they are still working in Africa or wherever. I am sorry that is a rather long answer, but it is a very important issue.

Professor Westby: One interesting scheme with which I had a very long association was run by the International Foundation for Science in Sweden. That provided relatively small amounts of money-$10,000 to $12,000-to encourage scientists, once they got their qualification, be it an MSc or more usually a PhD, to go back, have a bit of money to enable them to get going and get a bit of a reputation, and take forward the publishing that they did before.

It is also important to think about the type of research that researchers are doing if they come to the UK. Almost all of the scientists in my institute are involved in split-type programmes where they work on real problems related to their own countries. They get UK experience and the benefit of better facilities and things like that to advance their research, but they are able to spend time back in their own countries tackling real problems. That helps in the transition to become better researchers. Someone who has come here and worked on strawberries and goes back to a country that does not grow strawberries has the right research skills but they won’t become involved in the system. The fact they are involved in the system is just so important to doing it.

The other point is the extent to which those early career scientists can become involved in larger programmes and projects. From my own experience of the University of Agriculture in Nigeria, somebody came on a Commonwealth-funded split fellowship. He spent nine months with me. We built on that relationship over many years. He is now head of department. We have awards from the Gates Foundation to do ongoing work and things like that. His old university, largely through him, has come up the rankings. They were so pleased that they got into the top 30-something of African universities, which ain’t bad for an agricultural university in Nigeria. They are the third best in Nigeria. That is just one example. The danger of all these things is choosing the one example, but in many ways it can indicate the common things about the importance of this early career stuff. What John said about not getting too involved in administration and things like that too early is an issue, but it has to be for the universities themselves to work out.

Professor Costello: It is a kind of public-private partnership but keeping them within the system. The dilemma most people face is having a run-down, demoralised big teaching department on a low salary but some security versus going off to the attraction of very well paid freelance consultancy work. I was just in Malawi. There is a fantastic guy who should be head of education in Malawi, but he is a freelancer doing consultancies for every donor that comes along. That is a negative thing. The Wellcome Trust is trying to take its excellent research centres and embed them more within the national system so that there is private money coming into it, some of which is run autonomously and accounted for so that you do not have all the bureaucracy and red tape of a university, but you keep it within the system.

Q36 Stephen Mosley: Professor Westby, you just talked about a gentleman who was doing a split fellowship.

Professor Westby: I am sorry; he did a split fellowship 15 years ago.

Q37 Stephen Mosley: In the 2004 report done by our predecessor Committee we highlighted the advantages of studying for PhDs by doing a split site or distance learning. We have figures showing that only 4% of Commonwealth Scholarship Commission students did some sort of split scheme over the past four or five years. Should we do more to encourage distance learning or split schemes? If so, what should we be doing?

Professor Westby: I think there are two separate issues. One is the split site and the other is distance learning. I don’t mind addressing both. I am sure John would have something to add to both of those. As to split sites, from my perspective, the ability to work on real problems, learn new stuff here and build up linkages with the UK community, or whichever community they are in, and still maintain the linkages in their own community is very important. It is all about the relevance of research and things like that, which is really important.

As to distance learning, you have to look at it not purely at the PhD level. There is a gap in the market at present in terms of what distance learning could offer, particularly in the agricultural sector. For example, the Open university has very good programmes at present in primary school education and in the health sector, but currently it does not do anything in the agricultural sector, not PhD-level training but extension worker training and things like that. There is a gap in the market in terms of what distance learning could offer. Technology is taking huge steps in Africa. The missing bit that we went through has gone. We don’t bother with land lines any more; we go straight to mobiles. More and more information will be available to people through information centres and things like that. We need to be forward- thinking in terms of science and how those new technologies can be used.

I appreciate what John said about libraries and things like that, but in general, researchers in Africa are becoming much more connected with themselves in regions and also with outside partners such as us. Partnerships are also important. If distance learning is to work, it will be a partnership arrangement working with local institutions that are able to support it, not just throwing information out there and hoping someone uses it, but providing a local point of contact to use that information and move forward.

Dr Kirkland: Perhaps I may answer that question with particular reference to Commonwealth scholarships, which you raise. You are quite right that the 2004 report highlighted very favourably the innovation that was taking place. Since that time I think three things have happened. On distance learning, the numbers have risen considerably, typically at masters level rather than at PhD level, very much, as Professor Westby said, in collaboration with local partners. We don’t offer distance learning degrees on a one-off basis; we do it in groups through individual universities. Although it takes time to evaluate that-we have been doing it for eight years now-the pass rates are very encouraging.

Formal split site degrees have expanded less quickly for a number of reasons. First, because our bases tended to be just with the UK university, we have been very concerned about what is happening at the other end. Typically, our model of split sites, for legal reasons connected with the Commonwealth Scholarship programme, has been allowed to support only the UK end of the degree. That could be broadened. We are a little wary about having an island of a year in the UK, or whatever, and people not being able to complete quickly when they go back. But, again, the success rates of those degrees are now encouraging.

The big growth area, which is not too far from split site PhDs, is the proportion of students who are now undertaking very substantial amounts of field work in Africa or other developing countries as part of their degrees. Our regulations are changing on that almost on an annual basis. It is now not unusual for a student on a three-year PhD programme in the UK to spend six or nine months in a block, still under UK supervision, in-country doing work that is relevant in the home country. While the formal PhD structures have expanded relatively slowly, the distance learning model and the proportion of time spent back in developing countries is expanding much more quickly. I would say it is almost the norm.

Professor Costello: If that is true, I am staggered by your statistic. I would not take anyone unless they do a split site and have a very substantial component where they do field research overseas with us in partnership and our supervisors go out there. I am absolutely horrified to learn that only 4% are doing that. If that is the case, I think there should be a massive move towards that. You should have our researchers going out. That is what happens in our institution, so I am rather staggered to learn that so few do it.

Professor Leach: What we have here is a distinction between the formal definition of "split site", which needs to be looked at, and the real practice, as Dr Kirkland and Professor Costello have said, which is exactly the same in our institution and is standard in development studies. A student perhaps comes to the UK, spends nine months at Sussex doing literature work and then goes out to do, often, a year of field work for which they would be affiliated to a local institution-usually a university. Then they come back and write that up, but in the process longer-term relationships are established with that university department. Perhaps more could be done to formalise the capacity-building aspects and institutional links that are created so that it is not just the individual but supervisors more often go out. That might be a question of funding and some additional funds for supervisory visits in capacity building. I am sure that Commonwealth scholarships in that sense have been doing much more of this split site stuff informally than is actually registered. It has certainly been the case for the Commonwealth scholarship students I have supervised.

But I am also wondering whether DFID might be encouraged to talk further with the UK research councils, which are the major funding bodies for studentships for UK and European doctoral students. I have a research centre that has linked studentships, but we are constantly frustrated because they have to be British or from the EU. DFID, in partnership with the SRC, NERC and so on, is involved in a whole range of other areas, including the excellent DFID/ESRC poverty research scheme and a whole range of other cross-council interdisciplinary programmes. I wonder whether it might be time for DFID and the research councils to talk a little further about potential joint funding of PhDs under the same kind of model that would enable international students to access the kind of arrangement of which Commonwealth scholarship students have had the privilege for a significant time.

Dr Kirkland: Perhaps I may clarify the 4% statistic that you quoted. I make two points. First, I suspect that that figure is 4% of the total number of scholars we have at any one time, and we would count a split site only as one year rather than a three-year programme. Therefore, in terms of new awards I would be right in saying that we would offer about 30 split sites out of about 400 or 500, so the figure would be higher in terms of new awards each year.

Secondly, in terms of definition, as Professor Leach said, "split site" in that very precise sense would not be a degree awarded by a UK institution. Those split site awards would be students who were undertaking their PhDs at developing country universities and came to the UK for one year as part of that. It certainly would not include the number of students who come to the UK under the supervision of Sussex, UCL or whoever, and returned to their home country to undertake substantial field work of the kind Professor Costello described. I would be very happy to give any further figures independently that might be required on that.

Q38 Stephen Mosley: The figures we have are that between 2007 and 2010 there were 97 awards for split sites.

Dr Kirkland: That would be about right; there are 20 or 30 a year.

Professor Westby: When you talk about split sites, distance learning and things like that, which make the most of the UK science base in that way, what is important is building the capacity of the universities of African or developing countries such that the quality of their supervision and awards also improves. UK universities can play a major role in those partnership arrangements to try to do that. In the longer term it should not be about people coming here but people being able to get good education where they are. The Natural Resources Institute is committed to these longer-term partnership arrangements, which are so valued by the organisations with which we work. Very often the problem is funding them on a sensible basis and not cross-subsidising them from other things.

Q39 Stephen Mosley: I want to follow up Professor Leach’s reference to research councils. I will start not with Professor Leach but with Professor Costello, for obvious reasons. The UCL Institute for Global Health has stated that DFID’s research programme consortia model has not worked successfully. Professor Costello, would you agree with that statement? What are the problems with the model, and how could it be improved?

Professor Costello: I did not write that. It has its limitations. I have run three research programmes over 15 years, so I would not say it is a complete failure at all. There were aspects of the last call that worried us, such as the attention given to peer-reviewed research, the importance of rules of evidence and that kind of thing. Within DFID there appear to be two different views. There is an evidence element, which is very keen to bring an evidence-informed culture to the organisation, but also there are people who historically have seen short-term consultancies and working papers as what they call research. There are definitional issues there.

Another issue we have been concerned about is that it is too directive of what it should do. Also, there should be in-house capacity at DFID to look at the value of evidence, to be interested in it and to cope with it. There was a period in the past-it is now changing, certainly in the last 18 months-when the technical advice to DFID was very limited and it was very overstretched. By the way, there are some fantastic people in DFID. These things are changing. Michael Anderson, Chris Whitty and others are trying to bring about an evidence-informed culture to DFID in trying to change the system, but it is a slowly-moving juggernaut.

Dr Kirkland: I don’t have anything to say about whether the RPCs are working as such. One point I throw in is that there might be more interaction between the research commissioning side of DFID and capacity building through scholarships and other activities. As someone involved on the scholarship side, we would very much welcome that interaction. We have started in a very small way, as Professor Costello knows. RPCs are invited to nominate students for PhD studies through the Commonwealth Scholarship scheme. That kind of interaction could be expanded, because it would enable us on the scholarship side to ensure we got high quality relevant scholars and also, on programmes like split sites, to ensure that they have an environment in which to work when they are doing the developing country element of their work that is conducive to good research. That area could be considered further, but I would not want to make any comment on the effectiveness of the research programmes.

Professor Leach: I speak as a fan of the RPC model. My institute has been involved in five or six over the last decade or so. It is a really important way to build precisely that kind of long-term partnership and mutual capacity. If RPCs work well, the UK researchers learn just as much from working with Latin America and African partners as they do; so it is in itself a capacity-building process through the RPCs.

I emphasise that they need to be about research and quality of evidence, and that kind of emphasis is critical. To my mind, the recent round of RPCs has moved rather too far in terms of the requirement that 20% of funding goes to communications and towards the kind of directive, logframe, impact orientation that stifles the creativity and intellectual curiosity of researchers. Therefore, a little more freedom to enable researchers to pursue ideas that really matter and then to publish them in proper peer review places is critical.

There are also questions of power. There are good and bad RPCs, and the ones that have worked less well have been where a UK institution has ended up running the show too much and the power imbalance with respect to the capacity of the southern partners to help set agendas is incorrect. I know that DFID has been desperate to have some that are genuinely southern-led. That has proved quite elusive, partly because the administrative and managerial requirements of running an RPC are much greater than most African or even Asian universities can manage.

That creates just one more capacity-building angle to this, which is that capacity building in research management is quite an important area and will become more so if we want to see developing country institutions and universities taking the lead in some of these international consortium arrangements, as they absolutely should. It is a neglected area where the UK has built up a lot of capacity and new jobs and roles-RPC programme coordinators and so on-yet there is very little funding, incentive or arrangement through which they can transfer those increasingly important skills that African universities need.

Professor Westby: From my perspective working mainly in agriculture, there have not really been RPCs in agriculture; it is more in other areas. We have had some involvement and positive experiences in being involved in them. In theory, it sounds a good idea.

Q40 Stephen Mosley: Perhaps I may press you specifically on something else. I know that you were involved in SCARDA-Strengthening Capacity for Agricultural Research and Development in Africa-which ended in December last year. What were the main outcomes of that project, and how do you think DFID should support agricultural research in future?

Professor Westby: Those are two very different questions. SCARDA itself is very much a capacity-building initiative. It was supposed to be a pilot programme that could then be scaled up and scaled out beyond that. From that perspective it fits my criteria as being a long-term programme. It is just at the point of finishing. We are doing a systematic review for DFID on capacity building, which hopefully will inform its next investment along this line.

As a process and project it has been successful. There have been benefits at three levels: individual, organisational and institutional. Initially, the project worked on the basis of a report produced by FARA-Forum for Agricultural Research in Africa-saying there was a great capacity gap that needed to be filled. DFID was responding to a need. When the programme started, it worked with FARA and a number of sub-regional organisations, which then selected some focal countries. Within those focal countries a number of institutions were chosen on which the capacity building was focused.

There was then a needs assessment for those particular organisations. What did they need to improve the quality and amount of research they were then doing? On the basis of that, there were a number of different training workshops or experiential learning activities. MScs were supported; there were a number of short courses; innovation platforms were developed to scale out research on a small-scale basis to build organisations’ experience. It was very much demand-led and the training was linked in to what those organisations needed. For example, in The Gambia it was decided that they needed MScs in this, that and the other, but in Lesotho they said they did not need any more MScs but more experience of working with other partners in the so-called innovation-type platforms. There were different things in different countries.

You asked me about the benefits. There were a large number of researchers and actors in the agricultural research for development area who were trained. About 80 MScs were trained and the vast majority-over 70-were trained within Africa. There was a building of African capacity to deliver MScs, which is really important. Many of those award holders from overseas have gone back to their organisations. As a result of the training they have had and training in proposal writing, they have gained additional money from within their own systems to do research.

At organisational level, the external evaluator of the programme said that, from what he could see, there was greater efficiency in the focal institutions in how they were carrying out their research and their research management processes; there was an increase in the number of projects that they won externally and also an increase in the publications generated by those organisations. An example of that is the Crops Research Institute in Ghana.

At the institutional level, in southern Africa, in the SADC region-Botswana, Lesotho and Zambia-they have adopted new ways of working. Rather than doing the traditional research in an ivory tower, they are developing partnerships with known research actors, and that has been a facilitated, supported process. In Lesotho, a new agricultural research development forum has been developed, which decided to focus on smallholder poultry as a case study that they will work through together. Benefits have been very much at different levels. It is embedded within what those individual institutions need but supported at the regional level to get the regional learning as well, all within the context of what Africa is trying to achieve in terms of agricultural development within the country.

Chair: Before we move on, I go back to Mr Mosley’s questions about split sites. So that there is no confusion in definitions, it would be helpful if each of you could submit any additional evidence about the proper interpretation of the data as you see it so that there is clarity in our thinking on that.

Q41 Graham Stringer: To go back to DFID, how good are they at coordinating with partners in capacity building, particularly at the large end? Are they any good at reaching out to smaller entrepreneurs and businesses that might not be part of any formal structure?

Professor Costello: They are very stretched. One issue that has come up earlier is that there is a research group to DFID that is largely centrally based, and then there is the in-country group, and often there have been problems in trying to get the two to connect. That is an organisational problem, and it has been very difficult to crack. They are very well aware of it, and it has been around for 10 years or more. Sometimes it works very well. I was in Malawi last week. The country representative there is very sensitive to this, and we have a very good working relationship. By and large, I have had quite good experiences, but in other cases it has not worked very well.

As to your specific question about DFID reaching out to entrepreneurs, oddballs and people out of the loop coming up with good ideas, I doubt that they are very good at it, but I am not sure it is their role. I would have thought that is where the contractor, the university, college, NGO or international institute should be incentivised to do that kind of work. It is interesting, but I don’t think it happens much. I might be wrong.

Professor Leach: It is patchy. There is recognition within some bits of DFID that this is an important role. There are some areas where it is trying to do that and supporting it. The Future Agricultures Consortium, which DFID supports, is actively doing this through its consortium partners, but it is through an indirect route. There are some DFID country offices that actively do this kind of reaching out in particular areas, but often it depends on the personality of the individual and their particular interests.

The other mechanism within DFID that has done this to some extent is the senior fellow programme that it started three or four years ago. I believe that was a direct response to being very overstretched and the gap that often existed between the central research department and policy division. This initiative was taken basically to recruit, sometimes on a full or half-time basis, seconded UK academics to base themselves within DFID in particular areas of work over a period. Some of those individuals have done a great deal of work because of their particular interests in reaching out to small businesses and entrepreneurs in different areas. I know that David Grimshaw, who came in from Practical Action, was sitting in the research division on science and technology. He has done some fantastic pioneering work on small businesses and innovation in various countries. The answer is that there are little pockets of best practice in this area, but it depends on personalities and particular whims; it is not something that is institutionalised.

Professor Westby: Can I add another example to the cause? In the kind of research in which we as an institute have been involved that has been DFID-funded, our capacity building has run alongside our research activities, and it is sometimes very hard to separate them and say something is this or that. An example of that is the work done under the Research Into Use programme, which is trying to have impact. If you are trying to have impact on the ground, you need to work with multi-stakeholders, be they the civil society, the private sector and so on. It is those types of things that have worked very well. For example, not funded by DFID, currently I run a very large Bill and Melinda Gates Foundation project, which is doing exactly those things within a cassava value chain, building the capacity of the private sector and others to engage within a research-type process.

Looking at capacity building separately is difficult; you need to look at research and capacity building together to answer that problem more fully. It is something with which the EU is also struggling. On Monday I chaired a multi-stakeholder support project that the EU is running. It is a challenging process, which takes time, to bring research and nonresearch actors to work together and then win money on top of it. So it is a challenging process that does take some time.

Q42 Graham Stringer: Dr Kirkland, how can the Research Capacity Strengthening Group of the UK Collaborative on Development Sciences be used for greater collaboration in programme design and implementation?

Dr Kirkland: That group has been a great success in a fairly modest way. It has been an attempt to bring together various academic and organisational groups like ours. I am not sure it has the formal inroads into DFID that are needed. From my fairly limited experience of the group, it has led to some very good discussions and informal contact-making, which certainly affects, speaking for the ACU, some of the programmes we do. The libraries and journals programme I mentioned earlier, which has brought us far closer to bodies like INASP and others, is an example of that.

To answer your question about how it could be used more effectively, it might be helpful if it had more senior level representation among some of the organisations that come to it. I hesitate to say it, but maybe there could be a better formal channel to DFID, although, to be fair, DFID does send representatives to the groups. There is always a risk in losing informality, particularly when one has groups that are essentially bidding for funds together. A fine balance needs to be struck between the need to preserve informal useful interaction and the need to be able to feed in, as you say, to the design of policy.

Q43 Pamela Nash: Dr Westby, in the Natural Resources Institute’s evidence to the Committee it was suggested that DFID’s in-house expertise in agriculture science and technology was less than that of its predecessor in the ODA. Do you agree with that assessment? Do you think there has been any restoration of that quality of expertise since the chief scientific adviser’s post was created in 2004?

Professor Westby: Our feeling a few years ago was that that was the problem, in that you would go to an office in Africa and would be searching for someone who knew something about agriculture, and it would be called something else-the private sector or whatever. But there was certainly concern about whether DFID had that expertise. In the last year, which is probably a result of the chief scientist’s actions, I have seen much more emphasis on recruitment into those types of positions. We now see titles like food security adviser, and the whole link with climate change is coming through as being important. From my perspective, we went on a bit of a lull, but agriculture in general, in the whole donor community, went on a real lull. It was only the World Development Report of 2008 and the food price crisis, which everybody else remembers and has carried through to now, that has brought agriculture back up the agenda and made it far more important. To us, it has been a bit of a turnaround from how it was. I think this Committee in its last report played quite a big role in supporting that.

Q44 Chair: Is there a strong message there to the Foreign Office about making sure that the network of science counsellors is up to the job in all countries? It is usually very strong in countries where we have bilateral relations and a high level of science endeavour, but in developing countries historically they have not been the strongest.

Professor Westby: Is it the difference between the Foreign Office and DFID?

Q45 Chair: Yes. They are based at our diplomatic missions, aren’t they?

Professor Westby: They are, but more and more I am seeing separation between the two. You go to a DFID office or the high commission, and they are not necessarily the same organisations. I don’t know whether others have had the same experience.

Q46 Chair: But you would want to see emphasis on strengthening it.

Professor Westby: That would be important and sends the right message to the countries themselves. From where do countries take their lead? If someone is there saying, "Yes, science and agriculture are really important. We have someone based here who will support your national system in doing that," to me that sends a clear message.

Professor Costello: You don’t want to turn DFID into a university; you want its people at the highest level to understand about evidence. They can come from a different technical background but still understand the application of evidence to other areas. One thing DFID has done well in the past, and could do more of, is organising conferences where multi-disciplinary groups from policy and the research sector are brought together to talk. I have been to some great meetings it has run on that subject. You can also use its video conference facilities to link up with DFID offices overseas. At relatively low cost you can get some incredibly creative exchanges of ideas. I think it should be doing more of that.

With the appointment of the chief scientific adviser, currently there are lots of moves in DFID to try to sensitise all the different people from an administrative or economic background to evidence approaches. I think that will pay dividends over time.

Professor Westby: It is very important that DFID uses the UK science base. I do not think it uses it, certainly in agriculture, as much as it could. If it is not careful, it is not going to be there if it is not used and properly supported. There are still some issues on that side of things.

Professor Leach: I agree with that. DFID won’t be able to recruit enough people inhouse in London to be able to cover all the areas of expertise it needs to cover, so maintaining these linkages with the UK academic community is crucial. The senior fellow programme has helped with that. Some of the cross-council research programmes-the DFID/ESRC poverty scheme, some of the ESPA schemes, and the new one about to be launched on zoonotic disease and so on-are important in creating those connections.

It is not just about technical science expertise. I would like to see much stronger links between the social development advice area-the people who work on governance, human development, women’s empowerment and so on within DFID, whether it is in country offices or centrally in London-and those taking on the more technical areas, be it agriculture or health, because it is at the interfaces of these things where science, technology and development can work with one another. We have sometimes seen a disconnect between the social and technical within DFID that needs to be overcome.

Dr Kirkland: Can I make two general points? I am not qualified to answer in the particular context of agriculture or other disciplines. First, as Professor Leach said, it is very important that DFID uses the expertise within UK universities. I strongly endorse that. I am not sure that the structures exist to tap expertise in developing country universities outside particular projects. That might be something that could be looked at.

Secondly, a general issue about knowledge capture within DFID has come out of a number of comments by colleagues. Although, as Professor Costello said earlier, groups such as UCL, Sussex and so on have developed a lot of knowledge over a long period themselves, I am not sure that is happening within DFID itself. You may want to come back to that if you were going to ask about evaluation and so on in the long term. There is a danger that, perhaps because of artificial distinctions between what is project funding and what is administrative funding, DFID might disengage with a number of the projects it is funding, as a result of which long-term knowledge might not be captured within the organisation itself.

Q47 Pamela Nash: That is helpful. You have all touched on this, and I am sorry to pick apart the evidence that was already provided, but it is of some concern to me. Also, in NRI’s evidence, it is stated that there is "an absence of a clear strategy or view within DFID on capacity building for scientific and technology in developing countries". Obviously, this is quite a serious accusation. Do you agree that is a problem with DFID?

Dr Kirkland: I suspect DFID would cite a lot of examples. I don’t know whether it is joined together in a coherent approach. That is something I would say about universities. It is very welcome that DFID has now come back. There was a period, as you will know, a decade ago when universities and higher education were seen as virtually outside the development debate and not as a priority. Happily, that has now changed, and in many individual cases DFID has shown its recognition of that. Whether DFID has a policy for universities-I am not talking about a funding stream-is a different matter. Others will be better placed than I to know whether a similar comment could be made about science development more generally.

Professor Costello: I do not think we should over-criticise DFID. Equally, we take short cuts. I have worked through NGOs rather than universities because it was going to be much more feasible and pragmatic to get a particular project done. There is a growing awareness among all of us that these short-term benefits have a long-term opportunity cost and you have to restore the balance by trying to strengthen universities. Everyone in this room probably went to a university. Universities are absolutely key institutions for development. I know of no developed country that does not have a functioning university system to improve its governance, structures, regulation, education-everything-and that is what we have to try to invest in. It is difficult, and I am sure we have all cut corners in the past. I would not beat DFID over the head about this. A lot of people in DFID are well aware of that but also the problems and pitfalls.

Professor Westby: It sounds much more critical than we intended it to be, because we are very positive about a lot of what DFID does. The key point we were trying to make was about the degree of joined-up thinking. DFID has managed to mainstream capacity building across a lot of its initiatives, which is to be welcomed, but it is the joined-upness of that. You need additional capacity strengthening initiatives like SCARDA, for example, that bring together all these different things to an extent. DFID also needs to be a good member of the donor community in getting the joined-upness working between ourselves and what the EC, the World Bank and other bilateral donors are doing. There is some work to be done in the joined-upness of capacity building across the donor community of which DFID is a very important part.

Q48 Pamela Nash: Do you see this as improving, or is it a continuing problem?

Professor Westby: I made the presentation on capacity building at the recent G20 conference on agricultural research and development in Montpelier. Of the three things that the French presidency pulled out, to me the important one was capacity building. One of the main things to come out of the discussions among the G20 nations was the need to be more coordinated in what they were doing, because they were all doing slightly different things in different ways. How could you become much more effective in doing it? In addition, one point that DFID makes is to understand what works and what does not, and to have the evidence base there. The systematic review that we are due to finish during March will, hopefully, contribute quite a lot towards that.

Q49 Pamela Nash: I am sorry, because I feel I am picking apart everything that was sent in. Just looking at the memorandum submitted by the STEPS Centre, there was a call for greater integration and working together of development advisers within DFID, and perhaps a role for the CSA in doing that. Is the fact that that coordination does not exist something you would identify as a major problem at the moment?

Professor Leach: As I mentioned a few minutes ago, there has been a persistent divide between the more technical advisory areas in DFID to do with health, agriculture and so on and its very strong social development capacity, which looks at socio-economic issues, the governance aspects of poverty, questions about gender and so on. That is partly because, often, staff capacity is short; people have their own interests. The social development areas are ones where DFID very much leads the world and is recognised as doing so by other agencies. The technical areas have been deeply neglected and are now coming up, partly through the influence of the chief scientific adviser, but there is still a knitting together and dialogue that does not always happen. There are some examples where it is beginning to do so. For instance, in the area of climate and energy there are really good discussions that bring together the social and technical dimensions. I think more of that needs to happen, and it requires institutional mechanisms when new programmes are being designed and new policy areas-and, indeed, research programmes-are being designed to make sure the different disciplines are part and parcel of those discussions. It is coming; there is a recognition; but I think a further push as to why this is important is needed.

Q50 Pamela Nash: Dr Kirkland, in your submission you described the current arrangement where DFID’s chief scientific adviser is also head of the research and evidence division as being problematic. Will you clarify that and explain to us why you feel it is a problem?

Dr Kirkland: It certainly was not a comment on the post holder, who I think commands very wide respect, but it seemed to us that, given the huge and to a large extent welcome evidence on monitoring and evaluation within DFID, there was an issue about having the individual most concerned with commissioning research being involved in drawing the conclusions from that exercise. I do not think we meant anything more damning or critical than that. It seemed to us to be a curious conflation of roles, but it was not in any respect a comment about the current post holder.

Q51 Pamela Nash: You also mentioned that you felt there was a greater need for coordination between the DFID scholarship mechanism and the research and evidence division. Is that lacking at the moment?

Dr Kirkland: It was a point that came up earlier. The research programmes of DFID have as one of their key themes capacity building. The scholarship programmes are very successful in that respect, but there must be more that can be done together. That has started. The major long-term research centres that DFID funds now are invited to identify and nominate PhD students who can work with them. I think that is a step forward, but in the longer term, if the research programme is about capacity building, clearly more communication with the scholarship programme would be welcome.

Q52 Stephen Mosley: Following on from that, how do you monitor and evaluate whether the capacity building has worked?

Dr Kirkland: I can answer that in regard to scholarships. We have a programme that has been going for several years. I don’t think we claim to have discovered everything in the methodology. It is amazing how little international methodology there is in areas like this, but, for example, we now know where a very large proportion of our students-our alumni- are; we know what sectors they are in and, by and large, the levels of seniority they rise to; and we know what countries they are in. All of that evidence is encouraging.

We now need to address the very tricky questions of attribution, for example, to be sure your students are working on relevant projects in relevant countries, and to what extent that could have happened without Britain having a scholarship scheme. We need to drill down further into all those things, but our evidence in those areas, in particular the very large percentage of students who are working in their home countries, is a very good first step towards that. DFID has encouraged the scholarship programme in particular to develop that further over the next few years.

Professor Leach: This is a difficult area. Despite the challenges, it is rather easier when you are talking about individuals and tracking where they have got to. The big challenge is monitoring and evaluation first of institutional capacity development, where it will often be less tangible and measurable how a university or another institution has emerged over the years. There will also be multiple influences on that. It will be less easy to track that it was the UK’s inputs that enabled this university now to achieve its research ratings. It might have had other donors and so on. I think qualitative impact stories about tracking backwards from where an institution is now to the influences on it will be important.

The other area is new institutional arrangements that ought to be part of the way we monitor and evaluate. What new networks, initiatives or institutional fora happened as a result of UK capacity building that would not otherwise have happened? Some of that will be qualitative; it will be about a forum that did not exist before, but also a qualitative narrative-a story, as it were-about what things are happening for whose problems that did not happen before. This is a bigger challenge about measuring impact, which the UK research community more broadly is facing in the context of the research excellence framework and so on. Everybody is now expected to produce impact, and it is recognised that those impacts cannot always be monitored and evaluated with clear, easy numerical indicators. Impact and capacity-building stories will be part of the metric we have to deal with.

Professor Westby: I very much agree with what has been said previously. It is problematic. DFID has used and does currently use logframes as a method, which has milestoned indicators. Some of those are more difficult to quantify. If I revert to the answer I gave you on SCARDA and what SCARDA has achieved, I ended up telling you a story about the different things it had achieved at different levels. You referred to the fact that it was a combination of metrics in terms of the number of publications, PhDs, MScs and so on, but the other thing is what that has gone on to achieve. What impact has been achieved at the end of that? That is rather more difficult to get to, but we need to get there to justify the investments that need to be made in capacity building.

Professor Costello: The national institutions’ metrics are a bit like our own: grants in; papers out; first authorship; last authorship; being a principal investigator or co-investigator; whether south-south collaborations have started up; the number of PhDs within host institutions or masters programmes and the like; what their university ranking is; whether people stay in post or migrate. There are a lot of quantitative measures, but I agree with Melissa that there is also qualitative stuff, but you want to see an institution grow over time. That has been seriously neglected by everybody-all the funders-over the years and they now realise they have to address it, and it is not easy.

Dr Kirkland: In general, the emphasis on monitoring and evaluation is welcome, but there is a great danger that it will be monitoring and evaluating projects that are of two, three or five years’ duration, and we need to find much more sophisticated models to find out, for example, what is happening to DFID’s investments of 10 years ago or indeed what in 10 years’ time will be happening. At the moment the logframe and so on is designed very much on the project, which is a danger.

Professor Costello: Another thing is the impact on policy. Does a lot of this evidence influence policy? On that point, we have done work in Nepal, but evidence from Nepal will not influence the Indian Government. When we did it in India it had a powerful effect on Indian policy. You have to recognise the need for host champions and academics doing the work to influence their own policy. You can do fantastic research in an international institution, but if the nationals do not want to believe in it or have a sense of ownership it will not influence policy. There is a key element in getting that evidence into policy and practice stuff around building the host institution.

Q53 Stephen Mosley: All four of you when answering have responded in a very positive way. You have been talking about how you measure achievements. Surely, there are some projects that don’t succeed-some projects fail. We have heard evidence that sometimes it can be difficult to get a candid evaluation, because if you are going out there and doing research, and you turn round and say, "This didn’t work," it might be difficult to get a grant next time. How can we encourage researchers and funders to tell negative stories and to be honest about what didn’t work, as well as what did?

Professor Costello: Do you mean results of studies, because negative findings from good studies are really important, or do you mean studies that were done badly?

Q54 Stephen Mosley: A combination of both. There can be studies that have been done well but don’t have a positive effect.

Professor Costello: But they are really important. I have done some very important negative studies because they told funders and donors not to roll out programmes. In our portfolio of cluster randomised control trials of women’s groups we have had four, soon to be five, with very positive effects and one negative one, which taught us about the dose relationship. We realised that our coverage of groups was inadequate. We fixed that. We have run it for another two and a half years and we are now getting positive results, so you learn an awful lot. There should not be publication bias towards only positive results. On the other hand, your question might touch upon where you try to set up a study and the whole thing fails, and that is a different question. I think you are right; we should be very honest about that.

Dr Kirkland: There is some truth in what you say. If you hire a group of individuals to do a project, they will invariably want to say that the project has succeeded. There are two answers. First, evaluation has to be much wider than individual projects-that is too narrow a definition. Secondly, in the current environment I sense a danger in evaluation to be seen by DFID officials as being at the expense of engagement. Sometimes with the great constraints it has, particularly on administrative time and whatever, there is a danger that people say, "Our way of finding out whether or not this has been successful is to devolve everything to an evaluation mechanism", and so on. If we lose the notion that DFID officials, particularly on large projects, are engaged with the project as it develops, the danger you identify is a real one. DFID needs to be seen to engage as the project develops rather than simply relying on a mechanism five years afterwards to say whether or not it succeeded.

Professor Leach: I very much agree with that. We were talking earlier about the RPC model. Those are programmes of enormous challenges. There are failures and learning processes along the way, not just about evidence and results and their effects, but also about how you manage a partnership and deal with contracts and power and relationship questions concerning people from very different backgrounds who are now expected to work together. The best of the RPCs have been very self-critical and reflexive as they have gone along. They have encouraged among the participants, indeed sometimes with DFID minders involved in those programmes, some very self-critical reflection from which they can learn in a long-term process.

It is absolutely crucial, because in an area like capacity building there are all sorts of areas where there is no right and wrong; it is about trying it out and learning from it. We need to encourage a more reflective and adaptive approach and not castigate or penalise research groups or institutions for making those mistakes. What should be rewarded is recognising them, discussing them openly and learning from them. That should be a positive thing, not something that counts against you and is a bad score against you in receiving a future award. That is about mindset. I am glad you have brought it up because it is very important.

Professor Westby: I totally agree with those comments. I was in Brussels on Monday at a project meeting and a lot of the meeting was a discussion about what was not working. It was a steering group meeting, so it was being done at that level. It was the governance within the project with a bit of externality that was saying, "We are trying to build these multi-partner consortia and there is a problem because Europeans are reluctant to participate unless they can see a clear financial advantage at the end of their doing it." The project was going really slowly and that was the reason why. The question is how you disseminate that information in a positive way that is not seen to be critical of how you designed the project in the first place.

Q55 Stephen Mosley: What is the role of DFID? How effective is it at doing that? Does it produce guidelines to help you do what you need to do?

Professor Leach: It has been helpful to some extent, and could be more so, not so much in producing directive guidelines but encouraging across programmes to generate best practice and share dilemmas. This has happened across the RPCs. There were a number of fora where three or four got together with some DFID coordination to talk about common challenges they faced, whether it was about Evidence Into Use, managing partnerships and so on. Those were valuable fora, which generated best practice principles and exemplars. I think that is the role of DFID in a coordinating manner to encourage that learning to happen. It is not DFID’s role to produce a top-down, point-by-point guide, because I do not think anybody has the ability to do that; nobody has the all-powerful knowledge, but there is certainly an important co-ordinating role in generating knowledge from those engaged in trying to do this in a day-to-day way.

Chair: It has been a fascinating session. Thank you very much for your time this morning.

Prepared 13th February 2012