Select Committee on International Development Sixth Report


VI. MONITORING AND EVALUATION FOR ACCOUNTABILITY AND LEARNING

60. If a development agency is to complete the cycle of development policy and practice—making itself accountable, improving its performance, learning from its experience, and acting strategically—it must have effective systems for monitoring and evaluation.[69] Briefly put, monitoring is a necessary component of evaluating development interventions in terms of progress towards objectives. DFID, in common with other donors, faces challenges both in terms of monitoring and evaluation. We were told that within the international development system there are three tiers of monitoring and evaluation. The first tier is at the level of the international system, monitoring progress towards the MDGs, and is led by the Organisation for Economic Cooperation and Development (OECD) and the World Bank. A second tier involves the peer review of individual donor agencies undertaken by the OECD's Development Assistance Committee.[70] A third tier—which we examine in more detail below—focuses more closely on DFID's development activities.[71] The various tiers of monitoring and evaluation—of the international development system, of DFID as a whole, and of DFID's specific interventions—should be well-integrated, with monitoring of progress towards top-tier targets such as the MDGs informing bottom-tier data collection, and data collected at the bottom-tier feeding into the monitoring of top-tier targets. This is currently not the case.

Monitoring Progress Towards the MDGs

61. The Departmental Report highlights the importance of measuring and monitoring progress, including progress towards the MDGs, but acknowledges the problems which developing countries—and hence their donors—face in collecting data.[72] In addition to annex 2 which sets out achievements against DFID's PSAs, chapter 4 of the Departmental Report—"How we are doing: Moving towards the Millennium Development Goals"—contains some assessment of progress. However, this is couched mostly in terms of progress towards the MDGs at the continental or regional level, using World Bank data, with little information provided about progress in individual countries in which DFID is engaged.

62. The obstacles to effective monitoring of progress towards the MDGs are widely acknowledged, both within DFID, and by other commentators. In its recent report, the Public Accounts Committee stated that: "The Department's ability to provide a reliable view of their performance has been constrained by the poor quality of data they have had to use to measure progress against their Public Service Agreement targets".[73] In response to our questioning DFID wrote:

    "There are known problems with availability, reliability and timeliness of the data for monitoring progress on development outcomes. This is a characteristic of poor statistical systems in poor countries. Where possible DFID takes figures directly from the databases of the World Bank (for example their World Development Indicators database) or from the UN agency which has responsibility for the particular indicator. These are the most comprehensive and comparable sources of data for these indicators but in most cases they rely on weak national statistical systems to provide the country-level data. The process of data collation by international agencies sometimes adds a considerable timelag, often of 2-3 years or more, to the process. Thus DFID often supplements international data with data obtained directly from the countries themselves".[74]

63. A variety of issues affect the collection of data in and from developing countries, including: the time lag between data collection and publication; gaps in the collection of data; differences in technical definitions and methods of data collection which make it hard to make comparisons between—and sometimes even within—different countries; the complexity and cost of some technical aspects of data collection (e.g. the international comparison programme to collect the data to establish purchasing power parity dollars); and the absence of vital registration and census data which undermines the denominators for many statistics.[75]

64. The difficulties encountered in obtaining reliable and comparable data undermine both the policy-making process, and the use of the MDGs—or more precisely the PSA—as a management tool.[76] In recognition of this, and in order to build the capacity of developing countries to collect data, DFID is collaborating with other development agencies to help build the capacity of developing countries to collect and use statistical data. The main focus of this effort is a programme called 'Partnerships in Statistics for Development in the 21st Century'.[77] DFID reports that over the past year, regional workshops have been held in some 40 countries, encouraging cooperation and dialogue between users and producers of statistics in developing countries.[78] More specifically, DFID is seeking to ensure that there is sufficient data to enable it to monitor progress in relation to its PSA targets. ODI reported that: "Of the 31 indicators related to the 7 country applicable goals in the MDGs, DFID is currently focussing attention on ensuring that all countries are able to generate data on 6 key indicators: $/day income-poverty; enrolment; gender equity in education; under-five mortality rate; births attended by trained specialists; [and], HIV prevalence".[79]

65. DFID faces major problems in collecting the data needed to monitor progress against the MDGs, and has made the first step towards addressing this issue by recognising its importance. We urge DFID to continue its work with developing countries, other donors, the World Bank and the OECD, to improve the provision of statistics on international development in both the short and longer term, and to provide progress reports in future Departmental Reports.

Evaluating DFID's Development Interventions

66. The third tier of monitoring and evaluation is focussed on the effectiveness of DFID's development interventions, and includes assessment of DFID at a corporate level (the PSA targets), at a country level—"evaluating on the run" as Suma Chakrabarti described it[80]—and at the level of programmes and projects. Such evaluations might be conducted by DFID country teams, by DFID's central evaluation unit, or by external evaluators. Mark Lowcock explained to us that:

    "In terms of internal evaluation of activities we finance, we have three systems, really. For our major projects—which amount to 85 per cent of the bilateral portfolio—we look once a year in a snapshot at whether the activities are achieving their objectives and we score them. We have a PSA target, reflecting whether they are risky or not—we break down by risk in other words—to improve the quality of the portfolio. Then, at the end of spending on each of our major activities we do a Project Completion Report and we analyse the results of those and try to spread the lessons, and so on. Thirdly, we have a category of ex-post evaluations where, some years after spending has been completed, we go back and have a look at what the lessons are and if we achieved the objectives intended".[81]

67. Criticisms have been made of DFID's evaluation systems, and of the fact that evaluations have little impact on reviews of spending and policy. The OECD's development cooperation review of the UK, produced in late 2001, wrote that: "Although DFID's monitoring and evaluation systems have evolved significantly in recent years and are now closer in line to good practices adopted by other DAC [Development Assistance Committee] Members, a number of concerns remain".[82] These concerns centre around the use of monitoring systems, the ownership of targets, and the impartiality and independence of the evaluation function. Howard White, in appendix two of the NAO report, wrote that: "At present one must wonder on what data DFID management do base their decisions. There is no 'bottom up' system to indicate overall performance. And the International Development Target-related indicators embodied in the PSA are of little operational use".[83] The Public Accounts Committee reported that "The Department are failing to make the most of their evaluation studies to influence the design of aid projects and programmes because at project level, the results have not become available in time and few formal evaluations have related to country programmes", and recommended that DFID undertake joint evaluations of country programmes with other major donors.[84]

68. In simple terms, DFID spends less on evaluation—0.05 percent of its total expenditure—than other donors such as the World Bank (0.12 percent) and the Danish development agency DANIDA (0.14 percent).[85] Evaluation is crucial, both for accountability and in order to improve organisational performance through better decision-making and learning. As such we welcome the fact that DFID plans to increase its spending on evaluation from £1.136 million in 20001/02 to £1.625 million in 2003/04.[86] We are also glad to hear that DFID has commissioned an independent evaluation to address comprehensively for the first time the question of "How effective is DFID?", and look forward to its publication. But, evaluation must itself be made more effective. We join with the Public Accounts Committee in recommending joint evaluations, and in urging a closer relationship between country teams and the evaluation unit, to ensure that lessons are learnt and implemented quickly and effectively. In addition, to ensure the independence of evaluation—something which we have previously asked of the European Commission[87]—we urge DFID, first, to increase the proportion of evaluations conducted by external evaluators, and, secondly to ensure that the results of all of its evaluations are made publicly available without undue delay. Above all, careful thought must be given to the ways in which the monitoring of DFID's progress against the top-level MDG targets can be more closely integrated with the monitoring of DFID's individual programmes and projects.

69. More specifically, during the oral evidence session, no mention was made of DFID's Performance Reporting Information System for Management (PRISM).[88] This makes us question whether PRISM is being used effectively for the monitoring of DFID's development activities, and if not, when this system will be implemented fully.


69   According to the OECD's Development Assistance Committee Working Party on Evaluation, "monitoring" refers to the systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives. "Evaluation" is the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. Back

70   See http://www.oecd.org/oecd/pages/home/displaygeneral/0,3380,EN-home-67-2-no-no--no,00.html Back

71   Q4 Back

72   DFID Departmental Report 2002, chapter 2, page 33. See footnote 1 for web-site. Back

73   Forty-Eighth Report from the Public Accounts Committee, Session 2001-2002, Department for International Development: Performance management-helping to reduce world poverty,HC793, paragraph 4, vi. See footnote 2 for web-site. Back

74   Ev 30, paragraphs 31-32. Back

75   The Millennium Development Goals and the IDC: Driving and framing the Committee's work, ODI, July 2002, paragraph 97. Back

76   The Millennium Development Goals and the IDC: Driving and framing the Committee's work, ODI, July 2002, paragraph 95. Back

77   See http://www.paris21.org/ Back

78   Ev 30, paragraph 33. Back

79   The Millennium Development Goals and the IDC: Driving and framing the Committee's work, ODI, July 2002, paragraph 96. Back

80   Q4 Back

81   Q4 Back

82   Development Cooperation Review of the United Kingdom, OECD Development Assistance Committee, DAC Journal, 2001, volume 2, no. 4, I62. Back

83   National Audit Office Report, Session 2001-2002, Department for International Development: Performance management-helping to reduce world poverty, HC739, appendix 2, page 63. See footnote 8 for web-site. Back

84   Forty-Eighth Report from the Public Accounts Committee, Session 2001-2002, Department for International Development: Performance management-helping to reduce world poverty,HC793, paragraph 4, vii. See footnote 2 for web-site. Back

85   How effective is DFID? An independent review of DFID's organisational and development effectiveness. DFID Evaluation Report, EV639. Final pre-publication draft. Back

86   DFID Departmental Report 2002, annex 1, table 4, page 103. See footnote 1 for web-site. Back

87   Second Report from the International Development Committee, Session 2001-2002, The effectiveness of the reforms of European development assistance, HC417, paragraphs 62-63. See http://www.publications.parliament.uk/pa/cm200102/cmselect/cmintdev/417/41702.htm Back

88   Development Cooperation Review of the United Kingdom, OECD Development Assistance Committee, DAC Journal, 2001, volume 2, no. 4, I59-60. Back


 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2002
Prepared 14 November 2002