Monitoring Progress Towards the
MDGs
61. The Departmental Report highlights the importance
of measuring and monitoring progress, including progress towards
the MDGs, but acknowledges the problems which developing countries—and
hence their donors—face in collecting data.[72]
In addition to annex 2 which sets out achievements against DFID's
PSAs, chapter 4 of the Departmental Report—"How we
are doing: Moving towards the Millennium Development Goals"—contains
some assessment of progress. However, this is couched mostly in
terms of progress towards the MDGs at the continental or regional
level, using World Bank data, with little information provided
about progress in individual countries in which DFID is engaged.
62. The obstacles to effective monitoring of progress
towards the MDGs are widely acknowledged, both within DFID, and
by other commentators. In its recent report, the Public Accounts
Committee stated that: "The Department's ability to provide
a reliable view of their performance has been constrained by the
poor quality of data they have had to use to measure progress
against their Public Service Agreement targets".[73]
In response to our questioning DFID wrote:
"There are known problems with availability,
reliability and timeliness of the data for monitoring progress
on development outcomes. This is a characteristic of poor statistical
systems in poor countries. Where possible DFID takes figures directly
from the databases of the World Bank (for example their World
Development Indicators database) or from the UN agency which has
responsibility for the particular indicator. These are the most
comprehensive and comparable sources of data for these indicators
but in most cases they rely on weak national statistical systems
to provide the country-level data. The process of data collation
by international agencies sometimes adds a considerable timelag,
often of 2-3 years or more, to the process. Thus DFID often supplements
international data with data obtained directly from the countries
themselves".[74]
63. A variety of issues affect the collection of
data in and from developing countries, including: the time lag
between data collection and publication; gaps in the collection
of data; differences in technical definitions and methods of data
collection which make it hard to make comparisons between—and
sometimes even within—different countries; the complexity
and cost of some technical aspects of data collection (e.g. the
international comparison programme to collect the data to establish
purchasing power parity dollars); and the absence of vital registration
and census data which undermines the denominators for many statistics.[75]
64. The difficulties encountered in obtaining reliable
and comparable data undermine both the policy-making process,
and the use of the MDGs—or more precisely the PSA—as
a management tool.[76]
In recognition of this, and in order to build the capacity of
developing countries to collect data, DFID is collaborating with
other development agencies to help build the capacity of developing
countries to collect and use statistical data. The main focus
of this effort is a programme called 'Partnerships in Statistics
for Development in the 21st Century'.[77]
DFID reports that over the past year, regional workshops have
been held in some 40 countries, encouraging cooperation and dialogue
between users and producers of statistics in developing countries.[78]
More specifically, DFID is seeking to ensure that there is sufficient
data to enable it to monitor progress in relation to its PSA targets.
ODI reported that: "Of the 31 indicators related to the 7
country applicable goals in the MDGs, DFID is currently focussing
attention on ensuring that all countries are able to generate
data on 6 key indicators: $/day income-poverty; enrolment; gender
equity in education; under-five mortality rate; births attended
by trained specialists; [and], HIV prevalence".[79]
65. DFID faces major problems in collecting the
data needed to monitor progress against the MDGs, and has made
the first step towards addressing this issue by recognising its
importance. We urge DFID to continue its work with developing
countries, other donors, the World Bank and the OECD, to improve
the provision of statistics on international development in both
the short and longer term, and to provide progress reports in
future Departmental Reports.
Evaluating DFID's Development
Interventions
66. The third tier of monitoring and evaluation is
focussed on the effectiveness of DFID's development interventions,
and includes assessment of DFID at a corporate level (the PSA
targets), at a country level—"evaluating on the run"
as Suma Chakrabarti described it[80]—and
at the level of programmes and projects. Such evaluations might
be conducted by DFID country teams, by DFID's central evaluation
unit, or by external evaluators. Mark Lowcock explained to us
that:
"In terms of internal evaluation of activities
we finance, we have three systems, really. For our major projects—which
amount to 85 per cent of the bilateral portfolio—we look
once a year in a snapshot at whether the activities are achieving
their objectives and we score them. We have a PSA target, reflecting
whether they are risky or not—we break down by risk in other
words—to improve the quality of the portfolio. Then, at
the end of spending on each of our major activities we do a Project
Completion Report and we analyse the results of those and try
to spread the lessons, and so on. Thirdly, we have a category
of ex-post evaluations where, some years after spending has been
completed, we go back and have a look at what the lessons are
and if we achieved the objectives intended".[81]
67. Criticisms have been made of DFID's evaluation
systems, and of the fact that evaluations have little impact on
reviews of spending and policy. The OECD's development cooperation
review of the UK, produced in late 2001, wrote that: "Although
DFID's monitoring and evaluation systems have evolved significantly
in recent years and are now closer in line to good practices adopted
by other DAC [Development Assistance Committee] Members, a number
of concerns remain".[82]
These concerns centre around the use of monitoring systems, the
ownership of targets, and the impartiality and independence of
the evaluation function. Howard White, in appendix two of the
NAO report, wrote that: "At present one must wonder on what
data DFID management do base their decisions. There is no 'bottom
up' system to indicate overall performance. And the International
Development Target-related indicators embodied in the PSA are
of little operational use".[83]
The Public Accounts Committee reported that "The Department
are failing to make the most of their evaluation studies to influence
the design of aid projects and programmes because at project level,
the results have not become available in time and few formal evaluations
have related to country programmes", and recommended that
DFID undertake joint evaluations of country programmes with other
major donors.[84]
68. In simple terms, DFID spends less on evaluation—0.05
percent of its total expenditure—than other donors such
as the World Bank (0.12 percent) and the Danish development agency
DANIDA (0.14 percent).[85]
Evaluation is crucial, both for accountability and in order to
improve organisational performance through better decision-making
and learning. As such we welcome the fact that DFID plans to
increase its spending on evaluation from £1.136 million in
20001/02 to £1.625 million in 2003/04.[86]
We are also glad to hear that DFID has commissioned an independent
evaluation to address comprehensively for the first time the question
of "How effective is DFID?", and look forward to its
publication. But, evaluation must itself be made more effective.
We join with the Public Accounts Committee in recommending joint
evaluations, and in urging a closer relationship between country
teams and the evaluation unit, to ensure that lessons are learnt
and implemented quickly and effectively. In addition, to ensure
the independence of evaluation—something which we have previously
asked of the European Commission[87]—we
urge DFID, first, to increase the proportion of evaluations conducted
by external evaluators, and, secondly to ensure that the results
of all of its evaluations are made publicly available without
undue delay. Above all, careful thought must be given to the ways
in which the monitoring of DFID's progress against the top-level
MDG targets can be more closely integrated with the monitoring
of DFID's individual programmes and projects.
69. More specifically, during the oral evidence session,
no mention was made of DFID's Performance Reporting Information
System for Management (PRISM).[88]
This makes us question whether PRISM is being used effectively
for the monitoring of DFID's development activities, and if not,
when this system will be implemented fully.
69 According to the OECD's Development Assistance
Committee Working Party on Evaluation, "monitoring"
refers to the systematic collection of data on specified indicators
to provide management and the main stakeholders of an ongoing
development intervention with indications of the extent of progress
and achievement of objectives. "Evaluation" is the systematic
and objective assessment of an ongoing or completed project, programme
or policy, its design, implementation and results. Back
70
See http://www.oecd.org/oecd/pages/home/displaygeneral/0,3380,EN-home-67-2-no-no--no,00.html Back
71
Q4 Back
72
DFID Departmental Report 2002, chapter 2, page 33. See footnote
1 for web-site. Back
73
Forty-Eighth Report from the Public Accounts Committee, Session
2001-2002, Department for International Development: Performance
management-helping to reduce world poverty,HC793, paragraph
4, vi. See footnote 2 for web-site. Back
74
Ev 30, paragraphs 31-32. Back
75
The Millennium Development Goals and the IDC: Driving and framing
the Committee's work, ODI, July 2002, paragraph 97. Back
76
The Millennium Development Goals and the IDC: Driving and framing
the Committee's work, ODI, July 2002, paragraph 95. Back
77
See http://www.paris21.org/ Back
78
Ev 30, paragraph 33. Back
79
The Millennium Development Goals and the IDC: Driving and framing
the Committee's work, ODI, July 2002, paragraph 96. Back
80
Q4 Back
81
Q4 Back
82
Development Cooperation Review of the United Kingdom,
OECD Development Assistance Committee, DAC Journal, 2001, volume
2, no. 4, I62. Back
83
National Audit Office Report, Session 2001-2002, Department
for International Development: Performance management-helping
to reduce world poverty, HC739, appendix 2, page 63. See
footnote 8 for web-site. Back
84
Forty-Eighth Report from the Public Accounts Committee, Session
2001-2002, Department for International Development: Performance
management-helping to reduce world poverty,HC793, paragraph
4, vii. See footnote 2 for web-site. Back
85
How effective is DFID? An independent review of DFID's organisational
and development effectiveness. DFID Evaluation Report, EV639.
Final pre-publication draft. Back
86
DFID Departmental Report 2002, annex 1, table 4, page 103. See
footnote 1 for web-site. Back
87
Second Report from the International Development Committee, Session
2001-2002, The effectiveness of the reforms of European development
assistance, HC417, paragraphs 62-63. See http://www.publications.parliament.uk/pa/cm200102/cmselect/cmintdev/417/41702.htm Back
88
Development Cooperation Review of the United Kingdom,
OECD Development Assistance Committee, DAC Journal, 2001, volume
2, no. 4, I59-60. Back
