House of Commons - International Development

Select Committee on International Development Fifth Report

DEPARTMENT FOR INTERNATIONAL DEVELOPMENT: 1998 DEPARTMENTAL REPORT

TARGETS AND PERFORMANCE MEASUREMENT

34. The Treasury core requirements relating to targets and performance measurement, in each of the department's main activities, are to:

(b) make clear how each of the department's main activities relates to the department's aims, objectives and key priorities;

(c) provide future targets, milestones or plans for the main activities of the department, against which future performance can be measured;

(d) give information about achievements, quantified wherever possible, and set out in such a way that ready comparison can be made with the previous plans or targets and links made with the department's objectives;

(e) provide information on effectiveness and quality of service, including relevant evaluation findings. Information on final outputs should be given where possible; if this is not available, intermediate measures or indicators can serve; and

(f) indicate the efficiency of each area of activity by relating outputs to inputs (financial and staffing)."

35. In DFID's Departmental Report, only one of these requirements is met to a satisfactory degree: the requirement to state the broad aims and objectives of the Department. None of the other requirements are met. The failure of the Department to fulfill these basic requirements is particularly disappointing given its commitment to performance measurement as one of its key policy strands.[57]

36. The Departmental Report is structured clearly around the Department's Statement of Purpose. This places a proper emphasis on outputs in the structure of the Report. We welcome the objectives-based structure of the Departmental Report. The White Paper, in addition to the Statement of Purpose, had included an outline of DFID's 12 policy strands and nine "challenges" and "responses", yet the Departmental Report refers only to the Statement of Purpose. This raises questions about the status of policy statements other than the Statement of Purpose included in the White Paper, and makes it difficult to establish clear links between the White Paper and the Departmental Report. We recommend that DFID establish a clear framework of policies and objectives around which to structure all its various publications. This would ensure greater transparency and clarity, which are essential for proper scrutiny.

37. Whilst we welcome the objectives-based structure of the Departmental Report, we have some concerns about DFID's systems of performance measurement, which appear to place far too great an emphasis on inputs rather than outputs. This was evident in the Government's response to our Report on the White Paper,[58] in which the Government reiterated its commitment to reporting regularly on the Department's performance: "We are committed to reporting annually on progress against the strategy and targets set out in the White Paper".[59] We welcome this commitment to targets; however, the response went on to describe PIMS: "DFID also operates a Policy Information Marker System (PIMS) to track the targeting of bilateral commitments and expenditure on priority policy objectives of the aid programme. PIMS is being reviewed and revised to ensure that it reflects the new objectives set out in the White Paper".[60] As we discussed above, PIMS is useful to the extent that it measures expenditure according to the aims and objectives of the Department, but does not provide information about the performance of projects and programmes in fulfilling those objectives. This emphasis on inputs rather than outputs is a criticism which has been raised by other commentators than ourselves, including the OECD's Development Assistance Committee: "one of the central problems for British aid, as for many of its partners in the DAC, will be to place increased focus on outputs, particularly differentiated by beneficiary, with respect to poverty and other elements of the overall strategy".[61]

38. The Departmental Report reiterates the White Paper commitment to "measure the effectiveness of our efforts, alongside others, against the [international development] targets, including the aim of halving the proportion of the world's population living in extreme poverty by 2015".[62] This was elaborated during oral evidence, when Mr Vereker told the Committee "what we propose to offer the Committee is an annual report showing progress against the 21 indicators which have been agreed by the Development Assistance Committee of the OECD's expert statistical group[63] which are grouped by individual target".[64] These 21 indicators are simply more detailed versions of the international development targets themselves.

39. We do not dispute the value of the international development targets as a driving force for the international community, and we have welcomed the commitment of the UK Government to the targets in a previous Report.[65] However, the international development targets, and the 21 indicators associated with them, do not relate directly to the efforts of individual donors; they quite rightly relate to the performance of the international community as a whole. This renders them useless as instruments for measurement of the performance of individual donors. This problem is acknowledged in the Departmental Report,[66] and Mr Vereker mentioned the "conceptual difficulties about causality"[67] in passing during discussion in oral evidence. We appreciate that the Department may face some difficulties, but the fact remains that in failing to provide any meaningful or detailed information about its achievements, the Department has failed to fulfill the basic Treasury core requirements. Whilst developmental outcomes may not always be quantifiable, it should be possible in most cases to identify targets and projected outcomes for individual projects, to measure the performance of projects and programmes against those targets, and ultimately to aggregate the results to provide an overall picture of the performance of the UK's development assistance programme according to sectors, countries, and as a whole. We view the establishment of objectives against which the performance of the Department can be clearly measured as crucial to proper scrutiny of the work of DFID.

40. DFID does have systems for the evaluation of projects, the main two being Evaluation Studies and Project Completion Reports. The OECD's Development Assistance Committee noted in their 1997 report on the UK development assistance programme that "one can now consult a range of development assistance plans, Development Assistance Strategies for the main bilateral programmes, progress reports, evaluation reports and summaries, and a home page on the One World Online Website".[68] The problem is that the Departmental Report fails to use this information in a meaningful way. The Departmental Report includes only basic descriptions of the systems themselves, and almost no information at all on performance of projects and programmes against targets, lessons learned from evaluations, or details of planned performance measurement exercises.

Project Completion Reports

41. Project Completion Reports (PCRs) are produced for all geographical projects costing more than £500,000, and a synthesis study is produced summarising the results of all PCRs carried out in a calendar year. The information provided in the Departmental Report on the findings of PCRs is almost non-existent. We view Project Completion Reports (PCRs) as a valuable tool for accountability, and as such we recommend that more details about recent findings be given in future Departmental Reports. The study of 1996 PCRs provided a table summarising the results, followed by a table summarising the results of the 1995 study. This is extremely useful information, and we recommend that DFID use similar information in future Departmental Reports as the basis for their discussion on PCRs.

42. The Departmental Report cites the synthesis study of PCRs completed in 1996, which found that "around two-thirds of DFID projects were expected to achieve their immediate objectives".[69] At face value this appears to be a fairly poor rate of success. When we questioned Mr Vereker in oral evidence, he explained that "our target is for everything to be fully effective, but in the real world we have to recognise that we are operating in a rather high risk business and it is not surprising that particularly in some of these more difficult social sectors, poor country institutional building areas we do not hit all our targets. We have tried to bench-mark this against other development agencies. Comparable systems for evaluation used by other donors such as the World Bank yield rather similar results to our own so I can say in bench-marking terms we are up there with the best and given the difficult institutional context within which we are working it is fair enough".[70] We fully support bench-marking of the performance of DFID projects and programmes against those of other donors, and recommend that future Departmental Reports include such comparisons, as well as providing details of strategies and specific targets relating to improvements in the achievement of objectives.

43. Project Completion Reports themselves have a series of limitations which must be taken into account when considering their findings. These are summarised at the beginning of the synthesis study of PCRs carried out in 1996:

(b) annual collections vary in coverage, timing and duration, and none is truly representative of the overall aid programme;

(d) the existing rating systems tend to exclude the middle ground. They are based on a five-point scale, with the top two ratings regarded as broadly satisfactory and the bottom two broadly unsatisfactory. The middle rating, being ambiguous, is therefore excluded from the results, although in a number of cases this rating accounts for a significant proportion of the total, as will be easily deduced from the evidence.

(e) standards of completion remain variable, despite improvement.[71]

44. We recommend that in its response to this Report, the Government describe what steps have been taken and are planned to address the weaknesses in Project Completion Reports described in the 1996 synthesis study.

Evaluation Studies

45. Evaluation studies are carried out on 12 projects a year and involve a full retrospective impact evaluation.[72] The studies are now produced not by the Department but by individual authors, as Clare Short pointed out in a letter to the Chairman of the Committee following our oral evidence session with the Permanent Secretary.[73] We welcome the responsibility of individual authors, rather than the Department, for their own evaluation studies. DFID must, however, make clear its response to the studies if they are to fulfill their maximum value as instruments for scrutiny.

46. The Departmental Report provides very little information about the criteria against which projects are evaluated, the results of recent evaluation reports or plans for forthcoming studies. This information is doubtless available elsewhere, however there is no reference in the Report to sources of further information. In an annex to the Departmental Report, a brief summary is provided of recent lessons learned from evaluation studies.[74] This summary does not show whether the lessons resulted from positive or negative experiences, and there are no strategies for the implementation of the lessons. For example, the Departmental Report states that "in emergency aid interventions, there is a need for more capacity to deal with a large number of implementing partners; and a review of co-ordination arrangements between Whitehall Departments is required", and that "in projects directed to the strengthening of university institutions, it is vitally important to set agreed performance indicators at the outset".[75] It is not clear from the Departmental Report what DFID plans to do in response to these lessons. We recommend that future Departmental Reports include more extensive analysis of lessons learned from recent evaluation studies, with details of strategies for making improvements.

47. The recent review of the UK development assistance programme by the OECD's Development Assistance Committee commented that "one of the major recommendations of the 'poverty guidance' of 1991 is first to identify the extent, causes and trends of poverty in recipient countries. However, a Secretariat review of several country strategies showed little explicit evidence that they had been based on such an analysis. Similarly, a Secretariat review of evaluation summaries found little evidence that projects had been evaluated with respect to their impact on poverty reduction".[76] We recommend that in its response to this Report, the Government explain in detail what steps it is taking to ensure that the recommendations of the 'poverty guidance' of 1991 are taken fully into account in evaluation studies.

PRISM

48. DFID is developing a mechanism for the aggregation of findings from its performance evaluation systems: the Performance Reporting Information System for Management (PRISM). This is mentioned only in passing in the main text of the Departmental Report,[77] and is described briefly in an annex.[78] The system is due to be piloted in some divisions in October 1998, but no timetable is presented the various other stages leading to full implementation of the system across all DFID's departments, programmes and projects. We recommend that the Government produce a timetable for the full implementation of PRISM, and that future Departmental Reports provide details of progress made.

49. The OECD's DAC Report on the UK's development assistance programme shows that DFID is taking steps to improve its performance monitoring systems, and is positive about achievements so far: "On the whole, in the spirit of the White Paper, the British aid system needs to continue to improve its transparency and openness to establish full partnership with recipient countries, and to improve its cooperation with other donors. Since the government culture had been relatively closed in the past, this change in attitude and reflexes is difficult. However, recent progress made in improving dialogue, such as with NGOs and others outside the aid system, has been encouraging. It simply needs to be carried further".[79] We would bring these comments to DFID's attention.

50. It is our conclusion that DFID has broadly adequate performance monitoring systems, and is making valuable efforts to modify and improve those systems in the light of the White Paper. In the 1998 Departmental Report DFID fails to make full use of the information provided by these systems. The Departmental Report could fulfill a valuable role by filling the gap between the two extremes of project level reporting on the one hand, and reporting against international targets on the other, showing how the Department has performed against its own objectives at sector, country and regional levels.

51. Resource accounting and budgeting techniques will require the Department to produce more detailed information on its performance related to its expenditure in the form of an output performance analysis. We trust that DFID will use the opportunity presented by the introduction of the new system to review its performance analysis mechanisms, and the information relating to them which it will provide in future Departmental Reports, in order that the Treasury core requirements and the challenges presented by resource accounting and budgeting can be met.

Accountability of Multilateral Agencies

52. Half of the UK aid budget is now spent on contributions to multilateral agencies, and this fact alone makes it imperative that these agencies be capable of being held to account by those who make contributions to them. Mr Vereker told us "we are satisfied that these [multilateral agencies] are delivering reasonably useful information" on their performance.[80] We commented in a previous Report on the lack of transparency in World Bank and IMF activities,[81] and concerns about the accountability of the EU aid programme are well-documented. In our Report on the White Paper[82] we noted the Government's commitment to encourage the multilateral institutions, and in particular the European Union, to "set quantifiable targets for poverty reduction, and measure progress towards these" and "to devote more attention to evaluating and monitoring the output of their activities, and to harmonise their impact assessment systems",[83] and we looked forward to significant improvements under the UK Presidency. We recommend that DFID provide details of what progress was made during the UK Presidency of the EU towards improving the accountability of the EU's development programme, and outline in more detail its plans to improve the accountability of other multilateral donors.

53. The UK contributed £194 million to the World Bank and IMF, and £59 million to regional development banks in 1996-97, yet in the Departmental Report very little indication is given of DFID's assessment of the performance of these institutions in terms of their effective management or their contribution to poverty eradication. We recommend that in future Departmental Reports, a paragraph be included for each international financial institutions, including each regional development bank, describing DFID's assessment of the performance of each institution and outlining strategies for improvements where appropriate. These paragraphs should include clear references to sources of further information. Previous Departmental Reports of the Foreign and Commonwealth Office provided specific targets and performance information relating to the accountability of multilateral development agencies,[84] and we recommend that DFID resume this practice.

RESEARCH

54. The White Paper states that "knowledge, research and technology underpin all our work",[85] and that "without research, many development institutions would fail or be much less successful".[86] We agree with this analysis, and given this emphasis, we would expect the Departmental Report to contain details of what research has recently taken place, is currently being undertaken, and is planned for the coming year. The Departmental Report contains several references to current and planned research,[87] however there is no systematic approach. In future Departmental Reports, we recommend that discrete sections be included providing details (including costs) of recent, current and planned research relating to each of the Department's objectives.

DEPARTMENTAL ADMINISTRATION

55. The Treasury core requirements state that the Departmental Report must include specific pieces of information relating to departmental administration, including staffing, environmental policy, and efficiency. We are disappointed that some of these requirements have not been met in the 1998 Departmental Report. A key shortfall is the failure of the Department to provide information about equal opportunities recruitment. DFID should show the number of ethnic minority and disabled staff recruited to senior posts, overall representation, and provide examples of initiatives being taken to promote equal opportunities. This point relates closely to another of our concerns. We would expect the Departmental Report to include an analysis of staff employed in senior positions overseas, showing the number, grades and pay of locally-employed staff. We were disappointed to note that this information was not provided in the Departmental Report, the reason being that the information was not held centrally by DFID.[88] We are, however, encouraged to learn that DFID is currently compiling such a database, and we look forward to a summary being provided in the 1999 Departmental Report.

CONCLUSION

56. We are in this Report very critical of DFID's 1998 Departmental Report as an exercise in accountability to Parliament. It simply does not contain enough information for an accurate assessment of the Department's activities and performance. If DFID wants to engage Parliament and the public in the vital work of development, it must tell more of its story and tell it better. To do this, DFID must have a clear view of the purpose of the Departmental Report. Its present style inhabits an unhappy no-man's land between the White Paper and British Aid Statistics. The Departmental Report must contain a comprehensive and more detailed account of past, current and planned expenditure and a clear summary of the evaluations of performance against targets and objectives. We look forward to next year's Departmental Report meeting these requirements and trust our comments will contribute to this objective.

57 Departmental Report, p. 86. Back

58 Second Report from the International Development Committee: The Development White Paper, Session 1997-98, HC 330. Back

59 Second Special Report from the International Development Committee: Government Response to the Second Report from the Committee, Session 1997-98: The Development White Paper. HC 643 p. viii. Back

60 Ibid. p. viii. Back

61 OECD Development Cooperation Review Series: 1997 No. 25: "The United Kingdom", p. 18. Back

62 Departmental Report, p.5. Back

63 Evidence, pp. 23-24. Back

64 Q. 9. Back