3. Achieving a sustainable impact from
projects
19. Achieving a lasting beneficial impact is one
of the key objectives of development assistance. Between 1997
and 2002 the Department completed 193 dedicated water and sanitation
projects, which represented a commitment of £61 million.
The Department's evaluations of these projects showed that, where
assessments were available, three quarters had completely or largely
achieved their objectives (Figure 4).[65]
But there was a lack of available evidence to assess the extent
to which these projects were achieving a lasting beneficial impact.
Two thirds of project completion reports that included an assessment
of sustainability raised doubts and risks as to whether a sustainable
impact would be achieved.[66]
There are risks inherent in providing development assistance in
developing countries and the Department's achievements must be
considered within this context. The Department has also tried
innovative approaches in risky places in order to achieve development
outcomes.[67]
Figure 4: Achievement of objectives for water
projects completed between 1997 and 2002

Note: Output level rating is used to assess the project's
planned results. Purpose level rating is used to reflect whether
the intended changes have occurred in the host country as a result
of those outputs.
Source: NAO analysis of Project Scoring Information
20. The Department has responded to the need to achieve
a sustainable impact in a number of ways.[68]
The Department has recognised that unless the broader policy framework
in developing countries is strong, then individual projects are
not going to be sustainable.[69]
Improving governance has been a major focus of the Department's
bilateral aid programmes for a number of years as it has recognised
that building effective institutions is an important pre-condition
of achieving sustainable development.[70]
21. The Department also seeks to address sustainability
issues in project design. The National Audit Office found that
some recognised good practice criteria received less attention
than others. Most commonly, projects did not consider the availability
of replacement parts, the use of cost-effective technology and
hygiene awareness issues.[71]
The Department told us that it is possible to break down the
components of sustainability into five key factors (Figure
5).[72]
In the past, too many projects have been designed without addressing
all of these issues.[73]
The Department has published a series of strategy papers to inform
staff thinking on the priorities and methodologies that should
be considered when developing country programmes. These papers
include "Addressing the Water Crisis" and "Achieving
sustainability, poverty elimination and the environment".
However, little attention has been given to knowledge management
within the Department or the best approach to disseminating to
country teams the good practice and lessons learned from project
evaluations.[74]
Figure 5: The five key components of sustainability
Political support: ensuring that there is host government to addressing the issues in that sector.
Economic sustainability: ensuring that the appropriate charging structure is in place to ensure that revenue is raised for operation and maintenance issues.
Social sustainability: addressing the community's requirements are understood and addressed in the project.
Technical issues: ensuring technology is appropriate to the community's needs.
Policy and institutional issues: ensuring the institutional infrastructure is in place to manage the projects and deliver services.
|
Source: Ev 1, Q 8
22. The inability of local partners to manage donor-funded interventions
can lead to problems with sustainability. In the Two Regions Project
in Ghana, the partner government did not have the capacity to
address issues that emerged during project implementation, thus
leading to a reduction in the effectiveness of the Department's
funding.[75] The Department
accepts that, in the past, too many projects have given insufficient
attention to what happened after funding had ended and expertise
had been withdrawn. Greater attention is now being given to the
skills and institutional structure that are needed to manage the
project after the Department's involvement ends.[76]
23. Increasingly, the Department's water projects have focused
on strengthening the capacity of those bodies responsible for
water delivery and transferring knowledge from its advisers to
developing country administrations.[77]
The aim is to secure a longer-term impact, and the Department
believes that it has a comparative advantage over other donors
in providing this type of assistance.[78]
This has led to successful projects which have had a lasting
beneficial impact. For example, a Departmental water project in
Maharashtra in India led to the introduction of a new charging
structure which enabled the government to charge those who could
afford to pay and raise funds to maintain systems.[79]
24. The Department has a well-established approach to evaluating
its projects. Country teams are required to carry out project
completion reports for all bilateral projects with a commitment
in excess of £1 million.[80]
However, the Department's performance in completing such reports
has been poor. Project completion reports should have been carried
out for 32 of the 193 water projects completed between 1997 and
2002, but only 17 reports had been produced. In value terms, this
represented coverage of 56% of eligible water projects.[81]
The lack of available information means that it is not possible
to determine fully the value for money of the Department's assistance[82]
or undertake a comprehensive assessment to identify those factors
leading to project success. The Department has devoted considerable
efforts to improve compliance with its project evaluation procedures
and told us that twice as many projects were reviewed last year
compared to previous years.[83]
25. The main purpose of project completion reports is to assess
the extent to which the project goal and planned outputs have
been achieved. But there is less emphasis on assessing sustainability
or the wider impacts of projects, often because they are completed
too early to judge the longer-term impacts.[84]
The evaluation of impacts needs to be considered at the outset
of the project and, as necessary, appropriate baseline information
needs to be collected to assess outcomes. The Department has not
always given sufficient attention to the need to establish robust
monitoring arrangements at the project design stage.[85]
26. The Department also carries out a rolling programme of ex-post
evaluations to assess the long-term sustainability and wider impacts
of projects. However, due to the wide range of its activities,
the Department is only able to achieve partial coverage in any
one year.[86] The Department
is increasing the resources devoted to evaluations by 25% in each
of the next two years and is establishing a direct link to the
management board.[87]
Efforts are also being made to improve the quality of evaluations[88]
and make them more timely and relevant.[89]
As part of the increased evaluation effort, there is also scope
to revisit projects some years after funding has stopped to assess
impacts and learn lessons.[90]
65 C&AG's Report, para 7 Back
66
ibid, para 8 Back
67
Q 49 Back
68
C&AG's Report, para 1.13 Back
69
Q 19 Back
70
C&AG's Report, para 1.13 Back
71
ibid, para 1.15 Back
72
Q 8 Back
73
Q 8 Back
74
C&AG's Report, para 1.22 Back
75
ibid, para 1.10 Back
76
Qq 9, 76 Back
77
Q 9 Back
78
Qq 19, 53 Back
79
Q 32 Back
80
C&AG's Report, para 1.18 Back
81
ibid, para 1.8 Back
82
Q 11 Back
83
Q 11 Back
84
C&AG's Report, para 1.19 Back
85
ibid, para 1.20 Back
86
ibid, para 1.21 Back
87
Qq 11, 57 Back
88
C&AG's Report, para 1.18 Back
89
Q 11 Back
90
Qq 52, 57 Back
|