Annex A
QUALITY MEASURES
Switching from project assistance to programme
based approaches (PBAs)
1. Since most programme based approaches
are channelled through partner governments' financial management
systems, one of the obvious risks of spending funds in this way
is that of the diversion of such funds. Extensive work has been
done to mitigate this within DFID: mandatory procedures to manage
fiduciary risk are included in the Blue Book, and the application
of these policies is monitored centrally.
2. Another quality measure to address fiduciary
risk is monitoring the improvements in public financial management
in our partner countries. As we deliberately work with some of
the poorest countries in the world, there is a close correlation
between countries supported by DFID and the register of Heavily
Indebted Poor Countries (HIPCs). The International Monetary Fund
(IMF) and World Bank (WB) drew up a set of indicators and benchmarks
to monitor Public Expenditure Management (PEM) systems in HIPCs,
updating their assessment periodically. The last of these[1]
(April 2005) showed that PEM had improved for the countries assessed,
though the extent of progress was mixed across countries and indicators.
The HIPC measurements were superceded in June 2005, when a joint
initiative by the donor community (including the IMF, WB, DFID
and others) finalised a new tool to assess financial management
called the Public Expenditure and Financial Accountability (PEFA)
Public Financial Management Performance Measurement Framework.
The PEFA Framework consists of 28 high level indicators designed
to measure the performance of national public financial management
systems, processes and institutions; and three indicators which
measure the performance of donors in providing aid.
3. The individual country evaluations completed
since June 2005 will establish a PEFA performance baseline against
which future progress can be assessed. It is expected that PEFA
evaluations will be carried out every three to five years to allow
real changes in countries' systems to work their way through,
so conclusive evidence about progress will not be available during
the 2005-08 efficiency period.
4. A further risk is the reduced control
over what aid is spent on, leading to possible spending on areas
that are not focused on poverty reduction. Where this is a particular
risk, additional measures are put in place, such as designated
sectoral funding to ensure that a poverty focus is maintained,
while the long-term goals of sustainability and country ownership
are promoted.
5. It is also important to remember that
not all countries are suitable candidates for programme based
approaches, in particular fragile states or those with no commitment
to address systemic weaknesses in financial management systems.
Country selection therefore is based on a thorough evaluation
of both the current context and prospects for change.
Savings on procurement
6. DFID's procurement savings have been
achieved through post contract clarification savings. These have
been achieved through effective and informed negotiation with
successful bidders and will not have any adverse impact on quality
of output, since the contracts include specific quality levels
that need to be achieved. We have ensured that any reductions
in costs have not come about due to reductions in either quality
of personnel or timely and effective achievement of key milestones/deliverables.
7. We have achieved our procurement savings
by working with consultancy partners in removing "fat"
and rationalising "non value adding" transaction costs.
In the case of our use of reverse auctions we state the level
of quality to be achieved, and we achieve savings by bidders competing
to achieve lowest cost whilst still meeting these standards. Our
Procurement Group normally uses the most economically advantageous
terms (MEAT) as the basis for determining best value for money
whilst maintaining appropriate quality standards.
8. DFID also requires consultants to develop
and utilise their own quality assurance plans.
Improvement in the performance of DFID's bilateral
portfolio
9. In our Efficiency Technical Note we recognised
that there is a risk that focusing on targets around project/programme
success might mean projects/programmes which are more likely to
succeed are chosen—in other words projects/programmes that
are low risk, rather than projects/ programmes which provide the
best rate of return.
10. Projects and programmes are currently
classified into three broad risk categories—high, medium
and low. In order to ensure that the integrity of our risk profile
and appetite has not been undermined, we have monitored the proportion
of projects/programmes in each category. The distribution has
changed as shown in the following table since the baseline date.
This demonstrates that our success has been achieved even while
increasing our risk appetite.
|
| Risk category | March 2005
| Q2 2006-07 |
|
| High | 14% | 21%
|
| Medium | 61% | 61%
|
| Low | 25% | 18%
|
|
| |
|
11. We also recognised in our Technical Note that there
is a risk of projects being scored inconsistently. The National
Audit Office felt in their report of February 2006 that our self-assessment
of performance, though often undertaken by external consultants
or in conjunction with partner governments and other donors, as
well as being scrutinised by Heads of Office and our Internal
Audit and Evaluation Departments, was not sufficient to ensure
the impartiality of our assessment. We have therefore commissioned
consultancy work to assess the quality, consistency and accuracy
of DFID logframes, Project Header Sheets and project performance
reviews, and determine whether the quality has changed over time
by comparing reviews from two periods. The first period is prior
to the start of DFID's PSA and Efficiency Programme targets on
portfolio quality, and the second covers the period October 2005-September
2006.
Increased contributions to low income countries (LICs) via
EC aid
12. This target does not capture spend quality. So the
risk remains that the money spent in low income countries is badly
spent. The EC, as part of its reform process, has introduced Results
Oriented Monitoring (ROM), which assesses all the EC regional
programmes on a one to four scale in terms of relevance, efficiency,
effectiveness, impact and sustainability. These results are published
in the EC Annual Report and scores have been improving year-on-year
in all regions.
13. In ROM, projects and programmes are given scores
against internationally agreed criteria, using a well-structured
and consistent methodology.
14. In 2005, a more accurate calculation method for reporting
was introduced: the averages are not calculated on the rounded
final scores, but on the basis of the points given for each sub-criterion.
For each project, short explanations and recommendations on quality
improvement are addressed to all stakeholders, providing important
complementary information to internal monitoring by project stakeholders.
15. To ensure consistency of methodology, each of the
five criteria comprises individual sub-criteria which the monitor
has to consider, comment on and rate: scores for the criteria
are then calculated from the ratings for the sub-criteria. This
new and more exact method of calculation was introduced end-2004.
Consequently, while the original method of calculating from the
(rounded) scores for each criterion gives the result 2004(a),
the overall averages for 2004(b) and 2005 are now calculated from
the ratings for each sub-criterion, in order to give a more precise
evaluation.
16. Over the last four years, the overall ratings in
ROM show a steady and statistically significant improvement: from
2.62 in 2002 to 2.67 in 2003, and from 2.68(a) and 2.73(b) in
2004 to 2.76 in 2005.
17. This is a step in the right direction, but gives
a highly aggregated snapshot. DFID also continues to monitor progress
in this area. Although ROM's methodology has not been evaluated
independently, it has been presented and discussed in several
fora, including the last DAC peer review. In addition, the Commission
cross-checks results through its evaluations. Another element
of quality control is the "response sheet" where delegations
and other stakeholders are invited to react on the reports, their
quality and usefulness.
18. Public financial management systems in LICs will
also be monitored through the Public Expenditure and Financial
Accountability (PEFA) Performance Measurement Framework.
Administration costs
19. The challenge is to ensure that administration cost
efficiencies do not lead to reductions in the quality of our programmes
and policies. DFID has sought to achieve this by:
— transforming our corporate support processes to
enable staff to spend less time on general administrative tasks
and more time on front-line delivery through DFID's CATALYST programme.
The programme has already improved electronic information management
and has automated several basic administrative jobs (such as organising
travel);
— ensuring that administration cost reductions impact
mainly on support functions (particularly the Human Resources
Division and the Finance and Corporate Performance Division) and
not on Divisions involved in front-line delivery and policy development.
Africa Division has received a significant real terms increase,
Asia Division and International Division have received slight
increases, and other Divisions have flat or reducing budgets;
— developing and using an Administration Costs Allocation
Model that allocates administration costs between and within geographical
divisions on the basis of: size of programme; the policy environment;
and the types of projects and programmes; and
— developing and implementing a People Strategy
that outlines how DFID will lead, manage, develop and support
our people from 2005-08. This strategy aims to address the challenge
of managing a rising aid budget with a decreasing number of staff.
20. The quality of our project portfolio is the proxy
indicator that DFID uses to monitor overall programme quality.
This gives a good indication of whether we are managing administration
cost reductions in ways that do not impact on our work on the
ground. It also has the advantage of being an indicator on which
we already collect data. This indicator is considered alongside
the assessments of our effectiveness contained in:
— progress reports on the PSA (contained in the
Autumn Performance Report and the Departmental Report);
— select committee reports (particularly the annual
International Development Committee report on DFID's Departmental
Report); and
— other external reviews of DFID (such as the OECD
Development Assistance Committee's Peer Review of the UK that
was concluded in 2006 and the Departmental Capability Review).
1
http://www.imf.org/external/np/pp/eng/2005/041205a.html Back
| 
