2 MONITORING AND EVALUATION
7. In last year's Report we expressed our regret
that DFID had not made available its Monitoring and Evaluation
(M&E) Framework at the same time as its new Strategy was published,
in June 2008.[8] We found
the Strategy to be "strong on rhetoric but weak on communicating
how DFID will implement it." We noted that there were "few
measurable targets or indicators of how the Strategy's effectiveness
will be assessed" and stressed the need for the M&E Framework
to set out specific targets and indicators, to enable an assessment
to be made of whether DFID's HIV/AIDS programmes were achieving
their aims.[9] We highlighted
that, in its 2004 Strategy, DFID's indicators of success had been
linked primarily to funding targets rather than to outcomes. We
observed that, in the new Strategy, DFID was still concentrating
on funding levels rather than measuring effectiveness: "there
remains an emphasis on the amount of money which will be spent
rather than the impact which will be measured".[10]
8. DFID eventually published the M&E Framework,
Achieving Universal Access: Monitoring performance and evaluating
impact, in December 2008.[11]
The document states that the Department sees monitoring of the
performance and evaluating the impact of its HIV/AIDS activities
as a "central part of DFID's corporate performance systems".
It emphasises that:
Keeping track of the inputs, processes, outputs,
outcomes and impacts of DFID funded bilateral and multilateral
programmes, and UK activities to influence others, is key to ensuring
that the UK responds quickly to fill gaps in performance as well
as ensuring transparency and accountability.[12]
A further document, Achieving Universal Access—a
2008 baseline, was published in October 2009.[13]
This provides a snapshot of the global AIDS epidemic in mid-2008,
when the new Strategy was launched, and sets a baseline against
which the impact of DFID's HIV/AIDS commitments will be measured.
DFID says that "It is against these commitments that we will
be held to account in future biennial reporting."[14]
9. Several witnesses praised DFID for publishing
the Baseline to complement the M&E Framework. Sally Joss of
the UK Consortium on AIDS and International Development said that
it was "a massive move forward from the previous AIDS strategy
where there was no baseline and not really a monitoring and evaluation
framework to even start to measure what is happening". Mike
Podmore of VSO applauded DFID for linking its Strategy to global
targets and indicators and said that DFID was "leading well
at the international level in terms of global indicators".[15]
10. DFID has committed to reporting on performance
of its HIV/AIDS Strategy every two years, with the first progress
report to be published on World AIDS Day (1 December) 2010.[16]
The information for the biennial reports will be compiled from:
- biennially collated overviews of the AIDS response
from DFID country, regional, policy and multilateral representatives;
and
- information embedded in DFID corporate performance
systems.
This information will then be set in the context
of internationally agreed targets and indicators administered
and collated routinely by UNAIDS and partner countries.[17]
The overviews will be collated systematically every two years
and will describe the situation in-country, including the epidemic
status, the aid environment and the key actors. They will provide
details of how DFID is performing against each of the five priorities
set out in Achieving Universal Access:
- Priority 1: Increase
effort on HIV prevention; sustain momentum for treatment; increase
effort on care and support
- Priority 2: Respond
to the needs and protect the rights of those most affected
- Priority 3: Support
more effective and integrated service delivery
- Priority 4: Making
money work harder through an effective and co-ordinated response
- Priority 5: How we
will turn our strategy into action. [18]
A template showing the information that will be collected
from DFID country offices has been included as an annex to the
M&E Framework.[19]
11. Witnesses commended DFID for the process it had
followed in the development of the M&E Framework, commenting
that it had used a "groundbreaking approach in engaging civil
society in the monitoring process […] and in making the framework
more relevant for those involved in the implementation of the
Strategy".[20] At
DFID's request, the UK Consortium on AIDS and International Development
("the UK Consortium") set up an Indicators Working Group
(IWG). The International HIV/AIDS Alliance ("the Alliance")
said that:
Whilst the IWG was asked to focus [on] the development
of indicators, it was able to provide support and expertise to
inform other parts of the framework. DFID's commitment to the
process and the IWG was clearly shown through the continued engagement
of staff and the openness and honesty with which meetings were
conducted […] the discussion between the IWG and DFID led
to a more balanced approach to monitoring and evaluation of the
Strategy. It allowed DFID's efforts to be informed by recognized
good practice and the direct experiences of monitoring HIV responses.
The discussion and joint inputs have resulted in more requests
for qualitative information within the data collection tools,
which will facilitate documentation of good practice for knowledge
sharing and learning.[21]
However, there was some criticism of the process.
The Alliance pointed out that:
The short timeframes for review of draft documents
and provision of feedback, and the application of Chatham House
rules to the IWG proceedings, limited the ability of the IWG to
consult and engage the stakeholders it was representing. From
the outset there appeared to be lack of clarity of the purpose
of the group, with no efforts to agree on Terms of Reference for
the IWG or to clarify its role in the final decisions related
to the selection of indicators. IWG members were not assured endorsement
of the final product.[22]
Alvaro Bermejo, Executive Director of the Alliance,
told us that he "welcomed the initiative at the beginning
in the sense it was very innovative, it was one of the first times
that DFID was really involving civil society in setting up indicators
they were going to use".[23]
But he regretted that "both from civil society and from the
DFID side I think we were unable to see that translated into the
final product and many things slipped in that path. The one thing
we did not really achieve was to get a clear definition of what
success would look like."[24]
12. Witnesses remained concerned about the likely
effectiveness of the monitoring and evaluation process. World
Vision said that, before the publication of the M&E Framework,
there had been "high expectations" that answers to
important questions about how the Strategy would be monitored
would be provided. However, in their view, the question of what
the UK Government's contribution would be towards achieving the
Strategy's goals had yet to be answered:
While the Framework outlines how the collective
progress by the international community will be monitored, it
does not attempt to systematically measure the contribution made
by the UK Government. This impedes monitoring of the Government's
performance and evaluation of the impact of the Strategy on UK
Government policy.[25]
Sally Joss of the UK Consortium had similar reservations:
One of the difficulties with a lot of the present
AIDS strategy is that it is going to be very difficult to attribute
what DFID has done in the harmonised international efforts to
tackle HIV and AIDS and I think it will be very difficult to work
out exactly what DFID has contributed to the general battle against
AIDS.[26]
13. The Minister told us that he appreciated that
monitoring and evaluation was "not an easy task" and
that "complexity does provide us with a test". The AIDS
Strategy and the Baseline assessment were geared at the "strategic
end rather more than at a […] more operational level".
He stressed that DFID was keen to listen to the views of others,
but it did not want to "spend valuable resource measuring
for no benefit". It was necessary to balance "the need
for detailed measurement against using numbers and data that are
already available".[27]
Many witnesses believed that publication of the completed
biennial country overviews would make a significant contribution
to increasing transparency and accountability and would assist
those outside DFID to assess the impact of its HIV/AIDS programmes.
When we pressed DFID officials on their plans for making this
information public, Jerry Ash, Team Leader for AIDS and Reproductive
Health, said that DFID would "seriously consider publishing
the country returns in full".[28]
14. We welcome the innovative approach which DFID
used in drawing up the Monitoring and Evaluation Framework for
its HIV/AIDS Strategy. We look forward to the publication of the
first biennial report on World Aids Day 2010, and expect it to
provide valuable information on progress made by DFID against
its commitments in the Strategy. To further enhance transparency
and accountability, we recommend that DFID publishes, in full,
the completed biennial country overviews of progress against its
priorities for action. This will assist all stakeholders, including
ourselves, in assessing whether DFID is achieving its objectives
for its HIV/AIDS activities.
15. A challenge remains, however, in disaggregating
DFID's contribution from that of other partners in the global
AIDS effort. This is necessary to demonstrate to UK taxpayers
what the UK's substantial funding for HIV/AIDS is achieving. We
recommend that, in response to this Report, DFID provides us with
further information on its plans for measuring the specific contribution
its funding is making to tackling HIV/AIDS.
8 Twelfth Report of Session 2007-08, HIV/AIDS: DFID's
New Strategy, HC 1068-I, para 117 Back
9
ibid, Summary Back
10
ibid, para 112 Back
11
DFID, Achieving Universal Access-Monitoring performance and evaluating
impact, December 2008 Back
12
Achieving Universal Access-Monitoring performance and evaluating
impact, p 4 Back
13
DFID, Achieving Universal Access-a 2008 Baseline, October 2009
Back
14
ibid, p iv Back
15
Q 3 [Mike Podmore] Back
16
Achieving Universal Access-Monitoring performance and evaluating
impact, p 5 Back
17
ibid, p 6 Back
18
DFID, Achieving Universal Access, June 2008, pp 62-66 Back
19
Achieving Universal Access-Monitoring performance and evaluating
impact, pp 16-20 Back
20
Ev 51 Back
21
Ev 51 Back
22
Ev 51 Back
23
Q 2 Back
24
Q 2 Back
25
Ev 84 Back
26
Q 3 [Sally Joss] Back
27
Q 23 Back
28
Q 25 [Jerry Ash] Back
|
