National Audit Office's response to the
memorandum submitted by the Centre for International Public Health
Policy
CONCLUSIONS AND
RECOMMENDATIONS
1. The briefing note dated 18 October 2005
from the Centre for International Public Health Policy (CIPHP)
on the National Audit Office Report "Innovation in the NHS:
Local Improvement Finance Trusts" criticises the methodology
used by the National Audit Office. The note suggests that the
methodology used is based on surveys of stakeholders with a vested
interest in LIFT and that, other than these interviews, there
is no evaluation of value for money or the factors that underpin
it. The note says the National Audit Office Report marks a new
phase in a shift away from quantitative towards qualitative analysis.
The note also says that potential delivery problems arising from
new governance structures are identified in the Report but not
evaluated.
2. The National Audit Office has considered
these points carefully and responds to the methodological issues
made below. This note provides clarification on appropriate elements
of the National Audit Office's fieldwork. It also refers to the
results of further analysis carried out as a result of the hearing
of the Committee of Public Accounts on 17 October 2005.
USE OF
QUALITATIVE METHODOLOGY
3. The National Audit Office attaches importance
to both quantitative and qualitative analysis. In reports on subjects
such as LIFT, where an initiative is at an early stage, there
is room for judgement on how it will develop and whether its ambitions
will be fulfilled. The report carried out both qualitative and
quantitative analysis as set out in paragraphs 14-17 below.
EVALUATION OF
VALUE FOR
MONEY
4. The following methodological points were
made:
(i) There is no comparison of LIFT proposals
against other current or potential financing methods.
(ii) There is no examination of risk transfer.
(iii) There is some evidence that affordability
may be a problem but no analysis.
Comparisons against other methods
5. The National Audit Office Report compared
LIFT to both traditional public capital investment and third party
development (paragraphs 1.6-1.9). The reasons why the final Report
did not quantify the cost of the alternatives and compare them
to the cost of LIFT are expanded on below.
6. There was a lack of information on suitable
comparators procured using alternative mechanisms. The National
Audit Office considered comparing the cost of LIFT to third party
developments and noted the different financing and contract structures
in such developments, together with different whole life cost
profiles. Moreover, analysis of individual leases including rent
review clauses and the extent of further charges such as utility
and insurance services would have been necessary. The National
Audit Office met with several third party developers who were
unwilling to share detailed data.
7. The National Audit Office asked each
PCT and LIFTCo to provide examples of suitable primary care developments
by way of comparison. It found that public finance in the form
of central funding was rarely available and usually only for small
scale refurbishment and redevelopment. At present improvement
grants of up to £100,000 are available so long as there is
a commitment that the development will remain in the NHS for 10
years.
8. The CIPHP briefing paper refers to funding
under the London Improvement Zone (LIZ) which was available until
1999. It cites the example of the Fairfield Grove Health Centre,
Greenwich which was completed in 1996 for a contract value of
£1.5 million. Fairfield Health Centre offers integrated primary
health care facilities for general medical practice, community
services and other primary care practitioners. Its size and the
fact that it was completed nearly 10 years ago suggest that it
is not a good comparator for LIFT.
9. In the light of the interest shown by
the Committee of Public Accounts in such comparisons the National
Audit Office approached the PCTs involved in the case studies
in its report to try again to find meaningful comparators. Information
on a comparable third party development—the Lyng Centre
for Health and Social Care is provided in the supplementary memorandum
to the Committee. The PCTs could not find any comparable centrally
funded developments.
Risk transfer
10. The National Audit Office Report did
not set out details of the risk transfer arrangements in LIFT
as it identified no value for money concerns about the allocation
of risk under the LIFT model.
Affordability
11. The National Audit Office Report discussed
affordability—some tenants were concerned about the rental
costs they faced (paragraph 2.14). Representatives from the National
Pharmaceutical Association, the British Dental Association and
Local Authorities told the National Audit Office that they had
concerns over rental costs. The National Audit Office asked for
relevant background and received qualitative feedback rather than
quantitative data.
12. Wide consultation at the time the NAO
Report was carried out (paragraphs 16 and 17 below) did not reveal
any general affordability issues for PCTs. The business cases
reviewed for the case studies had considered affordability for
the local health economy. After the Report was published, in the
light of experience as developments came on stream, affordability
questions were starting to be raised. During a visit by members
of the Committee of Public Accounts to a LIFT development, arranged
by the National Audit Office after the Report was completed, this
issue was raised by Dr Kohli, a GP in the LIFT development. He
also raised the issue in the 17 October 2005 Committee Session.
As a result, the National Audit Office gathered relevant information,
which it has included in its supplementary memorandum to the Committee.
NEW GOVERNANCE
STRUCTURES
13. The National Audit Office Report found
some potential governance problems which it exposed in the Report
(paragraphs 3.11-3.15). It set out the potential implications
of these but it found no tangible delivery issues at that stage—possibly
because the LIFT schemes were at an early stage.
OTHER ISSUES
14. The National Audit Office's methodology
was set out in detail in the Report to allow scrutiny. The CIPHP
main criticisms are that the Report relies on surveys of interested
parties, it looks at only the first six projects completed and
that the Report is not structured around the four main high level
questions it identified. The independent quality review of the
Report commissioned by the National Audit Office concluded that
a range of appropriate methodologies had been used and gave the
Report a good mark—an average of 4.1 out of five. Further
information on methodology is set out below.
15. Questionnaires were issued to three
main interest groups, principally to gain an understanding of
the adequacy of the standardised procurement process devised by
Partnerships for Health. These were: all LIFT Project Directors;
shortlisted bidders across the 42 LIFT schemes extant at that
time; and the Local Pharmaceutical Committees in each case study
areas.
16. The National Audit Office aimed to obtain
a balanced view of the scheme by consulting widely across parties
with differing views on LIFT, although not all agreed to speak
to the National Audit Office team. As is normal in National Audit
Office studies, the team had access to a panel of experts who
were invited to comment on the audit plan and the structure and
draft of the Report, thereby providing quality assurance. The
panel for the LIFT Report included amongst others a dental practitioner,
a general practitioner and the Chief Executive of a PCT.
17. A key element of the evidence was provided
from analysis of six case studies (Appendix 1, paragraph 8). The
first six schemes to have completed the procurement process were
chosen for practical reasons and because they represented all
three waves of LIFT. These case studies were used to identify
best practice and lessons for future LIFT developments. A wide
range of stakeholders were interviewed in the local health economy
for each case study and key documents were reviewed. The National
Audit Office was aware of concern in some areas about LIFT schemes.
It flagged this up in the Report (paragraph 2.4) but the projects
were not advanced enough to be analysed as case studies.
18. The CIPHP note suggests that the National
Audit Office team was unaware of Department of Health guidance
"Learning from Post Project Evaluation" although this
guidance is referred to in paragraph 3.8 of the Report. This paragraph
of the National Audit Office Report went on to say "there
is no clear guidance recommending either its nature or its timing"
in relation to the post project evaluation of LIFT. This latter
reference relates to the National Audit Office's concern that
the Department and Partnerships for Health should issue detailed
guidance—as referred to in recommendation 9 of the Report.
| 
