Supplementary memorandum submitted by
the National Audit Office
After the hearing, Mr Richard Bacon submitted
a series of supplementary questions to the National Audit Office,
what follows is their response.
Q1. How many gateway reviews did the National
Audit Office see in relation to the National Programme for IT
in the NHS?
A1. The National Audit Office saw just under
30 Gateway reviews in relation to the National Programme for IT
in the NHS.
Q3. Why were the contents and some results
of gateway reviews mentioned in some detail in recent National
Audit Office reports on the Child Support Agency and on the ASPIRE
contract for HM Revenue and Customs, but not for the National
Audit Office's report on the National Programme for IT in the
NHS?
A3. The National Audit Office's aim in preparing
the report was to provide a reasonably succinct overview of the
National Programme overall. As with other reports, we have to
consider the competing merits of the information at our disposal.
In this instance, we did not consider it appropriate to bring
into the report the recommendations made through such a large
number of Gateway reviews. As the question indicates, we do sometimes
cover detailed results from Gateway reviews in our reports, but
this is by no means always the case.
Q4. Before producing its report on the NPfIT
(pursuant to Caroline Flint's parliamentary answer of 24 May 2006):
(i) Did the National Audit Office see
the results of internal audits on the National Programme for IT
in the NHS?;
A4. (i) Yes.
Q5. How many independent reviews have been
commissioned under contract (pursuant to Caroline Flint's parliamentary
answer of 24 May 2006) such as:
(i) The study by McKinsey to inform the
approach before the start of the programme;
(ii) from other suppliers to establish
the value for the National Health Service and taxpayer achieved
through the contracts; and
(iii) any other studies? And
(iv) before producing its report on the
NPfIT did the National Audit Office see the results of these independent
reviews? And if so,
(vi) what were the key results of each study?
A5. National Audit Office saw a number of
independent reviews commissioned by the Department as it conducted
the study, including the McKinsey review. Many of these are referred
to in the C&AG's report:
— Paragraph 3.2 refers to a report
commissioned from IT industry analyst Ovum, which compared the
prices achieved by NHS Connecting for Health with estimates of
the prices that could have been achieved by individual NHS organisations
purchasing the same services separately.
— Paragraph 3.20 refers to a review
commissioned in February 2005 jointly by NHS Connecting for Health
and BT which confirmed that the overall technical architecture
deployed by BT was sound and capable of supporting NHS Connecting
for Health's requirements.
— Paragraph 4.10 refers to the research
commissioned by NHS Connecting for Health from Ipsos MORI to track
awareness and understanding of the Programme across the NHS.
(Transferred to the Department of Health the
request for key results from each of these studies)
Q6. What is the most recent figure for total
expenditure on central administration of the NPfIT since 2002,
and does the National Audit Office consider this value for money
so far?
A6. (Transfer the request for the most recent
figure for total expenditure on central administration of NPfIT
to the Department of Health). Paragraph 4 on page 1 of the C&AG's
report states that "it will be some time before it is possible
fully to assess the value for money of the Programme, as this
will depend on the progress made in developing and using the systems
it is intended to provide". In the same way, the National
Audit Office is not in a position to conclude whether or not central
administration expenditure is value for money so far. The National
Audit Office can only judge the value for money of the National
Programme when there is a sufficient body of evidence of benefits
being achieved for patients.
Q8. Please provide a breakdown of the figures
referred to in paragraphs 1.20 to 1.25 of the National Audit Office
report showing how one arrives at a total of £9.2 billion
for forecast national expenditure
A8. The breakdown of forecast national expenditure
is shown in paragraph 1.20-1.25 of the C&AG's report. The
sum of the figures in paragraphs 1.20 to 1.25 is actually £9.078
billion, which rounded is £9.1 billion. (There is a typographical
error in the heading above paragraph 1.20, which give the figure
of £9.2 billion).
Q9. What was the total cost of the National
Audit Office report on the National Programme for IT in the NHS?
A9. The total cost of the National Audit
Office study was £578,000 (including £72,000 consultancy).
This is higher than the average cost of a National Audit Office
VFM study, but this reflects the scale and complexity of the subject.
For example, each of the ten major contracts is comparable in
size to a major PFI deal. The investigation also required interviews
and visits with more than 30 organisations across England, including
the programme regional clusters, suppliers, NHS Trusts, professional,
regulatory and academic bodies, and trade unions. The body of
knowledge built up on this study also leaves the National Audit
Office well placed to monitor further developments on the National
Programme.
Q10. Paragraph 1.26 of the National Audit
Office report refers to forecasts of local expenditure (made in
the investment appraisals carried out at the time of the award
of the main LSP contracts in late 2003 and early 2004) which total
£2.6 billion excluding PACS. Did the National Audit Office
subject the figure of £2.6 billion referred to in paragraph
1.26 to independent verification?
Q11. Is the National
Audit Office satisfied that the figure of £2.6 billion represents
a realistic estimate of total forecast local expenditure?
A10-A11. The National Audit Office did not subject
the figure of £2.6 billion to independent verification. As
is described in the report, this was the figure used in the Department's
investment appraisals carried out at the time of the award of
the main LSP contracts in late 2003 and early 2004. That is why
the National Audit Office report made the recommendation (recommendation
(e) page 7) that: "the Department, NHS Connecting for Health
and the NHS should commission a study to measure the impact of
the Programme on local NHS IT expenditure—both costs and
savings—where systems are now being deployed, and, together
with its quantification of financial and non-financial benefits
(recommendation (d)), use this to provide an up to date assessment
of the overall investment case for the Programme."
Q12. Given the significant changes in the
contracts reflected in the increase in forecast national expenditure
from £6.2 billion to £9.2 billion, does the National
Audit Office believe that the figure of £2.6 billion for
forecast local expenditure is still correct?
A12. Paragraph 1.26 of the National Audit
Office notes that in addition to the £2.6 billion a further
£775 million in local expenditure was estimated as being
needed as a result of the extension of the central contracts to
include PACS. This is the basis of the total forecast local expenditure
of £3.4 billion referred to in the heading to paragraph 1.26.
The National Audit Office recommended that the Department, NHS
Connecting for Health and the NHS should commission a study to
measure the impact of the Programme on local NHS IT expenditure
(recommendation (e) on page 7 of the C&AG's report).
Q14. What are the national costs of PACS?
A13-A14. The contracted central expenditure
on PACS is £245 million for the central data store over 10
years to 2013-14 (paragraph 1.20 page 25 and Figure 3 Page 17).
These contracts would involve local IT spending of an estimated
£775 million, to be paid for by local NHS Trusts (paragraph
1.26 and Figure 3 page 17).
(Otherwise, transfer to the Department of Health)
Q16. What assessment has the National Audit
Office made of the statement by Connecting for Health that: "It
is generally accepted in the IT industry that implementation costs
are some 3-5 times the cost of procurements. That is reflected
in the business case that was made for the National Programme"
A16. This is a statement attributed to NHS
Connecting for Health in response to media enquiries in October
2004. It was not considered as part of the National Audit Office
analysis.
Q17. What is the source for the statement
in paragraph 4.8 of the National Audit Office report that: "The
programme is the only major public sector IT project in the UK
where the responsible body has a dedicated website to provide
information on the progress of the project"?
A17. We are aware from our research that
there are other sites providing information on aspects of public
sector IT projects, such as, the Criminal Justice IT Programme.
However such websites are not a common feature of UK public sector
IT initiatives and so far as we are aware none provides as much
information as the NPfIT website.
Q18. In the National Audit Office report,
Patient Choice at the point of GP Referral, the National Audit
Office expressed concerns about the costs of interim systems that
will have to be replaced by full systems at a later date. Does
the National Audit Office have similar concerns about the costs
of interim systems being deployed in the NPfIT because of the
late delivery of core software?
A18. National Audit Office recommendation
9a on Page 6 of the National Audit Office NPfIT report highlights
the importance of ensuring a robust, engineering-based timetable
for delivery which NHS Connecting for Health is confident its
suppliers are capable of achieving.
Recommendation 9c on Page 6 recognises that
whilst some adjustment of suppliers' delivery milestones may be
a necessary response to suppliers' difficulties in delivering,
NHS Connecting for Health should not allow this to compromise
the eventual achievement of the vision of the fully integrated
care record service that was the objective of the Programme at
its inception.
Assessing the net impact of deploying interim
systems is complicated because both costs and benefits are affected.
We would expect the assessment of local costs and benefits we
recommend (recommendations 9e) to cover these issues.
Q19. What concerns does the National Audit Office
have about the long timeframe for the NPfIT, given that the NHS
and its demands for IT could change substantially between now,
when systems are being defined, and the time they are deployed?
A19. The NPfIT is a ten year programme.
Just as the last ten years have seen large changes in the NHS,
for example in organisation, medical techniques and information
technology, so further changes should be expected in the future,
even if the precise form of these changes cannot now be fully
foreseen. Paragraph 3.10 identifies the mechanisms put in place
by NHS Connecting for Health to help ensure continuing value for
money over the life of the contracts, such as ensuring technology
is continuously improved and refreshed so that systems continue
to meet the changing needs of the NHS throughout the contract
periods. The contracts with suppliers also include mechanisms
for incorporating changes in the NHS's requirements. It will be
important that NHS Connecting for Health uses such mechanisms
to make sure that the Programme adapts to the changing nature
and needs of the NHS.
Q20. Given the apparently long timeframes, at
what point will the National Audit Office judge whether the programme
has been value for money?
A20. Paragraph 4 of the C&AG's report
explains that "the main implementation phase of the Programme
and realisation of benefits is mainly a matter for the future
and it will therefore be some time before it is possible fully
to assess the value for money of the Programme, as this will depend
on the progress made in developing and using the systems it is
intended to provide." The National Audit Office will
be in a position to judge whether the Programme has been value
for money when there is a sufficient body of evidence on patient
benefits from the Programme.
Q21. In paragraph 3.5 of the National Audit
Office report the National Audit Office praises the work done
on requiring suppliers to produce a "Proof of Solution".
Why does the National Audit Office believe this good work has
not led to core systems being delivered on time?
A21. As reported in paragraph 3.5, "Proof
of Solution" tests were carried out in a simulated environment
to show whether potential suppliers' systems could meet a number
of scenarios devised by NHS Connecting for Health. They are helpful
because they provided a degree of confidence that the potential
contractors understood NHS Connecting for Health's requirements.
They therefore help to reduce risk but of course cannot on their
own eliminate it.
The main factors that have contributed to delays
in delivery are (i) additions and changes to the scope of the
Programme (paragraph 1.7), (ii) difficulties by some suppliers
in meeting agreed delivery dates (paragraph 1.11), (iii) a need
to pilot more extensively the clinical record before full scale
implementation (paragraph 1.11), (iv) system integration being
highly complex and taking longer than suppliers initially planned
(paragraph 1.9), and (v) a move away from the original structure
of phases and releases, with new functionality being delivered
in a different order from that originally envisaged (paragraph
1.11 and Figure 2 on page 12).
Q22-23. Before producing its report on the
NPfIT, did the National Audit Office see the document Delivering
21st Century IT Support for the NHS: National Strategic Programme?
If so, did the National Audit Office see:
(i) the version with four appendices including
Appendix 3 with the Project Profile Model completed by Sir John
Pattison showing a risk score of 53 out of a maximum possible
72 and estimated whole life costs of £5 billion; or
(ii) the version with only three appendices
from which the Project Profile Model with its risk scoring and
estimated whole life costs had been removed?
Q23. If the National Audit Office did see
the version including the Project Profile Model
(i) Why was no reference made to the PPM
scoring in the National Audit Office report?
(ii) What is the National Audit Office's
assessment of the PPM scoring broken down by category; and in
total?
A22-23. The National Audit Office saw the
published version of Delivering 21st Century IT Support for
the NHS: National Strategic Programme with three Appendices.
Whilst there may have been earlier drafts of this document, this
is the only substantive version definitively representing the
Department's views.
Q28. Why are there no detailed references
in the report to the numerous technical difficulties encountered
by users of systems deployed under the contracts?
A28. Paragraph 4 of the Summary to the C&AG's
report sets out the scope of the National Audit Office examination:
"we examined the progress being made in delivering the systems
against the original plans and the costs of the Programme; the
steps taken by the Department, NHS Connecting for Health and the
NHS to deliver the Programme; how the IT systems have been procured;
and how the NHS is preparing to use the systems delivered."
This National Audit Office study did not, therefore, include within
in its scope an examination of the technical difficulties that
users of deployed systems may have encountered.
The reason why we drew the scope of the report
in this way was that we considered it was too early to examine
whether, for each project in the Programme, these were signs of
insoluble problems from which the National Audit Office could
draw far reaching conclusions.
Q29. Why did the National Audit Office report
the July 2005 Mori poll and not the February 2006 Mori poll?
A29. At the time of finalising our report,
the analysis of the February 2006 Mori poll was not complete.
| 
