LESSONS
TO BE
LEARNED FOR
FUTURE MAJOR
INFORMATION TECHNOLOGY
PROJECTS
5. In his report, the Comptroller and Auditor General
identified various factors that contributed to the cancellation
of the Benefits Payment Card. In his view, the important reasons
were insufficient time for specifying the requirement and piloting;
the lack of a shared, open approach to risk management; and divided
control.[4]
6. The Comptroller and Auditor General reported that,
as a groundbreaking PFI project in the information technology
sector, there was little by way of precedent to provide guidance.
There was limited experience at the time as to the appetite and
ability of the purchasers or potential suppliers to accept important
risks, such as the liability for failing to prevent fraud. There
was also a perception that, because responsibility for delivery
could be transferred to suppliers, purchasers should be less concerned
with validating the supplier's internal arrangements and had less
need to know the detail of the supplier's solution.[5]
7. While Pathway had submitted narrowly the cheapest
bid, they ranked third of three bidders on eight of eleven technical
and management criteria. A decisive factor in their selection
was their greater acceptance of risk, making their bid the only
one that was PFI compliant. Pathway took on the liability to pay
up to £200 million for losses if their system failed to operate
or prevent fraud, at a cost of £20 million in their bid.
The other bidders priced this potential liability pound for pound.
The choice the purchasers felt they had was therefore either to
accept the Pathway bid or not proceed with the project at all.
In the event, Pathway lost £180 million in development costs
through cancellation, but the Department and the taxpayer bore
the impact of continued losses through benefit fraud.[6]
8. The Comptroller and Auditor General drew six key
lessons from the project about the procurement of complex information
technology projects (Figure 2).[7]
Figure 2: The procurement of complex IT systems
|
1 |
There is often understandable pressure on purchasers and potential suppliers to conclude a deal and to seize as soon as possible the benefits of the project. But it is never acceptable to sign a contract with fundamental "agreements to agree" the detail of the service in the future, even if as in this case, they are intended to be resolved quickly. Allowing realistic timescales for early planning and detailed specification will pay dividends in terms of overall project delivery and cost.
|
2 |
Departments undertaking IT procurement projects should fully understand the quality and quantity of resources available which actually will be committed by the supplier to deliver the agreed services. This is particularly important where new software development is required. It should be agreed during the competitive process how resource requirements can be achieved and measured, and the agreement should be drafted into the contract
|
3 |
For major, mission-critical, tailored and bespoke projects, there should be proper piloting of technical solutions to address the full service requirement, rather than reliance on part-functional demonstrations. Departments may have to consider part-funding such pilots and should also consider awarding separate contracts for the design and development of systems before contracting with the developer for full implementation of the successful pilot. This approach also allows keener pricing of the later service implementation and operation stages by suppliers because the risks to them are reduced.
|
4 |
There must be agreement between purchasers and suppliers at the outset of information technology projects on the extent to which new systems will either replicate the purchasers' existing systems, or re-engineer and simplify them.
|
5 |
After examining the scope to simplify their business processes, and given certainty as to the detailed requirement, Departments should examine with potential suppliers the scope to use generic and widely used system components where available. This process may in turn suggest modifying the initially proposed solution. A major risk of the Benefits Payment Card elements of the project turned out to be their "bespoke" nature. Building bespoke systems adds to the development costs and the risks of any solution.
|
6 |
Where there are major project developments which involve more than one system being developed in parallel, as was the case here with the Benefit card, the Department's Customer and Accounting Payment System and new Post Office systems, it is sensible to plan and monitor these jointly.
|
9. The Department and the Post Office saw the use
of PFI, and the way it was perceived at the time, as a major factor
in the problems experienced on the project. This was one of the
first PFI contracts. Everyone in Government and the trade was
trying to find new ways of getting investment into projects. The
philosophy then was for purchasers to design their business requirements
at a high level, and to allow the contractor to create the solution,
to innovate, without constraining them with detail. The expectation
was that the risk would then be passed to the supplier. It was
only after the contract had been placed, and there was a fixed
price and delivery date, that the project team and the contractor
worked through the detailed business requirements and the technical
design. It was at that point that the difficulties began to emerge.[8]
10. The Department acknowledged that, with hindsight,
the timetable for the project had been very over-optimistic. The
people involved at the time had seriously underestimated the scale
and complexity of the job. Everyone had been under pressure to
move the project along as fast as possible. The Department had
wanted the fraud savings; the Post Office had wanted automation;
and ICL had commercial reasons.[9]
11. ICL added that people had been looking for quick
results, and a lot of assumptions had been made by all three parties
as to what they were going to get out of the deal. For their part,
they had accepted an aggressive timescale on the assumption that
there was a mood for compromise on some of the 289 requirements
that had not been agreed at the contract stage. They had been
encouraged to think about innovation and enterprise, and expected
room for manoeuvre. The Department had attempted to simplify the
benefit payment process in order to ease implementation, for example
by streamlining arrangements for agents to collect benefit on
claimant's behalf. But ICL and the Department found many things
that were fixed, enshrined in various regulations.[10]
12. The Comptroller and Auditor General noted that,
at various stages of the project, each of the parties produced
registers of the key risks that needed to be managed, but these
registers did not always assess probability, allocate risks to
owners, or propose options for managing risks. Some risks were
not addressed sufficiently, for example the risks and costs of
slippage. Others were downgraded, because risk was seen to rest
with the supplier, even though the impact would be on the Department,
for example project slippage would delay delivery of anticipated
fraud savings. In addition, a shared, open approach to risk management
across the whole programme was not achieved. The Comptroller and
Auditor General drew out 6 key lessons to be learned from this
project about risk management (Figure 3).[11]
Figure 3: Lessons On Risk Management
|
1 |
For all projects, purchasers should maintain from the start of the procurement stage an assessment of the inherent risk of late delivery, and analyse before signing contracts the sensitivity of their business cases to major slippage and cost overrun.
|
2 |
Risks identified should be registered, assessed for impact and probability, assigned to a risk manager and used as a basis for subsequent management and contingency planning. Closed risks should be retained in a closed risk register and reviewed at regular intervals for "re-incarnation". Risk identification must be an ongoing activity, as new risks will occur throughout projects
|
3 |
Departments should appoint a permanent "risk scrutineer", independent of the project team and ad hoc input from consultants, to monitor how the project is handling risks and to report to senior management at regular intervals. This is a feature of the PRINCE 2 project management system widely used in government and in the private sector.
|
4 |
Contracts with suppliers, including Private Finance contracts, require detail and clarity about the reporting obligations of suppliers to support risk management and contingency planning by the purchaser. Contractual obligations must be underpinned by a recognition on all sides of the need for openness, extending beyond oral reporting to sharing their risk management documentation.
|
5 |
The project illustrates the importance of being able to clarify, quantify and allocate responsibility for risk very clearly if the Private Finance approach is to be a suitable contractual model. In the case of IT development projects in the public sector, this is particularly difficult. Ministers and officials cannot transfer responsibility for the overall service for which they are legally responsible and accountable to Parliament. Some risks, such as the delivery of benefits payments, on which many people depend, are too great for private sector suppliers to absorb and departments therefore must retain a direct interest and involvement in how the service is to be delivered.
|
6 |
It is vital that all bidders, and if necessary their parent companies, are clear about the extent of risk transfer proposed by the purchasers at the start of procurement rather than towards the end. Purchasers must ensure that the extent of risk transfer they propose is viable, and must evaluate the extent of risk that they retain. Difficulties in this area can result in the loss of otherwise valid bids.
|
13. The Department accepted that, because the parties
had not fully appreciated the real scale and complexity of the
project, and because there had been a misunderstanding about how
a PFI contract would work, some issues had not been identified
as major risks early on. But following procurement, and as the
scale of the project became apparent, their risk management methodology
started to spot these significant risks. It was also in the nature
of risk management that risks could arise at any stage, could
increase during a project, and could be taken off the risk register
and re-emerge later on, and its was part of their methodology
that they kept all risks under review. For example, at the procurement
stage they had identified the need to provide adequate security
as a risk. They had accepted Pathway's high level proposal and
had downgraded the risk to medium; but as they started to develop
the detail, the difficulties became clear and the risk was increased
to high. ICL confirmed that the devil was in the detail.[12]
14. As regards the sharing of data on risks, the
Department noted that risks were regularly discussed at project
boards and steering committees. However, they had not shared information
in risk registers as well as they should have. Again, they noted
that the view taken of PFI, about transferring risk to the people
best able to manage them, had been very much about leaving the
contractor to manage the risks from then on. The way PFI was being
interpreted at that time got in the way of collaborative work.[13]
15. The Comptroller and Auditor General concluded
that another important factor in the difficulties experienced
on the project, was that the Department and the Post Office each
had different aims, and different primary drivers. He drew 2 key
lessons for future procurement by more than one purchaser (Figure
4).[14]
Figure 4: Procurement by more than one purchaser
|
1 |
Joint procurement is always difficult, especially where purchasers have divergent objectives. It is better to let one purchaser take the lead with proper arrangements for information flow and reporting to the other. This requires a clear agreement, embodied in the contractual arrangements as well as in a memorandum of understanding, as to roles and responsibilities.
|
2 |
Incentives to deliver should pull the same way for both parties to a project: for example, financial and timetable incentives should be mutually supportive: and the parties should agree common objectives and "must-haves" at the outset, as these will influence future behaviour.
|
16. The Department confirmed that they were interested
in a fraud-free means of payment, which put an emphasis on controls
to ensure the secure handling of benefit payments by post offices.
On the other hand, the Post Office wanted to press ahead with
developing their services and secure throughput of customers in
local post offices, which would best be achieved by the speedier
handling of claimants. Together they had put together a programme
development authority to try to minimise conflicts of interest
as far as they could. However, the Department accepted that when
things were going well, those differences perhaps did not matter,
but when things started to go wrong, and there were difficult
choices to be made, those different priorities created difficulties.[15]
17. Since then, many lessons had been learned about
handling PFI for IT projects, and in early 2000 the Treasury had
issued guidance which drew on the lessons outlined in the Comptroller
and Auditor General's report. In addition to this, and the report
by Mr Ian McCartney, Minister of State at the Cabinet Office on
Major Government IT Projects, there was now additional expertise
available within the Treasury, and the Office of Government Commerce
to help Departments.[16]
18. The Department thought it inconceivable that
nowadays such a project would be tackled in one big bang. It would
be broken down into more manageable chunks. They would adopt a
phased acceptance programme. Under the new arrangements introduced
following the McCartney report, projects would now be subject
to independent reviews at key stages (the Gateway process).
In future that would ensure for example that sufficient effort
went into developing the technical design of the system before
contract for implementation. The Department also recognised that
for future projects of this nature there had to be a more collaborative,
partnership approach to solve problems as they arose, and they
would have a much more shared and open approach to risk. They
had taken a different approach to PFI in their new modernisation
projects, such as Child Support reform.[17]
19. ICL confirmed that the new guidelines very much
reflected their experience. There needed to be much greater clarity
at the outset about requirements, particularly where there are
two purchasers and complex legacy systems which are vital to the
smooth running of society. They supported the notion of an initial
phase of defining requirements and solutions, before going forward
to PFI delivery. They also preferred a more joined-up approach,
rather than operating in separate silos and dealing at the boundary
which did not work with such complex end-to-end systems.[18]
20. The Comptroller and Auditor General noted that
the contract was placed in May 1996 for delivery by 1999. Despite
a re-plan of the project in February 1997, the timetable for delivery
slipped continually, the benefits were delayed, and in late 1997
Pathway requested improved terms. In November 1997, the Department
took steps to preserve their legal rights to cancel the project.
In March 1998, Ministers set up an interdepartmental working group
to assess whether the project was technically viable, if so how
quickly it could be completed and at what cost to the government;
as well as the costs of cancellation and of any alternative available
to deliver the project's objectives. Because no decision had yet
been reached, the Secretary of State twice gave the Accounting
Officer of the Benefits Agency formal instructions not to terminate
the contract. The Government formally announced cancellation of
the Benefits Payment Card element of the project on 24th
May 1999.[19]
21. The Department said that the length of time it
had taken from November 1997 to May 1999, to devise a way forward
reflected the importance of the issues at stake. Cancellation
would affect every post office and 22 million potential benefit
claimants, as well as the future of the Post Office and its automation
and measures to reduce fraud. In addition, these issues involved
the Department for Trade and Industry, as sponsor of the Post
Office, and the Treasury. And there were commercial issues for
ICL, which was coming up to flotation. There had been a long process
of different kinds of review, inter-departmental groups, and a
long period of negotiation trying to find a way forward. There
had also been various changes of Ministers along the way. In the
Department's view these factors justified such a lengthy decision
making period.[20]
Conclusions
22. All the parties seriously underestimated the
complexity of the project and the risk of attempting to tackle
such a challenge in one go using relatively untried PFI arrangements.
They failed to achieve a shared understanding of what was to be
delivered and how, and the barriers to innovation contained in
benefit rules. The other major factors included inadequate contracting
and project management skills, and lack of clear ownership of
project delivery and risk management. These mirror many of the
failings referred to in our predecessor's January 2000 report
on lessons to be drawn from IT projects.
23. Pathway was selected because they were willing
to take on a level of risk for preventing benefit fraud which
the other two bidders would not accept, even though Pathway came
third on 8 out of 11 technical and management criteria. At the
time, there was a mistaken view that PFI bidders should compete
on the level of risk they were prepared to take on, rather than
achieving an optimum allocation of risk.
24. The various parties identified many of the risks
at various stages, but did not always share this information.
Risks were "cleared" without justification, and "cleared"
risks were not well monitored and so re-emerged.
25. Joined-up projects involving more than one purchaser
carry extra risks, and steps need to be taken to establish clear
leadership in purchasing and project management, and transparency
of information between the various stakeholders. The Department
and the Post Office took some steps in this direction during the
project, by revising the project management arrangements. The
McCartney report recommends one single responsible owner for each
major project in future; but the Benefits Payment Card also demonstrates
the delays and extra costs that can emerge when decisions on the
future of a project involve many stakeholders with conflicting
objectives, in this case, for example the Department, the Department
of Trade and Industry, the Treasury and the Post Office.
4 C&AG's report
(HC 857, Session 1999-2000), para 29 Back
5 ibid, para 15 Back
6 Qs 141-144 and C&AG's
report (HC 857, Session 1999-2000), paras 25-26, 1.36 and
3.46 and pp 14-15 Back
7 C&AG's report (HC 857,
Session 1999-2000) pp 14-15 Back
8 Qs 1, 20-23, 26-30, 60, 116-123 Back
9 Qs 1, 18-19, 24-25, 135-136 Back
10 Qs 22, 26, 107-116 and Evidence,
Appendix 2, p22 Back
11 C&AG's report (HC 857,
Session 1999-2000), paras 18-20, 27-29 and pp 13-14 Back
12 Qs 1, 19-25, 31-32, 43-51 Back
13 Qs 52-55, 124, 151-154 Back
14 C&AG's report (HC 857,
Session 1999-2000) paras 16-17 and p15 Back
15 Qs 3, 81-85 Back
16 Qs 1, 5-7 and Successful
IT: Modernising Government in Action-Review of Major Government
IT Projects, May 2000 Back
17 Qs 1, 56-57, 122-124, 170-172 Back
18 Qs 7, 55 Back
19 C&AG's report (HC 857,
Session 1999-2000) paras 1 and 1.11-1.29 Back
20 Q2, 33-36, 42, 168-169 Back
|