House of Commons - Public Accounts

Select Committee on Public Accounts Third Report

LESSONS TO BE LEARNED FOR FUTURE MAJOR INFORMATION TECHNOLOGY PROJECTS

5. In his report, the Comptroller and Auditor General identified various factors that contributed to the cancellation of the Benefits Payment Card. In his view, the important reasons were insufficient time for specifying the requirement and piloting; the lack of a shared, open approach to risk management; and divided control.[4]

6. The Comptroller and Auditor General reported that, as a groundbreaking PFI project in the information technology sector, there was little by way of precedent to provide guidance. There was limited experience at the time as to the appetite and ability of the purchasers or potential suppliers to accept important risks, such as the liability for failing to prevent fraud. There was also a perception that, because responsibility for delivery could be transferred to suppliers, purchasers should be less concerned with validating the supplier's internal arrangements and had less need to know the detail of the supplier's solution.[5]

7. While Pathway had submitted narrowly the cheapest bid, they ranked third of three bidders on eight of eleven technical and management criteria. A decisive factor in their selection was their greater acceptance of risk, making their bid the only one that was PFI compliant. Pathway took on the liability to pay up to £200 million for losses if their system failed to operate or prevent fraud, at a cost of £20 million in their bid. The other bidders priced this potential liability pound for pound. The choice the purchasers felt they had was therefore either to accept the Pathway bid or not proceed with the project at all. In the event, Pathway lost £180 million in development costs through cancellation, but the Department and the taxpayer bore the impact of continued losses through benefit fraud.[6]

8. The Comptroller and Auditor General drew six key lessons from the project about the procurement of complex information technology projects (Figure 2).[7]

Figure 2: The procurement of complex IT systems

1
There is often understandable pressure on purchasers and potential suppliers to conclude a deal and to seize as soon as possible the benefits of the project. But it is never acceptable to sign a contract with fundamental "agreements to agree" the detail of the service in the future, even if as in this case, they are intended to be resolved quickly. Allowing realistic timescales for early planning and detailed specification will pay dividends in terms of overall project delivery and cost.

2
Departments undertaking IT procurement projects should fully understand the quality and quantity of resources available which actually will be committed by the supplier to deliver the agreed services. This is particularly important where new software development is required. It should be agreed during the competitive process how resource requirements can be achieved and measured, and the agreement should be drafted into the contract

3
For major, mission-critical, tailored and bespoke projects, there should be proper piloting of technical solutions to address the full service requirement, rather than reliance on part-functional demonstrations. Departments may have to consider part-funding such pilots and should also consider awarding separate contracts for the design and development of systems before contracting with the developer for full implementation of the successful pilot. This approach also allows keener pricing of the later service implementation and operation stages by suppliers because the risks to them are reduced.

4
There must be agreement between purchasers and suppliers at the outset of information technology projects on the extent to which new systems will either replicate the purchasers' existing systems, or re-engineer and simplify them.

5
After examining the scope to simplify their business processes, and given certainty as to the detailed requirement, Departments should examine with potential suppliers the scope to use generic and widely used system components where available. This process may in turn suggest modifying the initially proposed solution. A major risk of the Benefits Payment Card elements of the project turned out to be their "bespoke" nature. Building bespoke systems adds to the development costs and the risks of any solution.

6
Where there are major project developments which involve more than one system being developed in parallel, as was the case here with the Benefit card, the Department's Customer and Accounting Payment System and new Post Office systems, it is sensible to plan and monitor these jointly.

9. The Department and the Post Office saw the use of PFI, and the way it was perceived at the time, as a major factor in the problems experienced on the project. This was one of the first PFI contracts. Everyone in Government and the trade was trying to find new ways of getting investment into projects. The philosophy then was for purchasers to design their business requirements at a high level, and to allow the contractor to create the solution, to innovate, without constraining them with detail. The expectation was that the risk would then be passed to the supplier. It was only after the contract had been placed, and there was a fixed price and delivery date, that the project team and the contractor worked through the detailed business requirements and the technical design. It was at that point that the difficulties began to emerge.[8]

10. The Department acknowledged that, with hindsight, the timetable for the project had been very over-optimistic. The people involved at the time had seriously underestimated the scale and complexity of the job. Everyone had been under pressure to move the project along as fast as possible. The Department had wanted the fraud savings; the Post Office had wanted automation; and ICL had commercial reasons.[9]

11. ICL added that people had been looking for quick results, and a lot of assumptions had been made by all three parties as to what they were going to get out of the deal. For their part, they had accepted an aggressive timescale on the assumption that there was a mood for compromise on some of the 289 requirements that had not been agreed at the contract stage. They had been encouraged to think about innovation and enterprise, and expected room for manoeuvre. The Department had attempted to simplify the benefit payment process in order to ease implementation, for example by streamlining arrangements for agents to collect benefit on claimant's behalf. But ICL and the Department found many things that were fixed, enshrined in various regulations.[10]

12. The Comptroller and Auditor General noted that, at various stages of the project, each of the parties produced registers of the key risks that needed to be managed, but these registers did not always assess probability, allocate risks to owners, or propose options for managing risks. Some risks were not addressed sufficiently, for example the risks and costs of slippage. Others were downgraded, because risk was seen to rest with the supplier, even though the impact would be on the Department, for example project slippage would delay delivery of anticipated fraud savings. In addition, a shared, open approach to risk management across the whole programme was not achieved. The Comptroller and Auditor General drew out 6 key lessons to be learned from this project about risk management (Figure 3).[11]

Figure 3: Lessons On Risk Management

1

For all projects, purchasers should maintain from the start of the procurement stage an assessment of the inherent risk of late delivery, and analyse before signing contracts the sensitivity of their business cases to major slippage and cost overrun.

2
Risks identified should be registered, assessed for impact and probability, assigned to a risk manager and used as a basis for subsequent management and contingency planning. Closed risks should be retained in a closed risk register and reviewed at regular intervals for "re-incarnation". Risk identification must be an ongoing activity, as new risks will occur throughout projects

3
Departments should appoint a permanent "risk scrutineer", independent of the project team and ad hoc input from consultants, to monitor how the project is handling risks and to report to senior management at regular intervals. This is a feature of the PRINCE 2 project management system widely used in government and in the private sector.

4
Contracts with suppliers, including Private Finance contracts, require detail and clarity about the reporting obligations of suppliers to support risk management and contingency planning by the purchaser. Contractual obligations must be underpinned by a recognition on all sides of the need for openness, extending beyond oral reporting to sharing their risk management documentation.

5
The project illustrates the importance of being able to clarify, quantify and allocate responsibility for risk very clearly if the Private Finance approach is to be a suitable contractual model. In the case of IT development projects in the public sector, this is particularly difficult. Ministers and officials cannot transfer responsibility for the overall service for which they are legally responsible and accountable to Parliament. Some risks, such as the delivery of benefits payments, on which many people depend, are too great for private sector suppliers to absorb and departments therefore must retain a direct interest and involvement in how the service is to be delivered.

6
It is vital that all bidders, and if necessary their parent companies, are clear about the extent of risk transfer proposed by the purchasers at the start of procurement rather than towards the end. Purchasers must ensure that the extent of risk transfer they propose is viable, and must evaluate the extent of risk that they retain. Difficulties in this area can result in the loss of otherwise valid bids.

13. The Department accepted that, because the parties had not fully appreciated the real scale and complexity of the project, and because there had been a misunderstanding about how a PFI contract would work, some issues had not been identified as major risks early on. But following procurement, and as the scale of the project became apparent, their risk management methodology started to spot these significant risks. It was also in the nature of risk management that risks could arise at any stage, could increase during a project, and could be taken off the risk register and re-emerge later on, and its was part of their methodology that they kept all risks under review. For example, at the procurement stage they had identified the need to provide adequate security as a risk. They had accepted Pathway's high level proposal and had downgraded the risk to medium; but as they started to develop the detail, the difficulties became clear and the risk was increased to high. ICL confirmed that the devil was in the detail.[12]

14. As regards the sharing of data on risks, the Department noted that risks were regularly discussed at project boards and steering committees. However, they had not shared information in risk registers as well as they should have. Again, they noted that the view taken of PFI, about transferring risk to the people best able to manage them, had been very much about leaving the contractor to manage the risks from then on. The way PFI was being interpreted at that time got in the way of collaborative work.[13]

15. The Comptroller and Auditor General concluded that another important factor in the difficulties experienced on the project, was that the Department and the Post Office each had different aims, and different primary drivers. He drew 2 key lessons for future procurement by more than one purchaser (Figure 4).[14]

Figure 4: Procurement by more than one purchaser

1

Joint procurement is always difficult, especially where purchasers have divergent objectives. It is better to let one purchaser take the lead with proper arrangements for information flow and reporting to the other. This requires a clear agreement, embodied in the contractual arrangements as well as in a memorandum of understanding, as to roles and responsibilities.

2

Incentives to deliver should pull the same way for both parties to a project: for example, financial and timetable incentives should be mutually supportive: and the parties should agree common objectives and "must-haves" at the outset, as these will influence future behaviour.

16. The Department confirmed that they were interested in a fraud-free means of payment, which put an emphasis on controls to ensure the secure handling of benefit payments by post offices. On the other hand, the Post Office wanted to press ahead with developing their services and secure throughput of customers in local post offices, which would best be achieved by the speedier handling of claimants. Together they had put together a programme development authority to try to minimise conflicts of interest as far as they could. However, the Department accepted that when things were going well, those differences perhaps did not matter, but when things started to go wrong, and there were difficult choices to be made, those different priorities created difficulties.[15]

17. Since then, many lessons had been learned about handling PFI for IT projects, and in early 2000 the Treasury had issued guidance which drew on the lessons outlined in the Comptroller and Auditor General's report. In addition to this, and the report by Mr Ian McCartney, Minister of State at the Cabinet Office on Major Government IT Projects, there was now additional expertise available within the Treasury, and the Office of Government Commerce to help Departments.[16]

18. The Department thought it inconceivable that nowadays such a project would be tackled in one big bang. It would be broken down into more manageable chunks. They would adopt a phased acceptance programme. Under the new arrangements introduced following the McCartney report, projects would now be subject to independent reviews at key stages (the Gateway process). In future that would ensure for example that sufficient effort went into developing the technical design of the system before contract for implementation. The Department also recognised that for future projects of this nature there had to be a more collaborative, partnership approach to solve problems as they arose, and they would have a much more shared and open approach to risk. They had taken a different approach to PFI in their new modernisation projects, such as Child Support reform.[17]

19. ICL confirmed that the new guidelines very much reflected their experience. There needed to be much greater clarity at the outset about requirements, particularly where there are two purchasers and complex legacy systems which are vital to the smooth running of society. They supported the notion of an initial phase of defining requirements and solutions, before going forward to PFI delivery. They also preferred a more joined-up approach, rather than operating in separate silos and dealing at the boundary which did not work with such complex end-to-end systems.[18]

20. The Comptroller and Auditor General noted that the contract was placed in May 1996 for delivery by 1999. Despite a re-plan of the project in February 1997, the timetable for delivery slipped continually, the benefits were delayed, and in late 1997 Pathway requested improved terms. In November 1997, the Department took steps to preserve their legal rights to cancel the project. In March 1998, Ministers set up an interdepartmental working group to assess whether the project was technically viable, if so how quickly it could be completed and at what cost to the government; as well as the costs of cancellation and of any alternative available to deliver the project's objectives. Because no decision had yet been reached, the Secretary of State twice gave the Accounting Officer of the Benefits Agency formal instructions not to terminate the contract. The Government formally announced cancellation of the Benefits Payment Card element of the project on 24^th May 1999.[19]

21. The Department said that the length of time it had taken from November 1997 to May 1999, to devise a way forward reflected the importance of the issues at stake. Cancellation would affect every post office and 22 million potential benefit claimants, as well as the future of the Post Office and its automation and measures to reduce fraud. In addition, these issues involved the Department for Trade and Industry, as sponsor of the Post Office, and the Treasury. And there were commercial issues for ICL, which was coming up to flotation. There had been a long process of different kinds of review, inter-departmental groups, and a long period of negotiation trying to find a way forward. There had also been various changes of Ministers along the way. In the Department's view these factors justified such a lengthy decision making period.[20]

Conclusions

22. All the parties seriously underestimated the complexity of the project and the risk of attempting to tackle such a challenge in one go using relatively untried PFI arrangements. They failed to achieve a shared understanding of what was to be delivered and how, and the barriers to innovation contained in benefit rules. The other major factors included inadequate contracting and project management skills, and lack of clear ownership of project delivery and risk management. These mirror many of the failings referred to in our predecessor's January 2000 report on lessons to be drawn from IT projects.

23. Pathway was selected because they were willing to take on a level of risk for preventing benefit fraud which the other two bidders would not accept, even though Pathway came third on 8 out of 11 technical and management criteria. At the time, there was a mistaken view that PFI bidders should compete on the level of risk they were prepared to take on, rather than achieving an optimum allocation of risk.

24. The various parties identified many of the risks at various stages, but did not always share this information. Risks were "cleared" without justification, and "cleared" risks were not well monitored and so re-emerged.

25. Joined-up projects involving more than one purchaser carry extra risks, and steps need to be taken to establish clear leadership in purchasing and project management, and transparency of information between the various stakeholders. The Department and the Post Office took some steps in this direction during the project, by revising the project management arrangements. The McCartney report recommends one single responsible owner for each major project in future; but the Benefits Payment Card also demonstrates the delays and extra costs that can emerge when decisions on the future of a project involve many stakeholders with conflicting objectives, in this case, for example the Department, the Department of Trade and Industry, the Treasury and the Post Office.

4 C&AG's report (HC 857, Session 1999-2000), para 29 Back

5 ibid, para 15 Back

6 Qs 141-144 and C&AG's report (HC 857, Session 1999-2000), paras 25-26, 1.36 and 3.46 and pp 14-15 Back

7 C&AG's report (HC 857, Session 1999-2000) pp 14-15 Back

8 Qs 1, 20-23, 26-30, 60, 116-123 Back

9 Qs 1, 18-19, 24-25, 135-136 Back

10 Qs 22, 26, 107-116 and Evidence, Appendix 2, p22 Back

11 C&AG's report (HC 857, Session 1999-2000), paras 18-20, 27-29 and pp 13-14 Back

12 Qs 1, 19-25, 31-32, 43-51 Back

13 Qs 52-55, 124, 151-154 Back

14 C&AG's report (HC 857, Session 1999-2000) paras 16-17 and p15 Back

15 Qs 3, 81-85 Back