Select Committee on Public Administration Written Evidence


Memorandum by the British Computer Society (CSE 04)

  The BCS welcomes the opportunity to submit evidence to PASC on their Inquiry into Civil Service Effectiveness.

  The BCS wishes to address specifically the issue raised in the Section "Civil Services Skills", Question 8, which seeks comments on the recent problems in government IT procurement and project management.

  We attach herewith a detailed analysis undertaken by one of our Expert Panels, the United Kingdom Computer Research Council (UKCRC), which addresses many of the technical issues associated with the handling of Complex IT Projects.

  We would also like to take this opportunity to address more broadly some fundamental issues which relate largely to the management of large IT-based projects. Here, we would draw attention to the recent report jointly compiled by Fellows of the Royal Academy of Engineering and the BCS, entitled the "Challenge of Complex IT projects".[11]The report noted that expenditure on IT in the public sector alone was in excess of £12.4 billion in 2003-04. It also raised the alarm that a significant number of IT-based projects still fail to deliver key benefits on time and to target in terms of cost and specification. It was recognised that whilst such projects' success rate may be improving, the associated challenges are also increasing rapidly. These challenges are fuelled in a large part by the exponential growth in the capability of hardware and communications technology and the corresponding inflation in people's expectations and ambition.

  It is important, however, to note that in the opinion of the team undertaking this extensive piece of work, there does not appear to be any significant difference between the success rate of IT projects in the private and the public sector, clearly indicating that the challenges are the same in both sectors. It is also important to acknowledge, as the report clearly states, that whilst there are some dramatic failures, there are some amazing success stories and, unfortunately, the press is often only too willing to highlight the failures, rather than praising the successes.

  The BCS has been delighted to be invited by various Government Offices to discuss aspects of their handling of IT-based projects and we are encouraged by their awareness of the complexity involved and the willingness of many senior civil servants now to take advice and to adapt their practices. We are, as you will know, engaged with and continue to be supportive of, the work undertaken by the OGC and particularly can commend its Code of Best Practice for suppliers.

  The Society believes that the fundamental issue underpinning the success of IT-based projects relates to the need to adopt a truly professional approach to each and every stage of any project, whether it involves IT components or not. Indeed, it is critical to note that there probably is no such thing as an "IT project"—IT is simply an enabling component in the solution to other business support programmes. In this light the Society is encouraged by the positive thoughts that are emerging from the very heart of government, particularly through the newly appointed Head of Government IT, Ian Watmore. There is clearly a realisation within government of the need to adopt IT professionalism which will cut across government bodies involved, as well as ensuring professionalism from suppliers of technology and organisations to whom IT systems are outsourced.

  Finally we would like to highlight the statement made in the joint report referred to earlier, which states that one of the most serious problems encountered in IT-based projects is the associated inflation of people's expectations and ambitions. In our meetings with senior civil servants it is clear that whilst they fully understand the limitations of the technology, they continually have to cope with the aspirations of their political masters who clearly do not understand what the limitations are. As was clearly stated in the Royal Academy/BCS report, if one was trying to build the Second Severn Crossing and required to achieve it as a single cantilever structure (ie supported from one end only) then any professional civil engineer would simply say that the job was impossible. Unfortunately, in many cases government's top IT persons are being instructed to build "bridges" such as this.

  The BCS is always most willing to provide independent assistance and professional advice whenever this is required.

APPENDIX A

  The UK Computing Research Committee (UKCRC), an Expert Panel of the British Computer Society, the Institution of Electrical Engineers and the Council of Professors and Heads of Computing, was formed in November 2000 as a policy committee for computing research in the UK. Its members are leading computing researchers from UK academia and industry.

  UKCRC has the expertise to address question 8 identified by the committee:

    —    How does the performance of the Civil Service compare with that of its equivalents in other countries? Who or what is mainly to blame for the recent problems in government IT, procurement and project management?

  A recent study of problems of large scale projects by the Royal Academy of Engineering[12] reported that poor project definition is one of the major contributors to project failure. The Academy also observed that there is a greater tendency for poor project definition in the public sector, where systems are intended to meet political ends, and the practicalities often have not been thought through.

  Recent work by the Office of Government Commerce has addressed the management failures that have contributed to failures of Government IT projects, with recommendations relating to the appointment of "senior responsible owners" and guidelines for Gateway Reviews. More recently, the OGC and Intellect (the trade association for companies in the IT industry) have collaborated on a Code of Conduct to address some of the worst examples of unprofessional behaviour by suppliers to Government. These are worthwhile initiatives and we support them, but in our opinion they do not get to the heart of the problem.

  A report published this month by the National Audit Office commends the OGC on the steps they have taken. In our opinion, the NAO report also fails to recognise that the OGC has failed to address a critical issue in software procurement.

  Developing new computer software is an engineering task—often of extraordinary complexity. The software for a modern air-traffic control centre, or to support a Government Agency, will involve a million lines of computer program (often, several million lines) and will typically require tens or hundreds of person-years of effort to design. It is beyond the power of humans to create structures of this size and complexity without making mistakes, so the focus of software engineering has to be on reducing the number of errors made, finding as many as possible before the software is delivered, and designing programs that are resilient to the faults that remain. Unfortunately (and, as we shall see, unnecessarily) most software developers deliver systems that still contain more than ten faults in each thousand lines of program. A large system will therefore contain upward of ten thousand faults, each waiting to cause a failure under the particular circumstances that cause that path in the software to be executed.

  Engineering is much more than applied science, although the science is essential; engineers also use mathematics, codified experience, and judgement. They use mathematics to build models of their designs and to explore the properties of these designs before investing in further development. They capture experience of design methods and management in a form that can be taught as best practice engineering processes. They codify this experience in the form of standards, such as the international quality management standard, ISO9001. The best software engineers recognise the parallels between software design and classical engineering, and adopt similarly rigorous methods. Unfortunately, most suppliers do not follow these principles, and most Government Departments are not knowledgeable enough to hold them to account for their failure to reach levels of professional competence and responsibility that are taken for granted in other branches of engineering.

  Much research has been done on software engineering, complementing the progress made in computer science. Yet, despite the many books and papers providing evidence of ways in which software (even very large and complex software) can be developed at low risk, most of industry has ignored the research and continued to waste huge amounts of money on projects that are late, over budget, and fail to deliver what the customer needs.

  Two surveys illustrate the scale of the problem. In 1995, The Standish Group[13] reported that the average US software project overran its budgeted time by 190%, its budgeted costs by 222%, and delivered only 60% of the planned functionality. Only 16% of projects were delivered at the estimated time and cost, and 31% of projects were cancelled before delivery. Later Standish Group surveys show an improving trend, but success rates are still low. The Standish Group's "Chaos Chronicles" report for 2003 analyzed over 13,000 IT projects, and estimated that nearly 70% either failed completely or were "challenged" (ie although completed and operational, exceeded their budget and time estimates, and had less functionality than originally specified). This led to their estimate that in 2002 the US "wasted $55 billion in cancelled and over-run IT projects, compared with a total IT spend of $255 billion."

  A UK survey, published in the 2001 Annual Review of the British Computer Society[14] showed a similarly distressing picture. Of more than 500 development projects, only three met the survey's criteria for success.

  The US National Institute of Standards and Technology published a report in 2002 that estimated the annual cost of poor quality software to the US economy as $60 billion[15]; we imagine that the cost to the European economy is of a similar order.

  Unfortunately, we do not know the detailed sampling methods used by these surveys, nor have we seen their raw data; we are therefore unable to comment on the statistical significance of the results. Even if the survey results exaggerate the situation somewhat, the general state of software development is depressingly amateurish.

  For example, few projects capture the users' requirements using notations and methods that make it possible to analyse them rigorously to find contradictions and omissions, even though these are among the most common causes of project delays and failures. Such analysis ensures that the requirements have been properly considered and that conflicts and contradictions are not hiding behind vague statements and jargon. It also has a powerful effect in persuading customers and suppliers not to be too ambitious, which is another frequent cause of failure.

  Across Europe, there are a few companies who deliver software projects significantly more quickly and more cheaply by using "formal methods"—software engineering methods that are soundly based in computer science-and mature engineering processes[16]. These companies are increasingly willing to offer warranties, because they know that they can deliver software with 0.1 to 1 fault per thousand lines, instead of the 10-20 faults per thousand lines that are more typical. Such companies are able to analyse the requirements that they have been given by their customers, to reveal the inconsistencies and to draw out the deeper requirements that would otherwise result in late changes to the project requirements.

  Late changes to requirements are very damaging. The customer has to renegotiate the project cost and timescales, usually at a point where competitive bids cannot be obtained. The project risk is transferred back to the Government Department, and the project slips and runs over budget. Naturally, some changes may be unavoidable: the world is a dynamic place and there may be reorganisations and changes to legislation or tax structures that affect a project between the time that the contract was signed and the time when the system was delivered and implemented. Nevertheless, very many changes to requirements are not of this nature: instead, they are changes that could have been discovered or foreseen before the contract was signed, if there had been a rigorous approach to requirements specification and adequate analysis of the implications of the stated requirements.

  The use of these formalisms also allows the engineer to specify precisely a useful degree of imprecision, and so plan the implementation so that (some reasonably foreseeable) late changes can be accommodated without impacting the sound structure of the product. Those that do require a major re-write (and that can never be completely avoided) can be identified rapidly, and can be accommodated by rescheduling.

  Mature engineering professions, such as mechanical, structural and electrical engineering, have developed mature processes and mathematically-based methods over very many decades, but the IT industry is already too important for Government to be able to tolerate its current slow rate of progress towards maturity. With political targets regularly put in jeopardy, £Billions wasted annually on poor quality software, and failing projects in the news most weeks, investment in transferring best practice in computer science and software engineering into routine use in industry could have an enormous payoff for the UK. There are models that could be followed: in the USA, the Department of Defense commissioned the Software Engineering Institute (SEI) at Carnegie Mellon University to develop an audit regime that would allow them to judge the maturity of their suppliers' software engineering processes. This "Capability Maturity Model" has been widely adopted, and a recent report from the Royal Academy of Engineering[17] proposes that an organisation similar to the SEI should be established in the UK.

  To summarise: the UK may be no worse than other countries in the ability of the Civil Service to procure effective IT. Unfortunately this does not mean that the situation is acceptable; the international state of practice is woefully inadequate, and the UK seems to be more complacent than, for example, the USA in believing that recent improvements in procurement practices have solved the problems.

  UKCRC would be happy to answer follow-up questions on any of these points.

Martyn Thomas

for UKCRC.

November 2004





11   It can be found at: http://www.bcs.org/NR/rdonlyres/3B36137E-C5FE-487B-A18B-4D7281D88EF7/0/complexity.pdf Back

12   The Challenges of Complex IT Projects, A report of a working group from the Royal Academy of Engineering and the British Computer Society, April 2004. Back

13   http://www.standishgroup.com Back

14   http://www.bcs.org.uk Back

15   The Economic Impacts of Inadequate Infrastructure for Software Testing, RTI Project Number 7007.011, Final Report, May 2002. Back

16   See, for example, http://www.sparkada.com/publications.html#industrial Back

17   The Challenges of Complex IT Projects, A report of a working group from the Royal Academy of Engineering and the British Computer Society, April 2004. Back


 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2005
Prepared 14 April 2005