Memorandum by the British Computer Society
(CSE 04)
The BCS welcomes the opportunity to submit
evidence to PASC on their Inquiry into Civil Service Effectiveness.
The BCS wishes to address specifically the issue
raised in the Section "Civil Services Skills", Question
8, which seeks comments on the recent problems in government IT
procurement and project management.
We attach herewith a detailed analysis undertaken
by one of our Expert Panels, the United Kingdom Computer Research
Council (UKCRC), which addresses many of the technical issues
associated with the handling of Complex IT Projects.
We would also like to take this opportunity
to address more broadly some fundamental issues which relate largely
to the management of large IT-based projects. Here, we would draw
attention to the recent report jointly compiled by Fellows of
the Royal Academy of Engineering and the BCS, entitled the "Challenge
of Complex IT projects".[11]The
report noted that expenditure on IT in the public sector alone
was in excess of £12.4 billion in 2003-04. It also raised
the alarm that a significant number of IT-based projects still
fail to deliver key benefits on time and to target in terms of
cost and specification. It was recognised that whilst such projects'
success rate may be improving, the associated challenges are also
increasing rapidly. These challenges are fuelled in a large part
by the exponential growth in the capability of hardware and communications
technology and the corresponding inflation in people's expectations
and ambition.
It is important, however, to note that in the
opinion of the team undertaking this extensive piece of work,
there does not appear to be any significant difference between
the success rate of IT projects in the private and the public
sector, clearly indicating that the challenges are the same in
both sectors. It is also important to acknowledge, as the report
clearly states, that whilst there are some dramatic failures,
there are some amazing success stories and, unfortunately, the
press is often only too willing to highlight the failures, rather
than praising the successes.
The BCS has been delighted to be invited by
various Government Offices to discuss aspects of their handling
of IT-based projects and we are encouraged by their awareness
of the complexity involved and the willingness of many senior
civil servants now to take advice and to adapt their practices.
We are, as you will know, engaged with and continue to be supportive
of, the work undertaken by the OGC and particularly can commend
its Code of Best Practice for suppliers.
The Society believes that the fundamental issue
underpinning the success of IT-based projects relates to the need
to adopt a truly professional approach to each and every stage
of any project, whether it involves IT components or not. Indeed,
it is critical to note that there probably is no such thing as
an "IT project"IT is simply an enabling component
in the solution to other business support programmes. In this
light the Society is encouraged by the positive thoughts that
are emerging from the very heart of government, particularly through
the newly appointed Head of Government IT, Ian Watmore. There
is clearly a realisation within government of the need to adopt
IT professionalism which will cut across government bodies involved,
as well as ensuring professionalism from suppliers of technology
and organisations to whom IT systems are outsourced.
Finally we would like to highlight the statement
made in the joint report referred to earlier, which states that
one of the most serious problems encountered in IT-based projects
is the associated inflation of people's expectations and ambitions.
In our meetings with senior civil servants it is clear that whilst
they fully understand the limitations of the technology, they
continually have to cope with the aspirations of their political
masters who clearly do not understand what the limitations are.
As was clearly stated in the Royal Academy/BCS report, if one
was trying to build the Second Severn Crossing and required to
achieve it as a single cantilever structure (ie supported from
one end only) then any professional civil engineer would simply
say that the job was impossible. Unfortunately, in many cases
government's top IT persons are being instructed to build "bridges"
such as this.
The BCS is always most willing to provide independent
assistance and professional advice whenever this is required.
APPENDIX A
The UK Computing Research Committee (UKCRC),
an Expert Panel of the British Computer Society, the Institution
of Electrical Engineers and the Council of Professors and Heads
of Computing, was formed in November 2000 as a policy committee
for computing research in the UK. Its members are leading computing
researchers from UK academia and industry.
UKCRC has the expertise to address question
8 identified by the committee:
How does the performance of
the Civil Service compare with that of its equivalents in other
countries? Who or what is mainly to blame for the recent problems
in government IT, procurement and project management?
A recent study of problems of large scale projects
by the Royal Academy of Engineering[12]
reported that poor project definition is one of the major contributors
to project failure. The Academy also observed that there is a
greater tendency for poor project definition in the public sector,
where systems are intended to meet political ends, and the practicalities
often have not been thought through.
Recent work by the Office of Government Commerce
has addressed the management failures that have contributed to
failures of Government IT projects, with recommendations relating
to the appointment of "senior responsible owners" and
guidelines for Gateway Reviews. More recently, the OGC and Intellect
(the trade association for companies in the IT industry) have
collaborated on a Code of Conduct to address some of the worst
examples of unprofessional behaviour by suppliers to Government.
These are worthwhile initiatives and we support them, but in our
opinion they do not get to the heart of the problem.
A report published this month by the National
Audit Office commends the OGC on the steps they have taken. In
our opinion, the NAO report also fails to recognise that the OGC
has failed to address a critical issue in software procurement.
Developing new computer software is an engineering
taskoften of extraordinary complexity. The software for
a modern air-traffic control centre, or to support a Government
Agency, will involve a million lines of computer program (often,
several million lines) and will typically require tens or hundreds
of person-years of effort to design. It is beyond the power of
humans to create structures of this size and complexity without
making mistakes, so the focus of software engineering has to be
on reducing the number of errors made, finding as many as possible
before the software is delivered, and designing programs that
are resilient to the faults that remain. Unfortunately (and, as
we shall see, unnecessarily) most software developers deliver
systems that still contain more than ten faults in each thousand
lines of program. A large system will therefore contain upward
of ten thousand faults, each waiting to cause a failure under
the particular circumstances that cause that path in the software
to be executed.
Engineering is much more than applied science,
although the science is essential; engineers also use mathematics,
codified experience, and judgement. They use mathematics to build
models of their designs and to explore the properties of these
designs before investing in further development. They capture
experience of design methods and management in a form that can
be taught as best practice engineering processes. They codify
this experience in the form of standards, such as the international
quality management standard, ISO9001. The best software engineers
recognise the parallels between software design and classical
engineering, and adopt similarly rigorous methods. Unfortunately,
most suppliers do not follow these principles, and most Government
Departments are not knowledgeable enough to hold them to account
for their failure to reach levels of professional competence and
responsibility that are taken for granted in other branches of
engineering.
Much research has been done on software engineering,
complementing the progress made in computer science. Yet, despite
the many books and papers providing evidence of ways in which
software (even very large and complex software) can be developed
at low risk, most of industry has ignored the research and continued
to waste huge amounts of money on projects that are late, over
budget, and fail to deliver what the customer needs.
Two surveys illustrate the scale of the problem.
In 1995, The Standish Group[13]
reported that the average US software project overran its budgeted
time by 190%, its budgeted costs by 222%, and delivered only 60%
of the planned functionality. Only 16% of projects were delivered
at the estimated time and cost, and 31% of projects were cancelled
before delivery. Later Standish Group surveys show an improving
trend, but success rates are still low. The Standish Group's "Chaos
Chronicles" report for 2003 analyzed over 13,000 IT projects,
and estimated that nearly 70% either failed completely or were
"challenged" (ie although completed and operational,
exceeded their budget and time estimates, and had less functionality
than originally specified). This led to their estimate that in
2002 the US "wasted $55 billion in cancelled and over-run
IT projects, compared with a total IT spend of $255 billion."
A UK survey, published in the 2001 Annual Review
of the British Computer Society[14]
showed a similarly distressing picture. Of more than 500 development
projects, only three met the survey's criteria for success.
The US National Institute of Standards and Technology
published a report in 2002 that estimated the annual cost of poor
quality software to the US economy as $60 billion[15];
we imagine that the cost to the European economy is of a similar
order.
Unfortunately, we do not know the detailed sampling
methods used by these surveys, nor have we seen their raw data;
we are therefore unable to comment on the statistical significance
of the results. Even if the survey results exaggerate the situation
somewhat, the general state of software development is depressingly
amateurish.
For example, few projects capture the users'
requirements using notations and methods that make it possible
to analyse them rigorously to find contradictions and omissions,
even though these are among the most common causes of project
delays and failures. Such analysis ensures that the requirements
have been properly considered and that conflicts and contradictions
are not hiding behind vague statements and jargon. It also has
a powerful effect in persuading customers and suppliers not to
be too ambitious, which is another frequent cause of failure.
Across Europe, there are a few companies who
deliver software projects significantly more quickly and more
cheaply by using "formal methods"software engineering
methods that are soundly based in computer science-and mature
engineering processes[16].
These companies are increasingly willing to offer warranties,
because they know that they can deliver software with 0.1 to 1
fault per thousand lines, instead of the 10-20 faults per thousand
lines that are more typical. Such companies are able to analyse
the requirements that they have been given by their customers,
to reveal the inconsistencies and to draw out the deeper requirements
that would otherwise result in late changes to the project requirements.
Late changes to requirements are very damaging.
The customer has to renegotiate the project cost and timescales,
usually at a point where competitive bids cannot be obtained.
The project risk is transferred back to the Government Department,
and the project slips and runs over budget. Naturally, some changes
may be unavoidable: the world is a dynamic place and there may
be reorganisations and changes to legislation or tax structures
that affect a project between the time that the contract was signed
and the time when the system was delivered and implemented. Nevertheless,
very many changes to requirements are not of this nature: instead,
they are changes that could have been discovered or foreseen before
the contract was signed, if there had been a rigorous approach
to requirements specification and adequate analysis of the implications
of the stated requirements.
The use of these formalisms also allows the
engineer to specify precisely a useful degree of imprecision,
and so plan the implementation so that (some reasonably foreseeable)
late changes can be accommodated without impacting the sound structure
of the product. Those that do require a major re-write (and that
can never be completely avoided) can be identified rapidly, and
can be accommodated by rescheduling.
Mature engineering professions, such as mechanical,
structural and electrical engineering, have developed mature processes
and mathematically-based methods over very many decades, but the
IT industry is already too important for Government to be able
to tolerate its current slow rate of progress towards maturity.
With political targets regularly put in jeopardy, £Billions
wasted annually on poor quality software, and failing projects
in the news most weeks, investment in transferring best practice
in computer science and software engineering into routine use
in industry could have an enormous payoff for the UK. There are
models that could be followed: in the USA, the Department of Defense
commissioned the Software Engineering Institute (SEI) at Carnegie
Mellon University to develop an audit regime that would allow
them to judge the maturity of their suppliers' software engineering
processes. This "Capability Maturity Model" has been
widely adopted, and a recent report from the Royal Academy of
Engineering[17]
proposes that an organisation similar to the SEI should be established
in the UK.
To summarise: the UK may be no worse than other
countries in the ability of the Civil Service to procure effective
IT. Unfortunately this does not mean that the situation is acceptable;
the international state of practice is woefully inadequate, and
the UK seems to be more complacent than, for example, the USA
in believing that recent improvements in procurement practices
have solved the problems.
UKCRC would be happy to answer follow-up questions
on any of these points.
Martyn Thomas
for UKCRC.
November 2004
11 It can be found at: http://www.bcs.org/NR/rdonlyres/3B36137E-C5FE-487B-A18B-4D7281D88EF7/0/complexity.pdf Back
12
The Challenges of Complex IT Projects, A report of a working group
from the Royal Academy of Engineering and the British Computer
Society, April 2004. Back
13
http://www.standishgroup.com Back
14
http://www.bcs.org.uk Back
15
The Economic Impacts of Inadequate Infrastructure for Software
Testing, RTI Project Number 7007.011, Final Report, May 2002. Back
16
See, for example, http://www.sparkada.com/publications.html#industrial Back
17
The Challenges of Complex IT Projects, A report of a working group
from the Royal Academy of Engineering and the British Computer
Society, April 2004. Back
|