Government and IT- "A Recipe For Rip-Offs": Time For A New Approach - Public Administration Committee Contents

6  Agile Development

76. Another problem with government IT identified during our inquiry lies in the way in which it develops new systems. The Institute for Government (IfG) report System Error discussed this issue at length, contrasting how the Government had traditionally run projects using "waterfall development"- with their preferred approach "agile development".

Waterfall versus Agile Development

77. In the waterfall model the design goes through a series of sequential phases with each phase being completed before moving onto the next. The main argument advanced in favour of this model is that time spent up front making sure requirements and design are correct saves time and effort later. Therefore each phase should be completed before proceeding to the next phase.

78. The IfG argues that this model would be successful "as long as the plan will not need to change mid-way through".[109] However, in the vast majority of cases, the complex nature of the problems that Government is faced with means that this criterion is rarely met. New priorities, policies and initiatives, as well as new technological options, often lead to changes being required both during development and after implementation.

79. The waterfall approach encourages users to try and define all their requirements in as much detail as possible during the initial specification stage, due to the high costs of making changes at later stages in the process. This can therefore result in unnecessary 'gold-plating' of requirements as the initial specifications are seen as the only opportunity to request functionality.

80. In contrast agile development is based on an iterative and incremental process, where requirements and solutions continuously evolve, this is the way many businesses acquire IT. Rather than specifying rigid system requirements and then attempting to design the whole programme at once, the design team will produce a partially complete design which goes through a number of iterations. The design is also broken up into a number of parts (modules) which can be worked on and changed independently. The design, often in the form of a working although incomplete prototype, is shown to the customer at the end of each iteration allowing them to experience, comment and feedback on the product; thereby ensuring much greater integration of the business into the design of the systems.

81. The figure below, from the IfG Report, summarises the main difference between the two approaches:

Figure 1: Waterfall v. Agile development approach. Source: IfG: System Error, p32

82. The IfG recently facilitated a project using agile development techniques that developed software to help the DWP and the Metropolitan Police Service share data on suspected fraudsters. Their report identifies the following benefits:

  • Reducing time-consuming decision-making processes;
  • Implementing those parts of the system that are ready, without having to wait for the complete system to be finished;
  • Encouraging SME involvement, by breaking programmes up into smaller modules; and
  • Greater flexibility by being better able to deal with changes in system requirements.[110]

Challenges to using Agile

83. The Government has responded positively to the IfG Report, and its ICT Strategy contains a commitment to apply agile methods to ICT procurement and delivery.[111] Mark Holden, HMRC, told us that there was not much in the way of barriers to the greater use of agile development. However, he did say that some types of projects - particularly those which aimed to develop or enhance legacy systems - might not be suitable environments for using this methodology.[112] Sir Ian Magee, IfG, made a similar point saying that the Institute's report was not claiming that Agile would be suitable for every project.[113]

84. The IfG identified three main barriers to using agile working methods in Government:

  • Governance issues. The current approval process looks for a detailed specification as a sign that a project has been properly thought out, but such specifications are not normally produced for Agile development as the specification is expected to change as the project develops;
  • Commercial processes. A preference for fixed price contracts to deliver a particular solution reinforces the tendency for both sides to demand a high level of detail up-front;
  • Cultural issues. A reluctance to delegate and assign high levels of responsibility at lower levels of the organisation, in addition to a more general wariness of change from inside Government.[114]

In addition, greater use of agile development is likely to necessitate behaviour changes within Government. As agile methodology requires increased participation from the business to provide feedback on different iterations of the solution, departments will need to release their staff, particularly senior staff with overall responsibility of the project, to allow them to participate in these exercises.

85. Alistair Maughan, partner and head of Technology Transactions at an international law firm, has argued that the fact that agile projects do not have a fixed cost at the outset combined with the hierarchical nature of civil service management structures meant that it would not able to be successfully used in government.[115]

86. We do not think that any of these challenges are insurmountable. For example the theoretical ability to acquire a "fixed price" under the current model has not prevented projects being delivered late or additional costs being incurred. However, it is clear that moving towards a greater use of agile development will require changes in the way government commissions and manages projects. For example, the "System Error" report identified the need to develop different governance and oversight mechanisms for agile projects; arguing that the current Gateway Reviews are unsuitable both due to the length of the review process and the fact that agile projects lack obvious Gateway points at which to conduct these reviews.[116]

87. Agile development is a powerful tool to enhance the effectiveness and improve the outcomes of Government change programmes. We welcome the Government's enthusiasm and willingness to experiment with this method. The Government should be careful not to dismiss the very real barriers in the existing system that could prevent the wider use of agile development. We therefore invite the Government to outline in its response how it will adapt its existing programme model to enable agile development to work as envisaged and how new flagship programmes will utilise improved approaches to help ensure their successful delivery.


88. We also heard specific concerns about the compatibility of agile development with current procurement guidelines. Alistair Maughan argued that under EU procurement law the Government is required to compare bidders "on a like-for-like basis, and deciding on best value for money" but that "Agile can't give you a clear specification of outputs up-front. Nor can it give a definitive up-font price" that is required to make Government to make a like-for like comparison.[117] Similarly the IfG noted that:

    The procurement process can undermine agile projects in two ways. First, when detailed specifications are used it can restrict the freedom to innovate. Second, the long timelines associated with the procurement process can restrict the ability to deliver production ready solutions rapidly.[118]

89. Kelvin Prescott, Newbury Management Consultants and Susan Atkinson, Legal Director, Gallenalliance Solicitors suggested a number of changes which they believed would help overcome these problems including:

  • A different contract structure which breaks projects down in several phases, with parties only committed initially to the first stages. This would allow the customer to adapt the process as development continues.
  • The creation of an agile procurement strategy based on a "fail early, fail fast" principle that provides the ability to terminate a project when failure looks likely and that provides access to a number of suppliers than can be rapidly appointed to projects, or parts of project.
  • A number of specific changes to the current Procurement Contract Regulations.[119]

90. The Government should examine how it can remove barriers to agile development as an integrated part of its wider efforts to reform the procurement process and increase the role of SMEs. The Government will have to bear in mind the need to facilitate agile development as it renegotiates the EU procurement directive and revises the associated guidance.

109   IfG, System Error,p28  Back

110   IfG, System Error, p 47 Back

111   Cabinet Office, Government ICT Strategy, para 27 Back

112   Q 334 Back

113   Q 97 Back

114   IfG, System Error, p 53-54  Back

115   "Agile will fail Gov IT, says corporate lawyer", Computer Weekly, April 26 2011 Back

116   IfG, System Error, p 72 Back

117   "Agile will fail GovIT, says corporate lawyer", Computer Weekly, April 26 2011 Back

118   IfG, System Error: Fixing the Flaws in Government IT, March 2011 p 68 Back

119   Ev w46-154 Back

previous page contents next page

© Parliamentary copyright 2011
Prepared 28 July 2011