SMART ACQUISITION: EFFORTS TO IMPROVE
PROCUREMENT PERFORMANCE
26. Smart Acquisition requires projects to produce
three-point cost and timescale estimates at 30 percent, 50 percent
and 90 percent confidence levels. In giving evidence to our predecessors
on the Major Projects Report 1994, the Department stated that
three-point estimating was used routinely on all projects and
undertaken thoroughly. Yet the National Audit Office found that
only two of the ten assessment phase projects had reliable three-point
cost and time estimates. Asked why this was so, the Department
could not justify its record and agreed that it was not satisfactory
that it had failed to fulfil its assurance to the Committee. But
it did point out that since 31st March 2000 (the datum
of the National Audit Office report) there were only three projects
without three-point estimates. As part of the changed procurement
strategy the Department had asked bidders for enough data to support
three point estimates and promised the Committee would see these
as soon as they were available.[39]
27. In response to questioning on whether there was
a major project that it was proud of, the Department listed Trident,
Tomahawk and the recovery of the Challenger 2 programme. When
it was suggested that Trident and Tomahawk were largely off-the-shelf
projects where the Department did not have a great deal of scope
to alter the specification the Chief of Defence Procurement responded
that the nuclear propulsion plant did not feel off-the-shelf to
him.[40]
28. The contract for the Type 45 Destroyer is arbitrated
under British law, whilst the contract for its PAAMS missile system
has been placed under French law. The Department was asked what
would happen if the two systems did not work together and it was
not clear which half was the problem. The Department admitted
that it carried the risk, but was confident that PAAMS was required
to conform to all of the interface specifications which had been
agreed with the ship project.[41]
29. In giving evidence on the Major Projects Report
1998 the Chief of Defence Procurement stated that there were "the
best industry people in the world" working on BOWMAN and
that "they were close to a cost-effective solution".
Since the competition has subsequently been re-opened, some 12
years after the requirement was first raised the Department was
asked why it had not seen the difficulties with the ARCHER industrial
collaboration earlier and why it had taken so long to decide what
to do. The Department said that ARCHER was completely re-energised
in 1999 as a result of the consortium being managed properly by
BAE SYSTEMS. They had brought in new staff and were supported
by their consortium members both from the United States and from
France, with the best people that could be found. The Department
had, however, been very keen to make sure that a decision on whether
or not to continue with ARCHER was based on their final bid and,
right until the last few days of the bid, it was not clear whether
it was going to be satisfactory or not. The Department said that
it was very careful about contractors' extravagant promises, particularly
when they were made near the deadline on the basis that it had
become a must-win project for them. The fact that ARCHER put so
many changes into their bid in the last few days had caused the
Department to look closely at whether they would be able to live
up to their promises.[42]
30. The Department was asked why technical delays
had emerged on the Hercules C-130J programme, since at the time
the contract was awarded there were categorical assurances from
Lockheed-Martin and also the RAF that there would be no delays.
The Department said that the Hercules C-130J was a new aircraft
with new engines, propellers and avionics and that Lockheed Martin
had underestimated the development content. The Department did
not understand why the company made an under-estimate because
they built its predecessor, the C-130K, and knew more about this
aircraft than anybody else in the world.[43]
The Department also said it recovered substantial liquidated damages
from Lockheed-Martin as they fell due, which had at least compensated
it for the running costs of the existing United Kingdom C-130K's.[44]
31. The Chief of Defence Procurement believed that
the Department needed to find better ways of incentivising contractors
to deliver on time. The Department had plenty of 'sticks' but
needed to balance them with 'carrots'. For example, the Eurofighter
programme included some incentives for doing things on time by
using the retentions on the progress payments on the contract,
which were planned to be withheld to the end. Now, if industry
delivered the first, eighth and final Tranche 1 aircraft on time
they would receive bonuses.[45]
Conclusions
32. Our predecessor's report on the 1994 Major Projects
Report recorded that the Department had stated that it used three-point
estimates on all major projects. It is therefore surprising that
only two of the ten assessment phase projects in the Major Projects
Report 2000 had three-point estimates for time and cost at 31
March 2000. The Department could not justify this failure, although
it did make progress quickly in generating three-point estimates
after the National Audit Office had highlighted the problem. We
also note the Department's assurance that three-point estimates
for Bowman will be available soon and expect all projects in the
Major Projects Report 2001 to have robustly generated three-point
estimates.
33. The Department highlighted the Tomahawk Land
Attack Missile and Trident missiles as successful projects. One
obvious common factor is that both of these projects were largely
off-the-shelf purchases from the United States, but the Department
should look to identify other common features which may enable
it to emulate these successes in future.
34. The Type 45 destroyer is an anti-air warship
being procured in two parts comprising the warship and the Principal
Anti-Air Missile System (PAAMS). The Department recognised the
risks but assured us that it was confident there would be no interface
problems between the warship and PAAMS. That would be a significant
achievement, and we look forward to hearing whether the Department
has achieved it.
35. The Department said that it was now more cautious
about accepting over optimistic bids from industry. For example,
it had not continued with the ARCHER industrial prime collaboration
contractor on BOWMAN because it believed that the company was
making extravagant promises that would not be deliverable. Such
scepticism is a welcome development.
36. The Department seeks to transfer the cost risk
of resolving technical difficulties and the delays it causes to
industry by using performance based contracts. For example, the
Department received substantial liquidated damages from Lockheed
Martin in respect of the delays to the Hercules C-130J. Whilst
liquidated damages may offset the costs of delays the Department
still bears the operational implications of the delays, a fact
which re-emphasises the importance of planning to and agreeing
realistic timescales at the outset.
37. The Department regards control of costs through
contracts as straightforward, but believes that incentivising
contractors to deliver on time is more difficult. The principle
of incentivising contractors to deliver on time is a good one,
but it is important that incentives are based on a thorough understanding
of what is achievable and are effective in securing added performance.
The Department needs to ensure that it does not give contractors
extra benefits for simply doing what they should be doing anyway
under the contract.
SMART ACQUISITION: LITTLE PROGRESS
IN MEASURING SUCCESS
38. The Department has not yet set targets for its
performance measures for the assessment phase and does not have
a measure to show whether risk is reduced sufficiently to enable
a sound decision to be taken at Main Gate. Asked why it had taken
so long to set these performance measures and when they would
be fully in place, the Department said that setting the measures
was difficult, and that the measures agreed would have to be used
for all assessment phase projects, not just those in the Major
Projects Report. However, the Department promised it would have
the measures in place by 1st April 2001.[46]
39. Following the changes to the approval process
associated with Smart Acquisition approvals the Department approves
the cost and in-service date of an equipment which it is 90 percent
confident of achieving, while basing its plans on a lower/earlier
figure which it is 50 per cent confident of achieving. Timescale
and cost performance is measured in the Major Projects Report
by comparing the current forecast in-service date or cost underpinning
the Department's plans (the 50 percent estimate) with the latest
acceptable in-service date or highest cost set at approval (the
90 percent estimate). This means that individual projects approved
under 'Smart' procedures will be forecast to enter service earlier
or cost less than their approved in-service date or procurement
cost unless all of the risks allowed for in the 90 per cent approval
materialise.[47]
40. The differences between the 50 percent and 90
percent figures can be very large, for example, the Multi-Role
Armoured Vehicle is reporting a 31-month negative in-service date
variation. In this case, the Department said it was unfortunate
that one of the first projects to appear under the new regime
was a two-nation collaborative programme which might soon become
a three nation programme if the Netherlands joined. This uncertainty
had increased risk, and that there were a number of other international
factors which would bedevil the programme through development.
If these uncertainties were added up, the 31 months did not look
like a generous allowance of time.[48]
41. Asked how large, on average, it expected the
risk provisions for time and cost to be at Main Gate, the Department
suggested that the absolute differences would be variable between
projects. The fact that one project approved under "Smart"
procedures, the Seawolf Midlife Update, had already breached its
90 percent in-service date approval by 3 months showed it was
not being institutionally generous to itself.[49]
42. All of the projects covered in the Report were
conceived before Smart Procurement, and the Department suggested
that it was still early to expect to see improvements in timescale
performance from the application of Smart Procurement principles
reflected in the Major Projects Report. Only when projects with
significant delays such as the C-130J begin to pass out of the
Report, and new 'Smart' projects entered the Report, would the
average delay decrease. The Department was confident that this
effect would start to become apparent from the Major Projects
Report 2002.[50]
Conclusions
43. The Department is not clear what level of risk
it expects will remain in projects at Main Gate and how this will
be measured. A much clearer definition of the acceptable level
of risk at Main Gate is needed and the Department must give priority
to defining this and how to measure it.
44. Under Smart Acquisition, the Department bases
its Main Gate Approval on the cost and in-service date of an equipment
which it is 90 percent confident of achieving. We note that the
SeaWolf Mid-Life Update project has already exceeded the in-service
date which the Department was 90 percent certain of achieving
by some three months. This does not bode well for the future.
45. The Department does not expect there to be big
improvements in the timescale performance reported in the Major
Projects Report until 2002, some four years after Smart Acquisition
was first introduced. Whilst the timescale performance reported
in the Major Projects Report will continue to reflect pre-Smart
Acquisition projects for some years, we will expect to see signs
of improvement as new "smart" projects come through.
1 This report is based on two days of evidence and is
complemented by another (6th Report of the Committee
of Public Accounts: MOD Major Projects Report 2000-The Role
of the Equipment Capability Customer (HC 369 (2001-02)) Back
2 C&AG's
Report (HC 970 Session 1999-2000), Executive Summary para. 1 Back
3 ibid,
Executive Summary para. 2 Back
4 At
this session the Committee also drew on evidence on the four case
studies from the C&AG's Report on the Major Projects
Report 1999 (HC 613 Session (1998-1999)) Back
5 6th
Report from the Committee of Public Accounts: MOD Major Projects
Report 2000-The Role of the Equipment Capability Customer (HC
369 (2001-02)) Back
6 C&AG's
Report (HC 970 (1999-2000)) para 1.21 Back
7 Q266 Back
8 Q265 Back
9 Q267 Back
10 C&AG's
Report (HC 970 (1999-2000)) para 1.4 Back
11 Q278 Back
12 C&AG's
Report (HC 970 (1999-2000)) para 1.16 Back
13 Q189 Back
14 C&AG's
Reports (HC 970 (1999-2000)) p31, and HC 613 (1999-2000)) Figure
20 Back
15 33rd
Report from the Committee of Public Accounts (HC 247 (1999-2000)) Back
16 Qs
9-10 Back
17 Q51 Back
18 Q140 Back
19 Q8
Back
20 C&AG's
Report (HC 970 (1999-2000)) para 3.10 Back
21 ibid,
paras 3.11 and 3.14 Back
22 Q18 Back
23 Q269 Back
24 Qs
28-31 Back
25 Q32 Back
26 Q54 Back
27 Q56 Back
28 C&AG's
Report (HC 970 (1999-2000)) para 3.31 Back
29 ibid,
paras 3.18 and 3.21 Back
30 ibid,
para 3.13 Back
31 Q38 Back
32 Q40 Back
33 Q2 Back
34 Q41 Back
35 Q3 Back
36 Q7 Back
37 Q5 Back
38 For
the detail of the fitting of the sonar to the Type 45 Destroyer
see the 6th Report of the Committee - Ministry of Defence:
Major Projects Report 2000-The Role of the Equipment Capability
Customer (HC 369 (2001-02)) Back
39 Qs
207-209 Back
40 Q209 Back
41 Qs
185-186 Back
42 Q8 Back
43 Q285 Back
44 Q286 Back
45 Q278 Back
46 Qs
202-203 Back
47 C&AG's
Report (HC 970 (1999-2000)) para 1.13 Back
48 Q204 Back
49 Q205 Back
50 Qs
280-281 Back