ANNEX A
REVIEW OF THE CONDUCT OF THE SHORT TERM
STRATEGIC AIRLIFT PROCUREMENT
National Audit Office Findings
Criterion 1: Was Air Foyle's proposal considered
as a serious alternative?
Source: No firm should have been invited
to tender unless the Ministry was prepared to place a contract
with them (the Department's Commercial Manager's Toolkit, Section
5, Tender Action—Competitive and Single).
Conclusion—Yes; criterion met.
The circumstances of the Short Term Strategic
Airlift were unusual in that the initial competition was terminated
and the procurement was taken forward through a competitive environment
phase. The Department engaged in significant dialogue with industry,
and at the way forward meeting with Air Foyle on 10 August 1999
at the start of the competitive environment phase, the Department
circulated for debate a revised Cardinal Points Specification
for the STSA procurement. This excluded operational requirements
from the original Cardinal Points Specification that the Antonov
could not achieve.
Subsequently, following comments from industry
and internally, the Department concluded that the Cardinal Points
Specification approach was unnecessarily complex, given that they
were fully aware of the capabilities of the An-124 from the earlier
competition[14].
Instead, with the agreement of the potential bidders, the Department
issued a simplified Request For Proposals (RFP) specifying the
number of Antonov aircraft required (two) and the timescales.
The Department, therefore, accepted the capability of the An-124
as a viable option.
We have not seen any evidence to support suggestions
that the Department did not mean what it said at the start of
the competitive environment phase and did not seriously consider
the An-124 as a viable alternative.
Criterion 2: Were the Department's criteria
for evaluating proposals explicit?
Source: The Invitation To Tender should normally
provide an indication of any key criteria to be listed in the
evaluation of bids. Some guidance to bidders on the performance
weighting scheme is necessary to ensure that the best result is
obtained from the competition (The Department's Commercial Manager's
Toolkit, Section 5, Tender Action—Competitive and Single).
Conclusion: No; criterion not met but
Air Foyle were aware of the lack of defined criteria.
The Department told us that during the "competitive
environment" phase, the Boeing bid was assessed against the
original Staff Requirement and the two Antonov bids were assessed
against the RFP with the winner being assessed against the C-17.
In broad terms, the RFP noted that it was essential that any STSA
An-124 100 solution provided a more assured level of availability
than the ad-hoc charter arrangements that were being used to supplement
the existing air transport capability and represented value for
money. However, the RFP did not explicitly state what criteria
were to be used to evaluate the bids or their relative weighting
in the evaluation.
Air Foyle were aware that the evaluation criteria
were not explicit when they submitted their proposal. They noted
in their proposal that the RFP sought a greater level of assurance
than was available from existing contracts together with value
for money but provided no guidance on how either assurance or
value for money would be determined[15].
Air Foyle also understood that "political assurance"—defined
by the Department as the customer being able to use the service
without political interference—was the main thrust of the
Department's concerns[16].
Their proposal recognised that the Department's requirement was
extremely difficult to meet, demanding a security of service from
the contractor which was ill-defined and political in scope[17].
Criterion 3: Were all the costs and risks
fully evaluated through a combined Operational Effectiveness and
Investment Appraisal (COEIA)?
Source: A full investment appraisal will
be carried out whenever a substantial investment is contemplated.
A COEIA is a vital part of the decision-making process. Questions
that scrutineers can be anticipated to ask include; have risks
been evaluated, and are the proposals to manage risks sound (Smart
Procurement Approvals, Edition 4, November 2000)?
Conclusion—Criterion partially met;
the Department comprehensively evaluated the costs of the options
and identified the risks but concluded that all of the risks could
not be quantified in a full COEIA.
The Department carried out an investment appraisal
of the options put forward for STSA which comprehensively assessed
the costs involved including support and other costs such as RAF
manpower and training, fuel, navigation and landing fees, and
the impact of flying hours. The results of the Investment Appraisal
seemed to show that An-124 options would be far more cost effective
than C-17 options, since they were both cheaper and more effective.
However, in evaluating the Investment Appraisal results, the Department
acknowledged the operational limitations of the An-124 as severe,
but not a "show stopper", and identified technical,
timescale, commercial, financial and political risks for both
the C-17 and An-124 options. The Department noted that the considerable
risks, most notably those of a political nature, associated with
the An-124 options could not be quantified reliably, so no account
had been taken of them in deriving the Investment Appraisal results[18].
In summary, the Department considered that the
aircraft options were so different that they could not be compared
on either a constant cost or a constant effectiveness basis. Therefore,
the Department decided that the formal COEIA process could not
be applied noting that, although with the information available
it was not possible to carry out a meaningful cost-effectiveness
comparison, the decision making process would need to take full
account of the risks.
In putting the case to Ministers for consideration,
the Department presented the costs and all the technical, timescale,
commercial, financial and political risks identified for the C-17
and An-124 options, together with the risk mitigation measures
proposed. It did not specifically recommend any of the options
but noted that it was evident that the C-17 or An-124 would be
physically capable of transporting all outsize elements of the
airlift bill within the required timescales. The Department highlighted
that the choice was between the relatively cheap, and large capacity
but risky enhanced An-124 100 charter, and the relatively expensive,
and less capacious but less risky lease or buy of C-17[19].
Criterion 4: Was the basis for the decision
transparent?
Source: Competitive tenders should on no
account be assessed, accepted or rejected on extraneous grounds.
Where the Invitation To Tender indicated the evaluation criteria
the bidder should be advised how his bid measured against these
criteria (The Department's Commercial Manager's Toolkit, Section
5, Tender Action—Competitive and Single).
Conclusion: Criterion partially met; Air
Foyle were debriefed but the lack of explicit evaluation criteria
and quantification of all risks hindered complete clarity. This
has not been helped by an inconsistent picture coming from the
Department.
Debriefing on the outcome of the procurement
was made difficult by the fact that the Department had not set
explicit evaluation criteria and had not been able fully quantify
all of the risks involved in the options, and because the decision
was finely balanced.
Air Foyle were debriefed in July 2000 following
the decision. The Department told Air Foyle that the basis for
the decision was a trade-off between operational capability, availability,
affordability, risk and international and industrial issues. However,
there has been confusion between the Department and Air Foyle
over the basis of the decision and particularly the weight given
to political risk. The Department has been inconsistent in the
picture they have presented which, together with the lack of explicit
and clearly weighted evaluation criteria for assessing bids and
the absence of risk quantification, have hindered complete clarity:
— in the debate following the announcement
of the selection of the C-17, the Secretary of State told the
House of Commons, that "what matters is not simply the capacity
of an aircraft. One must also have regard to the size of the airfield
where particular aircraft may land, and the number of times that
it may be necessary to fly an aircraft into and out of an airfield.
Sometimes, as we found in practice, the very large aircraft to
which the hon. Gentleman refers by implication cannot fly into
the airfield, and if they do fly into a particular airfield, it
is closed for a number of hours before the next aircraft can arrive.
A balance must be struck between the size and physical capability
of the aircraft in question, and the number of occasions on which
it is necessary to fly in aircraft"[20];
— in the debriefing meeting, the Department
told Air Foyle that the emphasis of risks, including political
risks, was different between the C-17 and An-124 solutions and
could not, therefore, be considered a deciding factor. When challenged
by Air Foyle, the Department confirmed their view that political
risk had not been weighted against the Air Foyle assured charter
proposal; and
— in evidence to the Committee of Public
Accounts on the Major Projects Report 2000 (which did not itself
cover the STSA procurement) and in response to a specific question,
the Chief of Defence Procurement said that "the key problem
with the Antonov is it did not provide secure, assured airlift".
The Deputy Chief of Defence Staff (Equipment Capability) supported
this view saying that "from the customer point of view that
was the important single factor".
In summary, the short Term Airlift decision
was left open to Ministers and the Department's Director Capability
Resources and Scrutiny told us that:
"the An-124 did not fully meet the requirement,
but the Air Foyle bid also retained greater political legal, and
technical risk (including the lack of assured financial viability)
than the Boeing proposals, despite the good work by the company
and the Defence Procurement Agency to try to reduce these risks
in the second competition. While the lack of operational flexibility
inherent in the An-124 aircraft might have been acceptable on
its own to the customer if the other considerable risks had been
mitigated, taking these and other factors into account (including
revised affordability guidelines), and the competing options,
Ministers preferred C-17 to An-124".
14 Letter from Tom Logan to Bruce Bird, 12 October
1999 (not printed). Back
15
Proposal for STSA Capacity, Air Foyle Limited, 28 October 1999,
para 1.1. Back
16
Proposal for STSA Capacity, Air Foyle Limited, 28 October 1999,
para 1.3. Back
17
Proposal for STSA Capacity, Air Foyle Limited, 28 October 1999,
para 1.11. Back
18
Hercules Rolling Replacement Tranche 2 and STSA COEIA, 4 February
2000, para 40. Back
19
Future Transport Aircraft (FTA) and Short Term Strategic Airlift
(STSA) Main Gate submission, 11 February 2000. Back
20
Hansard Column 157, 16 May 2000. Back
|
