Select Committee on Public Accounts Appendices to the Minutes of Evidence



National Audit Office Findings

Criterion 1:  Was Air Foyle's proposal considered as a serious alternative?

      Source: No firm should have been invited to tender unless the Ministry was prepared to place a contract with them (the Department's Commercial Manager's Toolkit, Section 5, Tender Action—Competitive and Single).

      Conclusion—Yes; criterion met.

  The circumstances of the Short Term Strategic Airlift were unusual in that the initial competition was terminated and the procurement was taken forward through a competitive environment phase. The Department engaged in significant dialogue with industry, and at the way forward meeting with Air Foyle on 10 August 1999 at the start of the competitive environment phase, the Department circulated for debate a revised Cardinal Points Specification for the STSA procurement. This excluded operational requirements from the original Cardinal Points Specification that the Antonov could not achieve.

  Subsequently, following comments from industry and internally, the Department concluded that the Cardinal Points Specification approach was unnecessarily complex, given that they were fully aware of the capabilities of the An-124 from the earlier competition[14]. Instead, with the agreement of the potential bidders, the Department issued a simplified Request For Proposals (RFP) specifying the number of Antonov aircraft required (two) and the timescales. The Department, therefore, accepted the capability of the An-124 as a viable option.

  We have not seen any evidence to support suggestions that the Department did not mean what it said at the start of the competitive environment phase and did not seriously consider the An-124 as a viable alternative.

Criterion 2:  Were the Department's criteria for evaluating proposals explicit?

      Source: The Invitation To Tender should normally provide an indication of any key criteria to be listed in the evaluation of bids. Some guidance to bidders on the performance weighting scheme is necessary to ensure that the best result is obtained from the competition (The Department's Commercial Manager's Toolkit, Section 5, Tender Action—Competitive and Single).

      Conclusion: No; criterion not met but Air Foyle were aware of the lack of defined criteria.

  The Department told us that during the "competitive environment" phase, the Boeing bid was assessed against the original Staff Requirement and the two Antonov bids were assessed against the RFP with the winner being assessed against the C-17. In broad terms, the RFP noted that it was essential that any STSA An-124 100 solution provided a more assured level of availability than the ad-hoc charter arrangements that were being used to supplement the existing air transport capability and represented value for money. However, the RFP did not explicitly state what criteria were to be used to evaluate the bids or their relative weighting in the evaluation.

  Air Foyle were aware that the evaluation criteria were not explicit when they submitted their proposal. They noted in their proposal that the RFP sought a greater level of assurance than was available from existing contracts together with value for money but provided no guidance on how either assurance or value for money would be determined[15]. Air Foyle also understood that "political assurance"—defined by the Department as the customer being able to use the service without political interference—was the main thrust of the Department's concerns[16]. Their proposal recognised that the Department's requirement was extremely difficult to meet, demanding a security of service from the contractor which was ill-defined and political in scope[17].

Criterion 3:  Were all the costs and risks fully evaluated through a combined Operational Effectiveness and Investment Appraisal (COEIA)?

      Source: A full investment appraisal will be carried out whenever a substantial investment is contemplated. A COEIA is a vital part of the decision-making process. Questions that scrutineers can be anticipated to ask include; have risks been evaluated, and are the proposals to manage risks sound (Smart Procurement Approvals, Edition 4, November 2000)?

      Conclusion—Criterion partially met; the Department comprehensively evaluated the costs of the options and identified the risks but concluded that all of the risks could not be quantified in a full COEIA.

  The Department carried out an investment appraisal of the options put forward for STSA which comprehensively assessed the costs involved including support and other costs such as RAF manpower and training, fuel, navigation and landing fees, and the impact of flying hours. The results of the Investment Appraisal seemed to show that An-124 options would be far more cost effective than C-17 options, since they were both cheaper and more effective. However, in evaluating the Investment Appraisal results, the Department acknowledged the operational limitations of the An-124 as severe, but not a "show stopper", and identified technical, timescale, commercial, financial and political risks for both the C-17 and An-124 options. The Department noted that the considerable risks, most notably those of a political nature, associated with the An-124 options could not be quantified reliably, so no account had been taken of them in deriving the Investment Appraisal results[18].

  In summary, the Department considered that the aircraft options were so different that they could not be compared on either a constant cost or a constant effectiveness basis. Therefore, the Department decided that the formal COEIA process could not be applied noting that, although with the information available it was not possible to carry out a meaningful cost-effectiveness comparison, the decision making process would need to take full account of the risks.

  In putting the case to Ministers for consideration, the Department presented the costs and all the technical, timescale, commercial, financial and political risks identified for the C-17 and An-124 options, together with the risk mitigation measures proposed. It did not specifically recommend any of the options but noted that it was evident that the C-17 or An-124 would be physically capable of transporting all outsize elements of the airlift bill within the required timescales. The Department highlighted that the choice was between the relatively cheap, and large capacity but risky enhanced An-124 100 charter, and the relatively expensive, and less capacious but less risky lease or buy of C-17[19].

Criterion 4:  Was the basis for the decision transparent?

      Source: Competitive tenders should on no account be assessed, accepted or rejected on extraneous grounds. Where the Invitation To Tender indicated the evaluation criteria the bidder should be advised how his bid measured against these criteria (The Department's Commercial Manager's Toolkit, Section 5, Tender Action—Competitive and Single).

      Conclusion: Criterion partially met; Air Foyle were debriefed but the lack of explicit evaluation criteria and quantification of all risks hindered complete clarity. This has not been helped by an inconsistent picture coming from the Department.

  Debriefing on the outcome of the procurement was made difficult by the fact that the Department had not set explicit evaluation criteria and had not been able fully quantify all of the risks involved in the options, and because the decision was finely balanced.

  Air Foyle were debriefed in July 2000 following the decision. The Department told Air Foyle that the basis for the decision was a trade-off between operational capability, availability, affordability, risk and international and industrial issues. However, there has been confusion between the Department and Air Foyle over the basis of the decision and particularly the weight given to political risk. The Department has been inconsistent in the picture they have presented which, together with the lack of explicit and clearly weighted evaluation criteria for assessing bids and the absence of risk quantification, have hindered complete clarity:

    —  in the debate following the announcement of the selection of the C-17, the Secretary of State told the House of Commons, that "what matters is not simply the capacity of an aircraft. One must also have regard to the size of the airfield where particular aircraft may land, and the number of times that it may be necessary to fly an aircraft into and out of an airfield. Sometimes, as we found in practice, the very large aircraft to which the hon. Gentleman refers by implication cannot fly into the airfield, and if they do fly into a particular airfield, it is closed for a number of hours before the next aircraft can arrive. A balance must be struck between the size and physical capability of the aircraft in question, and the number of occasions on which it is necessary to fly in aircraft"[20];

    —  in the debriefing meeting, the Department told Air Foyle that the emphasis of risks, including political risks, was different between the C-17 and An-124 solutions and could not, therefore, be considered a deciding factor. When challenged by Air Foyle, the Department confirmed their view that political risk had not been weighted against the Air Foyle assured charter proposal; and

    —  in evidence to the Committee of Public Accounts on the Major Projects Report 2000 (which did not itself cover the STSA procurement) and in response to a specific question, the Chief of Defence Procurement said that "the key problem with the Antonov is it did not provide secure, assured airlift". The Deputy Chief of Defence Staff (Equipment Capability) supported this view saying that "from the customer point of view that was the important single factor".

  In summary, the short Term Airlift decision was left open to Ministers and the Department's Director Capability Resources and Scrutiny told us that:

    "the An-124 did not fully meet the requirement, but the Air Foyle bid also retained greater political legal, and technical risk (including the lack of assured financial viability) than the Boeing proposals, despite the good work by the company and the Defence Procurement Agency to try to reduce these risks in the second competition. While the lack of operational flexibility inherent in the An-124 aircraft might have been acceptable on its own to the customer if the other considerable risks had been mitigated, taking these and other factors into account (including revised affordability guidelines), and the competing options, Ministers preferred C-17 to An-124".

    14   Letter from Tom Logan to Bruce Bird, 12 October 1999 (not printed). Back

    15   Proposal for STSA Capacity, Air Foyle Limited, 28 October 1999, para 1.1. Back

    16   Proposal for STSA Capacity, Air Foyle Limited, 28 October 1999, para 1.3. Back

    17   Proposal for STSA Capacity, Air Foyle Limited, 28 October 1999, para 1.11. Back

    18   Hercules Rolling Replacement Tranche 2 and STSA COEIA, 4 February 2000, para 40. Back

    19   Future Transport Aircraft (FTA) and Short Term Strategic Airlift (STSA) Main Gate submission, 11 February 2000. Back

    20   Hansard Column 157, 16 May 2000. Back

    previous page contents next page

    House of Commons home page Parliament home page House of Lords home page search page enquiries index

    © Parliamentary copyright 2001
    Prepared 17 January 2001