Select Committee on Public Accounts Minutes of Evidence

Examination of Witnesses (Questions 1 - 19)




  1. Today we are taking evidence on the Comptroller and Auditor General's Report examining how the Ministry of Defence accept equipment off-contract and into service. We have four witnesses: Kevin Tebbit, Permanent Under-Secretary of State; Sir Robert Walmsley, Chief of Defence Procurement; Mr John Oughton, Deputy Chief of Defence Logistics; and Vice Admiral Sir Jeremy Blackham, Deputy Chief of the Defence Staff (Equipment Capability). Nice to see so many of you here, gentlemen. I hope it is because you take it seriously and not simply you trying to out number us! Because there are long and involved issues in a number of these matters, we will have our normal session and at the end of it there will be a closed session—and I expect there will be from what I have seen so far—without the public here. Mr Tebbit, let's start in the conventional way as you are used to. Paragraph 2.5 records that in half of the cases examined by the National Audit Office where the equipment failed to meet the Staff Requirement your Department made concessions which removed from the contractor the obligation to make good the shortfall. Paragraph 4.10 states that, given the contractor will usually price his work at the outset against the contractual requirement, it is surprising that you have not pursued financial recompense for making concessions in more cases. Why have you not pursued financial recompense more vigorously?

  (Mr Tebbit) Can I first say the reason I have come with such large numbers is because my colleagues represent the new way of dealing with acquisitions and procurement of equipment. This is, as it were, the Smart procurement head team although I as Accounting Officer clearly accept full responsibility.

  2. I was simply teasing you.
  (Mr Tebbit) I thought I ought to explain that. I think the story is indeed, as you say, one where in some areas there were financial shortfalls, but I think it is only fair to put it in the context of the total study. The NAO looked at some £10.7 billion worth of equipment over a five-year period and of that the financial shortfall, or the area where we could have sought recompense and did not, amounts only to £60 million worth of that equipment. Two-thirds of that was in fact one system, the 966 radar[3], so against that background the issues and the areas where we did not seek direct financial compensation are, I think, very much the exceptions rather than the rule. There will be a variety of reasons for doing so. In some areas the contractors met the requirement and it was an obligation the Department accepted to conduct any further changes as were required, but there is no general pattern there. As I say, the pattern is of satisfactory acquisition rather than failures. I would be happy to discuss or with my colleagues to discuss any particular financial shortfall areas you are concerned about.

  3. I am sure other members of the Committee will want to pursue that matter. I will press on as I have a lot to cover today and I may come back at the end or in the closed session. I note from paragraph 3.5 that over half of the projects surveyed by the National Audit Office had difficulty in demonstrating aspects of equipment performance at the time of accepting off-contract. The same paragraph records that most projects then undertook additional trials before the equipment could be accepted into service and that in 40 per cent of these trials further problems emerged when comparing performance with the Staff Requirement. Why do you find it so difficult to design acceptance strategies which address both the needs of users and contractual concerns?
  (Mr Tebbit) These are, by and large, extremely complex systems that are being introduced into service and some of the equipments are by their very nature impossible to verify until one is trialling them in an initial acceptance period. Naval systems, for example, ships, have to be brought on in that way. You may also be pointing out I will not say weaker areas, but in the period which the NAO had covered, we did indeed have two rather separate processes, one acceptance off-contract, the other one acceptance into service. The one coming from, as it were, the manufacturer acquired by the Defence Procurement Executive, the other the frontline saying it met their needs or not. And it is that dichotomy or rather the relationship between the two that is at the heart of Smart procurement, Smart acceptance. That is one of the problems that we are indeed overcoming into the way in which we are developing equipment now. So it may be to some extent that the difficulties were because there were two different audiences within the Department involved. In the future there is only one audience in an integrated form with Admiral Blackham here as a central customer for equipment operating in conjunction with the Defence Procurement Agency and, indeed, with the in-service support logistics structure which Mr Oughton represents in an integrated project form, so we look at all of these things from start to finish looking at performance throughout the project rather than straightforward requirement acceptance.

  4. I hear what you say, Mr Tebbit. I must say both from what is in this Report and from what we have seen in previous Reports and indeed anecdotal evidence, the tolerance of under-performance strikes me, frankly, as a culture of low standards, not necessarily one that is just extant today but over a long time in the MoD. I do not want you to give me an answer to that at the moment but I would like you when answering other questions that I am sure will come up on that to explain how you are going to change what strikes me as a cultural low standard.
  (Mr Tebbit) I would be happy to do so.

  5. Let me move on. Paragraph 3.26 states that where some form of remedy or concession was needed, the Department's analysis of acceptance options was largely qualitative and consensus reached through discussion and argument between the main stakeholders. Given the problems that can occur where subjective judgments on acceptance have been incorrect, as happened with the radio interference problem on Sea King Mark3a helicopter detailed in paragraph 3.29, why do you not undertake more quantitative analysis in assessing whether to accept equipment off-contract and into service?
  (Mr Tebbit) A quick word from me and then I might ask for Robert Walmsley to comment in a bit more detail.

  6. Absolutely. I was waiting for your striker to come into play!
  (Mr Tebbit) It is the case now under our new arrangements that every aspect of procurement and acceptance will have built into it a plan as to how we are going to verify the equipment at all stages, not just initial entry into service but also through life maintainability. We will be operating from what we call integrated test evaluation and acceptance plans. So the point has been and is being addressed, as the NAO recognise. It was never the case that we only had qualitative judgments earlier and now quantitative ones, but there is a shift in the quantitative direction. Perhaps I can ask Sir Robert Walmsley to comment on the more specific case.
  (Sir Robert Walmsley) The first thing to say, Chairman, is that we are extremely careful to quantify any suggestion that the shortfall threatens safety. That is meticulously documented. I think you mentioned the Sea King radios and the Military Aircraft Release for those aircraft very carefully documents what those radio shortfalls are. What I think the Report makes clear, because it says it, is that these were "irritating effects". They are quite difficult to fix. Anybody who has driven a car and passes a transmitting station like this will have experienced exactly the same effect. It is very difficult to fix close to a high-powered transmitter. We do think we have got the solution. It is under way and we expect to be able to demonstrate satisfactory performance in due course. As to documentation, it is fully documented.

  7. Others will come back to this and I will certainly come back to one of the points you made in a moment. In taking evidence on the 1997 Major Projects Report—I think you were with us Sir Robert—we heard about the technical difficulties which you had in procuring TUL/TUM Land Rovers which pulled to the left on braking. I remember you brought us the rubber bush that caused the problem. Paragraph 3 of Box 4 details the problems which you had in the earlier procurement of another utility truck—the RB44—which again revolved around the problems with braking. In the RB44 case you spent almost £1 million modifying the braking system prior to acceptance. How is it that the braking problem re-occurred once the vehicle was in service and why do you seem to be unable to prevent such basic technical difficulties re-occurring? My own handwritten note here says it is not rocket science. Mr Tebbit?
  (Mr Tebbit) You have got one here which I agree is partly our fault but I will ask Sir Robert Walmsley to cover it in detail.

  8. There we are, a hot pass, Sir Robert!
  (Sir Robert Walmsley) It is not rocket science, Chairman, but it is an extremely complex interaction between suspension and the front axle and the effects of braking. I cannot explain why it is so hard to make trucks, whether medium or heavy, that brake in a straight line. As to your specific question as to why did we keep on thinking we had fixed it and then find we had not, I think there is a clearer story. First of all, after the initial production vehicles came off the line we tested them. We asked for it to be fixed. Because we had asked for a specific problem to be fixed we introduced extra tests and it was those extra tests which showed that there was still a problem. We then fixed that and put the vehicles into service; that was the first stage. Additional tests uncovered additional faults. The second stage was that in operation we discovered that the brakes required far too frequent adjustment in order to keep them up to scratch in terms of performance and it was not really a practical proposition for the Army to keep on doing this continual adjustment, so in some vehicles where they did not continually adjust the brakes the performance was not adequate, that is why we reverted to automatic break adjustment which had been in the original specification but which we had removed. The third problem is that a lot of these vehicles were kept in store. As part of the defence cost studies we decided it was not sensible to go on maintaining everything we kept in store as though it was in operational use. What we missed—I cannot explain it—is that some seals on the brake cylinders, which anyone who has tried to fix their brakes will know, perish if they are kept in store. If we had not conducted the maintenance these vehicles would be brought back into service and the braking problems exhibited. So they are sensitive to not following maintenance. We did not and that is why there was a problem. We follow the maintenance now. The vehicles now brake properly.

  9. I hear what you say. One has to say, however, that interactions between mechanical systems are the sort of problems that designers in the commercial vehicle industry solve all the time. As for storing vehicles, one often sees lots of vehicles stored in the open and they are sold afterwards and they are not expected to go wrong. I think we are going to find people coming back to this question because it seemed to me to be quite straightforward, as we discussed last time you were with us.
  (Sir Robert Walmsley) And they were commercial vehicle manufacturers, Chairman. It was them who did not understand the effect of the bushes and it was them who did not understand they needed a tie rod between the axle and the structure of the vehicle.

  10. It was your Department who set up the contract with them and you should have demanded that from them.
  (Sir Robert Walmsley) We do not specify how to build vehicles, Chairman, we specify the performance we want and the performance was found to be inadequate.

  11. I am sure others will want to come back to that one in some detail. Paragraphs 3-6 of Box 5 on page 36 explain the problems you had validating the performance of the software in the FADEC system on the Chinook MR2 helicopter. In most cases the people who understand an item of software the best are those who designed it. Even employing other firms of consultants will not necessarily give you the knowledge and understanding you require. Given the shortcomings in the processes followed on the Chinook MR2 programme highlighted in the box and the fact that most defence equipments are now more or less software dependent, what approach does the Department adopt to gain sufficient assurance that the software in equipments which you are accepting off-contract and into service will perform as required?
  (Mr Tebbit) We use our own people within the integrated project team. When a project is developed there is a specific emphasis on any software aspect of it, whether in embedded chips or with a more general IT application. That is taken as a discrete element in the project and is followed in detail by the team itself or through contracts to DERA, the Defence Research Establishment Agency[4] which we often use for these things, or external consultants, or industry itself all performing to standards of safety which are nationally and internationally recognised to make absolutely certain that what we bring into service actually works and to make absolutely certain that what we specify can be tested and proven. There is a whole industry which ensures that this happens, Mr Chairman.

  12. Let us read paragraph 4 of Box 5 on page 36 where it says: "On the basis of EDS-SCICON's work"—this is on behalf of Boscombe Down—"in October 1993, Boscombe Down advised the Department that they could not recommend Controller Aircraft Release for the Chinook Mk2 because of the `unquantifiable risk associated with the unverifiable nature of the FADEC software,' and concluded that rewriting the software was essential." Why was it not rewritten?
  (Mr Tebbit) It was not rewritten because the design authority for the system informed Boscombe Down that the technique that Boscombe Down was seeking to use, which is called static code analysis, was inappropriate for verifying or validating the software provided because it had been designed to a different philosophy and testing programme using dynamic testing methods. In other words, whereas the static code analysis that Boscombe had chosen to use was looking at the actual design, the Design Authority made it clear after this that they were testing the safety performance of the design and did so through 70,000 hours of dynamic testing. It is important to be clear about what this meant. Boscombe Down and SCICON and the Department did not say that the FADEC software itself was unsafe. What Boscombe Down said was that they were impressed by the handling and performance of the aircraft using this method. Their concern, as I say, was that they were unable to verify it using their preferred methods of testing, but it was made clear to the Department by the Design Authority that this was not the right method to be using and subsequently this was demonstrated partly by ourselves using a more dynamic method for verification which was indeed done, partly by the evidence of the US themselves in running a very similar system with the US Forces which had about 95 per cent compatibility of software which had been introduced in 1990 and partly the evidence that the system was recommended to be and subsequently proved to be ten times more reliable and efficient than the previous method of controlling these engines which was by hydrodynamic methods.

  13. What is that based on, that ten times more safe?
  (Mr Tebbit) That was the difference between the predicted improvement in reliability on the previous engines that Chinooks used in the way of managing those engines.

  14. Can you let us have the figures that you are basing that on?
  (Mr Tebbit) That is very simple, Mr Chairman[5].

  15. Do you know what a rare subroutine problem is?
  (Mr Tebbit) I am sorry?

  16. Do you know what a rare subroutine problem is?
  (Mr Tebbit) I cannot say I do.

  17. It is what happens when you have two subroutines in a piece of software which are both called "rarely". As a result, when you are testing that software they do not show up very often. If they interact badly when they do show up it can lead to an unpredicted error. As I understand it the static code analysis was a logical analysis to see whether there was a potential clash between parts of the software. The dynamic analysis is effectively a test programme. I read this and I asked the Director of EDS-SCICON to tell us what this meant to them. Let me just read you one component of a note sent to us. He said: "EDS would not have expected to find any Category 1 anomalies in the software they were testing and would have expected far less Category 2 anomalies than were listed. He noted that EDS-SCICON had been sufficiently concerned about the safety implications of their findings to stop testing and to raise their concerns immediately with Hawker Siddeley (who produced the software under sub-contract from Textron-Lycoming) rather than waiting to produce a final report having tested all the software. Mr Bullock told us that EDS-SCICON were unhappy with the response they received from Hawker Siddeley who dismissed the concerns as being trivial[6]." Are you still happy with what you said to me about the safety of the software?

  (Mr Tebbit) I repeat, Mr Chairman, neither EDS-SCICON or Boscombe Down found the software to be unsafe. What they said was that they could not verify using their preferred methodology. The Ministry of Defence then had to consider what to do about it. What it did was go back to the design authority and ask for their comments. Their comments were that this was a different philosophy and a technique which would not work with this system in the way intended to demonstrate safety. They assured us that their system was safe and there was the proven record with the US Forces already. The Ministry of Defence decided to proceed in a way which imposed a weight limitation on the Chinook just in case there was any possibility that there could be any difficulties with the engines. So they proceeded on the basis of a weight limitation of 18,000 kilograms which meant that even if one of the two engines failed it would still be perfectly safe to fly. As it happened none of those occurrences occurred and it was unnecessary to have done that. Nevertheless that limitation remained in place, partially at least, until four and a half years later when it was finally lifted in full. There have been no problems since.

  18. How many aircraft of the US fleet have this brand of FADEC?
  (Mr Tebbit) I would not know the answer about the US fleet.

  19. You were citing it as a reason to take this as a safe system.
  (Mr Tebbit) As I said before, the US fleet has got a system which is 95 per cent the same. There has not been one problem with the UK fleet and it has now been flown for something like 119,000 hours.

3   Note by Witness: Of the £60 million financial shortfall, we did not pursue financial compensation for three-quarters (not two-thirds) and that was in respect of one system, the Radar Type 996, not the Radar Type 966. Back

4   Note by Witness: It is in fact, The Defence Evaluation and Research Agency. Back

5   Note: See Evidence, Appendix 1, pages 31-42. Back

6   Note: See Evidence page 5; Appendix 5, page 64; and Appendix 1, pages 31-42. Back

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2000
Prepared 30 November 2000