Select Committee on Public Accounts Minutes of Evidence

Examination of Witnesses (Questions 1-19)




  1. Order, order. Welcome to the Committee of Public Accounts and welcome Sir Robert back to the Committee and Vice Admiral Sir Jeremy Blackham who are going to give evidence today on the Major Projects Report 2001. We are delighted to see the progress you are making with Smart Acquisition. As you know, this has long been a particular interest of this Committee. Can you tell us some of the key initiatives you are taking to ensure that Smart is succeeding and how are you going to build on the progress you have already made?

  (Sir Robert Walmsley) We start with a structural one of forming Integrated Project Teams. That has been right at the heart of Smart Acquisition. Secondly, we decided who our customer was and I am delighted to say he is sitting on my left. That produces a clarity as to who is entitled to commission articles from the Defence Procurement Agency and that has rippled out through into an approach which is called a through-life approach, which makes sure that we are far better constituted to take into account costs which arise after the capital acquisition. Beyond that I would just mention two things: one is better project management, which you should never cease to strive for and new techniques are always coming through and we try to learn them and apply them; learn from each other as well as learn from outside. The final point is to recognise that the single most dominant cause of delays is technical difficulties. We need to take a new approach to the rigorousness with which we assess the technical risk remaining in a project at the main investment point and the tool we have chosen to do that is called technology readiness level assessment.

  2. May I ask you a bit about slippage and in particular Figure 9, which is on page 12? Can you tell us a bit more what the Department can do to address these causes of slippage and when the measures you have in mind will take effect to slow the rate of slippage. Slippage is still taking place on all projects, not just some, but it is slowing. Perhaps you could tell us a bit more about that.
  (Sir Robert Walmsley) It only slipped on four projects this year, but that does not mean to say there is not a terrible history sitting there. If you look at Figure 9 you can see the top line says that technical factors, that is 331 months, sit at probably slightly more than half the total delay. I explained that I think that is because we take on too much technical risk in some of our projects. However, we are not the only ones, because there is a table further on in the report which shows that the Department of Defense in the United States are on the same concern as us and they had an assessment by an outside body who concluded that they were entering projects with technology at an insufficiently mature level. I am quite sure we need to do something about that. That is Table 12 on page 18. So that shows you that we are not alone in starting off on projects with technologies which are not sufficiently mature. We have introduced on some projects and will be introducing increasingly as we approve new projects, this concept of technology readiness levels. What that is, is a ladder which runs from one to nine and you are at level one when you have had an idea and you are somewhere about level two when you have shown in a laboratory that it can work. You work through trying to integrate that as part of a system which might take you half way up the ladder. If you had flight tested it, for example, you might be at technology readiness level six and if you had proved it in a system, you are at technology readiness level eight. Somewhere between six and eight is a pre-condition for embarking on a major new technology for a new project. That is the idea. It forces you to gather objective evidence as to whether the technology is mature enough, not just promise it because you think it is a good idea.

  3. If we look to paragraph 2.5 we see that you are not currently measuring risk reduction during the assessment phase. Can you tell us a bit more about the amount of time or whether you indeed spend the right amount of time and money on projects in assessment right now?
  (Sir Robert Walmsley) I think we do set the level about right, but I am concerned, as the report is, that we do not yet have an objective measure as to whether the assessment phase has accomplished its objectives. Most obviously, if we cannot make a decision to proceed through Main Gate, then it is reasonable to conclude that the assessment phase did not do enough. In a sense that is not quite good enough because the real proof of the pudding is whether the subsequent activity, that is to say after Main Gate, actually delivered a project on time and delivered performance within budget. A second point, obvious but still worth mentioning, is that the assessment phase should itself be conducted to budget and to time. If there are any exceptions to that, then they need to be explained. It does not mean they should be prohibited, but that if you discover something during assessment you should investigate it rather than pretend it did not happen in order to avoid a cost increase. The third point is the most sophisticated one. I shall try to explain it. Three-point estimates are a fundamental tool and give you an idea of the uncertainty remaining in a project: what is the lowest cost, that is a 10 per cent confidence level, the expected cost, which is a 50 per cent confidence level and the highest cost, a 90 per cent confidence level. It is quite logical to me that when you start assessment there should be quite a big spread there. By the time you get to Main Gate, that spread should be much narrower and the narrowing needs to be supported by objective evidence.

  4. May I ask about three-point estimates? We had been promised that you were going to have them for all projects. That has still not materialised. Can you explain why?
  (Sir Robert Walmsley) They have not materialised in the report because some of them were just at a critical point as we came through the back end of last year, through towards the datum date for this report, which was 31 March. I updated the Committee with one or two further three-point estimates in the summer. I am happy to say that as I sit here today I do have three-point estimates for all the projects in the assessment phase which are going to go ahead. We are completely reviewing the procurement strategy on TRACER following the United States' decision to pull out. I just want to make it clear that I am committed to three-point estimates, I have them here on a piece of paper sitting opposite the table which shows them incomplete and they are now complete.

  5. Let us go into that in a bit more detail and try to tie you down with a few questions. Presumably from what you have told me you agree that consequent on what the Committee of Public Accounts has advised in the past, all projects do have full three-point estimates at initial gate. Do you accept that?
  (Sir Robert Walmsley) I absolutely agree with that.

  6. How often do you think these estimates should be reviewed between Initial Gate and Main Gate?
  (Sir Robert Walmsley) A lot depends on the project. It seems to me that it is a fundamental management tool as to whether assessment is delivering the objectives. Every time you take a deliverable from a contractor, you should see whether that has had any impact on changing the spread of the three-point estimate.

  7. From what you have said I presume the answer to this is no. Are there any circumstances in which projects should not have three-point estimates?
  (Sir Robert Walmsley) There are no circumstances in which they should not have estimates. There are many circumstances in which those estimates should not be disclosed to the potential contractors.

  8. Should projects be using these estimates in their management of projects? Will all projects in the Major Projects Report 2002 have reliable three-point risk estimates for cost and time? I am asking you for a commitment now which I shall be interested to hear whether you can give. We will hold you to it.
  (Sir Robert Walmsley) I understand that. I shall just pull one word out from that question, to which I shall return, and say yes, unequivocal yes. The word I should like to take out is "reliable". You qualified it and said "reliable three-point risk estimates". Soundly based yes; for me to guarantee the reliability would be guaranteeing that what I predict will happen in the future will in fact happen. They will be properly based, they will be analytical, they are not holding a wet finger up in the air. They will be soundly based and yes is the answer.

  9. But not necessarily reliable.
  (Sir Robert Walmsley) "Reliable" is asking me to guarantee the future.

  10. That is fair enough. A lot of claims have been made that we are going to be saving £2.4 billion because of Smart Acquisition, but when we look at Figure 16 this shows that 70 per cent of Smart cost reductions are expected to arise on only 12 projects. We have to ask you why Smart Acquisition is affecting projects so disproportionately?
  (Sir Robert Walmsley) Smart Acquisition is a technique for looking to see whether there is a better way of doing things. What that means is that when we look at the cost estimate for a programme, we try to see whether there is a cleverer way, a smarter way of discharging a programme which will produce savings. Most of the projects, I hope, have very tight estimates; that is how people prepare estimates and therefore the scope, the assumption that there must be scope for saving money is something I want to be quite cautious about. Below and beyond this table in the great population of projects there are other savings which could properly be characterised as Smart. I pick an example which occurred to me as I read the table, the increased use of just-in-time delivery of sub-contracted supplies means that the value of the asset under construction, particularly in the case of a ship will not build up so early during its build period. What that means is that this new concept of paying interest on capital or assets in the course of construction to represent the opportunity cost of seeking to invest in something which is not delivering to the front line, if we deliver bits later, then there will be a smaller total interest on capital charge. The Astute submarine through a combination of later delivery and a number of other measures associated with optimising the construction of the submarine, will save in this LTC about £90 million. That has not been characterised as a Smart saving because the people doing it thought it was normal business. Beyond the list here there is a great number of them. I just start by saying that the huge saving attributed to Skynet 5 right in the first line is as a result of our confidence that we can execute the programme through the private finance initiative or public/private partnership. That has really compressed the timetable for delivering a satellite and has resulted in this large saving.

  11. A last question by way of light relief. Sting Ray. You will see what is said in Box 1 about Sting Ray. This is basically to try to stop you blowing yourself up if you bounce the thing on the deck. What are you doing to minimise the risks associated with handling Sting Ray?
  (Sir Robert Walmsley) Sting Ray is handled very carefully in frigates and there are set procedures for moving it on the flight deck at night and a number of other issues related to the arming of helicopters with Sting Ray torpedoes. The first thing we do is make all those who are responsible for handling Sting Ray aware of it. I have to say that the risk is absolutely minute and proper handling can eliminate the risk. That does not mean we should not be pursuing insensitive munitions as a subject.

  12. You are having to work on this despite the fact that the risk is absolutely minute because of Health & Safety at work and all the things which people worry about nowadays which they did not worry about to such an extent when both of you joined the Navy. How much is this all going to cost in making sure that Sting Ray cannot blow up if you drop it on the deck?
  (Sir Robert Walmsley) We are doing a study now to establish what it would cost to put in new design warheads and when we have done the study we shall know the answer to that. If you ask me to hazard a guess, and we were trying to do it across the total number of torpedoes involved, it would probably be in the region of £50 million, but that is just an informed estimate, informed in the sense that I am experienced with torpedoes. It is not informed by the results of the study. I just wanted to put it in context for the Committee.

  13. But the risk of this ever actually happening is infinitesimal.
  (Sir Robert Walmsley) It is infinitesimal.

  14. But you are now having to spend £50 million on tackling the problem.
  (Sir Robert Walmsley) You are leading me by the nose there. I did not say the £50 million was an established figure. I wanted to do that to put it in context. There are risks associated with fire, which we believe can be absolutely managed away, but nevertheless we have a duty of care to provide a weapon which is safe under all conceivable circumstances. Because that is now possible, these are new warhead design techniques and it is now possible to fix it, so we wanted to investigate the cost of doing so.

Mr Jenkins

  15. You have my sympathy. If there is a job which is impossible, it is probably yours. On the one hand you have a client who wants the best technology available and sometimes not available but in someone's head and you then have to deliver this by dealing with a set of contractors who really have all the power over you and you pay up what they want and you bear the risk. I should like to meet whoever called this Smart Procurement, because it is procurement but not very smart at times, is it? Your Department's strategic goal is to deliver 90 per cent of projects with approval, to time, cost and technical performance. Who picked the 90 per cent? Why not 80 per cent or 85 per cent? Normally 100 per cent is picked. Why is it going to take until 2005 to deliver this, because this is within your own hands. Why?
  (Sir Robert Walmsley) I would just make the point that the contractors do not have power over us. We have power over them because money is a tool: they want our money and in competition we have a very good handle over them. Even when there is no competition I believe you have a good possibility to exert the power of a normal commercial deal on them. I should like to come to the particular question. I said in answer to a point from the Chairman that we assess cost on this three-point estimating on the basis of a 10 per cent estimate, a 50 per cent estimate and a 90 per cent estimate. It is clear that there is always a remote possibility that something extraordinary could happen which would reduce the cost, or awful could happen and could knock the price beyond. In an analytical sense it simply makes sense to try to do three-point estimating on a 10/50/90 basis rather than 0/50/100. If you now imagine there is a snapshot in time where some projects have just been approved, when my guess is to say that they would be perfect, it would be very, very unlikely that a project approved yesterday would have bust its limits, as opposed to another project which was just about to reach its entry into service date, in which case you would expect 90 per cent of projects still to be within tolerance, because that is what a 90 per cent confidence limit does, you can see you will have a population where some are perfect and the worst are at 90. What that means is that on average they will be exhibiting the figures for about 95 per cent. It just so happens and you will recognise that from this report, that there is an enormous number of legacy projects which simply will not have worked their way out of the system by 2005 and the best we can do, provided every single project stays within limits is, having 90 per cent of our project within approval by 2005 and the long-term goal is to get to 95 per cent.

  16. I notice during the assessment phase of this system the Committee took evidence last year and the Department guaranteed to have performance measures in place by 1 April 2001. I may have missed something but are they in place at the present time?
  (Sir Robert Walmsley) We do have the performance measures in place which I explained to the Chairman. We expect to discharge the assessment phase to time and cost but we do not prohibit breaking out of that if we find the reason to do that. I have also explained the approach to narrowing three-point estimates and I have confirmed that they are all now available. I am beginning to see the glimmer of a system whereby I do not expect the spread on cost estimates to be any more than 10 per cent at the Main Gate approval, but we have not firmed up that number yet. I am really homing in on what the National Audit Office asked for, which is a quantified system for demonstrating the benefits of assessment. We are going to use the narrowing of the spread of the three-point estimates as one of the contributors to that.

  Mr Jenkins: I have individual questions on individual projects which I would have put to you, but I think the answers I am going to get are going to be to justify the slippage, costs, etcetera within this envelope of working within new frontiers, pushing cutting edge technologies, going down that road, so I shall take no more of your time.

Mr Osborne

  17. How do you achieve the balance of eliminating the risk from technology and also encouraging technological innovation? Defence procurement has been a major source of technological innovation in society. How do you achieve that balance?
  (Sir Robert Walmsley) We have a very substantial technological competence within the Department and I am including in that QinetiQ. One of the reasons Admiral Blackham commissions a very substantial applied research programme is to make sure that we have a very good understanding of the technical possibilities. That does not mean that they have to invent it themselves. It means the experts in QinetiQ have to be aware, as are many of my own engineers, of what is today technically in the pipeline and then think about whether it is a reasonable thing to pursue it. The joint responsibility of Jeremy Blackham and myself is to make sure that we spend enough money during the assessment phase to give a rational rather than just an enthusiast's opinion about the likelihood of that technology delivering on time to cost and to performance. I think I explained before that we need to discipline ourselves to take a more objective view. There is no shortage of ideas. The tendency seems to be that we take on board things which are frankly going to be tougher in the reality than we had hoped when we had the initial advice. I have to say some of that advice includes "promises" from industry and it is only when they are contracted promises that they get into trouble.

  18. There is no danger that if you move to these technology readiness levels you are going to over-compensate and only go with technology which is completely tried and tested.
  (Sir Robert Walmsley) I very much hope not, but I do recognise that is a risk. You could solve an awful lot of my problems, but not Admiral Blackham's, by just saying that is too risky. One of the ways round that is that we are aware of our competitors both industrially and in a defence sense and we are not going to let British defence equipment fall behind our competitors either industrially or in defence capability terms.

  19. That is good to hear. May I ask you about cost overrun? I take on board what the report says about the improvements being made on this. I am still right in saying, am I not, that according to the report of your 20 post Main Gate projects you are running at £2.6 billion cost overrun? I am new to this field. Can you set that in some sort of historical framework? Is it normal for the armed forces procurement to be running at such a high level of overrun.
  (Sir Robert Walmsley) The idea that it is normal would be intolerable to me. It happens to be a little bit less than it used to be. There are some reports which indicate that in other countries the average annual cost increase for defence procurement projects is around the figure of two per cent. So we are doing a bit better than that. If you look at other high technology projects, Jubilee Line, computer systems, Channel Tunnel, civil aviation projects, it seems to me we are not operating out of context with high technology projects or indeed perhaps we are. Perhaps we are doing a little better than some of those. I do not want to give anybody in the Committee the sense that I am complacent. Every single pound of those £2.6 billion is a pound we cannot spend on other requirements and we should fix it.

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2002
Prepared 4 July 2002