TUESDAY 9 JULY 2002

SIR JOHN BOURN, MR MARTIN SINCLAIR AND MR MICHAEL WHITEHOUSE

  40. And also, if the running costs—if you were not contracting out that work, presumably you would then argue that your human resources budget would increase?
  (Sir John Bourn) Yes.

  41. The 10 per cent budget increase will allow the production of 60 reports, as against 50 Value for Money reports previously. Has the National Audit Office considered the trade-off between quality and quantity in the number of Value for Money reports it produces and might it not be better to improve the effectiveness of the existing 50 reports each year, for example, by revisiting to check that improvements have really been made, than to produce 60 reports in future?
  (Sir John Bourn) Well, within the 60 reports, currently we have 14 reports in progress which do involve checking up and following up previous reports, PAC decisions and government acceptances. Three examples would be: we are doing a follow-up report on central government construction; a follow-up report on hip replacements; a follow-up report on hospital-acquired infection. So we do, within that 60, have quite a lot of follow-up. It is an important point that we should, in our reports, where this is appropriate and feasible, have a section on following up. We now provide in each of the Value for Money reports, references to previous reports in this area—what the PAC had recommended and what the government have come back to. So, in going for 60, that is not at the expense of neglecting follow-up. Follow-up is provided for.

  42. Would a report in itself suggest the need for follow-up?
  (Sir John Bourn) Well, sometimes it may well do. I think that the team that is working on a subject may, as they do the work, see the connection between their present activity and some allied activity and therefore bring into the study some follow up elements.

  43. Thank you. The NAO Corporate Plan mentions, on a number of occasions, its target of saving the taxpayer eight times its net cost. That is a bit like the effectiveness of Customs and Excise, and one gets the impression that the more money that is put in, the greater the savings would be. If the NAO was really saving the taxpayer eight times its net cost and this relationship would also hold for budget increases, the 10 per cent increase is clearly a good deal for the taxpayer. Could you provide more information to the Commission on these calculations and the extent to which they represent real resource savings to the economy?
  (Sir John Bourn) Yes, I am certainly glad to develop that point, because they are savings which come from our recommendations when endorsed by the Committee of Public Accounts, when accepted by the Government in the Treasury Minute, then costed out by the Department itself in conjunction with us. But it is not us saying "This is our view of what could be saved", this is the Department concerned saying "Yes, this is what the savings would be". And we have that looked at, and the methodology of that is looked at, by our internal auditors, who, going back to a previous discussion, are in fact an independent set of people, not employees of the NAO, reporting to the chairman of the Audit Committee, who, again, is not an NAO person. So, there is a proper methodology for doing this, and it means that the savings figures that we talk about are agreed by the Department and do represent real savings. Examples of the kind of savings that they represent is if a report points out that a Department owns more land and property than it is using, there are recommendations to dispose of it, sell it, lease it out.

  Another area which has produced a lot of savings is making people more sensitive to what they are paying for public utilities: gas, water, electricity and the telephones. Things you would think they might do themselves, but often are sadly neglected. In a number of cases it links up with your previous question. Departments are often bad at chasing up people who owe them money, and that is something that can be underlined. There are a whole series of areas in which these savings accrue but, it is not just us saying so, it is with the Department and subject to being looked at by a separate group of people who are not our employees.

  44. One of the things you always tell us to do on the Public Accounts Committee is see whether the civil servants whose projects go pear-shaped underneath them, actually bear the personal cost of that, ie their promotion is held back, or they do not progress in the Department itself. The LSE does these quite useful reviews of your Value for Money reports. I was wondering whether National Audit Office teams who produce not very good reports—according to the LSE or indeed in your own judgment—pay some sort of consequence in terms of their promotion and prospects?
  (Sir John Bourn) Yes, they do: they get less chance of getting promoted.

  45. And you are quite tough about that?
  (Sir John Bourn) Yes. I want to encourage them to do better so we will provide training, and we have a whole system of encouraging and helping people to be better. But certainly, those who do not respond do not do so well, both in terms of their earnings—they fall behind their peers—and certainly as far as promotion is concerned, they do not get it.

  46. And these LSE external evaluation reports are quite an important part of that process, or not?
  (Sir John Bourn) Yes, they are. This is a system of evaluation that we introduced ourselves. It had seemed to us that among the various ways in which you had your reports evaluated, it would be quite good to get a university of intelligent people coming at it from a slightly different point of view, and, first of all, Brunel did it—they were the people who won the first contract to do it—and when that was up we had a competition and LSE won it. And it is taken as an important matter, and it is interesting how these things work in organisations. The people in the NAO, above all, are concerned with what the PAC make of it and whether it goes down well with the committee; and also whether the government accept the recommendations. But it has been interesting to see the desire to get your ratings from LSE up as well. So, it has worked as a professional pride incentive to try and do better and show these people, coming at it, as I say, from a rather different point of view, that the teams can do well.

  47. Is it also used as an evaluation of their personal work?
  (Sir John Bourn) Yes, it is.

  48. I think other members of the Commission are going to talk more about, for example, the report on HM Prison Service. Although the scores from the LSE team were all right, there were some quite strong criticisms of the methodology, for example. I was wondering what happened to the team who produced this Value for Money report. Were they given more training; were they held back; have they been promoted; what has happened?
  (Sir John Bourn) They certainly were given the opportunity to have more training. I cannot recall off-hand all the names of the people concerned; but certainly, in the assessment of the annual performance and what that translates into money; on the Value for Money side, if you have not come up with a sensible methodology, if you have not shown some enterprise and ingenuity in developing it and gone out and found people who have an interesting line that can realistically be taken, you will fall back.

  49. Following on with that thought, looking at your own notes which you have prepared for us on the trend analysis of the LSE scores awarded, there is this graph, and this shows the percentages of reports scoring three or more, and three, I think, is recorded as "solid professional performance". How many of them fall below "solid professional performance", and how many above? Do you have that sort of information?
  (Mr Whitehouse) In the last financial year, 43 per cent of the NAO's reports scored a four or a five overall, which is very good or excellent/outstanding, using our categories. The remainder were three. We had no report that scored an overall two, as "below a professional performance".

  50. Looking at the overall column, which is the last column, and if you look at the dotted line which is 1998, and the solid black line which is 2000-01; in that final column the 1998 figure was over 10 per cent below the 2001 figure. What shortcomings did you have to put right to get those overall improvements?
  (Sir John Bourn) That did lead us to take various measures, which have had an impact on the scores in subsequent years, to improve the training in relation to the methodology, and that has taken a number of forms, in terms of the study guides produced by the Value for Money Development Team. It has also taken the form of a series of what we call "master classes", where we invite people to come to the office who have some methodological sophistication which seems relevant. We had, for example, the recent Cambridge Professor who won the Nobel Prize in Economics come to the office to give his views on how we could take account of his insight. As well as that, we have introduced with LSE in some cases not only the sort of cold reviews that come at the end, but hot reviews while the study is going on, so that LSE people could say before the study is set in stone, "What about your methodology in relation to this or that matter?" I think some of the recent reports do bring that out. For example, the report that we did on the government's gold sales was informed by Professor Binmore's expertise on auction theory. The report we have just finished on helicopter logistics has used a variety of operational research techniques relating to supply chain management. We have used aspects of economic modelling in the work we did on New Deal for Young People and what that meant in terms of employment created for them. So, it was an important point that LSE made. It is something that we recognise and something we are doing, and have done something about, to improve and make sure that we do use the best methodology.

  51. What is the expertise at the LSE public policy department? As far as I remember them, university departments are small, filled with people with a grievance against each other, who hate the outside world and would not know how to talk to a prisoner if they saw one. What is their qualification for talking about the report on prisoners?
  (Sir John Bourn) I am tempted to say that we make a contribution to LSE ourselves by giving them the opportunity to work together in this way; and with the public policy group which is a mixture of political scientists, economists, statisticians, operational research people, geographers, psychologists, and they bring in others to them. I think your comments about the social sciences in general are very well made. Certainly the LSE itself has set up a very wide range of interdisciplinary activities as a centre for economic performance, for example, as well as the public policy group. Maybe social scientists have come to see that if they are to prosper they have to find better ways of working together.

  52. They are not redoing the research; they are looking at your reports?
  (Sir John Bourn) That is right, yes.

  53. I see. Because some of it—they are marking the reports as if they were marking an undergraduate essay—is condescending stuff. Some of it is quite hard criticism. I think this is the public private partnerships, the "Airwave" project, which I thought had been a bit of a disaster, but I do not know. They are specifying scope as future prospects for VFM rather than actual performance to date. There was a let out, which lets them skirt over any hard-edged critical comments about the procurement. In the one on the prison service, it says, "The study team seemed too willing to rely on data provided by the Home Office on the Prison Service . . . and they should have looked at other sources". This is quite tough stuff.
  (Sir John Bourn) And it is quite right it should be tough.

  54. But, what happens to it? Is that incorporated in the report? Do you then go and look at the other sources?
  (Sir John Bourn) If the report is completed it is a comment related to the future work. But, if it comes out in the hot review, then we can take it into account. It is a good point to make, and it is one that we have sought to take into account in future studies.

  55. But unless it is either circulated with the report or incorporated in the report, it is just for internal staff. These are quite telling criticisms actually, which go to the nub of the issue. What happened to these two criticisms?
  (Sir John Bourn) They were criticisms which were taken into account in subsequent work. The report had been completed and published and therefore could not be altered at that stage. These were good points to make, but it does not dismiss the total and overall value of the report.

  56. I am glad to see the LSE back me up with one small matter, Sir John. In this prisoner re-offending report, under "minor points" they said: "The repeated use of the photograph of the female prison officer talking to the male prisoner on pages 2, 4, 5, 6, 7, 18, 26 and 36, . . .detracts from its overall appearance". We have had this discussion before, have we not?
  (Sir John Bourn) Indeed we have.

  57. In one sense they are quite hard hitting. They say on that: "More use should have been made of the data collected; a more evidence-based approach; a more specific and independent analysis of recommendations could have been pursued" and on "Airwave" they said, "There are some data weaknesses. We are concerned that at times this report is a bit vague and the study team withholds analysis".

  I see that we have only got these two reports from them. Do they ever criticise you for the fact that you have to agree these reports with the Departments—not your fault—and that may cut a bit of the edge away from them, and that the Department is constantly crawling over these reports with you, and saying, "Oh, leave that bit out" and, "That is a bit too harsh"? That is a difficulty that you labour under, is it not? And it makes our hearings better because, you know, you cannot wriggle away from it.
  (Sir John Bourn) As long as they understand the system that operates, that there is agreement as to the facts. Of course, there is not an agreement as to the conclusions and recommendations, which are ours to make, and which are not agreed with the Department. But very often if you agree the facts, it is difficult not to agree with the conclusions and recommendations, which is why so often, when you and other members of the Committee of Public Accounts ask the Accounting Officer if he agrees with the recommendations, they say, "Yes". I think that LSE, when they first did it, did comment on the need to agree the facts, but it is not a point that they dwell on now, because they see and understand that the value, as far as the Committee of Public Accounts is concerned, of having a report, where you do not have to spend the whole time arguing between the C&AG and the Accounting Officer as to what the facts are. They have signed up to those and that is the springboard to go forward.

  58. Final question from myself, looking at the BKR Haines Watts report on your VFM audit of your financial audit support team, "Technical Advice, Training and Development"; that is an important team, which is why it is being examined.

  In paragraph 1.9, it says:

  "It is important to consider the policy for staffing the support function. It is usual for individuals to stay in the team for around two years, normally as part of their career progression. However, they often retain other responsibilities, which serve to restrict their support work, and other demands further reduce the effectiveness of their tenures, with staff turnover being relatively high" .

  And if we look at paragraph 3.16 we find that, "relatively high" equals virtually 50 per cent in the year 2001-02, with the range being between seven months and seven years. Now for a very important group, that sounds a strange amount of volatility, and also an unnecessary distraction of their attention because of other responsibilities and other demands on their time. I regard that as quite an important criticism. It has not received a great deal of highlight in the report. What are you going to do about it?
  (Sir John Bourn) I accept it. I think it is a useful criticism and I am glad to have it. It is partly, of course, because this team has developed quickly. There is also an aspect of it that the members in the team enjoy the methodological expertise that goes with it, but they are men and women who are also keen to get to the front line. Perhaps I could ask Mr Sinclair, who runs the team, to say something about that?

  (Mr Sinclair) Yes, thank you. It is a combination of experienced staff and more junior staff. At the core of the team are three Audit Managers who have been in place for a very long time, and David Heald is very well aware of some of the quality that is embodied in those people. But it is a unit where we, quite deliberately, pick some of our brightest and best to spend a couple of years on the team, because it is a position which gives them exposure both to professional methodology issues, to the profession in terms of standards—

  59. But a couple of years is not, in fairness, criticised. It is the ancillary activities they have to do as well, and the fact that in the last year turnover was up to 50 per cent, with some people serving only seven months.
  (Mr Sinclair) There was an individual who served seven months because they were promoted out of the team. We are talking turnover within the office, we are not talking turnover external to the office, and it has been our policy to post them there for a couple of years, so turnover something like 50 per cent would be the natural consequence of that. As Sir John says, we are now looking very closely at the balance here, because it has been a question of balancing up the value of the central exposure, the very rapid, wide range of experience it gives those individuals, and the need to maintain continuity within the team. And we have listened to what the external auditors say and we are looking at extending the periods of appointment to that team.