Select Committee on Treasury Minutes of Evidence

Examination of Witnesses (Questions 80-99)



  80. I am not at all sure what the original inquiry wanted to know. You do say in the report that there was a concern about systematic drift and I think this Committee would like to have some knowledge about how we ascertain systematic drift and how we do anything about it and how we are going to respond to that.
  (Sir John Kingman) Exactly so. There is a lot of research, a lot of academics have been in touch with us with interest about this. There is a lot of educational research. It is not fundamentally a statistical question. Our concern has been to make sure that the results of this academic research are available to and used properly by the policy makers in the department. As far as we can tell that is happening but it is quite difficult to be sure of that. You cannot ask the Statistics Commission whether there is grade drift and so on, those are not statistical questions, they are educational questions which are extremely difficult. They can sometimes have a statistical component but it would be wrong for the Statistics Commission to pontificate about that.

  81. The next question I wanted to ask might perhaps not be for you. Of the statistical component would you say that it was significant enough as to be of concern or will I find that out from the note?
  (Sir John Kingman) I do not think I can really help on that, I am sorry. I think you are going outside the proper sphere of the Statistics Commission.

  82. In that case I will wait and see your note.
  (Sir John Kingman) I wrote to Len Cook on 1 October, I wrote several letters to Len Cook on 1 October, and noted that there was a report in 1999 which investigated accusations that test standards were being deliberately manipulated and concluded that there was no evidence of that, but I went on to say that we know that the Office for National Statistics is having a quality review on education performance statistics starting sometime next year and we are really encouraging him and the department to take this seriously. "I would be grateful if you could ensure that this area is examined in the near future and let me know what the timescale will be."

  83. Some of the questions I would have liked to pursue this afternoon perhaps were to do with how the statistics are compiled in the first place and what confidence we can have in them but if you are not ready to deal with that then I will have to be patient.
  (Sir John Kingman) I would like to offer you a note that summarises the various lines of correspondence that have happened already and obviously this letter will be a part of that.

  84. Can we move on to hospital waiting lists. You say that you are satisfied that there has not been any interference in the statistics and the presentation of the information about hospital waiting lists and that you are satisfied that this has all been done in the usual way. Can you describe to me what the "usual" way is. I will tell you why I am asking you that. When I was in my local hospital recently a surgeon said to me, "Look at those figures up there. They are all wrong, you know, I have not put mine in." What confidence can I have as a politician, if a surgeon can tell me that he has not put his figures in yet, that what I am getting at the end of the process is an accurate picture of what is happening?
  (Sir John Kingman) We do not have the resources to go hospital by hospital and check what is going on and what we are trying to do is to look at the ways in which the hospital statistics are being put together, the checks on them and so on, and also the work that the National Audit Office, which does have the resources, is doing on hospital waiting lists. There is obviously a possible overlap between the interests of the Commission and the interests of the National Audit Office because they are concerned that when these numbers are put out they do actually contain everyone's figures and there are not miscountings or deliberate deceptions about the figures. And it is certainly suspicious that there are still substantial seasonal variations in the waiting list which is linked to the dates when the performance statistics are produced. So there is room for concern and what we are trying to do is to work with the National Audit Office so that their big battalians can look at the sort of questions which worry us about the validity of the statistics themselves. That is something where the National Statistician also has an important interest, so the three of us are trying to work together.

  85. So the dip that occurred before the General Election, which I assume is what the media were looking at which prompted your interest, presumably from some concern about interference (and I may be wrong in that) if that was a seasonal dip that we might expect periodically, by this amount of time would we have expected to have seen yet another seasonal dip? If so, have we seen it?
  (Sir John Kingman) We have seen seasonal dips each year and we have not had an election each year, so it is clearly not just an election phenomenon.

  86. That is good to know.
  (Sir John Kingman) It may be a phenomenon—and I only say "may be" because this is still very much under study—to do with the dates that affect people's performance pay and that sort of thing. These are the sorts of hypotheses which the National Audit Office and we are interested in.

  87. Would you pursue evidence to support that hypothesis?
  (Sir John Kingman) I think if there were any serious suspicions that that sort of thing was happening, that is something we would be very interested in.

  88. As it is, you are assured it is perfectly acceptable information?
  (Sir John Kingman) No, I did not say that.

  89. What are you saying?
  (Sir John Kingman) What I am saying is we and the National Audit Office are working on it and that they are the people who have the big batallions that can investigate this at the hospital level.

  90. So your satisfaction only lies in the fact that the statistics were compiled in the usual way not that the the statistics themselves are satisfactory?
  (Sir John Kingman) That is right.

Mr Laws

  91. Just following up this particular area, I am pleased that you said earlier that you wanted to focus as a group on some of the key statistics that are of particular interest to people in the country and raise confidence about those statistics. We discussed this issue of the waiting lists and whether there was anything suspicions in the pattern. One of the issues, as you know, that is raised by the waiting lists is those treatments and types of medical wait that are excluded from them. Many of us see in our surgeries the whole time people who in many cases have a two-year wait to see a consultant and you discover when you go to the hospital that they are not counted as being on a waiting list that is looked at nationally. Consultant referrals, diagnostic tests, angiograms, MRI scans and endoscopies are not on that. Is that the type of issue which you as a group can be concerned about, the quality and the extent of the data, particularly as you want to reinforce its public credibility?
  (Sir John Kingman) This is a very important area because it is clear that the system of targets has come to stay in our political life. The Government will lay down targets and will tie Treasury funding to the achievement or otherwise of those targets and inevitably choosing particular targets which will be defined simply enough for people to understand will distort the service. Whatever you choose people will play to those targets rather than to other things which have not been included in the targets. The temptation will then be to have more and more targets and many of these will be very difficult to measure.

  92. But do you see it as part of your job to improve the quality of statistics such as these, for example, by including people in the statistics who are waiting for primary care treatments, by including people who are waiting consultant to consultant, including people who are waiting for angiograms, diagnostic testing? Do you think the credibility of these statistics about waiting times in the NHS is important, that it is a legitimate area of inquiry for your group to look at whether the statistics should be widened and amended?
  (Sir John Kingman) I think our main function, our core function, is to ensure that where a target has been laid down the statistics which decide whether that target has been met should be statistics in which people can have confidence. That has two aspects. It involves ensuring that the targets that are laid down are actually measurable and, secondly, ensuring, as with all government statistics, that the statistical outputs are high quality and not distorted. You are asking us whether we would want to move into a different area of actually criticising the targets themselves. Strictly speaking I do not think that is part of our terms of reference although it is almost impossible to exclude that sort of consideration when you start looking at targets.

  93. So if people raise that issue with you, if my constituent who is excluded from all these waiting lists, both inpatient and outpatient, and is waiting until August 2004 for a neurophysiology report, came to you and said "This is ridiculous, the Government is highlighting all these areas that are publicly visible but I am being left to wait for this consultant to consultant appointment", is that something as a broad issue that you as a group would look at?
  (Sir John Kingman) It is something about which every citizen would have concern. I do not know whether the Statistics Commission would be regarded as having a particular expertise in that area that would influence ministers. After all, the choice of targets is typically a highly political matter and we can give advice but whether our advice would be thought to have any value in that sort of case I do not know.

  94. But you could give advice?
  (Sir John Kingman) We can give advice on anything but we are better to concentrate on those areas where we have some demonstrable expertise.

  95. May I ask you then very briefly whether you would comment on two other things in relation to this same issue of hospital statistics. The first is that there are some hospital statistics about particular waiting times that may be kept by hospitals but are not collected centrally and reported nationally and they would be in the types of areas I have discussed. I will ask you the other question to save time. Is it the job of the Statistics Commission to comment on that sort of thing about the case for publishing that type of data nationally? In terms of some of these areas of waiting times there seems to be an inconsistency in the way that different hospital trusts record data. In a newspaper article earlier on this year about this type of problem they were talking about the way in which some trusts count endoscopies as part of the waiting list and some do not. They reported a letter to chief executives of trusts in the South East of England from the NHS Regional Director advising that endoscopies should be removed from the waiting list but saying that this should be done gradually to avoid some lurches in the figures. Is that something that would be of concern to the Statistics Commission that you might comment on?
  (Sir John Kingman) I have not seen that letter but that sounds to me a letter that would concern us because if statistical information is to be of any value it must be the real information and not information which is tapered in order to avoid lurches. I have not seen the letter. If I do, I think it is something the Commission ought to consider.

  96. If I can get you a copy, it is something you will look at?
  (Sir John Kingman) Certainly. We are charged with looking at the scope of National Statistics and there are some very curious border lines in the National Health Service area as to which figures are National Statistics and which are not. We have not done a lot of work so far on the scope of National Statistics simply because it was pointless to do so before we had the Code of Practice which said, effectively, what it means to be a national statistic. Now that we have that Code, there are a number of areas of which I think the Health Service will be one where we will want to look at the boundary between National Statistics and management information which is not National Statistics. Clearly every organisation collects numerical information, some of it of a very trivial sort which it would be ridiculous to put within the discipline of National Statistics, but where exactly you draw the line does need careful consideration and the sort of example you cite shows how important this is in the NHS area.


  97. Finally, could we turn to the Census on which you have commented in your report. The first Census result seemed to show that the population in the UK was about a million less than previously forecast which seemed to come as a bit of surprise to me. Was it a surprise to you?
  (Sir John Kingman) Yes, I think it was a surprise to everyone.

  98. Does that suggest there are some problems with the methodology of the Census or the methods used on the previous Census?
  (Sir John Kingman) It certainly produces very important questions that need to be answered. If a scientist measures the same thing in two different ways and gets quite different answers he should not say which is right; he should say what is the reason for the discrepancy and let us try and understand it. On the face of it, the Census results ought to be more accurate than the extrapolations that are done between Censuses. If that turns out to be the case when there has been more detailed study that will not be at all surprising and it will be a caution to us about using the extrapolation techniques. This, of course, has very important implications for the question of what the next Census, if any, should look like because clearly if our methods of looking at births and deaths and migration are not sufficiently accurate to give us a figure within a million, a true figure, then they should not be used over a 10-year period to make important policy decisions. But I think there are still a lot of questions to be answered about this last Census in relation to the people who may have been missed both by the Census itself and by the subsequent Census coverage survey. The estimates about missing people are normally taken from an assumption that there is independence between these two exercises. In fact, there are likely to be particular categories of people who are missed out both by the Census and by the coverage survey and whether that is part of the explanation of the million difference between the two figures is still a question for discussion.

  99. You have anticipated my second question on the Census. It is perhaps important to distinguish between the two different figures of a million, is it not, the million lower overall population and then the adjustable figure of around about a million to take care of people who were not there? Would you normally have expected that adjustment of a million to be of that order from the coverage survey?
  (Sir John Kingman) Yes, the fact that you did not get Census returns from a million people is not at all surprising and is in line with international experience. The techniques for what they call "imputation", although they are very easy to make fun of in the press, are in fact very well grounded in statistical principle. It is important to realise that the estimate of the total population size that results is a statistical estimate. The experts have estimated that it is a figure plus or minus 100,000, so that needs to be borne in mind in using the figures. Of course when you break those figures down to smaller areas you get larger proportional errors. That is all within the scope of standard statistical methodology, there is nothing to worry about there. I think the discrepancy between the estimate that arises from the Census and the estimates that have been made over the 10 years since the last Census do need very careful study. I suspect that some of them will turn out to result from mistakes, not mistakes but poor methodology in 1991 which formed the base of the extrapolations. If those were then, of course, the extrapolations from them are bound to be wrong. I think it is important, and ONS agree with this, to understand where these discrepancies come from and, of course, to break them down at the local level as well because they do seem to be concentrated in certain areas where there is particular difficulty.

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2003
Prepared 6 January 2003