Select Committee on Education and Employment Minutes of Evidence

Examination of witnesses (Questions 390 - 399)




  390. Can I welcome Bahram Bekhradnia and John Thompson from HEFCE. It is nice to see you again, as you have been in front of the Committee before on the previous part of our report. As I said to the previous people before us, we run these things reasonably informally as you know. We want to get the most out of them. Were you in the room when the evidence was being given just now?
  (Mr Bekhradnia) John was there for most of it; I was there for part of it.

  391. Can I start by picking up the theme that was developed by Claire Callender in the sense that here we are, HEFCE has been around some time now, but what they are saying is, "We have not got the data to make serious policy judgements on". What the hell has HEFCE been doing all these years if we do not have proper data with which to inform ministers in order to make good policy?
  (Mr Bekhradnia) I did not hear Claire say that, Chairman. First of all, may I thank you for this opportunity to join you. You know me because I attended, as you said, before. John Thompson is our Senior Data Analyst who is responsible for the production of the performance indicators and the other work that we have done in this area. I did not hear Claire say that and I hope that the way you describe what she says colours a little—

  392. I am trying to bowl you a pretty hard ball.
  (Mr Bekhradnia) I think it would be unfair and wrong to say that we have not really taken quite strenuous steps to (a) get the data and (b) use the data in a way that is really quite innovative.

  393. She said there were no longitudinal studies.
  (Mr Bekhradnia) It is not quite true to say there are no longitudinal studies. We would certainly like more longitudinal data. We have some but what we have done particularly energetically, and I think quite successfully, in fact uniquely in the world, is to link the data sets that we have. John will describe what we have done but we have been able, for example, certainly in this country better than ever before and probably better than anywhere else in the world, to get a handle on the level of drop-out that exists in the system here. I think we can say that we know that pretty accurately now—and I heard some of what was said about people who drop out and then drop back in again later on—and probably better than most other systems do because of the quite innovative work that has been done in this area by John and his colleagues. Bear in mind also that we are a small body with the resources that we have. We prioritise our work and we have put a lot of priority into this particular area as it happens over the years. We appeared here once before and you were urging us to give priority to other aspects of the work that we do, and of course we do that as well. We give priority to all the things that we can. John will describe to you, if that is of interest, Chairman, the work that has been done on the data linking and what benefit it has given us in terms of our understanding of drop-out. I think it is fair to say that we now have a far better understanding of drop-out, and we are still developing this, as we said in our submission. This is work in progress. We are developing our understanding of drop-out but I think we have a far better understanding of (a) the extent of drop-out and (b) the nature of drop-out and we are beginning to get a handle on some of the causes of drop-out. We cannot give you some of the colour and the depth of the qualitative evidence that you have had presented to you. What we can do is rather greyer perhaps but in some ways probably more valuable in that we have the quantitative evidence. We look system-wide. We cannot get into the depth. We do not know the depth of the individual experiences but, as Gordon Marsden says, some of the individual experiences may not give you an accurate picture of what the system as a whole looks like. We are getting that data and that information far better than we used to.

  394. Can I ask John to come in? Standing where you are in HEFCE are you more concerned than you were two years ago about student retention, about drop-out? Is it a problem that is getting worse? Is it much the same? What is your level of concern?
  (Mr Bekhradnia) One of the problems we have with all data in this area is that it lags. We have data now for students that entered the system in 1997/98, and 1998/99 is just coming on stream. It is not absolutely current. As we said in our submission to you, the evidence seems to be that up to that point drop-out was pretty stable. It had been going up but not by anything like as much as you might have expected given the widening of the participation. Given also what we know about the relationship between drop-out and previous educational achievement, A-level points and that sort of thing, you might have expected drop-out to have increased by rather more than it has. Given what we know about the level of drop-out overseas, one might expect a higher level of drop-out in this country. It is very difficult to say any of this without sounding complacent, which we are certainly not as you know from what we are doing. We are putting quite a lot of effort into improving drop-out rates. What we can say is that it could be a lot worse and we would be a lot more concerned if it were worse.

  395. Shall we give John a chance to come in now?
  (Mr Thompson) We have not got the richness of the data that they have in the U.S. I am not an expert on the situation in America although we did have a study visit by colleagues and they came back with not just survey data but also very rich data held by individual institutions in the States. Certainly we are not satisfied with what we currently have. I could outline very briefly some of the work that we have done. There are two sorts of data. There is the sort of data I describe as the aerial photograph where we are talking about large routine complete data collections which have if you like the advantage of being complete but what you can ask and what you can record at reasonable cost with such a collection is obviously sparser than what you can record on a specific research study, the research study on a sample basis which goes in and looks in detail and focuses. Really you need a marriage of both of those. You know the work that we sponsored some years ago with Mantz Yorke and I know you have interviewed him. It became clear that just to quantify the level of non-completion, never mind get behind the causes, the routine data collection systems we had which were not good enough to provide really decent answers to that. What we did were two things: first of all, set in reforms to the way the data was collected. This is a massive thing with hundreds of institutions, all with their management information systems. That was a long way down the line to get that. It is in place now so that the records are built in their collection into a longitudinal record. We also started by getting permission for sufficient personal details as a back-end process to link the records from one to the other. We do have a longitudinal record, if you like, from the limited set of data that is collected routinely for students whilst they are in higher education so that, if they take a year out and they come back, we can follow them through. That is the basis of the work that produces the performance indicators and, as we have indicated, it can also produce more information about the associated risks because we have put a lot of effort into enhancing that data by linking, for example, with more detailed examination results from schools. This takes a long time to get the permission; there are data protection issues and technical issues. We have been working hard to build as rich data as we can to do the kind of modelling that should come up with the risks. We are also looking for opportunities to do the other kind of detailed work. In fact, I have been in conversation with Professor Callender for some time. Her main efforts have been getting this report out which came out in December, but one of the things that I very strongly recommended at the beginning of the study was that as far as possible sufficient personal details and the data protection issues would be sorted out so that we could take the rich data on the 3,000 students that she has interviewed and follow them through subsequently. There are a lot of difficulties (if you have read the report you will know) with the data protection issues, but we are hopeful that with at least part of that survey we will be able to follow them through and we will be able to squeeze more information out of that study. We are doing the same with others as well. We are trying step by step to build a more complete picture and a more complete understanding of the associated factors in respect of completion. There is also another very important sort of data, the youth cohort study, which is another longitudinal data step which starts with 16-year olds and has sweeps that take you through to young people when they will be in higher education. We have had conversations with the Department about adding extra questions because now a substantial proportion of the youth cohort go into higher education and that standard longitudinal study is already used but maybe we could make more use of it in understanding higher education.

Mr St Aubyn

  396. In the Secretary of State's letter to Sir Michael Checkland of 29 November he talks about asking the Council to bear down on the rate of drop-out, and goes on to say that the evidence shows that there are unacceptable variations in the rate of drop-out which appear to relate more to the culture and workings within the institution than to the background or nature of the students recruited. Which elements do you think the Secretary of State is referring to there?
  (Mr Bekhradnia) I think he must be referring to our performance indicators, of which we have now produced a second edition for a second year, where we provide indicators of drop-out by institution. Against that we also provide a benchmark to show what level of drop-out might be expected given the characteristics of the students in that institution. It is fair to say that against their benchmarks there is far less variation between institutions than there is if you look at the raw data; you would expect that, but there is evidence that some institutions have similar benchmarks but nevertheless have rather different drop-out rates.

  397. In deriving those benchmarks are you looking at the personal details of each student of an institution or are you looking at the postcodes of the areas they come from? How are you determining the characteristics of the students?
  (Mr Bekhradnia) John will answer that in more detail. There are two essential features that we are looking at for those benchmarks. One is the educational background of the student as reflected through their previous educational A-level entry grades, and the second is the subject mix of the institution concerned because these are the two factors that explain most of the differences between institutions.
  (Mr Thompson) The way I put it, we have got those two things so we look at an institution and we say, "What would be the rate that that institution would have if the whole sector had that pattern of subjects and entry qualifications that are at that institution?" We also implicitly take account of the age of students because all our data is tabulated by young students, that is under 21, and mature students, and that is also very important. The way I would put it (because this is a pretty rough and ready benchmark, it is not like a modelling exercise, a research exercise) is that if an institution was, say, a long way above the rate of non-completion as expressed with a benchmark, it is rather like the red light that goes on in your car that says that there is something wrong with the car. It might be that the bulb is faulty. It might be that the benchmark is faulty. There could well be characteristics of that institution which are outside their control and perfectly understandable, but it would be unwise to carry on driving if a red light came on. What our message would be is, if at an institution you appear on this admittedly rough and ready benchmark to have much higher completion rates, you really ought, if you have not already, to begin the process to understand why your non-completion is higher than it would be expected to be.

  398. Just for the record, you are saying if an institution deviates from its benchmark in terms of drop-out rate, it should not be assumed that that means they are under-performing but that they should investigate their performance?
  (Mr Thompson) Correct.

  399. Secondly, in terms of educational background, because perhaps of the paucity of data, you are looking simply at the A-level achievements of the intake to their courses. Is that right?
  (Mr Thompson) Yes. It is not so much the paucity of data as the balance. Producing performance indicators regularly is different from doing a research project. There is a balance between wanting to get an effective benchmark and getting something that you do not need a degree in mathematics to understand. It is a balance between those things.

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2001
Prepared 14 March 2001