Examination of witnesses (Questions 390
- 399)
WEDNESDAY 31 JANUARY 2001
MR BAHRAM
BEKHRADNIA and MR
JOHN THOMPSON
Chairman
390. Can I welcome Bahram Bekhradnia and John
Thompson from HEFCE. It is nice to see you again, as you have
been in front of the Committee before on the previous part of
our report. As I said to the previous people before us, we run
these things reasonably informally as you know. We want to get
the most out of them. Were you in the room when the evidence was
being given just now?
(Mr Bekhradnia) John was there for most of it; I was
there for part of it.
391. Can I start by picking up the theme that
was developed by Claire Callender in the sense that here we are,
HEFCE has been around some time now, but what they are saying
is, "We have not got the data to make serious policy judgements
on". What the hell has HEFCE been doing all these years if
we do not have proper data with which to inform ministers in order
to make good policy?
(Mr Bekhradnia) I did not hear Claire say that, Chairman.
First of all, may I thank you for this opportunity to join you.
You know me because I attended, as you said, before. John Thompson
is our Senior Data Analyst who is responsible for the production
of the performance indicators and the other work that we have
done in this area. I did not hear Claire say that and I hope that
the way you describe what she says colours a little—
392. I am trying to bowl you a pretty hard ball.
(Mr Bekhradnia) I think it would be unfair and wrong
to say that we have not really taken quite strenuous steps to
(a) get the data and (b) use the data in a way that is really
quite innovative.
393. She said there were no longitudinal studies.
(Mr Bekhradnia) It is not quite true to say there
are no longitudinal studies. We would certainly like more longitudinal
data. We have some but what we have done particularly energetically,
and I think quite successfully, in fact uniquely in the world,
is to link the data sets that we have. John will describe what
we have done but we have been able, for example, certainly in
this country better than ever before and probably better than
anywhere else in the world, to get a handle on the level of drop-out
that exists in the system here. I think we can say that we know
that pretty accurately now—and I heard some of what was
said about people who drop out and then drop back in again later
on—and probably better than most other systems do because
of the quite innovative work that has been done in this area by
John and his colleagues. Bear in mind also that we are a small
body with the resources that we have. We prioritise our work and
we have put a lot of priority into this particular area as it
happens over the years. We appeared here once before and you were
urging us to give priority to other aspects of the work that we
do, and of course we do that as well. We give priority to all
the things that we can. John will describe to you, if that is
of interest, Chairman, the work that has been done on the data
linking and what benefit it has given us in terms of our understanding
of drop-out. I think it is fair to say that we now have a far
better understanding of drop-out, and we are still developing
this, as we said in our submission. This is work in progress.
We are developing our understanding of drop-out but I think we
have a far better understanding of (a) the extent of drop-out
and (b) the nature of drop-out and we are beginning to get a handle
on some of the causes of drop-out. We cannot give you some of
the colour and the depth of the qualitative evidence that you
have had presented to you. What we can do is rather greyer perhaps
but in some ways probably more valuable in that we have the quantitative
evidence. We look system-wide. We cannot get into the depth. We
do not know the depth of the individual experiences but, as Gordon
Marsden says, some of the individual experiences may not give
you an accurate picture of what the system as a whole looks like.
We are getting that data and that information far better than
we used to.
394. Can I ask John to come in? Standing where
you are in HEFCE are you more concerned than you were two years
ago about student retention, about drop-out? Is it a problem that
is getting worse? Is it much the same? What is your level of concern?
(Mr Bekhradnia) One of the problems we have with all
data in this area is that it lags. We have data now for students
that entered the system in 1997/98, and 1998/99 is just coming
on stream. It is not absolutely current. As we said in our submission
to you, the evidence seems to be that up to that point drop-out
was pretty stable. It had been going up but not by anything like
as much as you might have expected given the widening of the participation.
Given also what we know about the relationship between drop-out
and previous educational achievement, A-level points and that
sort of thing, you might have expected drop-out to have increased
by rather more than it has. Given what we know about the level
of drop-out overseas, one might expect a higher level of drop-out
in this country. It is very difficult to say any of this without
sounding complacent, which we are certainly not as you know from
what we are doing. We are putting quite a lot of effort into improving
drop-out rates. What we can say is that it could be a lot worse
and we would be a lot more concerned if it were worse.
395. Shall we give John a chance to come in
now?
(Mr Thompson) We have not got the richness of the
data that they have in the U.S. I am not an expert on the situation
in America although we did have a study visit by colleagues and
they came back with not just survey data but also very rich data
held by individual institutions in the States. Certainly we are
not satisfied with what we currently have. I could outline very
briefly some of the work that we have done. There are two sorts
of data. There is the sort of data I describe as the aerial photograph
where we are talking about large routine complete data collections
which have if you like the advantage of being complete but what
you can ask and what you can record at reasonable cost with such
a collection is obviously sparser than what you can record on
a specific research study, the research study on a sample basis
which goes in and looks in detail and focuses. Really you need
a marriage of both of those. You know the work that we sponsored
some years ago with Mantz Yorke and I know you have interviewed
him. It became clear that just to quantify the level of non-completion,
never mind get behind the causes, the routine data collection
systems we had which were not good enough to provide really decent
answers to that. What we did were two things: first of all, set
in reforms to the way the data was collected. This is a massive
thing with hundreds of institutions, all with their management
information systems. That was a long way down the line to get
that. It is in place now so that the records are built in their
collection into a longitudinal record. We also started by getting
permission for sufficient personal details as a back-end process
to link the records from one to the other. We do have a longitudinal
record, if you like, from the limited set of data that is collected
routinely for students whilst they are in higher education so
that, if they take a year out and they come back, we can follow
them through. That is the basis of the work that produces the
performance indicators and, as we have indicated, it can also
produce more information about the associated risks because we
have put a lot of effort into enhancing that data by linking,
for example, with more detailed examination results from schools.
This takes a long time to get the permission; there are data protection
issues and technical issues. We have been working hard to build
as rich data as we can to do the kind of modelling that should
come up with the risks. We are also looking for opportunities
to do the other kind of detailed work. In fact, I have been in
conversation with Professor Callender for some time. Her main
efforts have been getting this report out which came out in December,
but one of the things that I very strongly recommended at the
beginning of the study was that as far as possible sufficient
personal details and the data protection issues would be sorted
out so that we could take the rich data on the 3,000 students
that she has interviewed and follow them through subsequently.
There are a lot of difficulties (if you have read the report you
will know) with the data protection issues, but we are hopeful
that with at least part of that survey we will be able to follow
them through and we will be able to squeeze more information out
of that study. We are doing the same with others as well. We are
trying step by step to build a more complete picture and a more
complete understanding of the associated factors in respect of
completion. There is also another very important sort of data,
the youth cohort study, which is another longitudinal data step
which starts with 16-year olds and has sweeps that take you through
to young people when they will be in higher education. We have
had conversations with the Department about adding extra questions
because now a substantial proportion of the youth cohort go into
higher education and that standard longitudinal study is already
used but maybe we could make more use of it in understanding higher
education.
Mr St Aubyn
396. In the Secretary of State's letter to Sir
Michael Checkland of 29 November he talks about asking the Council
to bear down on the rate of drop-out, and goes on to say that
the evidence shows that there are unacceptable variations in the
rate of drop-out which appear to relate more to the culture and
workings within the institution than to the background or nature
of the students recruited. Which elements do you think the Secretary
of State is referring to there?
(Mr Bekhradnia) I think he must be referring to our
performance indicators, of which we have now produced a second
edition for a second year, where we provide indicators of drop-out
by institution. Against that we also provide a benchmark to show
what level of drop-out might be expected given the characteristics
of the students in that institution. It is fair to say that against
their benchmarks there is far less variation between institutions
than there is if you look at the raw data; you would expect that,
but there is evidence that some institutions have similar benchmarks
but nevertheless have rather different drop-out rates.
397. In deriving those benchmarks are you looking
at the personal details of each student of an institution or are
you looking at the postcodes of the areas they come from? How
are you determining the characteristics of the students?
(Mr Bekhradnia) John will answer that in more detail.
There are two essential features that we are looking at for those
benchmarks. One is the educational background of the student as
reflected through their previous educational A-level entry grades,
and the second is the subject mix of the institution concerned
because these are the two factors that explain most of the differences
between institutions.
(Mr Thompson) The way I put it, we have got those
two things so we look at an institution and we say, "What
would be the rate that that institution would have if the whole
sector had that pattern of subjects and entry qualifications that
are at that institution?" We also implicitly take account
of the age of students because all our data is tabulated by young
students, that is under 21, and mature students, and that is also
very important. The way I would put it (because this is a pretty
rough and ready benchmark, it is not like a modelling exercise,
a research exercise) is that if an institution was, say, a long
way above the rate of non-completion as expressed with a benchmark,
it is rather like the red light that goes on in your car that
says that there is something wrong with the car. It might be that
the bulb is faulty. It might be that the benchmark is faulty.
There could well be characteristics of that institution which
are outside their control and perfectly understandable, but it
would be unwise to carry on driving if a red light came on. What
our message would be is, if at an institution you appear on this
admittedly rough and ready benchmark to have much higher completion
rates, you really ought, if you have not already, to begin the
process to understand why your non-completion is higher than it
would be expected to be.
398. Just for the record, you are saying if
an institution deviates from its benchmark in terms of drop-out
rate, it should not be assumed that that means they are under-performing
but that they should investigate their performance?
(Mr Thompson) Correct.
399. Secondly, in terms of educational background,
because perhaps of the paucity of data, you are looking simply
at the A-level achievements of the intake to their courses. Is
that right?
(Mr Thompson) Yes. It is not so much the paucity of
data as the balance. Producing performance indicators regularly
is different from doing a research project. There is a balance
between wanting to get an effective benchmark and getting something
that you do not need a degree in mathematics to understand. It
is a balance between those things.
|
