Examination of Witnesses (Questions 164-179)
WEDNESDAY 27 NOVEMBER 2002
DR IAN
SCHAGEN AND
DR SANDIE
SCHAGEN
Chairman
164. Good morning, Dr Sandie Schagen and Dr
Ian Schagen. Welcome. Have you given evidence to a Select Committee
before?
(Dr Sandie Schagen) I understand you have been given
a number of our papers so I do not want to take up your time and
obviously you want to ask questions but, very briefly, we have
been working together on the area of diversity for the past year
or two, myself as a researcher and Ian as a statistician. We became
very interested in this subject and we have looked at not only
specialist schools but faith schools, selection, single sex schools
and so on, and we have looked at it by using national value-added
datasets so our research has been focused exclusively on academic
performance, and we need to acknowledge that obviously there are
other claimed advantages of different types of schools but we
have not looked at those. Neither have we looked at change over
time, improvement over time. We have been focused specifically
on looking at the difference in value-added performance and different
types of schools. With that introduction we are quite happy to
take your questions.
165. You will get some strange questions given
that you are academics but you will have to accept that! On specialist
schools, is the diversity agenda that was popular with the last
political party around this country and with this one working?
Is it good value for taxpayers' money? Should we continue with
it?
(Dr Sandie Schagen) That is not something we have
looked at.
166. So you have no evidence that this is a
good spend for the money?
(Dr Sandie Schagen) No. We have not looked at that.
167. In your conclusions in a paper that was
sent to us by Ian Schagen and Harvey Goldstein, you are having
a pretty tough go at David Jesson, are you not? Given your answer
just now, the fact is you are really saying, and my interpretation
of this is, that the research done by Professor David Jesson on
which much government policy is based was not reliable, and implicit
in that was, "Be careful, you are building government policy
on research that does not really prove what it is supposed to".
(Dr Sandie Schagen) That is true.
(Dr Ian Schagen) I think I had concern and it was
shared by Professor Goldstein that the quality of the research
and the type of analysis carried out was not of sufficiently high
standard to warrant some of the conclusions being drawn. That
is not to say that the conclusions are wrong but that they were
not backed up by the best quality research and the best quality
analysis of the available data. We now have very good comprehensive
data, national value-added datasets, which it is possible to analyse
in a sophisticated and more valid way which is the kind of work
we have been doing, and therefore it seemed that, if you like,
going back to doing something which was simple is not good enough.
I think one of my favourite quotations I believe is by Einstein
which is, "Keep everything as simple as possible but no simpler"—in
other words,to produce analysis which is based on a simple-minded
look at the data and which, to be honest, seemed to be mainly
polemical rather than trying to look at what is the data telling
us about different kinds of schools, and the possible impact they
may be having on pupils' performance in a value-added sense was
quite wrong. Having said that, the best statistical analysis in
the world can tell you what relationships appear to be there and
can tell you, for example, what the apparent impact of specialist
schools is but it cannot tell you why there is that impact; whether
it is a pre-existing effect; whether it is caused by the fact
that specialist schools are getting more money; or whether it
is caused by the fact that good schools are chosen to be specialist
in the first place. So we had concerns about methodology, if you
like, from an academic point of view and also from the point of
view that it seemed to be getting a lot of publicity and a lot
of government policy was being based on something which we felt,
from a statisticians' point of view, was not the best way of looking
at the data.
168. Are you agreeing with David Taylor from
Ofsted that, quite honestly, if you look to academics' key facts
to base your policy on you are looking in the wrong direction
because each academic will have a different interpretation of
the facts?
(Dr Ian Schagen) The job of academics is not the same
as the job of government departments or even Members of Parliament.
Our job is to try to understand as deeply as possible the underlying
processes and the data which we have. Our aim is, at the end of
that, to come up with evidence which is based as closely as possible
on the data and which is as objective as possible and which people
can then use to build policy. However, if you are looking for
quick and simple answers to complicated and dirty problems, you
may be disappointed.
169. But going back to Professor Jesson's paper
you are concerned that the government White Paper seemed to lean
very heavily on one piece of research?
(Dr Ian Schagen) Yes.
(Dr Sandie Schagen) Indeed.
Mr Simmonds
170. What impact has the availability of pupil
level data had on your research?
(Dr Ian Schagen) It has had a very great impact and
one of the reasons we got into this field was the availability
from DfES and QCA of linked value-added datasets initially from
Key Stage 2 to key stage 3 and then from Key Stage 3 to GCSE outcomes,
but more recently a full dataset going from Key Stage 2 right
up to GCSE 2001 has enabled us to use the methodology, which has
been around for a long time, to look at what are the impacts of
different kinds of schools when you take account of not only prior
attainment at pupil level, free school meals at the school level
as a kind of somewhat crude issue, as we discussed earlier, a
proxy for social deprivation, and trying to build models which
enable to us look in a great deal of detail at what is left over
when you control for everything else. The problem with looking
fairly crudely at the school level—at some schools getting
these sort of outcomes but some schools getting those—is
that that is telling you a great deal about the prior attainment,
the abilities of the pupils on intake to those schools, and not
a great deal about what the schools are doing with those pupils
that they have, and you have to look at what schools are doing
given the pupils that they have, and so that is why we do value-added
analysis while we control for prior attainment, but using multi
level modelling because some of the things we have to take into
account are at the school level and not at the pupil level. I
do not want to get technical about the reasons for that but we
now have the methodology and the data and you put the two together.
We have not answered all the questions and the models we are developing,
as we get more data, from PLASC, the Pupil-Level Annual School
census, will enable to us control for more things at pupil level
to see what is happening with different ethnic minorities, what
is happening at pupil level for those eligible for free school
meals or whatever. We hope we will be continuing and also to build
in the longitudinal dimension of how these things are changing
over time. It is quite an exciting time to be an educational statistician,
and to answer questions which people have bandied around for quite
a while.
171. You mentioned in that response free school
meals and we heard from Ofsted earlier in relation to free school
meals. Has your research shown any pattern of distribution of
pupils who get free school meals, who perhaps benefit from having
special educational needs, and perhaps have English as a second
language, in addition?
(Dr Ian Schagen) We have not got that data yet. We
hope to get it from PLASC. All we have from free school meals
is the percentage eligible for free school meals. But our research
over several years has shown that, even when you do value-added
analysis, when you control for prior attainment, there is still
an impact of free school meals. Schools with higher levels of
deprivation tend to make less progress, and if you do not take
that into account it can be confounded with the effects of different
school types.
172. Do you have sympathy with Mr Chaytor's
view that free school meals is not an equitable measure of deprivation?
(Dr Ian Schagen) I am sure it is by no means the perfect
measure of deprivation. There will be better ones, I would have
thought. Unfortunately it happens to be the one we have at the
moment and it does show a relationship with outcomes and therefore
we continue to use it, because not to would be worse than to use
it. But if you or anybody can come up with a better measure which
we can collect reliably for DfES and pass on then it would be
helpful.
173. Does your research take account of pupil
mobility or lack of it?
(Dr Ian Schagen) At the moment we have not factored
that in but it is an area we ought to be looking at. The models
that we build are in a sense always provisional. There are always
ways in which they can be improved and ways in which we can build
in extra factors, and pupil mobility is something of concern to
many people.
(Dr Sandie Schagen) It is worth mentioning the fact
that, because we are using matched pupil level data, for pupils
who move around a lot it is probably more difficult to collect
and match their results at different times, and therefore it is
possibly the case that more highly mobile pupils are not included
in our analysis, but hopefully that would be overall and therefore
should not have, one would hope, a confounding impact.
Mr Pollard
174. Your report was based on diversity and
looks at grammar schools and others. Was there any thought of
including Montessori and Steiner schools in that, because that
is proper diversity, rather than others which are variations on
a theme really?
(Dr Ian Schagen) I am sure that is something we could
do if we were able to identify the schools on the schools database.
It is not something that we have included at this stage. In a
sense because NFER is funded through the research it does for
other bodies mainly, we tend to do research, first of all, that
people like the LGA commission us to do. Secondly, we do some
of our own internal research because we feel it is of general
interest. If there was an organisation or we felt there was something
that we as a public body should do anyway then I am sure that
is something we should take into account for future modelling.
175. You also mention in your report that in
mathematics and science boys do better than girls but in everything
else girls do better. Is there something we can read into that,
that these matters are genspecific or that boys are more interested?
What conclusions might we draw?
(Dr Ian Schagen) I do not think it is a total surprise
that, even when you are doing value-added analysis, there are
still some subjects in which boys make a bit more progress than
girls. Contrariwise, most of the general indicators which are
not maths and science tend to show slightly more progress made
by girls. We did do some work looking at the impact of single
sex schools—
(Dr Sandie Schagen)—and that tended to slightly
reduce the sex stereotyping particularly in girls' schools in
terms of the subjects they were entered for certainly[1]
Chairman
176. Professor Gorard last week shot down the
research, or seemed to, that suggested there were significant
differential achievements between boys and girls. Have you looked
at his work on that?
(Dr Sandie Schagen) Not on that issue. We looked at
his work on social division—polarisation as he calls it.
Chairman: I may have been doing him an
injustice. It could have been Dr Taylor or even Professor Pring
who said that, but we will send you the transcript.
Jeff Ennis
177. Will you be releasing research on the impact
of excellence in cities' interaction on programmes?
(Dr Sandie Schagen) Evaluation of EIC is going to
be our biggest project at the moment.
178. Have you got results from that?
(Dr Ian Schagen) We have done the analysis of the
first year's round of data and some of those papers that we produced
last December are in the public domain, though obviously that
was very much a baseline. We are currently waiting on the final
version of the PLASC data from DfES so we can do the analysis
of this year's data. We decided there was no point in doing things
half-baked so we are waiting for the full dataset and then we
will do the analysis. Obviously there will be similar sorts of
analyses of national data to a lot of what we are reporting here,
but in addition we have been collecting from EIC school surveys
which give us a lot more information about pupil attitudes, and
part of the aim of the EIC evaluation is to look not only at outcomes
but at attitudes because we feel that, if an initiative is to
have an impact, the first sign of that will be changes in attitudes
and in what pupils are thinking and doing, and that may follow
on later to changes in performance. So that is where we are coming
from on that.
179. Is it your intention to compare the outcomes
from that programme of work with this programme of work to see
if we are getting better value for money, for example, from one
as opposed to the other?
(Dr Ian Schagen) When we do the analysis for national
data for the EIC project we include in that national dataset information
about specialist schools, faith schools, grammar schools and whether
they are in education action zones and so forth, so all that is
in the model and it is just a question of when we run them all
and get the outcomes—
1 Note by witness: Our research did not directly
compare performance of boys and girls. It showed, however, that
girls in single-sex schools performed better in science than girls
in mixed schools. Back
|
