Examination of Witnesses (Questions 180-199)
WEDNESDAY 27 NOVEMBER 2002
DR IAN
SCHAGEN AND
DR SANDIE
SCHAGEN
180. You have not got any conclusions you can
draw at this stage?
(Dr Ian Schagen) Not at this stage.
181. Going back to the PISA study, that suggested
that education systems with a high degree of social segregation
have large variations in the achievement of children at the real
level. Does your research prove and support that?
(Dr Sandie Schagen) Are we talking about grammar schools
or social—.
182. Either. Well, both.
(Dr Ian Schagen) Where you have a large amount of
between-school variation?
183. Yes.
(Dr Ian Schagen) I would say that, generally speaking,
England does have a large amount of between-school variation compared
with some other parts of the world. Partly this is due to the
differing prior attainment of people going to secondary school,
so there probably is more segregation in the intakes of schools.
Whether, if you could re-run the educational history of the last
200 years with a less segregated educational system, you would
get better results, it is hard to say.
Ms Munn
184. Following on the answer you gave to Jeff's
previous question and also picking up this point where you said
that changes in attitude will lead to improved results later on,
and I asked a very similar question to the people from Ofsted,
have you any views about how long it takes for those kind of things
to work through, and do you have any evidence which suggests that
the longer a school has been in a school improvement programme,
whether specialist schools or education action zones, the more
these improvements increase?
(Dr Sandie Schagen) We did not look at that when we
did the original research we are talking about here. We just included,
as specialist, schools that had become specialist by 1999 so it
was a yes/no factor. We did not include length of time in the
programme. Obviously that is one refinement we could incorporate
but we have not done so as yet.
185. So from what you are saying it was your
gut instinct rather than any evidence when you were saying that
attitude change is likely to lead to—
(Dr Ian Schagen) Yes, I think that is probably fair.
There is some evidence from a lot of the school improvement and
school effectiveness literature that there is a link between outcomes
and attitudes. In fact, some of the analysis we have done for
our value-added work for schools has shown that if you measure
attitudes and outcomes you can link the two together fairly clearly.
Chairman
186. Did your research findings surprise you?
Did you find particular aspects where you started off with a piece
of research and you were then surprised at some of the results
you brought in, and which were they?
(Dr Sandie Schagen) I think what surprised us most
was right at the start. Interestingly, it was in contradiction
to what was mentioned at the end of the previous session because
it was concerning grammar schools. What we found was that the
grammar schools seemed to work not by enhancing the performance
of the most able which is sometimes suggested but by greatly enhancing
the performance of what we call borderline children—those
who just managed to scrape into grammar schools. There are two
theories about borderline children: within a selective system
there is a view that they do better in secondary modern schools
where they can be at the top of the pile rather than struggling
at the bottom of grammar schools, but there is also the view that
they may get pulled up within a grammar school, and certainly
our evidence showed very strongly the latter. We were quite amazed
when we saw the difference in performance of children with the
same starting point, the same Key Stage 2 results, and what they
would get by Key Stage 3 in a grammar school compared with another
school. The difference was really quite staggering, and it did
quite amaze us and was really partly why we looked further into
these issues. That was probably the thing that did surprise us
most.
(Dr Ian Schagen) And it was obviously why we went
into the specialist and faith schools—firstly, because of
current government generally and, secondly, to see whether there
were any similar effects we could detect.
(Dr Sandie Schagen) We were then asked by the LGA
to do initially a literature review of the impact of specialist
and faith schools but we realised there was probably not a great
deal of literature available yet—indeed, faith schools have
obviously existed for a long while but there is not much formal
analysis of their impact so, having seen the results of the work
we had done on selection, I suggested that we did a similar analysis
for specialist and faith, and perhaps we can come back to your
point at the beginning: as regards specialist schools, and consistently
in our more recent work, we did find that there does appear to
be a positive impact of specialist schools on all of the outcomes
we looked at, so from that point of view we are not necessarily
disagreeing with David Jesson's findings. It is his methodology
we are concerned about. We also found a positive impact but I
think to be fair we need to say this: that the differences we
see are, firstly, that the impact we found is not as big as is
sometimes claimed. It is relatively small in terms of total point
score, for example. I think it was just under two points' difference
between pupils in specialist schools and pupils in other schools,
so it is not so big. The other point is making the inference which
we feel is not justified that because pupils in specialist schools
appear to do slightly better in value-added terms one cannot therefore
assume that that difference is due to the fact that they are in
specialist schools, because there are clearly other factors that
could be at work which, for various reasons, we have not been
able to take account of.
187. In an e-mail to this Committee Professor
Jesson said, ". . . when schools are compared on their GCSE
results using Ofsted's `BenchMark' frameworks, Specialist Schools
outperform others to a substantial degree. If this is further
refined to discover the overall `advantage' of Specialist schools
the analysis shows very clearly that these schools gain 5% more
on the major criterion (percentage gaining 5 or more A* to C passes)
than do other schools. Since `national improvement' is only around
1 percentage point a year, 5% represents a very significant additional
achievement, and goes some way to account for the `popularity'
of this particular initiative."
(Dr Sandie Schagen) I have to say he uses as his main
measure the 5 plus A-Cs and we do not at all; we regard it as
unstable because it can be influenced by the performance of just
a few children. We all know there are schools which deliberately
focus on children who are on the C/D borderline to try to push
up their results. There is no secret that that happens, and we
feel that from a statistical point of view the outcome measures
you use should be ones that reflect the performance of all the
pupils in the school and cannot just be influenced by the performance
of a few.
(Dr Ian Schagen) Picking up that point, I do not feel
happy as a statistician in using any single measure to look at
schools and say, "This is the measure by which schools are
judged". I think that schools are complex, multi-faceted
organisations. You need a range of outcome measures to look at
different schools which is why we use about seven or so and there
are a number of others. You can look at schools subject by subject
and find that certain schools are doing well in certain subjects
and not so well in others. That kind of information is more of
value in terms of driving school improvement than in publishing
league tables. If you can tell schools where they are doing well
and not so well compared with what you might expect given their
prior attainment and other circumstances you can give them a lot
of valuable information which can help them to improve, which
may do more for them in some ways than just saying, "Here
is some money; go and be a specialist school". But that is
a personal opinion.
Mr Chaytor
188. Pursuing the point of the original Jesson
research methodology you challenge, was that peer reviewed before
it was published?
(Dr Ian Schagen) Not to my knowledge.
189. Under whose aegis was it published?
(Dr Sandie Schagen) The Technology Colleges Trust.
190. So it was published by the main organisation
that was established to promote specialist schools?
(Dr Sandie Schagen) Yes, and he works for them on
a regular basis.
191. So, in your experience, is it normal for
government White Papers or Green Papers to use as their evidence
base research that has (a) not been peer reviewed and (b) published
by the very organisation that was publishing the policy?
(Dr Ian Schagen) I would not like to say what was
normal!
(Dr Sandie Schagen) To be fair we have to say our
reports which you have been reading have not been peer reviewed
either. We have been submitting papers, again in this area, to
academic journals which are being peer reviewed..
(Dr Ian Schagen) And which have been accepted for
publication.
(Dr Sandie Schagen) Yes, after peer review, but we
do not always wait for peer review before publishing. In fact,
certainly speaking for the NFER, that would be unusual generally
because our sponsors want the results and could not wait for that
lengthy process, so I do not think it would be fair to criticise
purely on those grounds.
192. So peer review is not the key issue necessarily?
(Dr Ian Schagen) No. It is useful and we try wherever
possible to get results published in academic journals where they
will be peer reviewed. However, as you know, because the needs
of the sponsored research community are such that we need to produce
results, we have internal review processes. Obviously everything
that goes out from the NFER is internally reviewed and there is
a quality assurance procedure, but I do not want to get in an
argument about whether our methods for review are better than
other people's.
193. Your paper with Professor Goldstein challenges
the methodology and argues that multi-level modelling should have
been used, but in the research you subsequently did on specialist
schools are you dealing with exactly the same body of information
as that that was dealt with by Professor Jesson?
(Dr Ian Schagen) To my understanding he was using
the same national value-added datasets that we were.
(Dr Sandie Schagen) There is one difference in that
in his analysis he excluded selective schools. He only looked
at comprehensive or secondary modern specialist and non specialist
schools. We did wonder at one point if that was one reason for
the difference between us, so we looked at the proportion of both
specialist and non specialist schools that were grammar schools
and they were roughly the same, so we concluded that that was
not sufficient to account for the difference between us.
194. On the total point score, does that not
depend on the individual policy of the schools and the number
of entries?
(Dr Sandie Schagen) Absolutely.
195. Is that a relevant factor?
(Dr Sandie Schagen) That is partly why we look at
lots of different outcomes rather than just one. For the original
research on specialist and faith schools we used five and on the
more recent one we used seven or eight, but the interesting thing
is to look at both total point score and average. People argue
for one or the other but, if you look at both together, it can
give you an idea of whether a good performance in total point
score is due to a good all-round performance, or to doing extra
subjects. In fact, there was an interesting point—that when
we did the original research, by comparing those two outcomes
it seemed to us that for both specialist and faith schools their
advantage was not enormous anyway but in terms of total point
score it was not matched by an equivalent advantage in terms of
average score, and that suggested that the advantage was at least
partly due to taking extra GCSEs. On our more recent research
when we updated we used as a specific outcome the average number
of GCSEs entered, and that confirmed what we had inferred from
previous research—that specialist schools in particular
do enter their children for more GCSEs. On average it was about
a quarter of GCSE extra compared with children in non specialist
schools.
196. Did you separate out the nature of the
extra GCSEs? Do you have, for example, any information about entering
children for GNVQ?
(Dr Ian Schagen) No. There are a lot of things that
are on the "To Do" list, so to speak, with this data,
and especially when we get fuller data. What we have not looked
at (though we aim to as part of the EIC evaluation) is to look
at performance in specialist subjects or specialist-related subjects
and whether there are any differences there, but I think the suspicion
for the faith schools was that it was probably likely to be RE.
(Dr Sandie Schagen) And we also thought with specialist
schools that it might be science. Again this is hypothesising
but they did well in terms of total science score, so we thought
maybe they were encouraging pupils to take an extra science, but
there is no direct evidence of that.
Mr Turner
197. Sir Keith Joseph used to say that the next
best thing to a magic wand is a good head. As far as all the measures
that you have used are concerned, which is the next best thing
to a magic wand in relationship to high quality performance?
(Dr Ian Schagen) What would be interesting and nice,
and what we have not yet achieved, would be to get Ofsted to provide
us with some of their numerical scales from their inspections
where they look at leadership and so forth, which was something
they were just talking about. What would also be nice, although
there are a number of technical problems not least to do with
the cycle of inspections, would be to try and relate some of those
Ofsted scales to the value-added datasets and include them in
multi-level modelling. I did some work with Ofsted a few years
ago looking at their numerical inspection database trying to pull
out factors which seemed to be related to high performing schools
in one sense or another, and certainly leadership was one of them.
There were others as well which it would be interesting to apply
to the latest national value-added dataset. The thing about doing
this kind of work is that there is an awful lot of noise—in
other words, that you have all this data for all these pupils
but most is down to individual pupils. You can predict what a
pupil will get at GCSE from what they get at Key Stage 2 but not
brilliantly. You can explain about 50% of the variants in technical
terms. In other words, about half of what somebody does at GCSE
is down to what they were like when they came out of primary school,
and the other half is down to what happens to them at school—and
obviously some people will go up and down but it shows what they
did. In all of that there is a small amount of school effect.
Schools do make a difference but compared with all the noise it
is quite small, and if you take away all the other factors that
you can account for you can explain something like 50% of the
variants between pupils and about 80-90% of the differences between
the schools, so if you have a league table with all the schools
spread out in terms of their outcomes, if you take account of
all the other things you can collapse that. So there is less difference
between schools than you would think when you take account of
other factors, but there are still significant differences between
schools, and there are significant differences between schools
with differentially different outcomes—some schools are
very good at English, some at science, some at mathematics. Obviously
what you want to do is try and look at what things within schools
make that difference like leadership management and ethos—maybe
those things that explain those differences between schools.
198. I am pleased to hear that some schools
make a difference. We had some evidence the other day that suggested
that some pupils actually regressed in education. Is it evident
to you that some pupils do not gain at all?
(Dr Ian Schagen) Changing the tack slightly, we have
been doing some work with Professor John Gray of Homerton, Cambridge
based on data collected over many years for QCA on the results
of the optional year 3, 4, 5 tests—going into primary education—and
because we have longitudinal data on what levels pupils achieve
in key stage 1—i.e year 2, year 3, year 4, year 5 and then
year 6—we can draw tracks of these pupils and some are quite
fascinating in that, unlike what people used to draw which is
nice steady progress, you get waves. It is like a bowl of spaghetti
sometimes. Some pupils are going down; they regress in year 3
and suddenly shoot up and do really well in years 4 and 5 and
then come down again in year 6, so the path taken by individual
pupils through education is not clear cut and simple. Yes, they
can regress. There can be periods in which they apparently, depending
on your measures, regress because normally we only measure the
start and the end and draw a straight line between them.
199. Finally, do you find that Key Stage 2 results
are broadly accurate? Do they reflect anything in particular about
how well a child is doing when they go into their secondary school?
(Dr Ian Schagen) If I was trying to predict an individual
I would find it amazingly difficult but since what we are dealing
with is getting on for half a million individuals and what we
are interested in is not the individuals themselves but what is
happening in the system as a whole and what is happening in different
types of schools, we can put up with a bit of noise that the Key
Stage 2 levels may not be accurate. They may not be accurate for
a particular individual but broadly they give you a measure which
ties down with prior attainment within a particular cohort. In
particular in the modelling, if you do not mind me getting slightly
technical, we use not just the average Key Stage 2 level but the
Key Stage 2 levels in English, maths and science, so three separate
indicators for individuals who have a profile. We also allowed
the relationship between Key Stage 2 performance and GCSE outcomes
to vary from school to school because we are interested to see
whether some schools are accelerating the progress of the most
able when they come in and not doing so well with the least able
and vice versa, and also looking to see whether that is related
to school types so we found a small amount of evidence to say
that specialist schools seem to have a slightly steeper slope—ie,
that they are doing best with the most able to a small degree.
So we find the Key Stage 2 data is fit for our purpose; obviously
if you want to look in more detail at what individuals are doing
you may want to have other tests of prior attainment because there
are issues about Key Stage 2 and how well it is transmitted from
primary to secondary school. I think there have even been rumours
of people cheating in those exams but hopefully that is going
to be at sufficiently low level not to disturb our analysis.
|
