Examination of Witnesses (Questions 200-219)
WEDNESDAY 27 NOVEMBER 2002
DR IAN
SCHAGEN AND
DR SANDIE
SCHAGEN
Chairman
200. One of the things the Prime Minister has
particularly highlighted is the fact that, despite the fact that
there are a number of schools that are similar in intake—or
broadly similar with the same numbers of free school meals, statemented
children and so on—they perform very differently. How does
your research help us pinpoint which schools do perform well above
what was anticipated?
(Dr Sandie Schagen) It could be used for that purpose.
We have not done so because that is not part of our remit, but
clearly you can do this kind of analysis—you could use it
to identify schools that are performing well above expectations
given other background factors.
(Dr Ian Schagen) What we have done for many years
is produce feedback to certain schools and LEAs on their value-added
performance. We have produced what I call school residuals which
is how well schools are doing above or below what you would expect
taking account of everything else, and we do it for a range of
outcomes and we put confidence intervals about those residuals,
because you come up with a number for a school but obviously because
it is a statistical model there is a range of uncertainty about
that number, so we produce for schools "In total score you
are significantly above expectations"; "For average
score you are not significantly different"; "In English
you are significantly below expectations", for example that
kind of information which they can then use for their self improvement.
So yes, in principle, the national value-added datasets can be
used in that way. In fact, I am supposed to be at a meeting of
what is called the Value-added Methods Advisory Group for DfES
on how to take forward the whole production of value-added measures
for schools, and I hope to be able to make a contribution to that.
201. So your foundation, if you were asked,
could provide that sort of data to the Department?
(Dr Ian Schagen) Indeed.
202. How quickly?
(Dr Ian Schagen) That would depend on what dataset
you want. If you want it on the dataset we have already we could
do it within a week or two.
(Dr Sandie Schagen) 2001 is the latest we have at
the moment.
(Dr Ian Schagen) Yes. We are still waiting for the
2002 data.
203. When is that coming through?
(Dr Ian Schagen) There have been some problems, I
understand, in publishing the 2002 data and we are still waiting
to receive it. As I say, as part of the Excellence in Cities evaluation
we are waiting for and we hope to be allowed to use it for wider
research. We will make the usual request to the appropriate people
in QCA and DfES to have access to that data for our research purposes.
Paul Holmes
204. Ofsted earlier on declined to make recommendations
to the Government—they said their job was to collect the
data and others could interpret it. You did a report for the LGA
and on page 47 you said, "Our findings do not indicate that
an increase in the number of specialist schools would necessarily
lead to an improvement in performance". Do you want to expand
on that?
(Dr Sandie Schagen) Yes, I think probably the words
were mine! I am not sure we would say that is making a recommendation—that
is putting it too strongly—but I deliberately phrased it
negatively in saying that we do not feel there is an assumption,
on which this diversity programme I understand is built, that
creating more specialist schools would improve performance across
the board. Our research does not appear to indicate that. It shows,
as we have said, that if you compare those specialist schools
designated up to 1999 against all other schools they are slightly
ahead but, first of all, it does not indicate the reasons for
that and, secondly, when we looked within LEAs at the specialist
and non specialist schools and then at whole LEAs taking into
account the proportion of pupils in each that were in specialist
schools, there was no evidence to suggest that LEAs with more
specialist schools did better.
205. And on a slightly different tack but, again,
looking at comparisons, earlier on you were saying that if you
could wind back the English system 100 years so there was not
stratification and selection you would be able to judge the comparative
systems. Have you or anybody done comparisons between Scotland
and England, because in Scotland you have a system that is 96%
comprehensive where far fewer children go to private schools than
in England. Is there a system there where we can do comparisons
within the UK?
(Dr Ian Schagen) It may be interesting once we get
results on some of these international comparisons. Part of the
problem is you have different outcomes between England and Scotland
so it is difficult to make a direct comparison. We have some data
from the PIRLS international study which is the reading literacy
study which we intend to analyse for England, and it may well
be that if we can get hold of international data we can do some
sort of comparison between England and Scotland. One of our problems
is we cannot just do whatever comes into our heads—well,
up to a point we can but there is a limit to how much we can do
unless someone is prepared to fund it.
Chairman
206. Why do you do it? What is the purpose of
your educational research?
(Dr Ian Schagen) It is to improve education and training
through research.
207. So you do have a remit to improve. It is
not just purely academic?
(Dr Sandie Schagen) Yes.
208. Do you not get a bit dispirited when Ofsted
say, "Well, these academics—there will be another Jesson
and Jesson will say something different". They were inferring
that what they did was the real stuff and what you did was a bit
peripheral. Did you feel a bit of resentment?
(Dr Sandie Schagen) Not really. We will deal with
them afterwards! Believe it or not NFER, as opposed to the two
of us, does go into schools an awful lot—
209. Is it a private foundation?
(Dr Sandie Schagen) It is an independent foundation.
210. Not based on any university?
(Dr Sandie Schagen) No. It is not linked to any university
and, contrary to what a lot of people assume, it is not linked
to government either. We are an independent body and 80%-85% of
our research is for outside sponsors, of whom the DfES is by far
the biggest, but we work for other government departments. We
work for a whole range of sponsors including private companies
like BT and Nat West, so basically we work for whoever pays us,
more or less.
(Dr Ian Schagen) Including the TCT.
(Dr Sandie Schagen) And the Local Government Association.
There is a regular fund from them and they chose to sponsor the
research we are talking about now.
211. I am sorry to ask you that but it is interesting.
(Dr Sandie Schagen) That is fine, but the point is
you have asked us to talk about a particular subject and particular
area of research that Ian and I have worked on and which has been
virtually wholly statistical, but research projects that NFER
gets involved in can vary an awful lot. They are not all heavily
statistical. We have a number of research projects that do not
involve any statisticians at all because they are based entirely
on qualitative research, going into schools, in-depth visits to
schools interviewing a whole range of staff, maybe talk to pupils—whatever
is deemed appropriate for the particular project. Other projects
may involve questionnaire surveys to schools, telephone interviews,
desk work research, looking at what other people have said, etc.
212. Do you have any other information that
would shine light on this present inquiry into secondary education?
(Dr Sandie Schagen) Ian mentioned that we worked for
the TCT. Again, that was a project within my department although
I was not personally involved and that was a qualitative project.
In fact, originally they essentially gave us lists of schools
that had been identified by them on the basis of David Jesson's
work as being particularly good specialist schools and we were
then asked to go into those schools and do in-depth qualitative
research to identify the reasons why those schools were good.
As I said, neither of us were personally involved in that research
although we were aware of it and I know what the outcomes were,
and it is quite similar in a way to some of the other research
which James Tooley and others did which I have mentioned in the
report and I know he was here last week, but they were looking
specifically at schools already identified as high performing
and the factors they came out with as reasons for those schools'
success were very similar to factors identified as generally being
good in terms of school improvement. In other words, they did
not seem to me to relate specifically to the fact that the schools
were specialist: they related to the fact that they were very
good schools. What would be interesting would be to take an equivalent
group of non specialist schools and see if the findings were the
same but obviously that has not been commissioned by the TCT.
Jeff Ennis
213. On this question of comparative evidence,
within the UK you have Wales where they say there is no way they
are going down the specialist school route; Northern Ireland where
they are scrapping selection plus grammar schools; and Scotland
where 96% of comprehensive schools have no intention of going
down the specialist route. Are those three parts of the UK basing
their decisions on any sort of research base?
(Dr Sandie Schagen) I do not think all policy is based
on research, much as we think it is right that it should be! I
do not think we can really comment on the basis for those decisions
which are essentially political decisions, but I was thinking
about when we were talking earlier about comparisons because you
probably have a better basis for comparison there with Wales and
England because they have the same outcomes and, as you say, they
do not have specialist schools.
214. So do you know, as experts in the field,
of any research that is done to compare the different parts of
the UK going down very different roads?
(Dr Sandie Schagen) No.
(Dr Ian Schagen) Not to my knowledge but that is not
to say there is not.
(Dr Sandie Schagen) Our official title is the National
Foundation for Educational Research in England and Wales, and
we have been known to venture into Scotland and Northern Ireland
occasionally but not very often. A lot of our projects span England
and Wales but those are commissioned by the DfES, or the LGA for
purely England, and I am certainly not aware of anyone doing equivalent
studies in Wales, but it would be quite possible for us to do
given the same data.
(Dr Ian Schagen) The Welsh Assembly is currently collecting
similar national value-added data to the data that has been collected
in England, but whether it would be available to general research
I am not sure.
Chairman: I am surprised the Scots are
not looking at comparisons against devolved areas.
Mr Pollard
215. If I can take you back 100 years, as a
living fossil from a grammar school, I was a borderline grammar
school boy, as were the girls and boys who went to the same school
as I went from, and we did not do very well at all, I have to
say. We were top of the pile when at our primary school and when
we got to the grammar school we pretty quickly realised we were
not anything special at all and switched off and became disruptive.
One of our number was almost kicked out of the school—in
fact it was me—because of disruptive behaviour, so it flies
in the face of what you were saying before that those who were
borderline were dragged up—and that is going back 100 years,
or 50 anyway. More importantly, going on to faith schools, does
your research support the government strategy for having more
faith schools and, secondly, there is a general belief that faith
schools are better disciplined and achieve better pastoral care,
but is that real or perceived?
(Dr Sandie Schagen) I think there are three questions
there in one and I will take them in reverse order, if I may.
On the last one, I know this will sound like a cop-out but I said
at the start that we looked specifically at achievement and not
at those other issues so it may well be and on the basis of our
research alone I could not deny this, that there may be other
differences between faith schools and non-faith schools, if I
can call them that. It may be that they are better in terms of
ethos and behaviour, and I know Andrew Morris has done some work
using Ofsted data on the basis of which he claims to find that
there are those kind of differences, and that seems to me a not
unreasonable supposition and a not unreasonable finding. What
I have to say, however, is that on the basis of our research,
looking exclusively at achievement, there is not any evidence
at all to suggest really that increasing the number of faith schools
will improve the level of achievement. We suspect that the reason
why, and this is why value-added is so important, there is certainly
a perception that church schools are doing really well because
they often do appear at the top of league tables that are based
on raw results, and that is why it is so important to look at
intake. Our finding is that basically, when you apply value-added
analysis, that advantage all but disappears, which suggests that
the difference is based on intake. Interestingly, you can hypothesise
that if they do have better ethos and better behaviour and so
on that would lead to better achievement, but we did not find
any evidence that that is so.
Chairman
216. The most depressing thing last week was
that the three Professors seemed to agree that even anti poverty
measures like Surestart would only have a temporary effect on
outcomes. In a sense, you have to take my hypothesis that in a
relationship like poverty or social disadvantage of some sort,
one of the great investments of this Government has been on anti
poverty programmes of one kind or another. Has any of your research
shed light on that? Are you as pessimistic as your other professorial
colleagues?
(Dr Ian Schagen) I do not think we have any evidence
about the Surestart and other poverty reduction measures. There
is a lot of evidence that there is a relationship between poverty
and education attainment—not a completely mechanistic one.
Everything is statistical on average so we may have anecdotal
evidence from people who come from very poor backgrounds and do
well, or go to grammar school and are borderline and do not do
so well, but on average there is a relationship so if you take
that relationship as being causal you would expect reductions
in poverty would lead to higher attainment. There are other things—there
is a lot of work in school effectiveness and school improvement
literature on how to counteract poverty, effects of class sizes,
etc, so I think there are things that can be done even with leaving
existing levels of poverty, but probably you might find that reducing
poverty would have as big as or even bigger impact than that.
Mr Chaytor
217. Coming back very quickly to the grammar
school value-added point, what you are saying appears to be in
agreement with Professor Jesson's analysis of performance in grammar
schools whereby the highest ability pupils do slightly better
in comprehensive schools, but you are identifying a strong value-added
factor for borderline pupils. Where on the percentile range are
the borderline pupils? Are these round about 24-25 percentile?
(Dr Ian Schagen) Probably a bit lower down than that.
Probably 30 or 40 depending on the selection in the LEA.
218. This is critical because many grammar schools
will not have pupils at that point in the percentile range, will
they?
(Dr Sandie Schagen) And the selection varies tremendously,
as I am sure you know, from an LEA where there is just one grammar
school and everyone else is in so-called comprehensives to some
LEAs which have up to or even slightly above 40% in grammar schools.
The range is huge, which is why we try to take some account of
that in research. Coming back to Mr Pollard's point, obviously
we are looking at children overall and there will be exceptions
and I am sure there are many others like you and I think I would
have thought that myself, which is why these results surprised
us so much. The difference between ourselves and David Jesson
is that his work on grammar schools specifically looked at GCSE
outcomes and, to be honest, our findings on GCSE outcomes were
not enormously different to his. Again, we did not look at the
5 A-Cs but as far as looking at total point score and so on our
findings were not that dissimilar. We also looked at key stage
3 as well because at the time we did the original research we
had to look at the two key stages separately, and it was Key Stage
3 where we saw this enormous impact overall of borderline children,
and that intrigued us so much that we then, in order to find the
reason for this, wondered if it was because of higher expectations,
which was a fairly reasonable hypothesis, and we tested that,
because there is information on the national value-added datasets
about entry to tiers at Key Stage 3, by logistic regression to
estimate the probabilities of children being entered for high
tiers in maths and science at Key Stage 3 controlling for their
prior attainment to Key Stage 2, and the results of that probably
shocked us even more because the chances of a child in a grammar
school being entered for a higher tier at Key Stage 3 varied with
the subject and the level of percentage of children in grammar
schools but it was in the range of 9-20 times higher than a child
of the same ability in a comprehensive school, and that was what
really made us think that this research, as we see it, is of interest
beyond the relatively small number of selective authorities because
there must be an issue there, within comprehensive schools, of
what can be done to ensure children of that particular ability
range—
219. So the key factor you are arguing, certainly
at Key Stage 3 if not at 4, is the exam entry policy of the school?
(Dr Sandie Schagen) That was one, obviously.
(Dr Ian Schagen) We are hypothesising something to
do with expectations—that if you are within a system where
you are expected to achieve a certain level, then by and large
that is what you will achieve.
|
