APPENDIX 2
6. Comments by Dr Ian Schagen on the Evidence
of Professor David Jesson (OP 49)
Most of my comments on this work are well summarised
in a joint paper between myself and Professor Harvey Goldstein
(Schagen and Goldstein, 2002). However, here in brief are the
main points I would wish to make about this evidence:
— the analysis was not carried out using
multilevel modelling, although data is available on individual
pupils within schools. There is a danger (see paper referenced
above) that analysis on aggregate data can give rise to misleading
results. I do not accept the argument that analysis has to be
kept simple so that it can be explained to non-statisticians—the
task of the statistician is to carry out the best possible analysis
and explain what they have done clearly to others;
— the analysis takes no account of other
background factors, in particular eligibility for free school
meals. On average, specialist schools have 14.1% of pupils eligible
for free school meals, against 16.1% for non-specialist schools.
Our analysis has shown that this difference can have a significant
effect on performance, even in value-added terms;
— I have very strong reservations about
the use of the "5+ A* to C" measure as an indicator
of school performance. It is sensitive to changes in performance
by a small number of pupils around the C/D borderline, while being
completely unaffected by pupils at the high and low end of the
attainment spectrum. My preference is to use a range of different
GCSE outcomes, each based on the performance of all pupils;
— I agree with Professor Jesson, in so
far as our analysis has also shown a positive (though small) gain
in value-added performance for specialist schools, but the main
disagreement is about how this should be interpreted. One interpretation
is that some or all of this may be attributed directly to the
impact of the specialist schools programme itself—this is
the conclusion that Professor Jesson immediately puts forward
as self-evident. However, other interpretations are equally or
more plausible: for example, that schools with better value-added
performance, more supportive parents or a more dynamic staff are
more likely to become specialist. A second, not necessarily alternative,
explanation, is that the extra funding helps to generate the improved
performance and that this would happen anyway without the specialist
status;
— if, as the final paragraph states, the
pressing issue for schools is to root out underperformance, then
it seems to me that this can best be achieved by detailed value-added
information about different aspects of each school's performance.
This calls for analysis on a whole range of subject-level outcomes,
not just on a single measure such as 5+ A* to C grades;
— in summary, I believe that Professor
Jesson's analyses are simplistic and that he is too ready to draw
particular conclusions from them without considering alternative
explanations. To be fair, his final sentence does state "it
is not possible to say that becoming a Specialist school, of itself,
causes enhanced performance". Overall, however, his paper
gives the impression of drawing in data piece-meal to support
a particular position, rather than considering all the evidence
impartially.
December 2002
REFERENCES
SCHAGEN, I. and GOLDSTEIN, H. (2002) "Do
Specialist Schools Add Value?—Some methodological problems",
in Research Intelligence, No 80, pp 12-15.
| 
