Supplementary memorandum submitted by
the National Audit Office
Question 95 (Mr Bacon): Methodology set out in
Appendix 3 of the C&AG's Report
The methodology in Appendix 3 of the C&AG's
Report was based on a complex analysis using multi-level modelling.
This firstly looked at the impact of various external factors
on academic performance by analysing the records of around 600,000
pupils each at key stage 3 and GCSE. The results of this analysis
are shown in table 4 on page 44 of the C&AG's Report. The
numbers in the table show the relative strength of influence for
each factor, and whether the influence is positive or negative.
Some external factors have a stronger influence
on academic achievement than others. Thus it can be seen that
prior academic achievement is clearly the external factor that
has the strongest positive link with academic achievement at both
Key Stage 3 and GCSE level, such that pupils with a high level
of academic achievement at the start of a key stage are more likely
to achieve good results at the end (paragraph 2.29 of the C&AG's
Report).
If the number is a negative (such as for the
percentage of pupils eligible for free school meals), this suggests
that the factor has a negative influence on performance, such
that pupils in schools with a high proportion of pupils eligible
for free school meals are, on average, less likely to achieve
good results.
With regard to interpreting the influence of
each factor in isolation, the figures for "Ethnicity—Indian"
(1.3 at Key Stage 3 and 2.4 at GCSE), for example, suggest that
when all other external factors have been taken into account,
pupils from this ethnic group on average make slightly more progress
than the average for all other pupils.
Using these results, analyses at school level
were then run in the following four stages giving two scores for
each school (at key stage 3 and GCSE) for each stage of the analysis:
— using measures of each school's pupils'
academic achievement only;
— including the pupils' prior achievement
at the start of the key stage;
— adding in other pupil-level background
variables (table 2 on page 40 of the C&AG's Report); and
— adding in school-level background
variables (table 3 on page 40 of the C&AG's Report).
The two sets of scores for the first and last
of these stages of analysis are depicted respectively in figures
14 and 15 on page 42 of the C&AG's Report (key stage 3), and
in figures 16 and 17 on page 43 of the C&AG's Report (GCSE).
In both cases, taking account to the influence on academic achievement
of external factors (figures 15 and 17) has a two-fold effect:
— it reduces the range of variation
between the performance of secondary schools (as explained in
paragraph 2.8 of the C&AG's Report); and
— it changes the performance ranking
of schools (as explained in paragraph 2.10 and illustrated in
figure 7 of the C&AG's Report).
Question 139 (Mr Williams): the correlation between
the NAO's analysis of performance taking account of the influence
of external factors, and that of the Department that just takes
account of prior academic achievement
The NAO commissioned the multi-level modelling
work from the National Foundation for Educational Research (NFER).
As part of its analysis the NFER compared the outcomes of the
Department's value-added measures of secondary school performance,
which took account of prior academic achievement, with the outcomes
of the analysis in the C&AG's Report, which took into account
other external factors as well as prior academic achievement.
The comparison showed a stronger relationship between the two
sets of analysis at GCSE level (correlation coefficient 0.772)
than at Key Stage 3 (correlation coefficient 0.655), although
for some schools there was a substantial difference in outcomes
under the two measures. The NAO ranking of schools ranked in the
bottom 20% based on the Department's value added measures is shown
in the figure below.
Question 140 (Mr Williams): Ranking of schools
that were in the top 20% of performers in terms of academic achievement
in 2002, after adjusting for external factors
Of the 624 schools ranked in the top 20% of
performers, in terms of academic achievement in Key Stage 3 tests
in 2002, 243 were ranked in the top 20% when academic achievement
was adjusted for the influence of external factors, with 33 ranked
in the bottom 20%. Similarly, of the 621 schools ranked in the
top 20% of performers, in terms of academic achievement at GCSE
level in 2002, 182 were ranked in the top 20% after adjustment
for the influence of external factors, with 76 schools ranked
in the bottom 20%.
February 2004
| 
