Annex A
Making Good Progress
RESPONSE OF
THE ASSOCIATION
OF SCHOOL
AND COLLEGE
LEADERS
A. INTRODUCTION
1. The Association of School and College
Leaders represents 13,000 members of the leadership teams of colleges,
maintained and independent schools throughout the UK. This places
the association in a unique position to see the progress measure
from the viewpoint of the leaders of both secondary schools and
colleges.
2. ASCL welcomes the opening of a debate
by the Secretary of State on progress measures. Helping every
student to make good progress is the function of the education
system and so consideration of policies aimed specifically at
supporting that is long overdue.
3. The use of pupil progress measures is
in principle a move in the right direction of intelligent accountability
for schools. Good teachers measure the performance of individual
pupils on progress made and it is right that the same principle
should be used to measure the performance of schools.
4. However, the proposals as set out will
not have the desired effect. There are several specific aspects
about which ASCL has major concerns. The association looks forward
to a period of consultation and piloting in which the best features
of the proposals can be developed and the worst amended or dropped.
5. To support such a process an alternative
measure is proposed in section E of this document which the association
believes would command much greater support, avoid the faults
of the measure proposed in Making Good Progress, and would
better lend itself to target-setting at all levels.
B. ASSESSMENT
AND TESTING
6. The document's clear statement of assessment
for learning and endorsement of it is welcome. ASCL shares the
belief that this approach, always used by good teachers to some
extent and in some form, can be usefully extended; and has for
some time championed it. However, it should be remembered that
high-stakes, externally marked tests are antipathetic to assessment
for learning. For testing to be supportive of learning it must
be kept closer to home, with frequent assessments (of all kinds)
devised or chosen by the teacher and marked by the teacher.
7. The association welcomes a move away
from age-linked testing. The further idea of "testing when
ready" is also welcome. Sadly, what the paper sets out is
not that but rather a proliferation of the existing testing regime.
Modern technology can surely lead us to aspire to forms of testing
that enable students to be tested whenever they are ready, not
on a given day in an examination hall in six months time. To propose
tests on the current model, but more often, is to miss an opportunity
to devise something better for our children, and potentially to
exaggerate the faults and costs of the present system.
8. In both of these areas the paper is hidebound
by the prevailing orthodoxy of testing, which has prevented any
genuinely creative ideas.
C. PERSONALISED
LEARNING
9. ASCL also welcomes the renewed emphasis
on personalised learning. It is closely bound to assessment for
learning, and is again not new; good teachers, and good schools
and colleges, have always tried to personalise their offering
to students.
10. This is recognised in Making Good
Progress and in the 2020 Vision report, one of the
good features of which was its recognition of the good practice
already in the current education system. The present document
somewhat loses sight of that by extracting (on page 14) a list
of approaches that schools "will need to adopt" as if
they were not all in the usual repertoire of school behaviour.
Some may need more emphasis in some schools.
11. The "personalised classroom"
as set out in the first paragraph of page 16 is an attractive
prospect, but for it to be realised it is imperative that the
teacher not only has ready access to the necessary data but also
can rely on it. The present high-stakes testing regime and the
weakness of the national curriculum tests prevent any such reliance.
Many schools make use of CAT, Midyis or other diagnostic tests
for example because they do not feel able to rely on the National
Curriculum tests, which were devised as summative tests, as a
good baseline for predicting the future performance of each pupil.
12. A more rapid response to pupils who
are falling behind is clearly welcome, provided that that does
not translate into ever more frequent, stress-inducing, external
tests. Our young people have become the most tested in the world,
and their stress levels have risen markedly as that has happened.
There is now a need to take greater care with their mental health
and normal development.
13. The document does recognise at this
point that teachers are already skilled at discovering the progress
of their individual pupils and tailoring their courses to their
needs.
14. ASCL welcomes the clear statement that
personalisation does not mean devising a separate plan for each
student, and the renewed promise of greater flexibility in the
secondary curriculum to allow schools room to be more creative
in devising programmes suited to their particular students.
15. There is a contradiction between the
idea of personalised learning, which recognises the different
needs and abilities of each person, and the setting of systemic
targets, which presupposes that all young people should ideally
travel the same path at the same rate.
16. The suggestion in the 2020 Vision
report that students from disadvantaged backgrounds should
receive additional support is welcome, and the document does no
more than reiterate this. However, at one of the DfES presentations
to stakeholders this was extended to an intention to provide 10
hours of individual tuition to students not "on trajectory",
possibly at home, at weekends or in the school holidaysprovided
by local authorities. ASCL cannot welcome this interpretation
of the 2020 suggestion. It would be very expensive, costing
far too much to administer as well as overlooking the possibility
of joint work with small groups of students in similar states
of learning and with similar needs. It would be very unlikely
to be good value for money.
17. If additional funds are available for
this type of support they should be delegated to schools, which
are closer to the individual students and will be better able
to apply them than local authorities.
D. MEASURES AND
TARGETS
18.ASCL strongly opposes the proposals in section
five of the document. In this section the proposals go badly wrong
in ways that would ensure that the good intentions of the earlier
sections could not be realised.
19. First, measures framed as "the
percentage of children who . . ." are bad measures of progress.
They concentrate the attention of teachers, schools, partnerships,
local authorities, inspectors, government and the media on those
children on the borderline of making the grade when we should
all be interested in the progress of all children.
20. Sensible measures in this area should
look at the distance travelled by each child related to how far
we might reasonably expect a child with that starting point and
that set of attributes (disadvantages for example) to travel in
the time.
21. Secondly, the proposal that every child
should be measured against an improvement of two national curriculum
levels is absurdly crude. It may be easy to understand, but will
mislead most of those who see it, and will create new perverse
incentives as damaging as those caused by the some of the present
measures. For every complex and difficult problem there is
a simple and straightforward solution . . . that is wrong.
22. That a child who is badly behind at
the start of a key stage, a child who is a high flyer, a child
with a strong leaning towards or away from a particular subject,
a child with every sort of support, a child with a profound disability,
a child with severe social disadvantage, a child simultaneously
learning English, a child who learnt English at a previous stage
should all somehow move on two levels bears no examination. In
fact, any research that has been done into these and other interrelated
factors is ignored here.
23. The information that is set out in Making
Good Progress points to a further weakness in the proposals;
that they would systematically favour selective schools and other
schools that have a more able than average intake. This is illustrated
clearly by looking at Key Stage 2 to Key Stage 3 mathematics:

This diagram is highly simplified, some children
make greater or less progress than indicated, but the arrows show
the most common movements during Key Stage 3. Note that on the
whole those children who start at lower levels mostly do not make
two NC levels of progress, whilst those who start at higher levels
mostly do. The effect at school level would be to create a measure
that would imply that a school with a "good" intake
is doing better than a less favoured school, almost independently
of the excellence of the schools themselves.
24. Thirdly, the key stages are of different
lengths, and the national curriculum levels not designed to be
of equal size. So it is inevitable that, quite apart from the
many individual differences between children, a target of two
levels per key stage will be much harder to reach at some stages
than at others. Figures in the document itself make it quite clear
that this is the case. Adopting such a set of measures would therefore
invite media attention of the most unwelcome and ill-informed
kind: "only 30% of children make acceptable progress in English
between age 11 and 14", for example.
25. Encouraging such misunderstanding cannot
be to the advantage of children, schools or the Government itself.
Many children will be given the false impression that they have
"failed", when in fact they have made perfectly normal
progress. Secondary schools will be painted as failing their pupils
even when they have made above average progress. And the Government
will be accused of allowing a systematic failure of education
at Key Stage 3.
26. Targets framed in this way will set
up new perverse incentives. Students who part-way through a key
stage have already clearly made their two level improvement, or
who clearly cannot do so, will not be targeted as intensively
as those who may or may not make that improvement.
27. A two key stage improvement may be an
appropriate aspirational target for many individual students,
but it is not appropriate for all, and it is certainly not appropriate
as an accountability measure for teachers, schools or Government
at Key Stage 3.
28. Fourthly, the relationship between the
national curriculum tests and GCSEs is not close and not well
understood. The document seeks views as to how a measure based
on a percentage of those moving up two levels on these two incommensurable
scales should be formulated. It should not be formulated at all.
Any such formulation will fail to measure anything meaningful
and will create perverse incentives of the worst kind.
29. Fifthly, this whole section is predicated
on the national curriculum tests as robust and reliable measures
of attainment. These tests are better now than in their early
days but are still not capable of bearing the weight of all the
many uses to which they are already put. It is not sensible to
erect a further edifice of measures and targets on them.
30. Finally, this section asserts that these
new measures should be added to all the existing measures and
not replace anything. This is simply wrong. The English education
system already has more tests and measures than any comparable
system, a larger proportion of scarce resources is diverted from
actual learning into setting tests, preparing for them, administering
them, analysing the results, reporting the results, and dealing
with the inevitable misunderstanding of them by children, parents,
governors, the media and others.
31. The assertion that nothing can ever
be removed from a bureaucratic system does not sit well with recent
attempts by Government to reduce bureaucracy and improve the intelligence
of accountability systems. In this case it rests upon a separate
assertion, made at page two, that it is the elaborate system of
tests, targets and performance tables that has driven up standards
in recent years. No evidence is brought to support that idea,
which has taken on the aspect of a dogma. Indeed, on the very
same page of the document it is undermined by the assertion that
it is the Government's increased investment in education that
has had the beneficial effect. This seems a more likely explanation:
our schools are better led, are better staffed, have better facilities
and are better resourced than before, and this has been reflected
in better progress. Pupils and teachers are also more experienced
at the tests, which is bound to have had a beneficial effect on
scores nationally.
32. ASCL urges as a matter of general principle
that initiatives should not be taken, in a system already at full
stretch, without indicating what it is that they should replace.
In this case progress measures are welcome but must replace some
of the alternative measures that have now had their day, and done
whatever good they may have been able to do. There are plenty
of candidates . . .
E. AN ALTERNATIVE
MEASURE
33. As already stated ASCL welcomes the
idea of a measure of progress. As a constructive response to Making
Good Progress what is set out in this section is an outline
of such a measure that would command the support of school leaders.
34. Any such measure should avoid the perverse
incentives inherent in "the percentage of students who .
. ." but should depend on the progress made by all students.
It should also intelligently reflect the starting point of the
student in question.
35. What is needed as a first step is a
more complete and careful analysis of a cohort of students, say
those that took the various tests in 2005 that will give an expected
outcome for each student based upon the starting point. This will
also answer the question, left open in Making Good Progress,
of what progress can be expected between Key Stage 3 and GCSE.
What ASCL proposes is the use of this statistical relationship
as a baseline for expected performance against which performance
in future years can be judged.
36. Thus each student's result can be compared
to the expected outcome and a positive or negative "residual"
determined. These residuals can readily be aggregated to give
an average for a class, school, local authority and the country
as a whole.
37. This should be familiar as the approach
taken by value-added measures. Like them it would avoid perverse
incentives and be based upon careful research into the actual
performance of real students.
38. Traditional value-added measures have
the drawback that they are cohort referenced*, meaning
that they relate an individual or a group with the averages for
the year group to which they belong. This has some disadvantages.
The individual's score is partly determined by the performance
of the peer group. Such measures do not really reflect change
from year to year; a teacher, department, school, partnership
or local authority can improve in performance but find that that
is not reflected because others have improved too. And in particular
they hide improvement in the system as a wholethe average
residual for the whole group must by definition be zero.
39. So what ASCL proposes is different in
a crucial respect: the performance of future students should be
compared not to their own peers in their own year-group, but to
the fixed 2005 reference group. In the sense in which we are using
the term here this would make the measure norm referenced*
rather than cohort referenced and therefore avoid the drawbacks
outlined in the previous paragraph. It would allow for year to
year comparison of the performance at all scales from the departmental
to the national.
40. In particular any improvement would
be reflected in the measure which could thus be used for any target-setting.
The average residual for the nation as a whole would no longer
necessarily be zero; if the education system improves new cohorts
of students will do better than the 2005 group and this will be
reflected in a positive residual. These residuals are expressed
as fractions of a National Curriculum level (or GCSE grade) and
are therefore relatively easy to understand if not to calculate.
41. There is at least one precedent for
such an approach in the SAT (formerly Scholastic Aptitude Test),
widely used and respected in the USA. Before 1941 this test was
cohort referenced but after that date comparison was made each
year with the cohort of 1941 thus making the test norm referenced.*
42. It is not likely that in the more rapidly
changing modern world a reference group could be retained for
more than sixty years, and the measure could be rebased from time
to time, but it is envisaged that the same base should be used
for, say, a decade at a time in order for systemic progress to
be observed.
43. As far as the secondary phase of education
is concerned ASCL suggests that the main measure of progress should
be that between Key Stage 2 and GCSE. This reflects the most common
patterns of secondary organisation with 11-16 and 11-18 schools.
In such schools measures involving Key Stage 3 are clearly subsidiary
and should be primarily for internal use rather than used to rate
the school as a whole.
44. ASCL will be pleased to help develop
these ideas further as part of its commitment to more intelligent
accountability.
F. PROGRESSION
PREMIUM
45. This idea is particularly unwelcome.
Teachers and their leaders are motivated by a desire to do right
by those in their charge, not by a desire for a bonus. Such a
premium, especially one built upon a measure that lacks full professional
confidence, would either reward in a capricious fashion or would
systematically reward those that need no such reward (ie those
schools teaching the best supported pupils with the fewest disadvantages).
46. ASCL would remind ministers that the
School Achievement Award was scrapped for very good reasons. They
should not seek to reintroduce a similar, but equally flawed,
reward.
47. ASCL strongly suggests that this idea
be dropped forthwith.
G. CONCLUSION
48. ASCL welcomes the basic idea of Making
Good Progress and its aspirations. School and college leaders
have always striven to help all their students achieve as much
as they can.
49. However, the actual proposals contained
in the document, especially those in section five, would not help
in any way to do this, and would in fact do far more harm than
good.
50. ASCL would strongly suggest that the
whole of sections five and six, and some of section three as outlined
above, should be set aside. There is a need for some genuinely
new thinking about these important matters so that the whole system
of assessment, testing, reporting and accountability can be amended
to better serve the worthy aims of Making Good Progress.
51. ASCL stands ready to contribute to such
thinking and trusts that the major amendments proposed above should
be made before the pilot begins.
January 2007
* Thanks to Professor
Dylan Wiliam, Deputy Director of the London Institute of Education
for the explication of the difference between cohort referenced
and norm referenced, and for drawing attention to the norm referenced
nature of the SAT. Norm referenced is often used as opposed to
criterion referenced to describe measures that would more properly
be called cohort referenced. To be clear: in this paper norm referenced
means comparing the performance of a child or group of children
with a fixed reference group (say those that took tests in 2005)
whilst cohort referenced means comparing a child or group of children
with those taking the tests in that year.
REFERENCE
Wiliam, D. (2007). Balancing dilemmas: traditional
theories and new applications. In A Havnes & L McDowell (Eds),
Balancing dilemmas in assessment and learning in contemporary
education (pp 269-283). London, UK: Routledge. (To be published
later this year).
|