Examination of Witnesses (Questions 140
- 159)
MONDAY 14 JANUARY 2008
DR MARY
BOUSTED, MICK
BROOKES, BRIAN
LIGHTMAN AND
KEITH BARTLEY
Q140 Mr Chaytor: May I put my first
question to Keith? The chief inspector may tell us in his annual
reports that we have the best generation of teachers ever and
that standards of teaching are rising year on year, but what is
the relationship between that and the testing regime? Since the
testing regime came in and the publication of league tables became
the norm, standards of teaching appear to have risen. How do you
explain that relationship?
Keith Bartley: First, I do not
think that it is a causal relationship. Secondly, I observe that
the implementation of the national curriculum in the late 1980s
had a significant wash-back effect on initial teacher training.
There is no doubt in my mind that teachers who are moving through
into qualified teacher status now have a higher level of teaching
proficiency within that curriculum because the curriculum is now
much more closely defined than ever before. Year on year, we have
seen a rise in the quality of the training experience. Running
alongside that, we are also seeing a rise in the levels of qualification
of teachers. Last year's cohort of qualified teachers was the
highest qualified that we had ever seen in our schools. As for
the question of why those higher standards of teaching and better
qualified teachers are not manifesting themselves in improved
levels of achievement, I would argue that during that period we
saw a significant increase in the value placed on the tests. At
the same time as we have a curriculum—indeed, there is a
very high level of encouragement from Government through their
curriculum policies to broaden and experiment with that curriculum—all
the time, and particularly at the ages of 11 and 14, youngsters
are being narrowed down by a very narrow system of testing. That
is the bit where we are not unlocking the achievable.
Q141 Mr Chaytor: Why do you assume
that an improved quality of teaching is not leading to improved
levels of achievement? Formal evidence from league tables, Key
Stage tests, GCSE and A-level results, and vocational qualifications
suggests the opposite.
Keith Bartley: What we have seen,
all the way through, is that shortly after testing has been introduced,
there is an increase in standards. Typically, in the first few
years after the implementation of testing for a key stage, we
see a steady increase in achievement and standards, and then we
see it tail off. That is very much the case with any performance
measure, whether it is to do with schools, or with the punctuality
of trains. What happens is that people work towards a performance
measure, and it quickly plateaus. That is why you need to have
a much broader band of assessments that can be used.
Mick Brookes: Performance at the
end of Key Stage 2 has been largely stuck for about five years,
and until we start to do things differently, it will remain stuck.
There have been improvements, not only in children's knowledge
and understanding, but in teaching. We must be careful not to
polarise the issue. Going from the 1970s and 1980s, there was,
indeed, a need for much greater veracity of thought and practice
in our schools. Although that might have been a painful process
for schools to go through, we have gone beyond it. We now have
sophisticated systems of assessment, which we should use, and,
according to Ofsted, we have better standards of teaching in our
schools. That worked then, but we must consider where we go next.
Mr Chaytor: Mary wants to come in.
Dr Bousted: I should like to answer
the question directly. Ofsted does not investigate the quality
of the tests. It takes the test level data and says that they
indicate a rise in standards. If you examine the recent research
by Peter Tymms, which was in the Robin Alexander review, he undertook
a long inquiry into how difficult it is to measure standards longitudinally.
In fact, two years ago, ATL published, Standards in English
Primary Schools: are they rising? by Colin Richards, which
also examined the issue. Tymms and Richards said that standards
in reading had risen marginally, but nothing like what the tests
state. Tymms makes it quite clear that when the Key Stage 2 reading
tests started, for the first four years they found it very difficult
to get a foundation level. The baseline against which they tested
whether standards had gone up or down changed. It is also clear
from a paper by Mary Hilton, on English in education, that three
years into the Key Stage 2 tests, the level of questions that
asked for an inferential response—not, "Did this happen
in the passage?" but, "Why did it happen; what were
the motivations?", which is a much harder question—went
down. Tymms argues that only since 2000 has the baseline level
against which you measure year on year, by each cohort, become
steady. If you consider the rise since 2000, it is nothing like
as great as in the first four years, so Ofsted will say that standards
have risen dramatically because it does not question the evidence
base. The question that we must ask is, "What is the validity
of the evidence base?"
Q142 Mr Chaytor: May I link that
to the point that you made earlier about 25% of pupils not getting
the grade that their ability merits at the Key Stage test? Surely
the question is, who defines what their ability merits? It must
be true in any form of testing and assessment that the subjective
judgment of the tester comes into it. There are bound to be errors,
surely?
Dr Bousted: There are, but you
must consider the level of confidence error that is acceptable.
Part of the problem that we have with the tests is that they test
a very narrow range, so it is particularly difficult in science
to get a valid pencil and paper written test that tests the key
concepts in science. They are taken on a certain day, at a certain
time, and you can have only very narrow test items. You could
have tests with different test items that tested other parts of
the syllabus, and they would be equally valid but give a completely
different result. In other words, our system has gone for high
reliability: it is likely as not—we will put a lot of effort
into it, and be as sure as we can—that the kids who take
the test will get similar grades if they have a similar level
of ability. We have problems with that, but they are also highly
invalid.
Q143 Mr Chaytor: May I move on to
the question of the role of teacher assessment? Perhaps the question
is directed to Brian. What does the evidence say about the relationship
between teachers' judgment of ability and the Key Stages 2 or
3 results? Would it be possible to abandon completely external
testing and replace it with an entirely teacher-assessed system?
Mick Brookes: Certainly, by moderated
or accredited teacher assessment, but I think using—
Q144 Mr Chaytor: Sorry, what do you
mean by accredited?
Mick Brookes: Accredited would
mean using a bank of tests, so that if a teacher said, "I
think 80% of children in my class have got Level 4, but how come
the test doesn't say the same thing?", you then have that
professional conversation with your member of staff, not about
those who scored the same, who are easy, but about those who scored
differently. You must ask, "What's happened here? Is it your
assessment? Is it that they were sitting next to Fiona, and that
is why they got a higher mark?" It needs to be brought back
into the school and discussed in an atmosphere where professional
integrity is encouraged. If we could do that we would have a system
that is fit for our children and does not completely subvert the
year 6 curriculum, for example. I could take you to a school where
they cancelled Christmas for year 6 kids because they were so
worried about—
Q145 Mr Chaytor: You have not told
the Daily Mail this?
Mick Brookes: They want to know
where it is, but I have not told them.
Brian Lightman: I absolutely agree.
You have to have a clear definition of the standard that makes
a Level 4 or Level 5. That is a problem: there is a big discussion
about what constitutes a Level 4. You can test a little bit of
it or you can do an overall assessment. To do that, you need to
be able to draw down appropriate, high-quality materials that
can be used to assess reliably where the students are. That sort
of thing needs to take place. That professional conversation has
been missing over the years, because people have relied almost
entirely on external tests, rather than sitting down together
as groups of teachers and saying, "In geography, which is
my subject, a Level 4 is x, y or z; this is what constitutes it,
and we need this evidence and that evidence to see that."
Part of that will be a test and other parts will involve looking
at students' written work and, perhaps, using oral assessments.
There will be a whole bank of things. All of that put together
and properly moderated will lead to a much more robust assessment,
which will go back to the formative side of things—we can
then advise students about how to improve.
Chairman: David, a last one.
Q146 Mr Chaytor: Finally, in terms
of high-stakes testing and the publication of league tables, if
there were a movement towards the greater involvement of teacher
assessment, should that information be published in the league
table list, or are you saying that they should be taken out completely?
Brian Lightman: The results of
all that assessment?
Mr Chaytor: Yes.
Brian Lightman: I think we have
to ask how much we need to publish in the form of league tables.
Mr Chaytor: I am asking you how much
you think we should be—
Brian Lightman: My argument would
be that there is far too much going into the league table. We
are adding more and more. Parents need and have a right to an
overall picture of how the school is doing and how their child
is doing individually. The move towards real-time reporting, where
parents can see how their child is doing individually, is exciting
and positive. At the other end, the whole school accountability
side of assessment—which is a completely different purpose—could
be published in much more effective ways than league tables, which
often do not compare like with like, talk about the context of
the school or look at raw data.
Chairman: I know that you all want to
answer, but if you have anything to say that is different from
what Brian said, that would be the most useful.
Dr Bousted: I want to go back
to the issue of teacher assessment. Part of this discussion is
predicated—or, rather, most discussions are, but not this—on
the idea that teacher assessment is unreliable and invalid and
that test assessment is good. Currently, at Key Stages 1, 2 and
3, you have teacher assessment scores against test scores. The
issue is that teacher assessment scores count for nothing: they
are not used by Ofsted and are not in the performance league tables.
Teacher assessment is, however, usually significantly and consistently
lower by 2 or 3 percentage points, so there is no reason to believe
that teachers would make things up in a properly trained and moderated
system.
Mick Brookes: May I say an unequivocal
no from the NAHT? That information should not be published in
the league tables that we have now. It simply tells you where
rich people live and, sadly, where poor people live as well. It
is deeply undermining for those communities, teachers and children.
I cannot understand why a Labour Government have allowed this
to go on for 10 years.
Chairman: Adam.
Q147 Adam Afriyie: Thank you, Chairman—or
acting Chairman. This is my first official outing on the DCSF
Committee, and I am delighted to be here. My background is that
I come from the Science and Technology Committee, so I have been
fascinated to hear some of the comments about how assessment works,
the baselines, the changes over time, and whether things can be
compared. I want to concentrate my two questions on the single
level tests, to examine them as briskly as possible, and to obtain
your views on the proposed progression tests and the league tables
that they will be placed in. Is it a good thing to use single
level tests, or is it better to stick with the key stages or take
another route for assessment?
Brian Lightman: The first thing
is that placing single level tests in league tables implies that
everyone will have to do them at the same time, which flies in
the face of personalised learning. If we are talking about personalised
learning, we must be able to apply those tests at the right time.
Trying to force the entire cohort through that at a certain time
makes nonsense of the idea. The other thing is that we can use
new technology for genuine testing when ready, which we must do,
and we then return to the idea of downloading a bank of tools.
Q148 Adam Afriyie: It is interesting
that you have made the point about testing when ready several
times, and I have heard it loud and clear. Would you propose something
more along the lines of allowing pupils to progress when they
are ready or hit a key stage, or move on fast when they are ready,
or is that just purely the testing format rather than what happens
with pupils in the year in which they sit in school?
Brian Lightman: The only thing
I have an issue with in the way you expressed that would be if
you are saying that they should be allowed to progress when ready.
We, as teachers, want to ensure as much challenge as possible,
so we do not want to tell children that they can take as long
as they like. We want them to progress. That is one of our issues
about two levels of progress within a key stage, because that
is an arbitrary decision. Key stages are different lengths. For
example, there is no comparison between Key Stages 2 and 3 in
length and time, so why talk about two levels? Some students could
progress three levels, but it would be unreasonable to expect
others to do so. Also, the levels are not equal. The idea is not
as simplistic as the music grade analogy that is sometimes used.
Dr Bousted: Briefly, we think
that making good progress is fraught with difficulties, and we
do not think that the Government have thought through the matter.
The single level test, taken up to four times a year, could lead
to a huge proliferation of the test, and all the issues arising
from that—narrowing of the curriculum and teaching to the
test—could be ratcheted up to four times the present level.
Q149 Adam Afriyie: So you think the
Government are just plain wrong.
Dr Bousted: Yes, just plain wrong.
There is also a question about the validity of the levels. If
a single level test is shorter, how valid is it and how worth
while is the information that it provides? We agree absolutely
with the Association of School and College Leaders. We all know
that a less able child will find it far more difficult than a
more able child to move two levels. That has been shown already
in the current progression rates. Schools where less able children
congregate are often in more challenging areas or where children
have English as their second language. They would be penalised
by a progression measure that they could never achieve. In schools
where children arrive with Level 5 they are much more likely to
be able to move two levels than those in schools where the majority
of children arrive with Level 3. It is much more difficult for
them to make that progress because they learn more slowly. Yet
that school might be doing an equally good job, but be penalised
by a completely inappropriate progression measure. That has been
scientifically determined by the levels, so why the Government
have chosen this measure, I do not know. Finally, a system of
repeated and proliferation of testing is like excellence and enjoyment.
The Government think that you can put in excellence and enjoyment,
and have a high testing regime and a broad curriculum, and that
just by putting them in the same title it will happen. They also
think that as long as assessment for learning and progression
level testing are in the same document, that is all right. You
will not get assessment for learning in the system. They could
have done so many things, but this is not the right thing for
them to be doing.
Q150 Adam Afriyie: Thank you. And
Mick, do you agree with that view?
Mick Brookes: I would like to
encourage the Government to move on from where they are. The concept
of having a bank of tests that are appropriate for a single level
is right, but it is the way in which it is being rolled out. To
do the right deed for the wrong reason is the greatest treason.
The tests genuinely have to be for schools' use in assessing their
children. That assessment is then reported to the right people:
the parents, the children themselves and, of course, the governing
body. The problem is trying to use the same tests to judge the
performance of schools. I think that we have made that point.
Q151 Adam Afriyie: I hear your point.
You are arguing for separate tests or assessments for teachers,
for the school and for the pupil—they are not necessarily
the same thing.
Keith Bartley: May I just put
a slightly different take on that? At the moment, our reading
of the Making Good Progress pilots is that they are still
about testing when the system, rather than the child, is ready.
Actually, if you were to place a greater emphasis on the validity
of teacher assessment, and if you were to provide teachers with
a bank of tests that they can draw on at an appropriate stage
to confirm their judgment of where a child is in a particular
aspect of his or her learning, you would change the system. The
learners' needs would drive the assessment and, therefore, the
subsequent teaching. Making Good Progress is a step in
the right direction, but we would particularly like a robust evaluation
of its impact and the possibilities that it is beginning to open
up.
Q152 Adam Afriyie: I think we already
had the answer to my final question in previous answers. Do you
consider that it is in any way possible to disconnect summative
assessment of the children from the monitoring and performance
of the school or the teacher? Is that possible?
Mick Brookes: Yes, I think it
is, by sampling. Of course, we must retain some idea of national
benchmarking to see where things are. That is important because
this is part of the public purse and therefore the public need
to know how well education is faring across the country, but that
could be done with sampling rather than going to every school.
Going back to the previous question, I think an analogy with the
driving test is a good one. Some people might need five lessons
before they are ready to take the test; others will need 20 and
some might need 120.
Brian Lightman: I would agree
with that.
Q153 Lynda Waltho: My feeling is
that testing and assessment is coming out as, "Yes, we need
it, but perhaps not in the way we are doing it." I do not
get the same glowing feeling about league tables. Without league
tables and other value-added measures, can you suggest a measure
of school performance that can be used by the Government for accountability
but also by parents for their decision making? Does that exist?
Is it possible?
Mick Brookes: I think that parents
are being grossly misled by raw, or league table, data, and, indeed,
by some Ofsted assessments that are driven by exactly the same
set of figures. That creates polarisation. When it comes to admissions,
people will travel and do all sorts of strange things to get their
child into the school of their choice because of that polarisation.
Actually, their local school may be very good, but the only way
they can find that out is by going there and having a look. We
think there should be an emphasis on parents, who should make
sure that they visit their local school and see the work that
it is doing. As an experienced head teacher, I could walk into
any school and within five minutes tell you whether it is a good
school. Parents then need to look at the results, but they need
to look at them in the context of their community. That is not
necessarily about poverty. It is also about the expectations of
the community in terms of education qualifications, which is something
that Ofsted does not take into account. The level of higher education
qualifications in the community is a key factor in whether the
children will expect to progress in education. We are doing the
job of moving those expectations on, and we need to keep doing
it. Parents must go and see.
Brian Lightman: What are the criteria
for a good school, and what are the criteria that parents look
for? I show parents around the school. I strongly encourage all
parents to come round to our school before they make their decision
about whether to send their child there. They want to know whether
they are going to be happy, safe and encouraged to make the best
progress they can. It is those types of things that they want
to know. Over and above that, parents look at our school prospectus.
They read the school prospectus, which contains, by statute, a
whole range of indicators against the school's performance and
they can see a detailed account of the school's performance. I
know I am talking from a different context, where I work, but
that is a much more valid approach than putting the information
up, as it was done in the paper last week, where the top 10 schools
in any authority—the best 10, they were called—were
published, whether it was an authority with 100 schools or one
with 10 schools. That is the way they published the league tables
last week. That does not tell you which ones are the best schools.
Looking at the broad picture of a school is the way in which you
can judge its quality—and I think there are plenty. There
is also the Ofsted report—I hope you know—and all
the information in the public domain. There is a vast amount of
information enabling a parent to see what a school is like.
Q154 Lynda Waltho: So we are never
going to be able to do it with a league table. Is that the feeling?
Brian Lightman: I do not think
that we are going to have the sort of accurate information, which
gives a true picture, with league tables the way they are.
Dr Bousted: The problem with league
tables is that you can write whatever you want, but in the end
the drop-down system for league tables is so pernicious in other
respects. Let us consider a school in a challenging area that
is working hard and making the good progress that we know a school
can make. Variation between schools is much less than variation
within schools. We know a school can make about 14% of the difference;
the rest of the determining factors on a child's achievement come
from their background, actually. And 14% is a lot; I am not undermining
what a school can do. However, that means that schools in the
most challenging areas have to work extremely hard to get the
results that they do. It is up to Ofsted to go into that school.
We have one of the most accountable systems in the world. Ofsted
goes in and looks at whether the school has sufficiently challenging
targets for pupils—whether it is sufficiently demanding
of their progress, whether the behaviour in school is appropriate
and whether the teachers are working well enough. Ofsted looks
at a range of issues to do with the performance of that school.
But if you are in that challenging school, doing very good work
and adding clear value to the children's lives in that school
and you are still, because of the nature of your intake, going
to be at the bottom of your local authority league table, that
is not a good thing. I will tell you another thing: it is not
just not good for the teachers, but it is not good for the pupils
going to that school, who need to feel good about themselves.
Keith Bartley: May I share some
of the research that has been commissioned from parents? MORI,
in 2005, discovered that parents attributed very low value to
league tables in assisting them to determine their choice of school.
That was partly because they felt that the information that they
portrayed was to some extent confusing. For others, it did not
match what parents saw as being the really important things about
schools. The list that Brian gave captured those things. We undertook
some further research through the British Market Research Bureau
about what parents found most helpful in determining the quality
of a school. They looked for a much broader range, particularly
in respect of contextual factors: the nature of the young people
that the school educated, the kinds of aspirations it set out
for those young people and the way in which it delivered against
those. That wider bank and portfolio is what parents said they
would find most useful. League tables, parents tell us, are questionable
in terms of their value. The further dimension is that as our
14-19 curriculum unfolds, and particularly as we seek to offer
young people a much broader range of both education and training
opportunities through to the age of 18 or 19, the influence of
an individual institution is going to be much harder to measure
through league tables, because young people will have been involved
in several institutions—maybe a college, maybe two schools
or maybe a training provider—so their validity will become
even less as our system better meets the needs of that older group
of pupils.
Q155 Lynda Waltho: In light of that,
although it is slightly unfair to mention this because possibly
only one of you may have read The Western Mail today—[Interruption.]
Okay. Actually, it refers to Wales. It published comments by Professor
David Hopkins, who has said that school tests and league tables
should be brought back. In fact, it said that, "Statistics
show children in England, where testing the two ages and league
tables remain in place, performed better last summer in key exams
like GCSEs and A-levels." Professor Hopkins said, "We
very much know that the performance at seven correlates to success
at 16 ... The previous system was too harsh but Wales went too
far the other way." That is also backed up, to a certain
extent, by Professor David Reynolds, who described Wales's PISA—Programme
for International Student Assessment—ratings as "awful"
and said they are falling behind England. I am sorry to zero in
on you, Brian, but obviously I know that you have a more direct
experience of this subject. In the light of that report and bearing
in mind your comments earlier, what would you say in response
to that?
Brian Lightman: I have not seen
that article this morning. However, one of the big discussions
that I have had with the Welsh Assembly Government about the way
that they have published the results this year is that I discovered
a few weeks ago that what they understand as five As to C and
what the English league tables describe as five As to C are completely
different measures. That is an example of the type of thing that
happens. In England, there are all kinds of qualifications that
count towards the Level 2 qualification and in Wales they have
not got round to including those qualifications. So, for example,
my school does the DiDA qualification—Diploma in Digital
Applications—that counts as four GCSEs here in England,
but in Wales it does not count. So you have to be very careful
about how you produce these figures and let them mean what you
want them to mean after one year. This is one year's results and
one indicator and we must be very careful about the way that we
read things into those results. Obviously, in Wales as in England,
we want our children to achieve the best possible standards. So
we must look at a range of measures and not just one thing. At
the moment, it is not possible to compare England and Wales. It
is not possible to compare national curriculum tests or assessments
in England and Wales, because the national curriculum for each
country is different. So there is a real danger that things will
be misinterpreted in order to put forward a particular point of
view. I would be very careful about that.
Mick Brookes: Of course, the mechanics
of this system are bound to want to defend it; David Hopkins was
one of those. But I think that he is absolutely wrong. I think
that it is absolutely right to say that comparability between
England and Wales is very difficult, but those mechanics need
to look at other things. For instance, they need to look at the
UNICEF report about the happiness of children in this country.
They need to look at all the work that has been done by Robin
Alexander on the primary curriculum. There is a lot of evidence
now that says that we are simply focusing on too narrow an area
and it is having an effect on children's lives and it is certainly
having an effect on the curriculum. We must try to measure the
things that we value, rather than valuing the things that we measure;
that is an old cliché, but I think that it is true.
Q156 Lynda Waltho: You talked there
about the comparison between England and Wales, but the Organisation
for Economic Co-operation and Development tests—
Brian Lightman: Sorry, which tests?
Q157 Lynda Waltho: Some 400,000 15-year-olds
across 57 countries were tested for the first time in 2007. Wales
came bottom in the UK for maths, reading and science, and trailed
at 33rd place for maths, beneath Latvia and Lithuania. That is
obviously a wider comparison, across a wider set of nations, in
which England was rated along the same lines. I am just wondering
about that test.
Brian Lightman: But, you know,
you get those results and you then have to start asking questions.
At the moment, there has not been time in Wales for anybody to
analyse the evidence base of that test. There will be questions
asked about that test, but I cannot draw a conclusion about the
results until we see the evidence base. It is the first time that
those assessments were used in Wales and we need to look at the
evidence base now and ask questions about it. I do not think that
it would be fair to draw on the results of that one test for the
first time and come to some conclusion that is perhaps making
two and two into five before we have analysed what the evidence
base is and what those results are saying.
Mick Brookes: And I could direct
you to Jersey, which has no SATs, no league tables and a totally
different inspection system and which would be top of that league.
I do not know whether it is in the table, but the results from
schools in Jersey and staying-on rates in Jersey are much, much
higher and it does not have the same system as we have, so it
depends where you look.
Chairman: Indeed, and Jersey does not
have the variety of population. These tests are exactly the kind
of tests that I have heard you arguing for. They are like the
APU—Assessment of Performance Unit—tests. They are
internationally moderated. Anyway, we will leave it there and
I will give Sharon a chance to ask her questions.
Q158 Mrs. Hodgson: One question sprang
to mind on the back of the decision-making that parents go through
and how they use the league tables to make that decision when
perhaps they should be visiting the school and making the decision
on a whole host of other things, one of which I believe should
be the distance that the child would have to travel. My daughter
has just started secondary school and is within walking distance
of the school. It is a good walk, but it helps to keep her fit
and she and my son do the walk together. One of my daughter's
friends is late about twice a week because she has a horrendous
journey. She probably passes between six and 10 other high schools
to go to that one. Obviously, her parents have chosen the school
for a whole host of reasons, but the experience that that girl—and
probably many other children throughout the country—is going
through must be affecting her learning in negative ways. I am
thinking of the stress of constantly worrying whether she will
catch the bus and getting into trouble when she gets to school
late. Eventually that might affect her whole learning experience
at school. My children were in school in London for a while and
some children commute to school from outside London. They have
a commuting distance that an adult would consider a chore. What
are your comments on that?
Dr Bousted: In London that is
endemic. I get on the train every morning and my carriage is delightfully
shared with schoolchildren commuting from one area to another.
A long commute to and from school must affect the learning ability
of children, notwithstanding what it is doing to the environment
and everything else, and of course that relates to the school
run as well. This is often about the parents' perceptions of a
good school, and the perception of a good school is often based
on the class of the intake.
Mick Brookes: I was going to say
the same thing. Rather than being based on the quality of teaching
and learning, the decision is sometimes based on the fact that
the school is full of the children of "people like us".
Dr Bousted: It nearly always is.
Q159 Mrs. Hodgson: So there are parents
who want to get their children into the school for that reason?
Dr Bousted: That is right.
Chairman: You wanted to ask about Making
Good Progress.
|
