Examination of Witnesses (Questions 200-214)
MONDAY 28 OCTOBER 2002
KATHLEEN TATTERSALL,
JOHN KERR
AND RON
MCLONE
200. Yes; in New Zealand, they said every school
thought they were assessing objectively, independently, but they
were not?
(Ms Tattersall) And the important thing, therefore,
is moderation; and, therefore, as an examination board, we would
certainly welcome more emphasis on the teacher involvement in
assessment, but it would have to be in a context where there was
a very clear framework of moderation provided by some external
body, of which we would be one of them, we hope, and we would
provide the exemplar material for teachers, we would do the training
of teachers, and we would moderate samples of the work which they
assessed.
Ms Munn
201. John Kerr said nobody really understands
what you do, as examining bodies, and Ron McLone said earlier
that OCR were doing things differently from the other two bodies.
Can you just explain, what you told us right at the outset was
that you complied with the Code of Practice in terms of setting
the grade boundaries, and just explain how you do it, so we understand
really very clearly how that is different from how OCR do it?
(Ms Tattersall) I think what Ron McLone was talking
about was the information that we provide to the awarding meetings
when they are making their decisions about grade boundaries. And
we provide, at the outset, both the candidates' scripts and some
statistical information which will help the awarders come to a
judgement about those scripts; and that statistical information,
as Ron McLone said, is GCSE mean grades, it is AS performance
of the candidates who are being judged on the A2, it is information
in normal circumstances relating to the previous examination,
in terms of how candidates performed, and we provide that from
the outset. So we provide parameters, which show very clearly
what the impacts of judgements are going to be and how they might
compare with, let us say, the GCSE mean grade data. We believe
that interaction between the evidence is important for people
actually to understand the scripts themselves; because there is
an awful lot of research evidence to show that if you simply present
scripts in a vacuum then people are not very sure exactly where
they relate to the different grades. So we provide as much information
as possible from the outset, and that is the difference between
ourselves and what I think Ron McLone was saying.
202. Just explain that to me again, because
what I understood Ron McLone to be saying was there is a clear
difference, and this is something which has been confused throughout
this debate between marking and grading, and he was saying, as
I understand it, that OCR mark the scripts and then use statistical
information to help with the grading and the grading boundaries.
Now you are saying something different?
(Ms Tattersall) No, I am not really. I am taking it
from the point of the grading boundaries. All of us have a very
clear procedure, in terms of the marking of the scripts, the standardisation
of examiners, they all have a meeting where they are standardised.
203. So when you talk about the awarding meeting,
that is the grading, that is not the marking?
(Ms Tattersall) It is the end of the process. All
the marking has been done, or should have been done, by that stage;
and then there is a group of people whom we call `awarders', there
will be a Chair of Examiners, who is the person who makes recommendations
to the Chief Executive, and there are the various chief examiners,
or principal examiners, for the different components of the examination,
and they will have made recommendations as to where they think
the grade boundaries should rest. And the awarders as a whole
will get that information, together with the statistical data
that I have just mentioned, together with a range of scripts,
which cover the various recommendations which have been made;
and, within that, the awarders have got to try to find the defining
mark between one boundary and another, and it is not easy. And
if you actually look at the range of decisions which awarders
make, some will believe that a mark of, let us say, 40 is the
mark, others will believe a mark of 39, others believe a mark
of 41, and so on, and somewhere somebody has to come down and
make a judgement on that matter.
Chairman
204. We understand that; but what happens when
a senior examiner, the most senior examiner, of a board, gets
in touch with this Committee and says, in the whole 30 years that
he has been in the examining business, he has never known a year
where, in the final meeting, after the marks have been agreed,
that they then are especially called back, as chief examiners,
to a meeting, to be told that all the marks in that subject have
to be changed? What is going on out there, when that can happen?
This is something that was communicated to this Committee, anonymously,
because the person, in terms of the chief examiner of that subject,
did not want to be identified. What is going on, if that happens?
(Ms Tattersall) I have to say, that did not happen
in my board, so I cannot actually account for what you are actually
describing; it does not happen in that way in my board, the recommendations
come through to me, as the accountable officer, and in the vast
majority of cases the recommendations stand, I accept them. In
the very small number where I say, "I am not quite certain
whether all the evidence has been properly taken into account,"
then the normal procedure in AQA is for that then to be discussed
with the Chair of Examiners, and some accommodation of view is
arrived at. In some instances, I might have to decide that a different
mark, and it is usually one or two marks that we are talking about
here, would prevail, and, as I said earlier, some of the decisions
which I took, in the very, very small number of cases where I
made a decision, the majority of my changes were actually in favour
of the candidates, they were actually to drop the mark, not to
raise it.
Ms Munn
205. I am just getting even more confused now,
because I am not sure how your process is different from OCRs,
it does not sound different?
(Ms Tattersall) Obviously, I cannot account for what
Ron is saying, in terms of it being different. I think what he
was saying was that some of the statistical information, which
we introduce at the very outset of the awarding meeting, because
we believe that to be transparent, open, above board, everybody
knows what is going on, might have been introduced into the OCR
procedure at a later stage.
206. And do you think there is something different
about script selection, which was the other bit, where it starts
to get very technical but which seems to be very important in
terms of grade boundaries; is it different?
(Ms Tattersall) The Code of Practice very clearly
lays down that the script selection is made by the awarding body
staff, in connection with the recommendations which have been
made by the principal examiners for the unit or the paper concerned;
so I doubt very much if there are differences really in how we
operate there, because there is a very clear statement in the
Code of Practice.
Paul Holmes
207. Can I ask you, again, individually, the
question I asked everybody collectively. We heard a few minutes
ago from OCR, we were talking about the grid, showing whether
40 and 60 adds up to 100, or 50 and 50 does, and we have heard
you talking about whether a boundary should be 39 or 40 or 41.
Should not the exam boards this year, or in the last two years,
have been saying, to the media, to the QCA, to the Government,
that, that thinking, really you have got to move on from that,
because there should have been a quantum shift upwards, as a result
of the new exam system that has been introduced, that it should
not just be measured within 1 or 2% against last year's and the
year before and the year before?
(Ms Tattersall) I think that really is precisely the
issue that, as awarding bodies, we took up with QCA in March,
when some language, which might have suggested that we ought to
be having the same percentage of candidates, was being used, and
we took up that issue very firmly and very clearly in the letter
that we sent on 22 March, and which then, in my judgement, was
resolved by the letter which we had back from William Stubbs.
In terms of the quantum shift up, as it were, you referred earlier
to a 4% rise, and I think you were suggesting that perhaps it
ought to have been a 9% rise, or whatever; now, if you actually
look at individual subjects, you will find that there are 8% here,
9% there. Four% is the general, overall, national shift across
the three awarding bodies; look at it in individual subjects and
you will find some very different patterns emerging. And we have
not done this analysis yet in AQA, but I am suggesting to my colleagues
that we do it, as to whether those shifts were different in those
subjects which were modular beforehand from those subjects which
moved to a modular system in 2000; and that is an analysis which
certainly we can have a look at, and we will be happy to provide
the Committee with information later on.
208. If the average pass rate this year was
a 4% increase, what were the sorts of averages increases over
the last four or five years?
(Ms Tattersall) It has been at round about the1%,
sometimes less than 1%, level, but it has been a very marginal
change over the years. But, again, if I can pick up on the modular
theme, if you go back to round about 1993, when many of the subjects,
particularly in the sciences, started to, as it were, go modular,
you did actually find the shift then at the Grade E and above
level, which was greater than the normal pattern in other exams;
and that was in a system where you had a greater facility for
retaking than now, because there was no limit on the retakes.
Chairman: I have asked Dr McLone to come
back, and he has very kindly agreed, because I think Meg Munn
was not happy that she quite fully understood the difference between
the two approaches of the two boards; so would you like to rephrase
your question, Meg Munn?
Kathleen Tattersall, Director-General, AQA, and
Dr Ron McLone, Chief Executive. OCR, were further examined.
Ms Munn
209. What I am trying to get at is understanding
the point that you made earlier, Dr McLone, which was about saying
that you were doing it differently. Now Kathleen Tattersall has
explained to us what they do; is that different?
(Dr McLone) I do understand what Kathleen
is saying, because we have had these discussions many times. They
are both within the Code of Practice; the whole thing about the
Code of Practice says that you have got a balance between examiner
judgement and statistical evidence. I have to say, I am going
back a few years now, back to the Midland Examining Group, which
was part of one of the first GCSE groups, along with the NEAB
and SEG, and so on; we always took a view then that what we wanted
to do was to make sure that the examiners had the first go and
talked about it and then looked at what the impacts were. It is
sort of very much a bottom-up process; in a sense, I think the
Midland Examining Group said it was an accountable process, because
you could see what was happening with the statistics, because
then it was evident. It is true, there are other ways of doing
it, and one of the other ways is, as Kathleen has said, to produce
a good deal more of the statistics to inform where the scripts
are selected in the first place. That, essentially, is where we
are at.
Ms Munn: Thank you. I understand.
Chairman
210. Can I come back on a question I gave, that
it was one of your examiners, chief examiners, I was talking about,
who approached this Committee, who approached me, as the Chairman
of the Committee, because, in the 35 years, I think it was, he
had been an examiner, and now chief examiner, of a subject, he
had never had the process that occurred this summer ever before,
to have had the final meetings of his exam board, to have come
to some conclusions about the marking, and then to be pulled in
by a conference of heads of examining boards to be told that grade
boundaries were going to be moved. Everything you have said today
has said it has been business as usual, it has never been any
different; but here is one of your chief examiners who said something
very different happened this year?
(Dr McLone) Indeed; and, without knowing the subject,
of course, I cannot actually directly comment on what an individual
would say. The difference this year has been, it is the first
time in 50 years to have such a fundamental change of A level;
it is not different in the practice, and it may well have been
that, in his subject, or her subject, I would not like to say
whether it is his or her, nothing has happened significantly,
but this year, in a few subjects, I have to say, in most of the
subjects, did not get such substantial issues that have arisen,
but in some subjects, obviously in this subject, there was this
difference which has come by looking at what they have suggested
against statistical evidence that has been more dramatic than
in the past. And we have said, and I say again, it is a major
shift this year. The way we do it obviously works very well in
circumstances when it is maintained year on year and it is a regular,
consistent standard, but if you are working, again, with an A2
standard, which, I still submit, we did not know, we had no exemplars,
that has provided the issues for some of our examiners, it is
absolutely right. Remember, I look across all subjects, he is
looking at his own subject.
211. So it is not surprising that some of these
people, that saw themselves as guinea-pigs, might now consider
themselves sacrificial lambs?
(Dr McLone) I think it is unfortunate that we had
no trialling done before we made such a major change.
Mr Chaytor
212. Yes, but, to Kathleen particularly, is
not the root cause of the problem the fact that A level has this
unique means of assessing the grade boundaries, we do not have
this in awarding university degrees, the degrees are not moderated
by students' performance at A level, we do not have it in the
standard attainment test; and do you not think there has to be
in the future a move to a criterion system for AS and A2?
(Ms Tattersall) We are not a norm referenced system,
I think that is the first thing that I would wish to say; we moved
away from norm referencing many, many years ago. I think, at some
point in the 1980s, A level ceased to be a norm referenced examination.
Nor are we fully a criterion referenced examination, but we have,
as it were, moved along the scale more to that by defining some
criteria to underpin the grades, and those criteria are defined
at A level overall. If we moved entirely to a criterion referenced
exam, then you have got to take the consequences of that; namely,
if you have not mastered whatever is determined for the grade,
you will not get that grade, however good you are.
213. But most parents would assume that should
be the case, would they not?
(Ms Tattersall) But what we do have is a system which
is a soft criterion referencing, for want of a better term, where
there is some compensation for a weakness in one skill area, with
strength in another, and, in that sense, you could say, it is
a little bit of a fudge, when it comes to the criteria. But it
is a system which does reward attainment at the more general level
than some very specific criteria would do, and I think it is a
system which has served students exceedingly well over the years,
and, indeed, if you look at GCSE it is exactly the same sort of
system, if you look at Key Stages 2 and 3 then I think what we
are talking about there is a pre-determined level of attainment,
which is only slightly moderated when the students have actually
done their SATs. Some of us would say that, in some ways, criterion
referencing is fine, but it is when the students actually do the
exam that some of the criteria begin to break down, because it
is not like that in the real world. So, in a sense, I would argue
very strongly for the soft criterion referencing system that we
have, provided we have a little bit more definition of those criteria,
but not so specific that we are going to cut people out of the
grades.
214. But do you think we have this soft criterion
referencing because we have this overemphasis on external assessment,
and if we had more internal assessment there would not be the
need to have the methodology for the external assessment that
was designed to compensate for any protection.
(Ms Tattersall) I think we would have exactly the
same issue, but we would have to have descriptors which enabled
teachers to mark work consistently; and the fact is that students
do not perform in consistent ways, and, therefore, there has to
be, as it were, some sort of compensation for the way in which
students strive to meet the criteria, and that is what our system
does. And I think it is irrelevant, whether it is internal or
external assessment, to actually apply the criteria that we have.
I am all for making the criteria more explicit, trying to reach
criteria which are better understood by everyone, but I really
do think that we would be in trouble if we tried to rely entirely
on criteria for our system.
Chairman: I think that we have to end
the session now. Thank you, all of our witnesses today, who have
taken the time also to enjoy a rather different format we have
played with today, and thank you very much for being so flexible.
Can I say, to quote John Kerr, perhaps a mission to explain, I
was thinking this when you were talking about, of course, everyone
knows we have moved from norm referencing to soft criterion referencing,
but there are a few people in my constituency who did not quite
realise that that had occurred. Perhaps it is part of the role
of the QCA and the examining boards to tell parents and students
that that is the case. Thank you.
|
