Examination of Witness (Questions 440-459)
WEDNESDAY 4 DECEMBER 2002
MR MIKE
TOMLINSON
Chairman
440. Mr Tomlinson, welcome. We thought when
we said goodbye to you as the Chief Executive of Ofsted that we
would not see you so regularly but we are obviously going to see
more of you than ever before! You are very welcome to this Committee
but are you not becoming a "man for all seasons", to
an extent? I was in the radio/television studio this morning and
they complained that the Tomlinson report had not given them enough
blood on the floor, and I am looking at this painting behind you
and there seems to be blood on the wall in this particular room!
Is there not a problem? Knowing you well, you have a personality
that is likeable, if I may say so, and you have come up with two
reports that do not say anything nasty about anyone. In a sense
people are perhaps saying—parents, students—that we
went through this terrible trauma during the summer yet when you
read Mike Tomlinson's report basically no one is to blame and
everyone has got off scot free. Is that a fair comment on you
being too nice to everyone?
(Mr Tomlinson) I do not think so, no.
I think I try to be fair in the sense of where the evidence allows
me to go rather than where my own personal views might want to
take me, and those are two different things. This inquiry was
seeking to get to the bottom of what happened. I think my report
pointed clearly to where there were inadequacies in the system
which allowed the position we reached this summer to occur. I
do not find that attaching personal blame is a particularly helpful
activity. The issue was about the systems and the behaviours that
those systems allowed, and nothing that was done this summer was
outside of the code of practice and the frameworks which govern
that.
441. But how do we get to such a state where
you come up with some remarkable recommendations for change and
they, as we have heard yesterday, are going to be mainly accepted
by the Secretary of State and implemented, and indeed you are
going to take a significant role in the improvement of the system?
How did we get to the state of what went wrong with the system,
the relationship with QCA and the examining boards? (Mr
Tomlinson) I think probably it is long coming in history but
the particular point really is that, first of all, the introduction
of AS and A2, as I said in the interim report, was rushed. A2
was not piloted which it should have been, and there was no script
material available to the QCA to inform and use with teachers,
lecturers and students. Secondly, I believe that, though the QCA
issued some guidance, that guidance in my view was not satisfactory
and sufficient to clearly define the standard of AS and A2 and
to exemplify it by material not only with reference to the criteria
but also to students' work. That was missing as well. Then we
get into a third area which has been going on for a long time
and that is the annual August frenzy that says, if more students
have achieved the standard then the only way that could have been
done is by somehow lowering the requirements they had to meet,
and I find that a very unsatisfactory situation. So it is a combination
of a whole range of factors, some of which have been with us for
a while and others of which are particular to Curriculum 2000,
and more broadly some of which are particular to the way we tend
to see the introduction of innovation and new policy requirements.
442. You will know that certainly the Chairman
of this Committee agrees with your comments on the summer frenzy,
and what this Committee is very keen on is maintaining confidence
in the system; that students who have worked so hard to pass their
exams feel confident that the qualification is a good one and
endures for years to come. But you heard Mr Porkess give evidence
to this Committee in the last half hour: here you have conducted
what we all assume is a thorough inquiry, in two parts, and there
is Mr Porkess, a respected and well known statistician, who says,
"Come on, you missed the point"? (Mr Tomlinson)
I do not think I did. First of all, the awarding committees do
make recommendations about mark grade boundaries for each and
every unit. Sometimes at those Committees they are specific to
a mark: sometimes they give a range of marks and do not come down
on a firm mark, and I am talking about the system as a whole—not
the syllabus with which Mr Porkess is associated. In the case
of the particular board that administers Mr Porkess' syllabus
there is a second stage, and that is something called the GEM
(Grade Endorsement Meeting) and that takes the recommendations
of the awarding committee and involves the chairman of examiners
of the subjects concerned. It has also available to it not only
scripts but other data about performance and it can make recommendations
on the movement of grade boundaries. Those committees are often
attended by very senior people in the board, sometimes indeed
the chief executive but not at that point acting as the accountable
officer, and then those recommendations go to the accountable
officer and are moved again. I think what is important to accept
is that there is nothing sacrosanct about the recommendation of
the awarding committee. It is their view and it is a respected
view and an important one, but to suggest that no changes can
being made to those mark grade boundaries flies in the face of
what has happened consistently over time and no doubt will continue
to happen in the future. So it was a new situation this year.
The other point that has to be stressed is that at the accountable
officer level, too, there is that one and only opportunity to
look across the suite of syllabuses. In mathematics there are
a number of syllabuses all under the heading "Mathematics",
and the necessity there of ensuring that an `A' in that syllabus
in terms of the standard of students' work and in a syllabus in
that suite is the same is a key role for the accountable officer.
443. So Mr Porkess is plain wrong? He is wrong
to believe there are thousands of students who had an injustice
delivered to them this summer, and he is plain wrong that there
are serious problems for the future? (Mr Tomlinson)
I do not accept some of the assumptions that he makes in his paper
and hence his calculations. I am not pretending, either, that
this year or any other year there may not have been students who
did not get the grade that they may have thought they got—or,
indeed, deserved. That is the nature of examining. It is not a
science, it is an art, and you make decisions about grade boundaries.
Now, that may sound shocking but it is the reality. We have a
criterion reference system but it is not a perfect one. Nothing
of a perfect criterion reference system exists, and you have each
year, when you have got the data and the results, to have a look
to see whether or not applying the criteria and judging where
the grade boundaries are is right. In many instances they do need
movement and those movements vary between syllabuses and between
boards, in part because the arrangements for the process are different
in themselves.
Ms Munn
444. One of the things that we have struggled
with to some extent in talking to the different examining boards
is understanding the whole process that the examining boards go
through in arriving at first the marks and then the grade boundaries,
and understanding that there is a lot of confusion around that.
When we had the three boards here, there was a discussion which
indicated that two boards came to their conclusions in one way.
What they said, if I recall, is that they introduced statistical
information at a different point. Now, OCR have helpfully given
us a memorandum which sets out their process and includes the
process you have just described but we have not got one from the
other boards so I am still at a bit of a loss as to how that happened.
Did you as part of your inquiry form a view about whether either
of those ways is better, or is it just that they are different? (Mr
Tomlinson) I came to the view they are different but would
not of necessity lead to necessarily different outcomes. I think
that the difference in terms of how much statistical data is available
at various stages is correct and certainly at the awarding stage
in the AQA and Edexcel there appears to be more statistical data
available at that point than in OCR, but that additional data
becomes available at the GEM stage and even more at the final
stage—more in the sense that the accountable officer is
looking across all the suite of syllabuses in a particular subject,
which is not something easily done at the other two stages.
445. Would having that statistical information
earlier, as the two exams boards do, in your view mean there would
be more likely to be a positive or negative influence on people's
thinking in terms of where the grade boundaries should fall? (Mr
Tomlinson) If I take the balance of opinion of the chairs
of examiners that I have spoken to then I think the provision
of as much data information as possible at that awarding stage
is regarded as beneficial to their work. That is their view and
I respect their view as very experienced chairs of examiners.
446. So by bringing it in later what is the
effect upon the OCR process, in your view? (Mr Tomlinson)
I think it could lead to mark grade boundary changes which are
more numerous and potentially more in number than at the other
stages, and that was certainly the evidence I was presented with
by the three boards when I asked for their most recent 2001 data
movement in mark grade boundaries.
447. But what you said earlier still would hold
true—that the outcome is not better or worse; it is just
a different process? (Mr Tomlinson) It is different.
There are some studies being done by Professor Carol Fitz-Gibbon
in Durham which looks at the performance of different boards with
students of equivalent GCSE grades and what they get at A-level,
and certainly mathematics shows a close correlation between the
results of mathematics across the three boards, which is reassuring.
448. Is one of the outcomes of your report that
the process should be standardised across all the boards so that
the pointing of fingers in terms of "more grades are moving
here", which is what you seem to be saying is not justified,
would not happen, or can we live with two different processes? (Mr
Tomlinson) I think we can live with two different systems
as long as at each stage and particularly at the final stage changes
to mark grade boundaries are not made without recourse to discussion
with the Chair of Examiners, who of course has been intimately
involved in looking at students' work and therefore brings that
important dimension to that discussion. That is something, of
course, which following the interim report the QCA has moved to
make a requirement. There were a number of cases this last summer
that I investigated where those changes made had not been discussed
and agreed by the Chair of Examiners. In the case of Mr Porkess'
syllabus both the Chief Examiner and the Chair of Examiners had
agreed the mark grade boundary movements that were recommended.
If I go and investigate them, they are going to say, "I agreed
with these for the following reasons", and how do I gainsay
that they were wrong without going back and marking every single
paper myself, which is clearly impracticable.
Mr Pollard
449. The essence of all this is about resotring
credibility which we all support. Mr Porkess very clearly in his
evidence a few minutes ago indicated that others may be keeping
quiet. You have spoken to lots of people. Is there any evidence?
Are you confident that Mr Porkess is a lone voice in this? (Mr
Tomlinson) I would never put my head on the line and say he
is the "lone voice". I think I should remind the Committee
that I asked for the boards to relieve the Chairs of Chief Examiners
of the confidentiality clause. They were free to speak to me and
to offer me written and oral evidence if they so wished, and the
confidentiality clause did not count, and a large number of them
did submit evidence to me. In some cases it was very supportive
of what had happened and their belief that it was correct; others,
as you well know, did not agree. So the boards have not sought
to gag anyone at all. There are some issues which came out of
the inquiry which some examiners, and indeed some schools, continue
to feel concern about and I referred to a number of those in my
report of yesterday—in particular the fact that in one syllabus
the marks separating "A" and "U" were very
small in range and therefore gave rise to some difficulties. Now,
like Mr Porkess, I am surprised by that because, of course, not
only had that whole assessment proposal to go through the board
itself but it also had to go through QCA, and it raises some questions,
shall we say. There are schools still worried about that—and
quite rightly so. But the problem is it is not about the grading
issue but about the whole marking and assessment arrangement.
Those are being tackled by QCA in conjunction with the board and
there will be changes not only to the psychology but the English
literature syllabus, which suffered in a similar way, for the
examinations next year. So there are people concerned about those
issues and it did spark off concerns about the grading issue.
450. Mr Porkess could keep niggling away whilst
everybody else is trying to draw a line and move forward and restore
the credibility that everybody needs. If you just keep niggling
away, does that not undermine what you and others are trying to
do? How do we close that gap? (Mr Tomlinson) Well,
it is not going to help, is it, and certainly it does run the
risk of undermining efforts to restore credibility which I think,
and I have said in my report, is absolutely paramount: that people
feel—students, their parents and teachers—that next
year's examinations are absolutely secure and they are going to
get the grade their work deserves, and I have every confidence
that what is happening in the QCA, with the boards and others
means we are going to be quite clearly able to say that next year,
and I hope I will be able to say that. I have spoken once with
Mr Porkess and we have had a number of telephone calls. I might
suggest to him that with the OCR and the QCA we sit down and look
at this and see if we can find a solution which is acceptable
to all parties. I do not mean a fudge—I think there is a
need here to understand better and to have all the evidence in
front of people such that we can make sensible decisions.
Chairman
451. So you are suggesting a meeting? (Mr
Tomlinson) I am suggesting perhaps a meeting with the QCA,
the OCR board and particularly with the chair of examiners.
452. And Mr Porkess? (Mr Tomlinson)
And Mr Porkess, to look at this issue as clearly as we are able
to.
Chairman: "Blessed are the peacemakers"—and
I mean this Committee!
Mr Simmonds
453. Do you think your report would be more
complete and have a greater holistic approach had you considered
movements in all grade thresholds and not just in extreme ones? (Mr
Tomlinson) No, because as I have already indicated movements
of mark grade boundaries have been something which are part and
parcel of the examination systems—and justifiably so. You
cannot set a paper year on year which has the same level of demand
or difficulty. It is not humanly possible to do that. Therefore
you have to look at the marks and compare them with the past and
ask yourself whether you are still pitching at the same standard.
So there is always going to be mark grade boundary movements.
In terms of my inquiry I got the data for all of the mark grade
boundary movements for every unit done by all three boards this
year, and I asked them what the mark movements had been in the
previous year, 2001, which was the only basis because that was
the first of the AS systems as well—for most. Mr Porkess
quite rightly says there have been modular syllabuses for some
while. I equally wanted to know whether those changes had been
agreed after discussion with the Chair of Examiners whose responsibility
it is, and I did not want an assurance from the board but a written
assurance from the chair of examiners that that had been the case,
and I got those assurances in the very large majority of cases.
Where I did not, it was part of the stage 1 regrading exercise.
454. So with the exception of Mr Porkess, who
we have heard from this morning, there is no evidence you have
come across either directly or anecdotally that suggests there
was greater movement—not in the extreme grade thresholds
but the smaller ones—this year than in previous years? (Mr
Tomlinson) No, no evidence whatsoever. Remember that Mr Porkess
is a principal examiner; there is a chief examiner and there is
a chair of examiners for the subject, and the chair of examiners
is the key person who takes forward the views of the awarding
committee and certainly, as you have seen from the response, he
was party to the discussions about movements in that particular
set of syllabuses—and supported them. Now, if I had gone
to him and said do you agree with the boundary changes that were
made, then the answer would have been, "Yes, and this is
why"—and where would I have turned?
Chairman
455. If you had gone to him—what do you
mean? (Mr Tomlinson) If I had gone to Dr Seager and
included those units he would have said "I agreed those changes
because. . .", and he would have produced the evidence for
me. So I believed I did do all that was possible to identify where
movements were made which were outside the norm and had not been
agreed with the Chair of Examiners.
Mr Simmonds
456. In your report you recommend that the examiners
are professionalised. Where do you see those new professionals
coming from? Out of the existing teaching profession or as a new
graduate intake as professional examiners from the day they leave
university? Will this not impact on the teaching profession by
extracting numbers from it? (Mr Tomlinson) No. I am
not talking about a separate cadre of people; I am saying I want
to professionalise the examining process which at the moment is
quite rightly dealt with largely by teachers in our system—both
in schools, in colleges and in universities for that matter. What
concerns me at the moment is that their work gets little or no
credit: their training is, I believe, not as thorough and as consistent
as I would hope it ought to be, and what I am looking for is good
quality training to be provided for examiners and for examination
secretaries in schools and colleges who have a significant role
in all of this, and that that training should be properly accredited
and that that accreditation should be part of the individual's
career and professional development, and I think it would be quite
right to think in the future that a head of department in a secondary
or a head of faculty in a college should be someone who has had
experience of examining who can advise his or her colleagues and
new teachers in what is a very important activity—not just
in the public examination sense but in the internal school examination
sense as well. I am not looking to pull teachers out of school;
I am looking to give teachers a real professional status as an
examiner in the system.
457. Many heads of department in secondary schools
say they have quite enough to do as it is without laying more
professional work on to them through this examination process
you are talking about. (Mr Tomlinson) My reaction to
that is to say at the moment that is where the vast majority of
our examiners come from each year. I also have met a number who
are no longer examiners, and their reasons for not doing it any
longer are very much along the lines that it just does not get
the credit it deserves, and if we do value our examination system—and
I think we should—then we should ensure that the people
doing it receive the credit that it deserves and the training
and support that they need to do the job effectively, a job which
is changing quite significantly as time passes.
458. And paid? (Mr Tomlinson) And
paid too, yes.
Chairman
459. Some of us might say that if you had come
from Mars and made these comments we would understand but, come
on—you have been a senior education official for many years
and Chief Executive of Ofsted. All the time you were in Ofsted
and in other senior education roles, did you never worry or have
concern about the professionalism, and the way in which you ran
out examining and examination training for examiners? (Mr
Tomlinson) Yes, not in recent times because certainly Ofsted
did not have access to the examining process, but when Ofsted
was created in 1992 we continued then alongside QCA to have involvement
in monitoring the examination system, and certainly I was very
much involved at that stage in the work that was done in the reports
produced following the introduction of GCSE and indeed also at
A-level. We were, and our reports then were, critical of what
was happening at that point in time so it is not a new call. I
think it has become heightened, however, by the expansion in the
number of examinations that are sat and marked and upon which
so much depends for both schools and individual students.
|
