Examination of Witness (Questions 327-339)
WEDNESDAY 6 NOVEMBER 2002
SIR WILLIAM
STUBBS
Chairman
327. Sir William, welcome back to the Committee.
There has been a lot of water under the bridge since we last met
in May when we had a very good session as I recall. Can I first
of all not only welcome you but also say that this session is
about learning and about how we make our system work better rather
than worse, to learn some of the lessons from past experience
and see how we move positively into the future. We do not really
want to trawl over particular things of personal concern to you,
and everyone here will know what I mean by that. We want to learn
about how we get that relationship between the QCA and the examining
boards and between the Department and those organisations on a
better footing. I know from our previous session in May that you
had some pretty clear ideas about that then. You will maybe have
seen some of the evidence that was given to this Committee by
the examining boards last week. I am not sure you were in the
room when I asked my opening question to the people who have just
given evidence, but what I was asking them was, in what ways can
you move to learn from that past? Everyone knows that when a new
exam comes in there is going to be a certain amount of transition
difficulty and some might say that perhaps we have had less than
we could have had, but the Chief Executive of the QCA last week
was very pessimistic about having more problems in the coming
year, which rather concerned the Committee. Given the events of
this transition, I wonder whether you have any particular recommendations
for the Committee on how we could improve the system?
(Sir William Stubbs) Thank you, Chairman, and thank
you for that clarification, although I would just like for the
record to say, as we are in a new session and a new topic under
discussion, that I was Chairman of the QCA for five years. Throughout
the entire period I was part-time and for four of the five years
I was unpaid. I only was paid from the time I retired from my
full time employment about a year ago and I was asked to increase
my involvement from two days a week to four days a week by Estelle
Morris and, looking back at the letter, she said she wanted me
to provide strong leadership. That is where I come from in all
this. What I would like to do at the beginning, and the Clerk
very kindly sent me a copy of the transcript of the interviews
with the Awarding Bodies and the QCA, is to say something about
the A-level crisis, a term which has been used both in the Committee
and further afield. It seems to me the word "crisis"
in relation to A-levels has two possible meanings: either the
operation of the awarding system is so significantly defective
as to give good grounds for concluding that the main outcomes
are invalid, or, alternatively, confidence in the validity of
the system has been so diminished that there is widespread anxiety
among students, parents, universities and employers. The first
meaning is clearly inappropriate on the basis of the evidence
that was available five weeks ago and the evidence that has been
uncovered since then. The system is sound and indeed some of the
evidence I heard this morning confirms that. But a national exam
system as complex as the one that we now have available relies
significantly on trust in the overall process, trust in the markers
and examiners, trust in the integrity of the exam boards, and
trust in the integrity and independence of the regulator. Therefore
at the outset, Chairman, I have to say that in recent weeks each
of these elements of trust has been significantly and quite unnecessarily
weakened. Therefore the challenge for those responsible for those
matters in the future will be to restore that trust, but they
do so on the basis that the underlying system is sound, and that
is an enormous strength. So that is where we are coming from because
I think it is important when the word "crisis" is used,
it is a crisis of confidence rather than anything else. At some
stage, and I know you said you do not want to go into too much
of the past, but one cannot understand the future without the
past and I think in some of these discussions there will be something
328. The Chairman was merely trying to be reasonably
sensitive about these things.
(Sir William Stubbs) I know and I appreciate it.
329. Feel free to cover any subject you wish.
(Sir William Stubbs) At some stage I would like to
talk about the maintenance of standards in A-level but not necessarily
in this opening statement. All I wanted to do at the beginning
was say let's conduct a discussion on the basis of terms that
we understand and that is the way I understand "crisis",
and I think that is the probably the way you understand it in
the light of what has happened over the past few weeks.
Chairman: At the beginning of the summer
in the early days of the so-called turmoil both you and I appeared
on the same programme saying there was not a crisis and dampening
down the suggestion.
(Sir William Stubbs) Absolutely, Chairman.
330. However, let's move on. One thing that
came through from the evidence this morning was that one failing
of the QCA in the minds of those people who are the consumers,
in a sensethe colleges and schoolswas this inability
to set parity of standards across the piece. It seemed to be a
pretty valid criticism that QCA did not really do that. What would
you say to that criticism?
(Sir William Stubbs) I have got a little bit in reply,
Chairman. I think the chronology starts from April 1998 when the
then Minister responsible for qualifications, Baroness Blackstone,
in agreeing the new system and saying this was the Government's
policy, in 1998 said: "We are determined to ensure that A-level
standards are safeguarded and that all students study to rigorous
standards." From the outset at the time of the change continuity
of A-level standards was absolutely in the Government's thinking.
A year later in March 1999 a letter was sent to all schools from
the department "no compromise on A-level standards".
In August 1999 David Blunkett speaking as Secretary of State said,
"I can assure you that there will be no reduction in A-level
standards under this Government." In April 1999 a DfES official:
"Ministers place the standard of the A-level examinations
as the priority." There is absolutely no doubt where the
Government was coming from. In May 1999, as we started to develop
the intricate arrangements for the examinations, HMC wrote to
the Minister responsible and said: "It would appear that
the awarding bodies are contemplating various statistical treatments
to ensure that the first set of A-level results for the new system
will be very similar in outcome to the current percentage gaining
each grade. We would maintain there should be a small but definite
increase in the numbers passing and gaining higher grades under
the new system." As an aside, the outcome last year was a
4.5% increase in the pass rate and a 2.1% increase in grade As.
I would put it to you that that is exactly what HMC, GSA and SHA
were asking in May 1999 of the Minister. On the basis of that,
in June 1999and this seems to me absolutely significantthe
QCA then published a statement on standards which was subsequently
on their web site"broadly speaking, the proportion
of grades awarded in the current A-levels and those awarded to
candidates completing the new A-levels will be expected to be
similar. Where, however, on the basis of the quality of candidates'
performance and changes in the nature of the candidature"and
as we know it did change"a more substantial change
in proportions is justified, this will be acceptable, provided
the reasons for the change are fully justified and the standard
of the full A-level is maintained." There was correspondence
taking place at that time between HMC following an exchange with
Tessa Blackstone whom I referred to and Nick Tate who was the
Chief Executive of QCA, and they wrote to David Hargreaves, who
had by then become the Chief Executive and this is what HMC said:
"We cannot accept the lack of action over proper definition
of grade boundaries for the new awards. Standards must be defined
and some anchoring device must be established. Whilst it is good
that the awarding bodies will provide examiners with a comprehensive
package of statistical information, we would very much wish to
know whether they are going as far as to establish grade boundaries.
We would suggest it ought to be possible to use historic data
on regression lines to ensure that the various boundaries will
map on to a predicted grade boundary on a completion of A2."
That is HMC. The reply they received from David Hargreavesand
this is my last statement on this chronology, Chairmanleaves
absolutely no doubt on the record: "We are not clear why
you suggest there has been a lack of action over a proper definition
of grade boundaries in the new awards. A vast amount of work has
taken place throughout the development of the new specifications,
sample assessment materials and detailed statistical modelling
of the new awards. The Joint Council is involved in an extensive
programme of research to ensure that when the first awarding bodies
meetings take place next year the examiners are provided with
the most comprehensive set of statistical data that will ever
have been used in our public examination system. Historic data
on regression lines between GCSE and A-level are central to the
work that has taken place and the mapping you describe has been
going on for many months. You say that Nick Tate's statement `the
establishment of standards in any qualification is complex and
the prediction of grade profiles cannot be precise' is unacceptable.
No examination system which provides for an element of examiner
judgment and a changing cohort would allow the precise prediction
of grade profiles. This would be possible only with a completely
non-reference system. You may be arguing for such an approach
but that would represent a fundamental change in the way qualifications
are awarded and a step away from equitable treatment of candidates
over time." That, Chairman, effectively ended the correspondence
on standards between the heads associations and the QCA. There
have been since then, I am told, and I was not involved, something
between 30 and 40 technical meetings to flesh out the arrangements.
So I am in no doubt from the record that there was a clear understanding
of standards, recognising that we did not have past papers. Standards
are not like the metre where one could in the 18th century go
and hold something against it. It is not like that; it is a combination
of judgments made every year against criteria, against specifications
and against the evidence of previous performance. I believe it
would have helped to have had a run of exemplar A2 examinations
beforehand, in other words pilots. That was just not possible
in the time available and, indeed, would have been very complex
because to be good pilots they would probably have had to have
taken place after the AS examinations and you would have had to
draw on the AS experience so you would then have an interregnum.
I am not sure exactly how one could have run those terribly smoothly.
We did not have that. As far as the standards were concerned,
recognising we did not have past papers, there was a comprehensive
understanding and indeedand this is what I find utterly
bafflingthe results of two of the awarding bodies, having
been held up to the daylight more than once and scrutinised, have
come through with flying colours in judgments that I find must
be exceedingly difficult for the chief executives to make. I think
they have done a splendid job and we should be congratulating
them. Edexcel came from its knees. When I last saw you Edexcel
was in intensive care and indeed the board of Edexcel had decided
as a matter of policy that it wanted to abandon A-levels and cease
to award it as an awarding body and was in the course of discussion
in the spring on selling that off to a private company. Yet through
the valiant efforts of officials in Edexcel and colleagues in
QCA, they came through in the summer and produced an unflawed
system. I have said this to the Secretary of State not once but
twice that I believe, like you, there is no evidence of widespread
failing. There is evidence of shortcomings in one awarding body
but even there in only part of the judgments made by that awarding
body. We now know that the chief executive of OCR miscalled it
16 or 18 times out of the several hundreds of judgments he had
to make and he made a mistake. When I say made a mistake, when
fellow professionals are called in and asked to look at it, they
took a different view. I do not think there is a walk of professional
life where, when a professional judgment I've taken, whether it
be law, medicine or whatever, and held up to scrutiny by an independent
second opinion that you will be guaranteed you will get them all
confirmed as the view of the first opinion. In this case his judgment
was found to be wanting, but it was confined to a relatively small
number. What has caused the worry for not just tens of thousands
but hundreds of thousands is they thought their certificates from
other awarding bodies and from the unflawed part of OCR were invalid,
and for that there is not a shred of evidence. I believe it is
a scandal; it should never have happened. On standards thereforeand
that is where we startedI am saying there was evidence
there on the QCA web site and there have been plenty of technical
meetings, but there is nothing on the record over the last two
years from bodies that I have seen about it, although they all
recognise that this was a difficult transitional year and I think
in the main they have done well.
Chairman: Sir William, that has been
a most helpful chronology and explanation to the Committee. Now
we will begin the questioning. Meg Munn?
Ms Munn
331. When you came to see us in May you were
very confident that the quinquennial review would be a very positive
one and that it would say basically that you were doing a good
job. Are you satisfied with what was in the quinquennial review
and the conclusions that they came to? Do you think they were
fair?
(Sir William Stubbs) How does one say one is satisfied?
If we were graded, it was beta plus or alpha minus or something
like that; it was a good report. Indeed, one of the reasons last
time why there was an interregnum about the chief executive, whom
I hope you will find a very good colleague to work with in the
future was that we wanted to wait in the making of that appointment
until that quinquennial was out the road. I think it confirmed
in an area where 99% accuracy is not acceptable that the QCA,
in the observation of most correspondents to that inquiry, is
doing a good job. What it has got, though, and I heard from the
heads just 10 minutes ago and I saw in the evidence from Ken Boston,
is an accretion of tasks that are not central to its purpose but
were given to QCA because there was no other body in town that
the government could trust to do it, and that is of course running
key stage tests. They are a huge exercise, they are politically
highly significant, with great involvement by DfES officials (in
my view too great an involvement) and a way has to be found to
deal with that and to distance it from QCA. You could give it
to the awarding bodies but I think that would be unfair because
it is different to their main tasks, but a way has to be found
to get some kind of clear water between QCA and the key stage
tests.
332. One of the recommendations in the report
is that QCA should strengthen its capacity for intelligence gathering
as regards standards and then adopt a more visible and authoritative
public stance. I think this goes perhaps to the heart of the matter
you were just discussing, where in terms of creating confidence
in the examination system, in terms of trying to get past this
situation every summer where we have this "Are standards
dropping?" what this is suggesting is that QCA itself could
play a much more important role in creating that public confidence.
Do you agree with that?
(Sir William Stubbs) I think that is absolutely right,
Chairman. There the quinquennial report was echoing the comments
in the report that was published in January by the international
panel that looked into A-level standards and confirmed that QCA
was doing as good a job as could be expected of it but it should
do more to educate people about the system. I absolutely agree
with that, I think that is one of the big tasks. To some extent
there was evidence of success in that because when the results
came out in August and there was a significant increase in the
overall pass rate as well as the grade As, I think the amount
of carping that took place in the press this year, if colleagues
will forgive the expression, was less than had happened in previous
years. There was more of an element of celebration about it. Students
had worked hard and done well, I think we need to build on that.
So yes, I do agree that more work needs to be done on that, but
it is a complex matter to explain.
333. One of the other recommendations is around
the relationship between QCA and the examining bodies and saying
that both QCA and the DfES should actually look at the issue of
greater quality assurance of awarding bodies and less involvement
in the details of individual qualifications. Do you think that
would be a helpful way forward?
(Sir William Stubbs) It depends on where you see the
boundary. I think if I were sitting here now in the context of
QCA and it had not been involved in some of the detail of the
awarding bodies you would be highly critical of QCA and say, "Look,
you should be much closer to the action." I think what the
Review were saying was you could validate the awarding body and
give it a three-year licence and then it gets on and does its
task. There may be a place for that in some respects in some qualifications,
but for the high stake qualifications I think the QCA as the regulator
has got to be fairly well-informed and closer to the three principal
exam bodies.
Jeff Ennis
334. When Dr McLone, the Chief Executive of
OCR, gave evidence to the Committee he said we needed to make
the exam system "more transparent" and also "to
bring it into the 21st century". Do you agree with that statement
and how can we achieve that if you do?
(Sir William Stubbs) The transparency goes back to
the earlier question, that we need to explain it more. This year,
as a result of the crisis of the nature that I described, there
has been more independent observation of the grade boundary setting
by the awarding bodies. It is not done within a closed room. I
think that is absolutely healthy and I think one could build on
that. So to that extent, I believe that we need to do more. At
the end of the day, however, one has to see that for thousands
of young people and for hundreds of teachers they are having to
cope with partial success. Young people have put themselves forward
in a demanding situation and some have got higher grades than
others and indeed, sadly, some, but not many, fail entirely. They
would all like to see themselves doing better but the system is
designed to have rigorous standards, and some do not meet them.
So there is always going to be an element of disappointment around
but, yes, I think we could do more on that. But at the end of
the day judgments still have to be made because this is about
personal judgments, we are not dealing with a mechanised system,
and there could be mistakes there. On your question, and it came
up in your meeting the first time, I think "cottage industry"
was the phrase
Chairman
335."Victorian cottage industry".
(Sir William Stubbs)Victorian cottage industry!
We have not got a system of computerised examination as exists
in some colleges and universities in the United States, which
largely comprise multiple choice questions and which are computer
marked. I cannot see that the A-level system would fulfil its
expectations if it went down that route. It is going to rely on
individual judgments to a significant extent, but it is possible
through the development of new software to see how in five years'
time there could be a greater contribution from IT in the mechanics
and logistics of handling the process. By the way, someone in
the last meeting said all QCA was doing was behaving like Consignia.
I took that as a bit of an insult because the number of first-class
letters that get lost every day is quite high! You can through
the use of IT scan and transmit the papers to markers quickly
and indeed to selected markers on selected subjects and then bring
them together and aggregate them. That needs money and indeed
that was one of the reasons why Edexcel considered earlier on
in the year they might have to give up A-levels. I am not sure
whether this is the place to disclose it but I did speak to the
Secretary of State about that and said that I thought the Department
should invest significant sums of money running into 10 of millions
in order to assist the awarding bodies develop computerised systems.
Without that investment I think it is quite unrealistic to think
that they could do it themselves.
Jeff Ennis
336. You mentioned in your earlier remarks that
elements of the trust within the exam system have been weakened
over the last few months. This is to some extent echoed in the
submission from the Secondary Heads Association to the Tomlinson
inquiry when in one of their recommendations they says "SHA
recommends that the government should place greater trust in the
professionalism of teachers and thus recommends that internal
summative assessment should play a greater part in the examination
system." Do you agree with that?
(Sir William Stubbs) I am not sure there is quite
a yes or no to that. Yes, in parts. SHA has for some time been
proposing the idea that teachers should be eligible to become
certificated examiners or markers, and I support that, I think
that would be a very sensible development. In my first meeting
with David Miliband when he became the Minister responsible for
qualification and examinations I said to him I did not think this
system could be sustained over the next five years without increasing
the risk of significant failure, by that I meant not just A-levels,
I meant GCSEs, key stage tests, advanced extension awards and
the whole gamut. By the way, I heard the bit about ISB and that
would require more markers there. That is just another world if
we went down there. So I think a way has to be found to recognise
the professionalism of teachers and give them a greater place.
In Australia they find it possible to do that and have an external
check on the teacher' judgments, so there is not too much of a
halo effect in the school about the individual students. If we
are going to retain that same profile of examinations, Chairman,
we will have to do something about that, so to that extent I agree
with what is being said.
337. One final question, in your earlier remarks
you mentioned that there has only been one examination board,
the OCR, that has had major problems with the transition to the
new system. How confident are you that they will overcome these
problems next year?
(Sir William Stubbs) Just for the record, I said five
weeks ago there was only one awarding body with a problem. You
have found that that is the case. In other words, it is not just
me saying it now, the evidence has said it. It is only in OCR
and only in a minority of subjects. Do I think next year we are
going to have the same problems in the system? No, I do not take
quite such a pessimistic view at all. I think we will now have,
as HMC was saying this morning, real exam papers and real scripts
there to guide the teachers, guide the awarding bodies, guide
the markers and so forth. There is a greater understanding about
what is expected and some of the uncertainties surrounding course
work, which by the way Chairman, was the big crisis five weeks
ago. Where it is now I ask you. It is not there, although further
work needs to be done involving the people sitting behind me on
a greater understanding of what is expected about course work.
I think they can do that and I think they will be engaged in discussions
with QCA about how to bring that about. So the only problem facing
us not so much in January but certainly in the summer next year
(because the scale is so much bigger in the summer) is whether
they can get enough markers. Being a marker now is quite a demanding
task, Chairman, because your work can be discovered. Students
get scripts back and their parents and teachers can see it and
if there is a mistake they can, quite rightly, challenge it. It
is something that is truly a very professional task. When there
is all this confidence crisis around I think Ken Boston was right
to say to you there might be some doubt as to whether they can
get the markers. I know that some people say extra pay could help.
Maybe extra pay would help but that was tried in another place
a few years ago and there was not a quantum leap in the number
coming forward. I think as part of the professional development
of teachers, if they see it as something they do in order to understand
their subject and the learning process better, then there is a
way forward.
Chairman
338. You are being rather kind about Dr McLone,
saying that he made a bad call in just a small number of subjects,
but he described in our session the whole exam system as flawed.
Everything you have said to us this morning runs counter to that.
What would you say to him?
(Sir William Stubbs) I would say first of all I have
read the evidence from last week, I did not hear it all this morning
but I heard a bit this morning and, as far as I know, he is the
only person to come before you and say the system is flawed. No
one else has said that and he is only saying it is flawed because
of this notorious 40/60 50/50 split and you had a long and rather
complex discussion about that at your last meeting. That decision
was made a few years ago, I do not think it is going to be re-visited,
I do not think it should be re-visited, and we move forward. I
disagree with him. I think the system is now sound and we should
not change it. Lord help me, the amount of training of teachers
and the amount of new understanding by markers and examiners,
the new expectations to which young people would have to adjust
if the system were changed markedly are beyond comprehension.
This system needs to be allowed to settle down. I predict quite
confidently in a year or two years' time that we will be seeing
great strengths from it. One of the great strengths of it is the
anchor point of AS. It has proved to attract more young people
to continue their studies into the sixth form than many of us
thought was possible and it is showing encouraging but not convincing
signs of encouraging some young people to broaden their studies
in the lower sixth.
Mr Baron
339. Can I return to the line of questioning
I pursued earlier with what many of us see is a question of standards
versus statistics. To use a very brief analogy, when I was a platoon
commander in Germany before the Wall came down we were always
told that quality will outdo quantity any time, to which we retorted
under our breaths that quantity has a quality all of its own.
Has this not happened here in the sense that, in the absence of
standards, statistics became the standard because guidance was
given that certain statistics had to be met and that is what is
at the core of the whole problem?
(Sir William Stubbs) Neither of those statements is
true. You said standards did not exist; yes they did. No one has
said that there were no standards. Of course there were standards.
We would all have liked them to be clearer. We are using the statistics
of this year to try and make them clearer. That is the first statement
that is not true and the second statement that is not true is
that statistical information from last year had to be applied
rigorously. That is not true at all. So both of those statements
are invalid. What we have been mandated by the Governmentand
I gave you the chronology of itis "the A-level standard
is here to stay under this Government and you must make sure as
the regulator that that applies. That means you cannot ignore
previous years and the achievement of previous years". Statistical
information from previous years, I concede to you to some extent,
must come into play. Indeed, I reminded the awarding bodies that
there should be no grade drift or benign changes of the marking
system that were not justified in the actual achievement of candidates.
Those letters have been held up to scrutiny now and I am pleased
to say that the Chairman of the Joint Council said those were
perfectly reasonable letters and it was a perfectly proper view
for the regulator to take. Indeed, Mike Tomlinson himself said
that. Yes, there were always going to be difficulties in moving
to a complex new system but we very nearly got it completely right.
If it had not been for a small number of miscalls, I think you
would be exploring another topic this autumn.
|