Select Committee on Education and Skills Minutes of Evidence


Examination of Witness (Questions 327-339)

WEDNESDAY 6 NOVEMBER 2002

SIR WILLIAM STUBBS

Chairman

  327. Sir William, welcome back to the Committee. There has been a lot of water under the bridge since we last met in May when we had a very good session as I recall. Can I first of all not only welcome you but also say that this session is about learning and about how we make our system work better rather than worse, to learn some of the lessons from past experience and see how we move positively into the future. We do not really want to trawl over particular things of personal concern to you, and everyone here will know what I mean by that. We want to learn about how we get that relationship between the QCA and the examining boards and between the Department and those organisations on a better footing. I know from our previous session in May that you had some pretty clear ideas about that then. You will maybe have seen some of the evidence that was given to this Committee by the examining boards last week. I am not sure you were in the room when I asked my opening question to the people who have just given evidence, but what I was asking them was, in what ways can you move to learn from that past? Everyone knows that when a new exam comes in there is going to be a certain amount of transition difficulty and some might say that perhaps we have had less than we could have had, but the Chief Executive of the QCA last week was very pessimistic about having more problems in the coming year, which rather concerned the Committee. Given the events of this transition, I wonder whether you have any particular recommendations for the Committee on how we could improve the system?
  (Sir William Stubbs) Thank you, Chairman, and thank you for that clarification, although I would just like for the record to say, as we are in a new session and a new topic under discussion, that I was Chairman of the QCA for five years. Throughout the entire period I was part-time and for four of the five years I was unpaid. I only was paid from the time I retired from my full time employment about a year ago and I was asked to increase my involvement from two days a week to four days a week by Estelle Morris and, looking back at the letter, she said she wanted me to provide strong leadership. That is where I come from in all this. What I would like to do at the beginning, and the Clerk very kindly sent me a copy of the transcript of the interviews with the Awarding Bodies and the QCA, is to say something about the A-level crisis, a term which has been used both in the Committee and further afield. It seems to me the word "crisis" in relation to A-levels has two possible meanings: either the operation of the awarding system is so significantly defective as to give good grounds for concluding that the main outcomes are invalid, or, alternatively, confidence in the validity of the system has been so diminished that there is widespread anxiety among students, parents, universities and employers. The first meaning is clearly inappropriate on the basis of the evidence that was available five weeks ago and the evidence that has been uncovered since then. The system is sound and indeed some of the evidence I heard this morning confirms that. But a national exam system as complex as the one that we now have available relies significantly on trust in the overall process, trust in the markers and examiners, trust in the integrity of the exam boards, and trust in the integrity and independence of the regulator. Therefore at the outset, Chairman, I have to say that in recent weeks each of these elements of trust has been significantly and quite unnecessarily weakened. Therefore the challenge for those responsible for those matters in the future will be to restore that trust, but they do so on the basis that the underlying system is sound, and that is an enormous strength. So that is where we are coming from because I think it is important when the word "crisis" is used, it is a crisis of confidence rather than anything else. At some stage, and I know you said you do not want to go into too much of the past, but one cannot understand the future without the past and I think in some of these discussions there will be something—

  328. The Chairman was merely trying to be reasonably sensitive about these things.
  (Sir William Stubbs) I know and I appreciate it.

  329. Feel free to cover any subject you wish.
  (Sir William Stubbs) At some stage I would like to talk about the maintenance of standards in A-level but not necessarily in this opening statement. All I wanted to do at the beginning was say let's conduct a discussion on the basis of terms that we understand and that is the way I understand "crisis", and I think that is the probably the way you understand it in the light of what has happened over the past few weeks.

  Chairman: At the beginning of the summer in the early days of the so-called turmoil both you and I appeared on the same programme saying there was not a crisis and dampening down the suggestion.

  (Sir William Stubbs) Absolutely, Chairman.

  330. However, let's move on. One thing that came through from the evidence this morning was that one failing of the QCA in the minds of those people who are the consumers, in a sense—the colleges and schools—was this inability to set parity of standards across the piece. It seemed to be a pretty valid criticism that QCA did not really do that. What would you say to that criticism?
  (Sir William Stubbs) I have got a little bit in reply, Chairman. I think the chronology starts from April 1998 when the then Minister responsible for qualifications, Baroness Blackstone, in agreeing the new system and saying this was the Government's policy, in 1998 said: "We are determined to ensure that A-level standards are safeguarded and that all students study to rigorous standards." From the outset at the time of the change continuity of A-level standards was absolutely in the Government's thinking. A year later in March 1999 a letter was sent to all schools from the department "no compromise on A-level standards". In August 1999 David Blunkett speaking as Secretary of State said, "I can assure you that there will be no reduction in A-level standards under this Government." In April 1999 a DfES official: "Ministers place the standard of the A-level examinations as the priority." There is absolutely no doubt where the Government was coming from. In May 1999, as we started to develop the intricate arrangements for the examinations, HMC wrote to the Minister responsible and said: "It would appear that the awarding bodies are contemplating various statistical treatments to ensure that the first set of A-level results for the new system will be very similar in outcome to the current percentage gaining each grade. We would maintain there should be a small but definite increase in the numbers passing and gaining higher grades under the new system." As an aside, the outcome last year was a 4.5% increase in the pass rate and a 2.1% increase in grade As. I would put it to you that that is exactly what HMC, GSA and SHA were asking in May 1999 of the Minister. On the basis of that, in June 1999—and this seems to me absolutely significant—the QCA then published a statement on standards which was subsequently on their web site—"broadly speaking, the proportion of grades awarded in the current A-levels and those awarded to candidates completing the new A-levels will be expected to be similar. Where, however, on the basis of the quality of candidates' performance and changes in the nature of the candidature"—and as we know it did change—"a more substantial change in proportions is justified, this will be acceptable, provided the reasons for the change are fully justified and the standard of the full A-level is maintained." There was correspondence taking place at that time between HMC following an exchange with Tessa Blackstone whom I referred to and Nick Tate who was the Chief Executive of QCA, and they wrote to David Hargreaves, who had by then become the Chief Executive and this is what HMC said: "We cannot accept the lack of action over proper definition of grade boundaries for the new awards. Standards must be defined and some anchoring device must be established. Whilst it is good that the awarding bodies will provide examiners with a comprehensive package of statistical information, we would very much wish to know whether they are going as far as to establish grade boundaries. We would suggest it ought to be possible to use historic data on regression lines to ensure that the various boundaries will map on to a predicted grade boundary on a completion of A2." That is HMC. The reply they received from David Hargreaves—and this is my last statement on this chronology, Chairman—leaves absolutely no doubt on the record: "We are not clear why you suggest there has been a lack of action over a proper definition of grade boundaries in the new awards. A vast amount of work has taken place throughout the development of the new specifications, sample assessment materials and detailed statistical modelling of the new awards. The Joint Council is involved in an extensive programme of research to ensure that when the first awarding bodies meetings take place next year the examiners are provided with the most comprehensive set of statistical data that will ever have been used in our public examination system. Historic data on regression lines between GCSE and A-level are central to the work that has taken place and the mapping you describe has been going on for many months. You say that Nick Tate's statement `the establishment of standards in any qualification is complex and the prediction of grade profiles cannot be precise' is unacceptable. No examination system which provides for an element of examiner judgment and a changing cohort would allow the precise prediction of grade profiles. This would be possible only with a completely non-reference system. You may be arguing for such an approach but that would represent a fundamental change in the way qualifications are awarded and a step away from equitable treatment of candidates over time." That, Chairman, effectively ended the correspondence on standards between the heads associations and the QCA. There have been since then, I am told, and I was not involved, something between 30 and 40 technical meetings to flesh out the arrangements. So I am in no doubt from the record that there was a clear understanding of standards, recognising that we did not have past papers. Standards are not like the metre where one could in the 18th century go and hold something against it. It is not like that; it is a combination of judgments made every year against criteria, against specifications and against the evidence of previous performance. I believe it would have helped to have had a run of exemplar A2 examinations beforehand, in other words pilots. That was just not possible in the time available and, indeed, would have been very complex because to be good pilots they would probably have had to have taken place after the AS examinations and you would have had to draw on the AS experience so you would then have an interregnum. I am not sure exactly how one could have run those terribly smoothly. We did not have that. As far as the standards were concerned, recognising we did not have past papers, there was a comprehensive understanding and indeed—and this is what I find utterly baffling—the results of two of the awarding bodies, having been held up to the daylight more than once and scrutinised, have come through with flying colours in judgments that I find must be exceedingly difficult for the chief executives to make. I think they have done a splendid job and we should be congratulating them. Edexcel came from its knees. When I last saw you Edexcel was in intensive care and indeed the board of Edexcel had decided as a matter of policy that it wanted to abandon A-levels and cease to award it as an awarding body and was in the course of discussion in the spring on selling that off to a private company. Yet through the valiant efforts of officials in Edexcel and colleagues in QCA, they came through in the summer and produced an unflawed system. I have said this to the Secretary of State not once but twice that I believe, like you, there is no evidence of widespread failing. There is evidence of shortcomings in one awarding body but even there in only part of the judgments made by that awarding body. We now know that the chief executive of OCR miscalled it 16 or 18 times out of the several hundreds of judgments he had to make and he made a mistake. When I say made a mistake, when fellow professionals are called in and asked to look at it, they took a different view. I do not think there is a walk of professional life where, when a professional judgment I've taken, whether it be law, medicine or whatever, and held up to scrutiny by an independent second opinion that you will be guaranteed you will get them all confirmed as the view of the first opinion. In this case his judgment was found to be wanting, but it was confined to a relatively small number. What has caused the worry for not just tens of thousands but hundreds of thousands is they thought their certificates from other awarding bodies and from the unflawed part of OCR were invalid, and for that there is not a shred of evidence. I believe it is a scandal; it should never have happened. On standards therefore—and that is where we started—I am saying there was evidence there on the QCA web site and there have been plenty of technical meetings, but there is nothing on the record over the last two years from bodies that I have seen about it, although they all recognise that this was a difficult transitional year and I think in the main they have done well.

  Chairman: Sir William, that has been a most helpful chronology and explanation to the Committee. Now we will begin the questioning. Meg Munn?

Ms Munn

  331. When you came to see us in May you were very confident that the quinquennial review would be a very positive one and that it would say basically that you were doing a good job. Are you satisfied with what was in the quinquennial review and the conclusions that they came to? Do you think they were fair?
  (Sir William Stubbs) How does one say one is satisfied? If we were graded, it was beta plus or alpha minus or something like that; it was a good report. Indeed, one of the reasons last time why there was an interregnum about the chief executive, whom I hope you will find a very good colleague to work with in the future was that we wanted to wait in the making of that appointment until that quinquennial was out the road. I think it confirmed in an area where 99% accuracy is not acceptable that the QCA, in the observation of most correspondents to that inquiry, is doing a good job. What it has got, though, and I heard from the heads just 10 minutes ago and I saw in the evidence from Ken Boston, is an accretion of tasks that are not central to its purpose but were given to QCA because there was no other body in town that the government could trust to do it, and that is of course running key stage tests. They are a huge exercise, they are politically highly significant, with great involvement by DfES officials (in my view too great an involvement) and a way has to be found to deal with that and to distance it from QCA. You could give it to the awarding bodies but I think that would be unfair because it is different to their main tasks, but a way has to be found to get some kind of clear water between QCA and the key stage tests.

  332. One of the recommendations in the report is that QCA should strengthen its capacity for intelligence gathering as regards standards and then adopt a more visible and authoritative public stance. I think this goes perhaps to the heart of the matter you were just discussing, where in terms of creating confidence in the examination system, in terms of trying to get past this situation every summer where we have this "Are standards dropping?" what this is suggesting is that QCA itself could play a much more important role in creating that public confidence. Do you agree with that?
  (Sir William Stubbs) I think that is absolutely right, Chairman. There the quinquennial report was echoing the comments in the report that was published in January by the international panel that looked into A-level standards and confirmed that QCA was doing as good a job as could be expected of it but it should do more to educate people about the system. I absolutely agree with that, I think that is one of the big tasks. To some extent there was evidence of success in that because when the results came out in August and there was a significant increase in the overall pass rate as well as the grade As, I think the amount of carping that took place in the press this year, if colleagues will forgive the expression, was less than had happened in previous years. There was more of an element of celebration about it. Students had worked hard and done well, I think we need to build on that. So yes, I do agree that more work needs to be done on that, but it is a complex matter to explain.

  333. One of the other recommendations is around the relationship between QCA and the examining bodies and saying that both QCA and the DfES should actually look at the issue of greater quality assurance of awarding bodies and less involvement in the details of individual qualifications. Do you think that would be a helpful way forward?
  (Sir William Stubbs) It depends on where you see the boundary. I think if I were sitting here now in the context of QCA and it had not been involved in some of the detail of the awarding bodies you would be highly critical of QCA and say, "Look, you should be much closer to the action." I think what the Review were saying was you could validate the awarding body and give it a three-year licence and then it gets on and does its task. There may be a place for that in some respects in some qualifications, but for the high stake qualifications I think the QCA as the regulator has got to be fairly well-informed and closer to the three principal exam bodies.

Jeff Ennis

  334. When Dr McLone, the Chief Executive of OCR, gave evidence to the Committee he said we needed to make the exam system "more transparent" and also "to bring it into the 21st century". Do you agree with that statement and how can we achieve that if you do?
  (Sir William Stubbs) The transparency goes back to the earlier question, that we need to explain it more. This year, as a result of the crisis of the nature that I described, there has been more independent observation of the grade boundary setting by the awarding bodies. It is not done within a closed room. I think that is absolutely healthy and I think one could build on that. So to that extent, I believe that we need to do more. At the end of the day, however, one has to see that for thousands of young people and for hundreds of teachers they are having to cope with partial success. Young people have put themselves forward in a demanding situation and some have got higher grades than others and indeed, sadly, some, but not many, fail entirely. They would all like to see themselves doing better but the system is designed to have rigorous standards, and some do not meet them. So there is always going to be an element of disappointment around but, yes, I think we could do more on that. But at the end of the day judgments still have to be made because this is about personal judgments, we are not dealing with a mechanised system, and there could be mistakes there. On your question, and it came up in your meeting the first time, I think "cottage industry" was the phrase—

Chairman

  335.—"Victorian cottage industry".
  (Sir William Stubbs)—Victorian cottage industry! We have not got a system of computerised examination as exists in some colleges and universities in the United States, which largely comprise multiple choice questions and which are computer marked. I cannot see that the A-level system would fulfil its expectations if it went down that route. It is going to rely on individual judgments to a significant extent, but it is possible through the development of new software to see how in five years' time there could be a greater contribution from IT in the mechanics and logistics of handling the process. By the way, someone in the last meeting said all QCA was doing was behaving like Consignia. I took that as a bit of an insult because the number of first-class letters that get lost every day is quite high! You can through the use of IT scan and transmit the papers to markers quickly and indeed to selected markers on selected subjects and then bring them together and aggregate them. That needs money and indeed that was one of the reasons why Edexcel considered earlier on in the year they might have to give up A-levels. I am not sure whether this is the place to disclose it but I did speak to the Secretary of State about that and said that I thought the Department should invest significant sums of money running into 10 of millions in order to assist the awarding bodies develop computerised systems. Without that investment I think it is quite unrealistic to think that they could do it themselves.

Jeff Ennis

  336. You mentioned in your earlier remarks that elements of the trust within the exam system have been weakened over the last few months. This is to some extent echoed in the submission from the Secondary Heads Association to the Tomlinson inquiry when in one of their recommendations they says "SHA recommends that the government should place greater trust in the professionalism of teachers and thus recommends that internal summative assessment should play a greater part in the examination system." Do you agree with that?
  (Sir William Stubbs) I am not sure there is quite a yes or no to that. Yes, in parts. SHA has for some time been proposing the idea that teachers should be eligible to become certificated examiners or markers, and I support that, I think that would be a very sensible development. In my first meeting with David Miliband when he became the Minister responsible for qualification and examinations I said to him I did not think this system could be sustained over the next five years without increasing the risk of significant failure, by that I meant not just A-levels, I meant GCSEs, key stage tests, advanced extension awards and the whole gamut. By the way, I heard the bit about ISB and that would require more markers there. That is just another world if we went down there. So I think a way has to be found to recognise the professionalism of teachers and give them a greater place. In Australia they find it possible to do that and have an external check on the teacher' judgments, so there is not too much of a halo effect in the school about the individual students. If we are going to retain that same profile of examinations, Chairman, we will have to do something about that, so to that extent I agree with what is being said.

  337. One final question, in your earlier remarks you mentioned that there has only been one examination board, the OCR, that has had major problems with the transition to the new system. How confident are you that they will overcome these problems next year?
  (Sir William Stubbs) Just for the record, I said five weeks ago there was only one awarding body with a problem. You have found that that is the case. In other words, it is not just me saying it now, the evidence has said it. It is only in OCR and only in a minority of subjects. Do I think next year we are going to have the same problems in the system? No, I do not take quite such a pessimistic view at all. I think we will now have, as HMC was saying this morning, real exam papers and real scripts there to guide the teachers, guide the awarding bodies, guide the markers and so forth. There is a greater understanding about what is expected and some of the uncertainties surrounding course work, which by the way Chairman, was the big crisis five weeks ago. Where it is now I ask you. It is not there, although further work needs to be done involving the people sitting behind me on a greater understanding of what is expected about course work. I think they can do that and I think they will be engaged in discussions with QCA about how to bring that about. So the only problem facing us not so much in January but certainly in the summer next year (because the scale is so much bigger in the summer) is whether they can get enough markers. Being a marker now is quite a demanding task, Chairman, because your work can be discovered. Students get scripts back and their parents and teachers can see it and if there is a mistake they can, quite rightly, challenge it. It is something that is truly a very professional task. When there is all this confidence crisis around I think Ken Boston was right to say to you there might be some doubt as to whether they can get the markers. I know that some people say extra pay could help. Maybe extra pay would help but that was tried in another place a few years ago and there was not a quantum leap in the number coming forward. I think as part of the professional development of teachers, if they see it as something they do in order to understand their subject and the learning process better, then there is a way forward.

Chairman

  338. You are being rather kind about Dr McLone, saying that he made a bad call in just a small number of subjects, but he described in our session the whole exam system as flawed. Everything you have said to us this morning runs counter to that. What would you say to him?
  (Sir William Stubbs) I would say first of all I have read the evidence from last week, I did not hear it all this morning but I heard a bit this morning and, as far as I know, he is the only person to come before you and say the system is flawed. No one else has said that and he is only saying it is flawed because of this notorious 40/60 50/50 split and you had a long and rather complex discussion about that at your last meeting. That decision was made a few years ago, I do not think it is going to be re-visited, I do not think it should be re-visited, and we move forward. I disagree with him. I think the system is now sound and we should not change it. Lord help me, the amount of training of teachers and the amount of new understanding by markers and examiners, the new expectations to which young people would have to adjust if the system were changed markedly are beyond comprehension. This system needs to be allowed to settle down. I predict quite confidently in a year or two years' time that we will be seeing great strengths from it. One of the great strengths of it is the anchor point of AS. It has proved to attract more young people to continue their studies into the sixth form than many of us thought was possible and it is showing encouraging but not convincing signs of encouraging some young people to broaden their studies in the lower sixth.

Mr Baron

  339. Can I return to the line of questioning I pursued earlier with what many of us see is a question of standards versus statistics. To use a very brief analogy, when I was a platoon commander in Germany before the Wall came down we were always told that quality will outdo quantity any time, to which we retorted under our breaths that quantity has a quality all of its own. Has this not happened here in the sense that, in the absence of standards, statistics became the standard because guidance was given that certain statistics had to be met and that is what is at the core of the whole problem?
  (Sir William Stubbs) Neither of those statements is true. You said standards did not exist; yes they did. No one has said that there were no standards. Of course there were standards. We would all have liked them to be clearer. We are using the statistics of this year to try and make them clearer. That is the first statement that is not true and the second statement that is not true is that statistical information from last year had to be applied rigorously. That is not true at all. So both of those statements are invalid. What we have been mandated by the Government—and I gave you the chronology of it—is "the A-level standard is here to stay under this Government and you must make sure as the regulator that that applies. That means you cannot ignore previous years and the achievement of previous years". Statistical information from previous years, I concede to you to some extent, must come into play. Indeed, I reminded the awarding bodies that there should be no grade drift or benign changes of the marking system that were not justified in the actual achievement of candidates. Those letters have been held up to scrutiny now and I am pleased to say that the Chairman of the Joint Council said those were perfectly reasonable letters and it was a perfectly proper view for the regulator to take. Indeed, Mike Tomlinson himself said that. Yes, there were always going to be difficulties in moving to a complex new system but we very nearly got it completely right. If it had not been for a small number of miscalls, I think you would be exploring another topic this autumn.


 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2003
Prepared 14 April 2003