House of Commons - Education and Skills

Select Committee on Education and Skills Third Report

ANALYSIS OF THE ISSUES

33. In August 2001, QCA published Managing Curriculum 2000 for 16-19 students. This guidance was sent to schools, colleges, LEAs and Connexions services. It clearly states that A2 should be more demanding than the overall A level standard.[38]

34. The Tomlinson inquiry found that there was "no clear, consistent view among awarding body officials and many examiners and teachers about the standard required at AS and A2 unit levels in order to ensure that the overall GCE A level standard is maintained."[39] More worryingly, the report notes that these concerns were highlighted, in relation to the standard of the AS, in the review of Curriculum 2000 carried out by Professor David Hargreaves, then Chief Executive of the QCA, in 2001, months before the A2 examination problems.[40] Mr Tomlinson reported that different interpretations of the AS and A2 standard existed.[41]

35. Tomlinson's interim report concluded that "the lack of a common understanding of the standard associated with AS and A2 units, along with the challenges associated with aggregation of the units, given all had equal weighting, played a significant part in the problems experienced by the three examination boards during the grading this year".[42]

HOW THE SCHOOLS AND COLLEGES COPED

36. Mr Hopkins said that his college had worked extremely hard to implement the new curriculum and to understand the standard required. He said "we did a huge amount of training. We kept in constant dialogue with the boards. One of the advantages of Hampshire is that we have ten large sixthform colleges and we got together and we put on our own training, we encouraged our staff to become examiners. Every one of those colleges had an examiner in some subjects somewhere and we got togther and trained each other."[43]

37. Mr Neal argued that his school had dedicated considerable time and effort in providing training for staff teaching the new curriculum: "All the training took place and all the teachers were involved in that. The teachers moved heaven and earth to make the system work, but throughout that period the contradictory messages were coming back about standards. There was a lack of exemplar material, so it actually was quite difficult for teachers to have a clear understanding of what the standards were that were being aimed for, of what the assessments were going to look like. That was a genuine difficulty throughout AS and A2."[44]

38. Mr Hopkins emphasised the advantages of working with local colleges to combine training resources and share expertise: "We got together, we worked together and collaborated. We made sure that we had examiners in the boards from each of the colleges and we found things out. It was not spoon fed to us, I have to say."[45]

39. Mr Gould was frustrated that OCR appeared to have "set their own standard with A level minus one for AS level and A level plus one for A2. Nowhere is that in the code of practice, nowhere is that standard defined, nowhere has that standard been relayed to schools, teachers or examiners beforehand."[46]

The challenge that faced the examination boards

40. The A level results released in August 2002 were the first A levels taken under the new curriculum. The major awarding bodies, AQA, Edexcel and OCR, were faced with the task of marking these new examination papers.

41. The marking of A levels has become increasingly complex in recent years. The introduction of the compulsory AS level in 2000 effectively doubled the amount of work undertaken by the English awarding bodies, mainly AQA, Edexcel and OCR. [The bodies have frequently voiced concerns that they were unable to recruit sufficient numbers of staff.] The increase in work has put a strain on the examination boards. Edexcel was heavily criticised in January 2002 for errors concerning examination scripts. The then Secretary of State asked the QCA to investigate, and the QCA's Director of Quality Audit was placed inside Edexcel to monitor and prompt rapid action by its management. The supervision of Edexcel had been agreed earlier but was brought forward to 22 January 2002 following the exam boards' problems. Edexcel has since been given a clean bill of health and was not at the centre of concerns last summer.

42. Mr John Kerr, Chief Executive of Edexcel, told us that "The exam system is still essentially Victorian, it is a large number of pieces of paper; in our own exam board, it is ten million marks, five million pieces of paper, scripts, in a large warehouse, there is little technology that has been applied there".[47] He warned that without investment the examination boards would continue to make "errors and mistakes, which clearly we will strive to minimise, and it is important that we do so, but there are limited reserves within the exam boards, as charitable organisations".[48]

43. Dr Boston, the Chief Executive of QCA, highlighted the problems which continue to pervade the examination system: "It is the shortage of examiners, and I think that is going to be exacerbated this year by many people not wishing to examine again, or perhaps examine for one board again; the sheer volume of the assessment that occurs across the country. I do believe examinations here are probably the most excessive in the world for young people, and that we could get equally valid measurements of student performance and progress with less examination. The reliance so strongly on external examinations, rather than some component of it, at least, being internally examined. The notion of having internal assessments externally moderated, which the Secondary Heads Association is advancing in the form of chartered examiners, is, in fact, the norm for many examinations in many western countries and produces valid results.... The technology that we use is very simple, and it was the subject of some comment in the report, Maintaining A level Standards, that Eva Baker chaired earlier this year. Our scripts are all marked by single markers, no script is marked by two markers; the scripts from centres move by post to a marker's home, usually, we do not use, although we have trialed, as a general rule, marking centres, where markers are brought in to mark under supervision, and one marks questions 5a and 5b, and another marks questions 6a and 6b, and you get consistency that way. Very little application of technology. We are running here a 21st century education system on a huge cottage industry, in the marking process, and it is just going to fail, unless we move to change the way that operates. Now that cannot be done for the summer examinations next year, we do not have that capacity to move that quickly; but that is the longerterm issue, we have got to get the examination system logistically and technically on a much firmer basis."[49]

THE AUGUST CHALLENGE

44. In Summer 2002, the awarding bodies faced a particularly complicated challenge. A level course material had been split in two, with the less demanding content in AS and the more demanding content in A2. There had been considerable discussion and confusion regarding the level of attainment required at each level. A2 examination papers often used a more challenging style of question. Despite the difference in difficulty, AS and A2 examinations were each worth 50% of the final mark.

45. In previous years, examiners were shown example scripts from former examinations, which highlighted the preceding years' standards and associated grade boundaries, at awarding meetings to illustrate expected levels of performance. In 2002, the A2 was a new examination which had not been piloted, and consequently exemplar scripts of A2 examinations were not available.

How the Awarding Bodies tried to define the standard

46. Mike Tomlinson was clear that "nothing that was done this summer was outside of the Code of Practice and the frameworks which govern that".[50] However, there was a substantial debate on standards following the publication of the A level results, and at the heart of this debate was confusion about the differing approaches of the awarding bodies AQA, Edexcel and OCR in awarding grades.

47. The Tomlinson inquiry initially focused on the grading decisions made by the awarding bodies. Although questions were raised about the number of subjects requiring verification he decided, in consultation with the awarding bodies, that a very small number of papers would be re-graded. There was considerable concern that the bodies had altered grades inconsistently with the expected A level standard, and had acted in an attempt to limit artificially the increase in the number of A level passes in August 2002.

48. We discussed with many of our witnesses the difficulties the awarding bodies experienced with grading. We established that the awarding bodies had taken divergent views on standards and were surprised to note that, despite those divergent views, Mike Tomlinson stated that none of the bodies had acted outside the Code of Practice. Indeed Dr Boston, Chief Executive of the QCA, subsequently acknowledged that "the key to that lies in the revision of the Code of Practice.... which should remove the capacity for different approaches in that way."[51]

THE JOINT COUNCIL FOR GENERAL QUALIFICATIONS

49. Ms Kathleen Tattersall, the Chair of the Joint Council for General Qualifications [JCGQ] and Director-General of the AQA, told us of the work undertaken by the awarding bodies: "we met over the period of the four years, or so, leading up to the new A levels, on several occasions, there is the Joint Council for General Qualifications, that is the forum in which we meet together, and also with QCA, to try to establish all those difficult technical issues which have to be resolved when the new qualification comes into being. And this was a qualification which was quite different from the qualification that went before it; here we have a qualification made up of two parts, the AS examination and the A2 examination, AS being a qualification in its own right, and A2 being the second half that makes up the A level. I believe we worked as best we could to try to establish those standards, and it is only really in retrospect that some of these problems now begin to emerge, which at the time were not seen as real issues."[52]

50. Ms Tattersall told us that the JCGQ had had several meetings with Sir William Stubbs, the then Chairman of the QCA, to clarify the standard of the A2 examinations. She told us that "awarding bodies are charged with the maintenance of standards, year on year, ...and the issue is, how do you measure those standards". She said that "one measure... is the percentage of candidates who receive a given award in a given year, the outcomes, I will call them, and the discussion that we had on 12 March focused very much on the outcomes, the expectation being that in 2002 they would be very similar to 2001".[53] Ms Tattersall told us that this "worried many of us, because, clearly, many will see the outcomes only as indicators, not as real examples of standards, and the issue is what are the standards". Ms Tattersall then wrote to Sir William "really setting out our position, as awarding bodies, that we judged the standards from the evidence, and the prime evidence is the candidates' work, and the subsidiary evidence is the statistical information".[54] Sir William confirmed in writing that he concurred with that view. He told us that "the Chairman of the Joint Council said she was quite satisfied with the letters that she had got clarifying it in April and she thought as far as the meeting in July was concerned there was no pressure put on to go to any artificial targets and that has been echoed, indeed Tomlinson found that [was the case]".[55] Ms Tattersall believed that the correspondence had dealt with the awarding bodies' concerns.

51. Ms Tattersall told us of a further meeting, on 26 July, called as the examination boards had recognised that "the pattern of the outcomes was going to be very different in 2002 than it had been in 2001. What I was anxious to ascertain was whether this was something peculiar to AQA, or whether it was something which my fellow chief executives were also experiencing in their awards; and so we called a meeting of the boards, we ascertained that we were all experiencing the same sort of pattern of results, and we identified the reasons for that pattern of results." She said that the awarding bodies had concluded that "one of the major reasons being that there is a big dropout rate between the old AS and the full A level, people who had performed to the best of their ability at the AS level and then not gone on to take it at A level. And so, as a result of that meeting, we were very comfortable that the results we were seeing were indeed representing truly the true standards that we were expecting, the carrying forward of standards, and we then shared that information with QCA. There was no pressure from QCA to intervene and change the results after that point."[56] Sir William Stubbs told us that the guidance provided to the awarding bodies was to ensure that "any increase in the numbers passing or any increase in those getting the higher grades had to be rooted in the evidence of what the candidates did".[57]

52. Ms Tattersall said: "I did not see that as a pressure to actually bring in awards at a particular level, once we had clarified that we were talking the same language, and we were not actually saying that the outcomes for 2002 had to be exactly the same as the outcomes of 2001".[58]

AQA

53. Ms Tattersall told us that for AQA "the job that we have done in this first year of A level is exactly the same job that we have done in all the previous years of the old A level". AQA was asked by the Tomlinson inquiry to examine only two of the 1,008 boundaries which it set at A level. "The inquiry... has reaffirmed the boundaries which I set as a result of looking at the Chair of Examiners' recommendations. So AQA, I believe, can be very proud of its record of bringing in the new A level, and, of course, as a board, we are responsible for something like 45 % of the grades awarded in A level this year."[59]

54. We asked Ms Tattersall if she had concerns regarding the QCA's guidance to the awarding bodies. She reminded us of the JCGQ's correspondence with the QCA saying "as far as AQA was concerned, that clarified the issue, we were all talking the same language; we were not talking about outcomes being the same, we were talking about judging the evidence on the basis of what candidates actually did in the examination."[60] Ms Tattersall noted that "the main pressure on us is to be able to demonstrate that the standard of our awards is commensurate with the standard of previous awards; and in the first year of an examination that inevitably is difficult, because the syllabuses are different, the structure of the examination is difficult, you do not have the same reference points as you had in the past. But that is the sort of pressure that I would describe, but it is a pressure of which we were very aware, even without QCA saying it."[61]

EDEXCEL

55. Mr Kerr told us "I would not claim that Edexcel has not had its problems in the past; but, for this particular year, I am very confident we set the grades professionally, we set them accurately and we set them in accordance with the Code of Practice"[62].

56. Mr Kerr told us that "the letter of 19 April [from Sir William to the JCGQ] did not reassure me, I felt the pressure, I am sure the integrity was clearly there but the pressure put on by QCA was inappropriate".[63] He said that "to link grades this year back to legacy A levels was only one factor; the most important factor, from Edexcel's point of view, was the student performance, and to depress students' performance based on Government statistics would be unethical".[64]

OCR

57. Dr McLone said "we changed 18 out of 1,012 [units], which is a very small number. But, yes, we did, and it is a matter of doing it in a different context; we had a different context, we had different people present, we were making judgements. The judgements that we made, on the evidence, and [in] the summer, stand, the judgements that we made were done in a different context at this particular time, and I judged it right to be able to make the amendments I did in the 18 units that I did, but, nonetheless, that is quite a small number."[65]

58. Dr McClone told us "We have always worked to [get] the examiner judgements first and then [look] at statistical evidence, to make sure that we can compare year on year that we are getting to the right overall standard. I think I do go back to the question of AS and A2; we did not know exactly, all of us, where exactly A2 was. There is a real tension between trying to set boundaries at A2 and yet carrying forward a standard which is not A2, since we do not have any archive evidence at A2, there is nothing of that kind, but we do have to carry forward the A level standard, which is a combination of the AS and the A2. So therefore it has been a tension, in trying to establish all of that. The setting of the standard is actually QCA's job, of course."[66]

59. Dr McClone said "The system was flawed, if I may, and I think we are all trying to operate in a flawed system...[I have] great confidence in Ken Boston [the Chief Executive of the QCA]".[67] He believed that Dr Boston was attempting to " right what was not done in the past."

60. Dr McLone recognised that OCR applied a slightly different awarding process to the allocation of grades, compared to the other boards; however, he was adamant that standards had been kept at a very similar level across all of the examination boards. He said "if you take a look at where our boundaries have been set, compared with, say, [AQA's] boundaries, you will probably find them in very much the same place".[68]

38 QCA: Managing Curriculum 2000 for 16-19 students, Annex 1: level of demand. Back

39 A level standards - interim report, Mike Tomlinson, paragraph 14. Back

40 Ibid, paragraph 14. Back

41 Ibid, paragraph 15. Back

42 Ibid, conclusions. Back

43 Q.286 Back

44 Q.287 Back

45 Q.290 Back

46 Q.274 Back

47 Q.180 Back

48 Ibid. Back

49 Q.265 Back