4 Other lessons learned
Exam board communication with schools and colleges
91. Ofqual concluded in its November 2012 report that "communications between exam boards and schools could have been better, and more focused, most especially on the dangers of assuming grade boundaries". It suggested that exam board communication to schools about the possibility of changes to grade boundaries "may have seemed like pre-flight safety demonstrations, referring to a possibility so remote that it attracted almost no attention".
92. Interviews with schools and colleges conducted by Capgemini for Ofqual found that schools "did pay attention to warnings from awarding organisations about the scope for grade boundaries to change, but did not anticipate the size of shift that occurred." The research also found that advice given orally by exam boards was "less constrained." This is echoed by evidence we heard from school leaders. Kenny Frederick, Principal of George Green's School, told us that "we were working towards the grade boundaries we had with an expectation that [those] might vary between two or three points". Brian Lightman of ASCL stated that "we have plenty of schools that were told by boardsand it is not just one board but all boards that there would be little change from the January boundaries. They were told that in conversations that they had between January and June".
93. Capgemini's research for Ofqual found that teachers were under considerable pressure to provide accurate predictions for grades in GCSE English, with pressure coming from Ofsted, from school leadership teams and from parents. The survey found that schools had developed "highly sophisticated grade prediction and tracking mechanisms" which, prior to summer 2012, had proved highly accurate in predicting grades for GCSE English. Headteacher representatives confirmed to us in September 2012 that schools are "extremely strategic" in the way they allocate resource and prepare young people for exams, employing sophisticated prediction processes, data analysis and intervention strategies. As Ofqual has noted, many teachers consider the ability to predict grades accurately to be "a fundamental part of their role as a professional".
94. Exam board representatives acknowledged to us that communication with schools is a significant and ongoing challenge. Ziggy Liaquat of Edexcel explained that "we thought we had explained it well, but it clearly had not landed, and those grade boundary movements were a surprise [to schools] [...]we should be relentless in communicating the fact that these grade boundaries can move". Andrew Hall of AQA reported that although marks rather than grades had been issued for the January 2013 GCSE English exams, some schools had been using last year's grade boundaries to make assumptions about grades their students would achieve. He welcomed the input from the teacher associations which had "rallied behind this and are telling their membership that that is exactly the wrong thing to do, because grade boundaries have not been set".
95. Ofqual has acknowledged that it needs to improve its own communication with schools and colleges about how exams are marked and graded and its role in this. Results from Ofqual's most recent survey of perceptions of GCSEs and A levels suggest that confidence in GCSEs and in Ofqual has been damaged as a result of events with the 2012 GCSE English results, particularly among teachers and head teachers. Glenys Stacey told us that Ofqual is trying to address this through developments on its website and through engagement with teacher representative groups. She admitted that "the experience of GCSE English engendered a great deal of mistrust, and we have a job to do to explain as clearly as we can what we do, how we do it, and why we do it [...] I think there is never going to be enough that we can say to explain what we do".
96. As part of its monitoring of the January 2013 GCSE English exam series, Ofqual commissioned an independent review of exam boards' activities, which included their communication with schools and colleges. The review found that exam boards' communication is improving, but Ofqual has asked the exam boards to take some specific actions to make further improvements.
97. Exam boards must take very seriously the need to communicate better with schools and colleges, in order to improve trust and confidence in the exam system. We welcome signs that this communication is improving, but, as exam boards have acknowledged, this is a significant and ongoing challenge, and there is more to be done. We also welcome Ofqual's greater scrutiny of this area and recommend that Ofqual continues to be more vigilant in its monitoring of communication between exam boards and schools, as this area is closely linked to public confidence. We also welcome Ofqual's plans for improving its own communication with schools and colleges about marking and grading in the summer 2013 exams, in order to help restore confidence in Ofqual's regulation of exams, particularly among teachers and head teachers.
Grade awarding and the comparable outcomes approach
98. The difficulties experienced with GCSE English in 2012 have drawn attention to how the exam boards and Ofqual use the comparable outcomes approach in GCSE and A level grade awarding. One of the key issues in the judicial review was how the approach was used to maintain standards and how it influenced grade boundary setting for English GCSEs.
99. Ofqual has a statutory objective to ensure that standards in the qualifications it regulates are consistent over time and across exam boards, as well as consistent with attainment required for comparable qualifications in other countries. It has also a statutory objective to promote public confidence in qualifications such as GCSEs and A levels. As Lord Justice Elias noted in his judgement the two objectives are closely related, as maintaining the currency of the standard is an important element in maintaining public confidence in the system.
100. Grade awarding is a technical process, involving a blend of the professional judgement of examiners with a range of statistical data. The comparable outcomes approach is a statistical tool, whereby exam boards use data on Key Stage 2 attainment to predict the outcomes for each GCSE cohort. Ofqual states that it uses the approach for two key reasons: "to make sure that students taking new qualifications are not disadvantaged in the first years of those qualifications (to allow for teachers being unfamiliar with the qualification), and to make sure grade standards between exam boards are in line so that students sitting with one board do not have an advantage over others". It describes the approach as "a check at qualification level". Ofqual's position is that results in 2012 stayed steady overall at a national level, in part because exam boards applied the comparable outcomes approach to awarding, as the approach "helps to maintain standards over time".
101. Ofqual has told us that exam boards have used the approach for some years in GCSE awarding to improve comparability between exam boards. A change in recent years has been that Ofqual has strengthened its requirement for exam boards to justify increases in results outside tolerance. Ofqual has used the comparable outcomes approach at AS and A level since 2009 and 2010 respectively and for GCSEs since 2011. This coincides with an end to year-on-year increases in GCSE and A level results at national level.
102. The alliance which made the application for judicial review accused the exam boards and Ofqual of a "statistical fix", contending that the performance of students had not been fairly reflected in their grade because the results had been unjustly moulded to reflect predicted performance and that statistics had overly dominated the grading process.
103. In his judgement, Lord Justice Elias found that there was "nothing improper" in predicted outcomes playing a very significant role in grade awarding, concluding that "it was legitimate for Ofqual to pursue a policy of comparable outcomes, ensuring a consistent standard year on year, and assessing marks against predicted outcomes was a rational way of achieving that objective". He acknowledged that Key Stage 2 predictions were "widely thought to be the most reliable statistical evidence currently available for the purpose of comparing performance year on year" and noted that a common approach was important to ensure consistency across exam boards.
104. Exam board views on comparable outcomes vary. AQA is generally very supportive of the approach. Its chief executive officer, Andrew Hall, told us that "AQA has used past performance data for a number of years" and that "Key Stage 2 data is the best available for GCSE". Ziggy Liaquat of Edexcel described comparable outcomes as "not perfect as a system" but "as good as any system I have come across". By contrast, Gareth Pierce of WJEC criticised Ofqual's "over-deterministic" use of a single predictor model, and called for "balanced use of a range of relevant statistical information."
105. Glenys Stacey rejected the suggestion that Ofqual was unduly reliant on Key Stage 2 data, but acknowledged that she takes "a different view" to WJEC. There are some limitations to the comparable outcomes approach, which Ofqual accepts. For example, it does not work as well for subjects with a high performance element, such as drama. There are also issues with regard to Wales, where students do not sit Key Stage 2 tests and a different statistical tool, which does not rely on Key Stage 2 predictions, is favoured. Particular difficulties arose for WJEC GCSE English Language in 2012 when a majority of WJEC candidates were in schools in England.
106. Ofqual also acknowledges that the comparable outcomes approach "is not well understood by schools and colleges, and not generally trusted." Andrew Hall of AQA referred in oral evidence to misconceptions about the approach, saying that "one of the things potentially misunderstood at some stages was that the Key Stage 2 data was generating a quota. That absolutely was not the case".
107. The comparable outcomes approach has also attracted criticism from the teaching profession on the grounds that the approach makes it difficult for schools to demonstrate genuine improvements in performance. Ofqual itself voiced concerns to this effect in a letter to the Secretary of State, stating that "one consequence of this approach is that it can make it harder for any genuine increases in the performance of students to be fully reflected in the results". More recently, Ofqual has qualified this view, telling us that the approach does allow genuine improvements in performance to be reflected in results, but exam boards have to "present evidence of achievement above and beyond expectations".
108. Ofqual has taken advice from its Standards Advisory Group on its use of the comparable outcomes approach. Glenys Stacey told us that "comparable outcomes is regarded by assessment experts as the best possible model at the moment, and it looks good when compared with how these things are done in many other countries [...] there is no known better way internationally". Ofqual has confirmed that it will continue to use the comparable outcomes approach in GCSE and A level grade awarding in summer 2013, but that it will keep the approach under review.
109. As we noted in our 2012 exams report, there are significant challenges for Ofqual and the exam boards in explaining the technical difficulties surrounding grade awarding. The problems experienced with GCSE English in 2012 have brought many of the issues to the foreground, with the judicial review scrutinising the grade boundary setting process in considerable detail. There are, however, lingering misconceptions and concerns about unfairness and, as Ofqual has acknowledged, the experience engendered a great deal of mistrust. Work is needed to improve understanding of and restore confidence in the system, particularly with regard to grade awarding and Ofqual's role in maintaining standards.
110. Ofqual must continue to make greater efforts to explain the complexities of awarding and standards setting, and its role in this, to schools and colleges, in order to improve confidence in the comparable outcomes approach and in Ofqual's work as regulator. Ofqual should also keep the comparable outcomes approach under review and be prepared to adapt it in the light of experience and/or expert advice.
118 Ofqual second report, paragraph 1.50 Back
119 Ibid.,1.33 Back
120 Report of findings from Centre Interviews, Ofqual second report, Appendix 1 pp100-101 Back
121 Q46 Back
122 Q51 Back
123 Report of findings from Centre Interviews, Ofqual second report, Appendix 1, p94 Back
124 Q30, Q37 and Q59 Back
125 Ofqual second report, paragraph 6.10 Back
126 Q168 and 169 Back
127 Q220 Back
128 Perceptions of A levels, GCSEs and Other Qualifications-Wave 11 Summary Report, Ofqual, 3 May 2013 Back
129 Q339 Back
130 Ev 69 Back
131 Education Act 2011, section 22 and ASCL Act, section 28 Back
132 Judgement, paragraph 21 Back
133 Ev 67 Back
134 GCSE English Awarding 2012, Ofqual Supplementary Memorandum to Education Committee, 11 September 2012, paragraph 33 Back
135 Ofqual second report, paragraph 1.29 Back
136 Tolerance limits for overall outcomes in a qualification are agreed between Ofqual and the exam boards. These are typically +/-1 per cent. Exam boards can justify improvements in grades beyond these tolerance limits but must provide evidence of improvement or deterioration in performance. Back
137 Judgement, paragraphs 9-11 and 149 Back
138 Judgement, paragraphs 78 and 149 Back
139 Ibid., paragraph 37 Back
140 Q173 Back
141 Q174 Back
142 Q243 Back
143 Q345-6 Back
144 Annex D to Ofqual's second report, Report from OCR, p11 Back
145 Ev 76. Due to the majority of candidates being in England, it was agreed that in 2012 WJEC would report their projected outcomes using Key Stage 2 predictions for GCSE English Language and GCSE English (which is not taken by candidates in Wales) only. Back
146 Ev 68 Back
147 Q173 Back
148 Letter from Glenys Stacey to Secretary of State, 22 August 2012 Back
149 Q354 (Glenys Stacey) Back
150 Q350 Back
151 Ev 67 and Q350 Back
152 Q339 Back