Education CommitteeFurther written evidence submitted by Ofqual
This note is an update on Ofqual’s work relating to GCSE English and English language since the Chair, Chief Regulator and acting Director of Standards appeared before the Committee on 11 September 2012. We have set out our work under key headings but in reality these different strands of work have been part of a programme of work that has been a high priority and importance for Ofqual.
Final Rport on Summer 2012 Awarding
Following our interim report on 31August, we continued to investigate GCSE English. We looked more closely at exam board awarding and interviewed the Chief of Examiners in English at each board. We interviewed representatives from 100 schools and colleges to understand fully the concerns being expressed, and to identify common features and common approaches to teaching, learning, intervention and assessment. We required exam boards to conduct their own analyses of results. We developed systems in Ofqual to analyse data at school level (in the past our analysis has been limited to the national picture). We evaluated exam boards’ analyses as well, and we sought advice from experts in assessment.
In our final report, published on 2 November 2012 [http://bit.ly/TQ4m6p and http://www.ofqual.gov.uk/files/2012–12–10-gcse-2012-erratum.pdf], we confirmed the findings of our interim report: Grade boundary setting in exam boards had happened as it should, but nevertheless there was an unexpected variation in school results when compared to school expectations, and when compared to more established qualifications.
We were able to show that overall, results were almost unchanged, year on year. The issue was not overall outcomes, but variations, school by school. A significant minority saw large changes, about equally up and down. We went on to conclude that variable results were to be expected to some extent. We showed that results are always more variable when qualifications change, and we explained that these qualifications had changed significantly. This was the first time the new qualifications had been awarded. Some grade boundaries in January 2012 and in 2011 had been generous, but this had not been obvious to examiners or exam boards at the time.
We also concluded that the new qualifications were exceptionally complex and difficult to award, and especially susceptible to pressures, as teachers strove for the best possible outcomes for their students and school. We concluded that putting so much weight on one grade in one subject as part of accountability and performance measures created perverse incentives for schools in the way they marked controlled assessment and led to over-marking. It is our view that no teacher should be forced to choose between their principles on the one hand and their students, school and career on the other.
Fairness
It became clear during our investigation that there had been some unfairness, and that it was not possible to identify which students had been treated unfairly. Ofqual Board considered whether anything at all could be done to redress that unfairness. It considered all known options, including directing grade boundary changes, and concluded that there was no way in which to improve fairness: any action other than supporting an early resit opportunity, which was the option we announced in the August report, would have created greater unfairness.
Judicial Review
Legal proceedings were started against AQA, Edexcel and Ofqual by a consortium of local authorities, schools and students a few days before we published our final report. The claimants were not willing to await our report. The hearing was expedited, and the matter was heard in mid-December in a three day hearing.
The court delivered judgement in the New Year. It concluded that Ofqual had acted properly, that Ofqual’s decisions were cogent and rational, and that we had “grasped the nettle”. The claimants’ claims were dismissed robustly. The Court decided there was no case for Ofqual to answer. Subsequently, our costs and the costs of AQA and Edexcel have been awarded.
November 2012 Re-sit
With our agreement and support, exam boards offered resits free of charge in an exceptional sitting in November 2012. Over 46,000 students entered one or more units. 36,000 of those were students who had achieved a grade D in summer 2013. Approximately 30% (9400) of those students gained a C in the November re-sit.
Contrary to normal practice, we arranged for JCQ to publish a full set of results for these re-sits [http://www.jcq.org.uk/examination-results/gcses] in the interests of transparency.
Actions to Strengthen Future Exams
Our November 2012 report identified flaws in the way the qualifications are designed. We cannot quickly change some of those, but we have taken action, where we can do so safely, to protect the qualification in future exam series, as follows.
1. For November 2012 onwards we agreed reduced tolerances for moderation of controlled assessment marks—that is the difference that can exist between the teacher marks and the moderator marks without any adjustment being made.
2. We published a special condition for exam boards which prevented them from grading the January 2013 units in English and English language. This means that teachers will not know the January grade boundary marks when they are marking controlled assessment work for submission in June. Students who entered units in January 2013 received their raw marks on March 7th but did not receive a grade. Grading for both January and June units will take place in July 2013.
We and the exam boards communicated this to schools well ahead of the March results day. We scrutinised exam board arrangements for implementation of these new arrangements, as there were some system risks for exam boards to manage. We also scrutinised their communications with schools and colleges.
To date we have received 17 complaints into our helpdesk specifically about GCSE English since results were issued following January exams. Seven of these were from parents, five from schools and five from candidates. These are all complaints rather than enquiries. This is a comparatively low rate of enquiry and complaint, in our experience.
3. We had already implemented changes to make GCSEs linear in England for summer 2014. January 2013 was therefore the last January series for GCSE. In future all GCSE qualifications will be taken in a linear way—all units at the same time—in June. The only exception we made to this was to make provision for a re-take opportunity in November for English, English language and mathematics, because of the importance of these qualifications for progression to Level 3 qualifications or to the labour market.
We now know that the pressure to achieve a C in English and mathematics is so great that it sometimes distorts teaching and learning. So we have made clear that from 2014 the November exam series will only be available to students who are re-taking the qualifications. It will not be available for first time entry.
4. We are discussing with exam boards changes that might be made for teaching from September 2013. We know that there are particular challenges in moderating the teacher marks for speaking and listening. We are planning to consult over the next few months on changes that would mean that marks for speaking and listening did not count towards the overall GCSE grade. We have been working with exam boards to model the effects of that change.
GCSE English Language in Wales
The approach to awarding in summer 2012 had been agreed between the three regulators in England, Wales and Northern Ireland, as had been the case in previous years. In Wales in September 2012, after the results had been issued, the Minister directed WJEC to re-grade GCSE English language (GCSE English is not available in Wales) so that the overall outcomes matched those for Welsh students in summer 2011. This decision resulted in more favourable treatment for the 2,300 students in Wales (compared with students in England) who were upgraded.
The Welsh Government recently imposed on WJEC a special condition for summer 2013 that requires WJEC to hold separate award meetings for Wales students for January and June 2013, and to provide different question papers for students in England and Wales in June 2013. Different question papers will necessarily mean different grade boundaries. We do not think this is a satisfactory situation. It damages the credibility of, and confidence in, the GCSE brand for all students, but especially for those in Wales. On 6th March we wrote to the Welsh regulator to ask for a meeting to discuss it, but so far we have had no response.
We have been talking to the Welsh regulator’s officials with the aim of finding an approach to summer 2013 awarding that would provide us with the assurance that grade standards in WJEC English language in England and Wales will be comparable. To date we have not been able to gain assurance that common grade standards will be maintained.
To aid decision-making we have agreed a principled approach to regulating qualifications that may be subject to more than one set of regulations (see Appendix 1). If we enter summer awarding without the necessary assurance, we will make sure that it is clear to students and to users of qualifications that common standards are being set for these qualifications taken in England but that the standards in Wales are different. For the future there are likely to be increasing divergences between GCSE qualifications taken in England and Wales. We are working with the Welsh regulator and staff to make sure that users of the qualifications are not misled.
The brand GCSE is jointly owned by the three regulators, Ofqual, Welsh Government and CCEA (for Northern Ireland). When this was agreed it was assumed a common standard could be agreed. This position was tested to the limit in 2012. Looking ahead we will take the steps that are necessary to distinguish the GCSE qualifications that we regulate (and where we can assure the standards), from any that may be awarded at a different standard.
Our Approach to Awarding GCSEs in Summer 2013
We have been clear that we have used the comparable outcomes approach since 2009 when the first new AS awards were made (and it is a development of the approach that has always been used for GCSE awarding). It was not new for summer 2012. We have been clear that we use it for two key reasons––to make sure that students taking new qualifications are not disadvantaged in the first years of those qualifications (to allow for teachers being unfamiliar with the qualification), and to make sure grade standards between exam boards were in line so that students sitting with one board do not have an advantage over others. It provides a statistical check that would indicate if awards were out of line with expectations, in order that the exam board can consider whether there is evidence that this is justified.
We have discussed this approach with our Standards Advisory Group and we are clear that it is currently the best available approach. We will continue to use this approach for summer 2013 and we have published further detail on this [http://www.ofqual.gov.uk/standards/summer-exams-2013/].
Lessons Learned
The design of these GCSE English and English language qualifications is too complex. Moving to linear assessment helps: it enables safer awarding and eliminates the so-called “route effect’—the likelihood that students will do better if they enter in a modular way. The proportion of controlled assessment—60%—is hard to justify in assessment terms, and we are considering the place of controlled assessment in new GCSE English language qualifications due for first teaching in September 2015.
Qualifications and school accountability measures need to sit well together. We welcome the thrust of the Government’s current consultation on the secondary accountability framework. The current proposal is that a threshold measure is retained in GCSE English and maths. We understand the rationale for that. If that is confirmed, then we will need to ensure that new GCSEs in those subjects are sufficiently resilient.
The views of technical assessment experts in exam boards matter, and they need to be consulted as new qualifications are designed. There is some evidence that this did not happen sufficiently or at the right time when these GCSEs were being designed by our predecessor body. We will certainly make sure that this happens for the new GCSEs now being planned.
More generally, the English 2012 experience has told us a lot about how qualifications are used in schools and the strategies that are used by many teachers, as well as how they interact with the accountability system. That experience—sometimes sobering—has been and will be invaluable as we develop our regulatory arrangements for the future, plan for the implementation of reformed qualifications, and advise Government on its policies on qualifications, accountability and other related areas. For example, it heavily influenced our advice in the autumn to Ministers about the proposals for English Baccalaureate Certificates.
Regulation was effective in ensuring comparable outcomes nationally—preventing students in 2012 from being disadvantaged by the fact that they were taking a new qualification, and in reducing unjustified discrepancies in results between one exam board and another. The High Court subjected our approach in general, and our actions in 2012, to intensive scrutiny, and found they were justified—we welcome that. However the approach is not well understood by schools and colleges, and not generally trusted. The proper steps we took to challenge Edexcel and WJEC’s preliminary results appear to have taken some people by surprise, the suggestion being that Ofqual was fixing results. We have a big task ahead, to explain our approach, and to make it and our actions more transparent. We have already published information about our approach to the awarding this summer and we will publish more information in the run-up to the summer exam series.
Appendix 1
OFQUAL’S PRINCIPLES FOR REGULATING QUALIFICATIONS WHICH MAY BE SUBJECT TO MORE THAN ONE SET OF REGULATORY REQUIREMENTS.
1. Our main priorities are the qualifications taken by learners in England (and Northern Ireland for vocational qualification), the interests of these learners and the interests of others who use these qualifications. We will, however, work to avoid disadvantaging learners taking regulated qualifications in other countries where possible.
2. We should aim, where possible, for the same qualifications to be available in England, Wales and Northern Ireland. We will not pursue this aim where this would prevent us meeting our statutory objectives, and in particular securing standards, or where this is explicitly contrary to the education policy of the respective governments.
3. Where a qualification we regulate is also assessed or awarded in jurisdictions other than England (or for vocational qualifications England or Northern Ireland), the same standard must be maintained, whatever the location of the learners. If this cannot be assured, the same qualification should not be available outside England (or England or Northern Ireland for vocational qualifications). Any alternative qualification should be uniquely identified and distinguishable from the qualification we regulate.
4. We should avoid, where possible, putting recognised awarding organisations in a position whereby they cannot simultaneously satisfy our regulatory requirements and those of the regulators in Wales and/or Northern Ireland (with the result that an awarding organisation may not be able to offer the same qualifications in England and Wales and/or Northern Ireland).
5. We should prioritise managing the risks to the safe awarding of qualifications to candidates in England (and Northern Ireland for vocational qualifications), including where those risks arise from the activities of another regulator.
6. We should explain our principles to awarding organisations and to the public.
March 2013