Getting the grades they’ve earned: COVID-19: the cancellation of exams and ‘calculated’ grades: Response to the Committee’s First Report

Ofqual Response

Ofqual response to Education Select Committee Report: ‘Getting the grades they’ve earned: Covid-19: the cancellation of exams and ‘calculated’ grades.’

Introduction

This is Ofqual’s full response to ‘Getting the grades they’ve earned: Covid-19: the cancellation of exams and ‘calculated’ grades.’

We welcome the Committee’s report and its ongoing inquiry and we are committed to providing information to assist it. The report contains a number of conclusions and recommendations. Our response addresses each of the recommendations relevant to our responsibilities in turn.

We provided responses to some of the recommendations in our letter of 6 August (attached at Annex 1) when we confirmed the steps we had taken ahead of results days to respond to the Committee’s report. These are repeated here where appropriate. Some of the recommendations relate to the approach we had planned this summer rather than the final approach taken, but we recognise the importance of addressing each recommendation in turn to explain what happened and why we took particular decisions.

We note that the Department for Education is responding to the Committee on the recommendations relating to its responsibilities.

Of course, the approach to issuing results that was planned at the time of the report changed after A level results were initially released. We provided a written statement to the Committee on 1 September (attached at Annex 2) setting out our view of events this summer ahead of our appearance before the Committee on 2 September.

We made clear in that statement and at our appearance before the Committee that, above all else, we are sorry for what happened this summer: the distress and anxiety it caused for many students and their parents; the problems it created for teachers; and the impact it had on higher and further education providers.

As we also made clear in our statement, we must now learn the lessons from this summer and establish a way forward that can command public confidence and give students what they need to progress, even in difficult circumstances. Our immediate focus is on the safe and effective delivery of the 2020 autumn exam series and the work needed now to prepare for the summer 2021 exam series.

Response to recommendations

Recommendation 2: Ofqual’s evaluation must include comprehensive data on attainment, by characteristics including gender, ethnicity, SEND, childrenlooked after, and FSM eligibility, providing full transparency on whether there are statistically significant differences between attainment this year compared with previous years. (Paragraph 20)

Our interim technical report, published on A level results day, included equalities analyses of calculated results at both centre level and student level. We would have published a similar report on GCSE results day had calculated grades been the sole method of issuing grades.

We remain committed to a programme of work to evaluate summer 2020, including an updated equalities analysis on the final grades issued, the results of which we will publish as soon as possible.

Our initial equalities analyses of calculated grades examined the extent to which the relationship between grade outcomes and student background variables in 2018 and 2019 would have been maintained in 2020. Our analyses show no evidence of bias in calculated grades. For calculated grades, changes in outcomes for students with different protected characteristics and from different socio-economic backgrounds were similar to those seen between 2018 and 2019 and generally showed no widening of the gaps in attainment between different groups of students.

We are carrying out further analyses of the 2020 outcomes (centre assessment grades (CAGs) or calculated grades, whichever is higher) to consider whether there was any impact on students who share particular protected characteristics or socio-economic status. Early indications suggest that there are no material differences in the attainment gaps between different groups of students for the grades given to students this year compared with the calculated grades that would have been issued. Before publishing the findings of this work, we plan to discuss them with equalities groups, to explore suggestions for further analysis.

Recommendation 4: Ofqual must identify whether there is evidence that groups such as BAME pupils, FSM eligible pupils, children looked after, and pupils with SEND have been systematically disadvantaged by calculated grades. If this is the case, Ofqual’s standardisation model must adjust the grades of the pupils affected upwards.
(Paragraph 27)

We have explained our approach to equalities analyses above.

Along with the Committee and others, we had been particularly concerned to make sure arrangements for this year did not exacerbate the attainment gaps between different groups of students seen in previous years.

We were pleased to confirm on 21 July, ahead of results days, that our initial analyses of GCSE, AS and A level calculated grades were reassuring, suggesting that there would generally be no widening of the gaps in attainment between different groups of students. We published information about this here. This was also seen in the analysis we carried out on the final calculated grades, described above.

Therefore, it was not necessary to adjust the calculated grades of particular groups of students as suggested in this recommendation.

While our analyses of calculated grades showed no evidence of systemic bias, we recognised the potential for individual concerns about bias or discrimination. We explained how any such concerns should be raised in our student guide to appeals and malpractice or maladministration complaints.

Recommendation 5: Ofqual must be completely transparent about its standardisation model and publish the model immediately to allow time for scrutiny. In addition, Ofqual must publish an explanatory memorandum on decisions and assumptions made during the model’s development. This should include clearly setting out how it has ensured fairness for schools without 3 years of historic data, and for settings with small, variable cohorts. (Paragraph 28)

We sought to be as transparent as possible at all stages about the approach we took this summer.

We consulted between 15 and 29 April on the aims of statistical standardisation; the approach we would use—including the sources of evidence on which we would draw; how we should consider issues relating to schools and colleges with improving or declining historical results; and potential bias in CAGs. We published the details of our approach on 22 May. In particular we confirmed that a “statistical approach will mean that an individual student’s grade will be informed by their position in the centre’s rank order for that subject, their prior attainment where available, and the centre’s past performance.” We also confirmed that for small cohorts in schools and colleges for which “the data would be weaker”, we would “ensure that the standardisation model is sensitive to the size of error in the statistical predictions”.

We published further information on 21 July, after the deadline had passed for schools and colleges to submit their CAGs and rank order information. To provide more information earlier, mid-way through the period in which schools and colleges were submitting CAGs and rank order information, could have risked making teachers anxious, either because they thought they needed to take account of the information in their submission, or because they had already submitted their data and therefore could not take account of the extra information. There was also the risk that centre behaviour would change if additional information was published. On 21 July, we provided a detailed fact sheet and film and as well as information about the anticipated results this year.

We chose not to publish the full model ahead of A level results day, because we felt it was important that everyone finds out their results on results day. Early publication of the final details of the model could have allowed some centres to work out results early or cause unhelpful anxiety if the information was misunderstood. However, the information we published prior to that point provided significant detail on the approach and allowed a detailed understanding of the standardisation process. The only things we did not publish before results day were the algebra that was used in the model and the definition of what constituted a small centre. Although we can see why the Committee and others were keen to see the statistical model earlier, we waited until results day in the interests of fairness to students.

We published our technical report on A level results day (13 August). The report and associated documentation confirmed the decisions and the assumptions we had made during the model’s development, the model itself, and provided a detailed technical account of the approach to awarding GCSE, AS, A level, Advanced Extension Award and Extended Project Qualifications. The report also included metrics on the accuracy of results.

As the report makes clear, for schools or colleges with a small number of entries in a subject, and for those schools and colleges that did not have historical results data, the statistical standardisation process operated differently. This was necessary because the statistics were less reliable for smaller cohorts. In the interests of fairness to individual students, the model moved from statistical evidence as the primary source of evidence for determining students’ calculated grades to placing greater and, in some cases, sole weight on the CAGs. Our report sets out how we determined that for cohorts of fewer than 15 students progressively less weight should be placed on the statistical evidence and that for cohorts of 5 or fewer, or where there was no historical evidence, the CAG would be the student’s grade.

Recommendation 6: Ofqual must collect and publish anonymised data at the conclusion of the appeals process on where it received appeals from, including, as a minimum, type of school attended, region, gender, ethnicity, SEND status, children looked after (including children supported by virtual schools), and FSM eligibility. (Paragraph 29)

We publish statistics on appeals for GCSE and AS and A level qualifications each year. The statistics for the last academic year (2018/19) are available at this link. These statistics are published as Official Statistics and meet the principles of the Code of Practice for Statistics.

To comply with the Code, we must make sure that the statistics we report are trustworthy, are of high quality and have value for those who use them. This year we plan to publish statistics on appeals in GCSE and AS and A level qualifications for the summer 2020 series in December 2020. Very few appeals will be outstanding at the time of publication so the statistics will be largely complete.

The statistics we publish include the number of appeals received and upheld, the number of grades challenged and changed, the average number of days taken to complete an appeal and a breakdown of the different reasons for appeals. We plan to publish similar statistics this year.

We are considering publishing additional data breaking down appeals, for example by geographical region, gender, ethnicity and centre type. However, we will be mindful when publishing such breakdowns—where numbers and groupings become small—to make sure individual candidates cannot be identified. This will guide us as we decide on the granularity with which we can publish appeals data. This will only be clear to us once we receive the final appeals data from the exam boards in the third week of November.

Recommendation 7: As part of its evaluation Ofqual must publish comprehensive data on vocational and technical qualifications, by characteristics including gender, ethnicity, SEND, children looked after, and FSM eligibility, providing full transparency on whether there are statistically significant differences between attainment this year compared with previous years. (Paragraph 33)

We published some information about results issued for vocational and technical and other general qualifications at Levels 3 and 4 on 13 August. We evaluated whether, within the cohort, different demographic and socio-economic groups appear to show different patterns of outcomes compared to previous years. We concluded that in most cases attainment gaps have not increased over time between different demographic groups. There were some specific cases where attainment gaps have appeared to change, although changes were small in real terms.

We have very recently received revised data from awarding organisations (covering those results that were regraded) which we will use to complete further analysis in the coming weeks. This will form a part of our evaluation of summer 2020 awarding, as referenced above.

Separately, we will continue to publish quarterly information about vocational and technical qualifications. We will publish information covering April to June 2020 on 17 September.

Recommendation 8: Where calculated grades are used to award vocational and technical qualifications this year, Ofqual must identify whether there is evidence that groups such as BAME pupils, FSM eligible pupils, children looked after, and pupils with SEND have been systematically disadvantaged by calculated grades. If this is the case, Ofqual’s standardisation model must adjust the grades of the pupils affected upwards. (Paragraph 34)

Unlike GCSEs, AS and A levels, there is no overarching statistical standardisation model for vocational and technical qualifications (VTQs) and other general qualifications. To cater for the breadth of the VTQ landscape, we designed our extraordinary regulatory framework to allow awarding organisations flexibility to determine the most suitable approach for their qualifications. This has been operationalised by 147 awarding organisations for around 15,000 qualifications. In all cases, awarding organisations have been responsible for designing assessment models, within the framework of our rules, that allow assessments to be made in the most fair and valid way for the qualification type.

Different approaches to calculating results were taken depending on the evidence available and the nature and structure of a qualification. Whichever approach was taken, our rules required awarding organisations to:

Recommendation 10: Ofqual must urgently publish the evidence thresholds for proving bias or discrimination, clearly setting out what evidence will be required, including example case studies. This must be communicated to parents and pupils in advance of results day. (Paragraph 42)

We published, ahead of results days on 6 August, a guide for students. This explained what they should do if they had concerns or questions about their grades.

It detailed the appeals process for qualifications this summer and how students could complain about bias, discrimination, or other potential examples of malpractice or maladministration.

The guide made clear that, in the first instance, students should ask their school or college to check whether it made an administrative error when submitting information to the exam board. Such errors could include a clerical mistake or a failure to take into account relevant information about a student’s likely performance. Where schools and colleges can provide evidence of such mistakes, they can be quickly put right.

The guide explains how a student concerned that their grade was affected by some sort of wrongdoing or a lack of care by their school or college, including any concerns of potential bias or discrimination, should complain. This would be considered as a complaint of malpractice or maladministration. The guide includes information to help students understand whether they had cause to complain. We explained that students would need to show evidence of something specific or surprising in relation to the grade submitted by their school or college and gave examples of how a student might look for evidence that something had gone wrong. Of course, in practice, circumstances will vary and we made clear the examples we gave were not exhaustive.

The guide advised students to raise any concerns with their school or college in the first instance, although we recognised that in some cases a student might wish to go directly to the exam board. We have made clear that the process for making such a complaint was not part of the appeals process, therefore is not subject to deadlines for appeals and was not directly affected by the proposed changes to the appeals process on the basis of mock results. The guide provided details of our helpline, and of the Exam Results Helpline, who students and parents could call to discuss these issues. We also provided details of the Equality Advisory and Support Service who students could contact if they thought they had evidence of discrimination. We publicised this information through a wide range of stakeholder groups and through social media, as well as on our website.

Recommendation 11: Where pupils with SEND, or their families, have concerns about their grade, they must be allowed to see the evidence, such as past work or mock exams, used by teachers to arrive at their calculated grade. If appropriate access arrangements were not in place for the work used, or if their school did not use evidence from SEND specialists if this was appropriate, the pupil must be allowed to appeal on the basis of malpractice or maladministration. (Paragraph 43)

As explained above, we provided information for students concerned about malpractice or maladministration, including concerns that a reasonable adjustment or evidence relating to a student’s disability was not taken into account. We would expect the student’s school or college would explain how their CAG was determined and would make available to the student the evidence used where available.

If a school or college identifies that it made a mistake that had disadvantaged a student, for example by not taking into account their reasonable adjustment, it should raise this with the exam board under the appeals arrangements. It is only in cases where the school or college does not believe it made an error that a student would need to complain about malpractice or maladministration. We cannot require schools and colleges to provide students with particular forms of evidence. However, students have a legal right to request information held about them by their school or college as set out here by the Information Commissioner’s Office. Exam boards might decide to request further information and explanation from school and colleges as part of a malpractice or maladministration investigation.

Recommendation 13: Ofqual must ensure advice and support is easily accessible for all pupils unhappy with their grades. Both the helplines provided by Ofqual and the National Careers Service must be freephone lines. These must both be staffed by dedicated professionals with the training to provide sound and impartial step-by-step advice and support on appeals. (Paragraph 45)

We recognise the importance of students having clear and accessible information and that they can speak to someone when they are unhappy with their results. We provided full training for colleagues staffing our helpline to take calls in the run up to, on and after results days. We also supported the training of the careers advisers who staffed the government’s national Exam Results Helpline. We and the Exam Results Helpline were ready to receive larger than normal volumes of calls, having significantly increased our capacity to do so. The Ofqual helpline is charged at standard network rate and the Exam Results Helpline is a freephone number.

We also arranged for the Equality Advisory and Support Service to take calls from students on their helpline if they had evidence of discrimination.

Recommendation 14: Ofqual must issue guidance to schools and colleges about the options available for pupils unhappy with their results. Letters to pupils should be sent out by schools and colleges, to ensure they are aware of their options, including the standard of evidence required to bring an appeal on the basis of bias or discrimination. Schools and colleges should clearly communicate to pupils and families which staff member will be their point of contact for discussions about next steps. (Paragraph 46)

We issued a number of guidance documents aimed at students and schools and colleges. The documents we published on our website this year received over one million downloads and we have had over four million impressions on our posts to social media.

As part of our commitment to support students during this difficult year, we published a student guide to post-16 qualification results on 27 July providing information about the exceptional arrangements in place. We also published a student guide to appeals and malpractice or maladministration complaints on 6 August, as discussed above. We provided BSL translations of these guides.

We wrote to all schools and colleges on 27 July with information about this year’s arrangements. In this letter, we provided links to the student guide and other resources for students, parents and carers. We also provided hard copies of these documents to schools and colleges for them to give to students, parents and carers. Under our current regulatory framework, we could not require schools and colleges to provide this information to all their students.

We also published question and answer style guidance documents for teachers, students, parents and carers about the arrangements for general and vocational and technical qualifications throughout this period, from 3 April onwards. We updated these as information changed and to make sure they included answers to questions we received via our helpline and our Public Enquiries email address.

We updated our student guide to post-16 qualification results and our student guide to appeals and malpractice or maladministration complaints on 19 August after the change to awarding grades was announced on 17 August. We updated our question and answer style guidance document for teachers, students, parents and carers with a new document on 20 August.

We published further summary guidance on appeals, malpractice and maladministration complaints for GCSE, AS and A level grades on 26 August to respond to questions we received about appeals arrangements in light of the changes made to the approach to issuing results for these qualifications. We sought views from the school and college leaders that are members of Department for Education’s ‘Appeals taskforce’ as we developed that guidance and we welcomed their helpful input.

We publicised our information documents on our own website and via social media, as well as providing a wide range of stakeholder organisations, including equalities organisations, with links to the documents for them to publicise with their networks. We sought to promote the information as widely as possible.

Recommendation 18: Ofqual must urgently publish dates for the autumn exam series and end uncertainty for pupils, teachers, schools and colleges. (Paragraph 53)

On 9 July the Joint Council for Qualifications, which represents the exam boards, announced that the timetable will run as follows:

The deadlines for entry are:

Recommendation 20: We support a short delay for exams in summer 2021 as preferable to modifications to exam content. Any delay must be a matter of weeks, not months. Ofqual must publish details of the 2021 exam series as soon as possible, and before the end of the summer term. (Paragraph 56)

We consulted from 1 to 16 July on arrangements for the 2021 summer exam series and we asked questions about the desirability of delaying the GCSE, AS and A level exam timetable, perhaps so it started after the May half term. We received almost 29,000 responses. We published a summary of the responses and the decisions we took in light of the feedback on 3 August.

68% of those who responded to the consultation were in favour of a delay to the GCSE exam timetable and 60% in favour of a delay to the A level timetable.

However, support decreased to 51% and 42% respectively if, as a result of delayed exams, results days were also delayed. A decision to delay exams would have consequences across the education sector, for schools, further education colleges and higher education. It would also have consequences that extend beyond England. As we explained to the Committee at the hearing on 2 September, while we understand the benefits a short delay to the start of the exam timetable would bring, delaying exams would introduce risks to the timely completion of marking. These risks would need to be understood and mitigated, as far as possible, before a decision was taken to change the timetable.

We have proposed to the Secretary of State that we work with DfE and the exam boards to agree the timetable and any delay to this year’s exams as a matter of urgency.

In addition, following our consultation, we have asked exam boards to make some modifications to the way in which they assess their qualifications ahead of next summer.

Some of the changes relate to non-exam assessment arrangements in a number of GCSE, AS and A level subjects and are designed to allow assessments to be undertaken within public health provisions. In a few subjects there will be consequences for the exams, notably in geography. Given the difficulties schools and colleges would have arranging fieldwork for their students in the current academic year, the exams will be changed so that students do not need to draw upon their own fieldwork experience in the exams.

The government, which is responsible for the content of GCSEs, AS and A levels, has agreed that students taking GCSEs in English literature, history and ancient history should not be examined on the full range of content, allowing teachers some choice of the topics or texts they teach. Whilst the changes will reduce the volume of content that needs to be taught, it will not impact on the standards of the assessment that students will take—the sorts of questions that students will be asked or the way in which they will be marked. The assessments taken by students in 2021 will otherwise be in the familiar format.

Conclusion

We welcome the opportunity to give written evidence to the Committee in addition to appearing before the Committee. We are committed to learning the lessons from 2020 whilst making sure that we work to ensure that all regulated qualifications are delivered safely and to rebuild public confidence in them.





Published: 24 September 2020