The administration of examinations for 15-19 year olds in England - Education Committee Contents


Annex 1: Note of the Committee's seminar with examiners, 14 December 2011


These notes are a general account of the opinions expressed by a group of examiners, who met with the Committee for an informal discussion. Examiners represented all three exam boards, a range of subjects and a variety of examiner roles.

The examiners were sifted from over two hundred applications received by the Committee in response to an advertisement in the Times Educational Supplement and online. All examiners not selected to attend were invited to complete a short questionnaire covering similar areas to those discussed at the seminar.

Comments in double inverted commas indicate a direct verbatim quotation, although these are not attributed.

EXAMINER RECRUITMENT

There was general agreement that "lots of assessment requires lots of examiners" and that this has led to shortages in some subjects.

Some examiners felt that recruitment requirements have become less stringent in recent years. Previously, teachers needed at least 3 years' teaching experience before they could become an examiner. Some examiners reported that they were recruited in their first year of teaching or even during postgraduate training. Others argued that recruitment requirements are stringent enough as it is "easy to get rid of" poor examiners. However, one senior examiner expressed doubt as to whether all weak examiners are dismissed in shortage subjects, due to difficulties in recruiting sufficient numbers.

Some concern was expressed that too many examiners are retired rather than practising teachers. There was general consensus that it is important to have a balance between practising teachers and retirees. However, the main obstacle for practising teachers is time. The summer exam series coincides with a busy period in school for many teachers and examining (at least at more junior levels) is not sufficiently well paid to make it worth their while. Some suggested that being an examiner should be a recognised part of teachers' continuing professional development.

Senior examiners recounted from experience that it is very difficult to combine senior examining roles with leadership roles in schools or full-time teaching. This means that many question paper setters are no longer teaching. This was felt to be a shame as it is important that examiners are sensitive to the impact in the classroom of the examinations they are setting.

QUESTION PAPER SETTING

Examiners generally agreed that the question paper setting process is rigorous. It was pointed out that some of the errors in 2011 papers were introduced in the later stages of question paper setting (eg changes to graphs with accompanying text not updated).

ONLINE STANDARDISATION

Examiners were unanimously of the view that the loss of face-to-face standardisation meetings was "a big mistake". There was consensus that face-to-face standardisation meetings helped examiners to explore questions about the mark scheme in depth and get a feel for how the Principal Examiner was interpreting the mark scheme. Senior examiners felt that face-to-face standardisations meetings were useful for "talent spotting" as they helped them to identify potentially very good new examiners.

CONTROLLED ASSESSMENT

Examiners were generally negative about controlled assessment, although one senior examiner felt that controlled assessment had worked well in his subject (Drama). Many examiners were critical of inconsistencies between the exam boards in the rules for controlled assessment. Several complained that controlled assessment was hard to manage, dominating much of the school year for pupils in years 10 and 11 and also that it was impossible to be sure that a piece of work was a child's own. Generally, examiners felt that controlled assessment had failed to address the problems with its predecessor, coursework, and that, in some respects, it had generated problems of its own.

GRADE AWARDING

Senior examiners felt that there were many public misconceptions about how grades are awarded and that few people understood how data are used. There was some variation in examiners' views on the importance attached to statistical information and examiners' judgement of candidates' work in awarding. Some examiners felt that it was right that awarding was data-led and that statistical information on the previous attainment of candidates and school predictions should be more significant that the judgement of a small group of examiners on candidates' work. Others were concerned that candidates should be judged primarily on the standard of work achieved.

COMMERCIAL ACTIVITIES OF EXAM BOARDS

Several examiners (also practising teachers) suggested that exam boards spend a lot on marketing to schools, often very aggressively, in order to win market share. Some reported that market researchers from exam boards tried to determine which elements of a subject were the most popular (e.g. which period of history, novel in English) and incorporate these into their syllabuses.

TEXTBOOKS

There was general agreement that endorsed textbooks are problematic, although views varied on whether they should be allowed to continue. Some examiners felt that exam board/publisher partnerships, such as AQA/Nelson Thornes, were wrong and that exam boards should not make money from publishing agreements. Examiners suggested that the Committee could usefully explore how much money exam boards make from textbooks, compared to the core business of running exams.

Many examiners held the view that exam boards should not endorse particular textbooks and that publishers should not market books as "written in line with" a particular syllabus. Some felt that it was very misleading for exam boards to suggest that their endorsed or, in one case, own textbooks were all that schools needed, as well as potentially damaging to teaching and learning (by encouraging teaching to the test/book).

Some participants were also textbook authors. Several reported pressure from publishers to write to a particular examination, although they recognised the pressure on publishers and exam boards to supply what teachers wanted. There was disagreement about whether examiner authorship represented a conflict of interest, with examiners having insider knowledge. Some agreed with this, others argued that as senior examiners deal with so many exam questions across different exam series, insider knowledge is not an issue.

One examiner author said that he hoped his textbook gave an insight into the subject which would apply to all syllabuses. It was suggested that endorsed textbooks are more of an issue in some subjects than others. For example, English syllabuses are broadly similar across all exam boards, whereas in science there is more variation between exam boards, with textbooks linked to specific syllabuses and chapters for each module.

Some participants questioned the quality of some endorsed textbooks, pointing out that good examiners do not necessarily make good writers and that "children deserve good textbooks". Others noted that it was difficult in some subjects for teachers to find a textbook not linked to a particular exam board. There was also concern about the cost of textbooks and the problem that "if you change syllabus, you need new textbooks".

TRAINING

Examiners stated that they now have more contact with teachers than they did 20 years ago. Exam boards run more feedback meetings and there is greater transparency about the system with more materials available.

It was pointed out that many senior examiners are self-employed and may use textbooks and training to supplement their income from examining.

There were reports of some senior examiners (not participants) using their position to promote training to schools offered by private training providers, with courses entitled "changing your Ds to Cs" and "getting an A* grade" or intensive A level preparation courses. It was generally felt that this was inappropriate. By contrast, delivering general training on GCSE which covered all exam boards, was felt to be acceptable. Some pointed out that OCR contracts prohibit examiners from using their status as examiners to promote training and that Pearson is making changes to its examiner contracts from April 2012. However, it was pointed out that is difficult for examiners not to divulge their position as many teachers will know the names of senior examiners for their syllabus.

COMPETITION BETWEEN EXAM BOARDS

It was suggested that commercial companies will always market aggressively and find a way of playing the system.

Examiners expressed concern about the comparability of standards between exam boards in some areas, suggesting that there are "pockets where the differences are significant". Several agreed that "Ofqual has a case to answer" in terms of exam boards competing on standards.

REFORM OF THE EXAMINATION SYSTEM

There was no agreement between examiners on whether the exam system should be reformed or how. Some were in favour of a single state-run exam board (with regulation), others favoured retaining the multiple exam board system. One examiner expressed a preference for franchising by subject.

Several examiners felt that teachers wanted a choice of syllabuses and being able to switch exam boards was a helpful tool at teachers' disposal. Innovation and development were also cited as arguments in favour of a multiple board system. However, some examiners felt that the multiple board system carried the risk of competition on grade standards and that in this respect "Ofqual has a case to answer".

Examiners suggested that other countries where exams were run by a single state provider (e.g. Scotland, South Africa) might prove useful comparators, although others cautioned that systems which work well for countries with much smaller populations may not translate well into the English context and that our experience of a top-down national system with national curriculum tests has, on occasions, been "catastrophic".

Examiners observed that a franchised system would lose the elements of choice currently available to teachers and that system failure would affect larger numbers of candidates. However, some suggested that it could prove helpful to children moving schools as the same syllabus would be taught in all schools.

There was general agreement that better regulatory oversight by Ofqual was vital to the success of any model, whether a single board, franchising by subject or multiple board system.

IMPACT OF THE ACCOUNTABILITY SYSTEM

Examiners acknowledged that schools are under immense pressure to improve their performance against accountability measures and that this drives behaviour in schools. Several observed that teachers do their best to optimise the performance of their students at the C/D boundary, often at the expense of weaker pupils. Some suggested that the system also encourages teachers to mark internally assessed work as generously as possible.

One examiner observed that with schools under pressure to improve performance in league tables and exam boards competing for business, everyone is happy with increasing numbers of high GCSE and A level grades "except the pupil when he tries to get a job" (and finds that employers no longer value his high grades).

BEST FEATURES OF THE CURRENT EXAM SYSTEM

Examiners were asked to identify what they felt was the best feature of the current exam system. The most commonly cited feature was choice provided by different exam boards offering a variety of syllabuses. Transparency of the current system was also noted as a strength. Some examiners commented positively on the use of professional expertise and rigorous question paper setting, delivery and awarding procedures that support standards. Online assessment and innovation and development were also cited as positive features.

RECOMMENDATIONS FOR CHANGE

Examiners were asked to identify one thing that they would change about the current system. There was unanimous agreement that face-to-face standardisation meetings should be reintroduced. Several examiners suggested that the most important issue for the exam system is ensuring comparability of standards and preventing competition on standards between exam boards, although opinions varied on how best to address this. Several felt that commercial competition and profit-making should be removed from the examination system altogether.

Some suggested that the current pace of change is too rapid and leads to instability. One examiner was critical that changes in exams have often failed to take account of evidence of the impact of previous changes. He also suggested that the benefits of change should be weighed up against the cost of disruption. Several examiners felt that pressures from the accountability measures distort the exam system. One examiner suggested that all schools should have some staff involved in examining and that examining should be made part of professional development for teachers.


 
previous page contents next page


© Parliamentary copyright 2012
Prepared 3 July 2012