Annex 1: Note of the Committee's seminar
with examiners, 14 December 2011
These notes are a general account of the opinions
expressed by a group of examiners, who met with the Committee
for an informal discussion. Examiners represented all three exam
boards, a range of subjects and a variety of examiner roles.
The examiners were sifted from over two hundred applications
received by the Committee in response to an advertisement in the
Times Educational Supplement and online. All examiners not selected
to attend were invited to complete a short questionnaire covering
similar areas to those discussed at the seminar.
Comments in double inverted commas indicate a direct
verbatim quotation, although these are not attributed.
EXAMINER RECRUITMENT
There was general agreement that "lots of assessment
requires lots of examiners" and that this has led to shortages
in some subjects.
Some examiners felt that recruitment requirements
have become less stringent in recent years. Previously, teachers
needed at least 3 years' teaching experience before they could
become an examiner. Some examiners reported that they were recruited
in their first year of teaching or even during postgraduate training.
Others argued that recruitment requirements are stringent enough
as it is "easy to get rid of" poor examiners. However,
one senior examiner expressed doubt as to whether all weak examiners
are dismissed in shortage subjects, due to difficulties in recruiting
sufficient numbers.
Some concern was expressed that too many examiners
are retired rather than practising teachers. There was general
consensus that it is important to have a balance between practising
teachers and retirees. However, the main obstacle for practising
teachers is time. The summer exam series coincides with a busy
period in school for many teachers and examining (at least at
more junior levels) is not sufficiently well paid to make it worth
their while. Some suggested that being an examiner should be a
recognised part of teachers' continuing professional development.
Senior examiners recounted from experience that it
is very difficult to combine senior examining roles with leadership
roles in schools or full-time teaching. This means that many question
paper setters are no longer teaching. This was felt to be a shame
as it is important that examiners are sensitive to the impact
in the classroom of the examinations they are setting.
QUESTION PAPER SETTING
Examiners generally agreed that the question paper
setting process is rigorous. It was pointed out that some of the
errors in 2011 papers were introduced in the later stages of question
paper setting (eg changes to graphs with accompanying text not
updated).
ONLINE STANDARDISATION
Examiners were unanimously of the view that the loss
of face-to-face standardisation meetings was "a big mistake".
There was consensus that face-to-face standardisation meetings
helped examiners to explore questions about the mark scheme in
depth and get a feel for how the Principal Examiner was interpreting
the mark scheme. Senior examiners felt that face-to-face standardisations
meetings were useful for "talent spotting" as they helped
them to identify potentially very good new examiners.
CONTROLLED ASSESSMENT
Examiners were generally negative about controlled
assessment, although one senior examiner felt that controlled
assessment had worked well in his subject (Drama). Many examiners
were critical of inconsistencies between the exam boards in the
rules for controlled assessment. Several complained that controlled
assessment was hard to manage, dominating much of the school year
for pupils in years 10 and 11 and also that it was impossible
to be sure that a piece of work was a child's own. Generally,
examiners felt that controlled assessment had failed to address
the problems with its predecessor, coursework, and that, in some
respects, it had generated problems of its own.
GRADE AWARDING
Senior examiners felt that there were many public
misconceptions about how grades are awarded and that few people
understood how data are used. There was some variation in examiners'
views on the importance attached to statistical information and
examiners' judgement of candidates' work in awarding. Some examiners
felt that it was right that awarding was data-led and that statistical
information on the previous attainment of candidates and school
predictions should be more significant that the judgement of a
small group of examiners on candidates' work. Others were concerned
that candidates should be judged primarily on the standard of
work achieved.
COMMERCIAL ACTIVITIES OF EXAM BOARDS
Several examiners (also practising teachers) suggested
that exam boards spend a lot on marketing to schools, often very
aggressively, in order to win market share. Some reported that
market researchers from exam boards tried to determine which elements
of a subject were the most popular (e.g. which period of history,
novel in English) and incorporate these into their syllabuses.
TEXTBOOKS
There was general agreement that endorsed textbooks
are problematic, although views varied on whether they should
be allowed to continue. Some examiners felt that exam board/publisher
partnerships, such as AQA/Nelson Thornes, were wrong and that
exam boards should not make money from publishing agreements.
Examiners suggested that the Committee could usefully explore
how much money exam boards make from textbooks, compared to the
core business of running exams.
Many examiners held the view that exam boards should
not endorse particular textbooks and that publishers should not
market books as "written in line with" a particular
syllabus. Some felt that it was very misleading for exam boards
to suggest that their endorsed or, in one case, own textbooks
were all that schools needed, as well as potentially damaging
to teaching and learning (by encouraging teaching to the test/book).
Some participants were also textbook authors. Several
reported pressure from publishers to write to a particular examination,
although they recognised the pressure on publishers and exam boards
to supply what teachers wanted. There was disagreement about whether
examiner authorship represented a conflict of interest, with examiners
having insider knowledge. Some agreed with this, others argued
that as senior examiners deal with so many exam questions across
different exam series, insider knowledge is not an issue.
One examiner author said that he hoped his textbook
gave an insight into the subject which would apply to all syllabuses.
It was suggested that endorsed textbooks are more of an issue
in some subjects than others. For example, English syllabuses
are broadly similar across all exam boards, whereas in science
there is more variation between exam boards, with textbooks linked
to specific syllabuses and chapters for each module.
Some participants questioned the quality of some
endorsed textbooks, pointing out that good examiners do not necessarily
make good writers and that "children deserve good textbooks".
Others noted that it was difficult in some subjects for teachers
to find a textbook not linked to a particular exam board.
There was also concern about the cost of textbooks and the problem
that "if you change syllabus, you need new textbooks".
TRAINING
Examiners stated that they now have more contact
with teachers than they did 20 years ago. Exam boards run more
feedback meetings and there is greater transparency about the
system with more materials available.
It was pointed out that many senior examiners are
self-employed and may use textbooks and training to supplement
their income from examining.
There were reports of some senior examiners (not
participants) using their position to promote training to schools
offered by private training providers, with courses entitled "changing
your Ds to Cs" and "getting an A* grade" or intensive
A level preparation courses. It was generally felt that this was
inappropriate. By contrast, delivering general training on GCSE
which covered all exam boards, was felt to be acceptable. Some
pointed out that OCR contracts prohibit examiners from using their
status as examiners to promote training and that Pearson is making
changes to its examiner contracts from April 2012. However, it
was pointed out that is difficult for examiners not to
divulge their position as many teachers will know the names of
senior examiners for their syllabus.
COMPETITION BETWEEN EXAM BOARDS
It was suggested that commercial companies will always
market aggressively and find a way of playing the system.
Examiners expressed concern about the comparability
of standards between exam boards in some areas, suggesting that
there are "pockets where the differences are significant".
Several agreed that "Ofqual has a case to answer" in
terms of exam boards competing on standards.
REFORM OF THE EXAMINATION SYSTEM
There was no agreement between examiners on whether
the exam system should be reformed or how. Some were in favour
of a single state-run exam board (with regulation), others favoured
retaining the multiple exam board system. One examiner expressed
a preference for franchising by subject.
Several examiners felt that teachers wanted a choice
of syllabuses and being able to switch exam boards was a helpful
tool at teachers' disposal. Innovation and development were also
cited as arguments in favour of a multiple board system. However,
some examiners felt that the multiple board system carried the
risk of competition on grade standards and that in this respect
"Ofqual has a case to answer".
Examiners suggested that other countries where exams
were run by a single state provider (e.g. Scotland, South Africa)
might prove useful comparators, although others cautioned that
systems which work well for countries with much smaller populations
may not translate well into the English context and that our experience
of a top-down national system with national curriculum tests has,
on occasions, been "catastrophic".
Examiners observed that a franchised system would
lose the elements of choice currently available to teachers and
that system failure would affect larger numbers of candidates.
However, some suggested that it could prove helpful to children
moving schools as the same syllabus would be taught in all schools.
There was general agreement that better regulatory
oversight by Ofqual was vital to the success of any model, whether
a single board, franchising by subject or multiple board system.
IMPACT OF THE ACCOUNTABILITY SYSTEM
Examiners acknowledged that schools are under immense
pressure to improve their performance against accountability measures
and that this drives behaviour in schools. Several observed that
teachers do their best to optimise the performance of their students
at the C/D boundary, often at the expense of weaker pupils. Some
suggested that the system also encourages teachers to mark internally
assessed work as generously as possible.
One examiner observed that with schools under pressure
to improve performance in league tables and exam boards competing
for business, everyone is happy with increasing numbers of high
GCSE and A level grades "except the pupil when he tries to
get a job" (and finds that employers no longer value his
high grades).
BEST FEATURES OF THE CURRENT EXAM
SYSTEM
Examiners were asked to identify what they felt was
the best feature of the current exam system. The most commonly
cited feature was choice provided by different exam boards offering
a variety of syllabuses. Transparency of the current system was
also noted as a strength. Some examiners commented positively
on the use of professional expertise and rigorous question paper
setting, delivery and awarding procedures that support standards.
Online assessment and innovation and development were also cited
as positive features.
RECOMMENDATIONS FOR CHANGE
Examiners were asked to identify one thing that they
would change about the current system. There was unanimous agreement
that face-to-face standardisation meetings should be reintroduced.
Several examiners suggested that the most important issue for
the exam system is ensuring comparability of standards and preventing
competition on standards between exam boards, although opinions
varied on how best to address this. Several felt that commercial
competition and profit-making should be removed from the examination
system altogether.
Some suggested that the current pace of change is
too rapid and leads to instability. One examiner was critical
that changes in exams have often failed to take account of evidence
of the impact of previous changes. He also suggested that the
benefits of change should be weighed up against the cost of disruption.
Several examiners felt that pressures from the accountability
measures distort the exam system. One examiner suggested that
all schools should have some staff involved in examining and that
examining should be made part of professional development for
teachers.
| 
