Annex 2: Examiners' questionnaire responses
The Committee received over 200 expressions of interest
in response to the advert in the Times Educational Supplement
(TES) and online for the examiners' seminar held on 14 December
2011. A copy of the questionnaire was sent to all those who were
not selected or who were unable to attend on 14 December.
The questionnaire covered areas similar to those
discussed at the seminar. These were: examiner recruitment and
performance, question papers, marking, grading and the commercial
activities of the exam boards. In common with the seminar, examiners
were asked to identify one thing which they would change about
the current system.
The Committee received 45 completed questionnaires.
It should be noted that this is a self-selecting sample with very
small numbers, so the findings must be treated with caution. Some
examiners did not respond to all questions or sub-questions. The
responses capture a "flavour" of examiners' views about
the system, in much the same way as the seminar. Some points raised
in questionnaire responses echo those made by examiners at the
seminar and this is noted below where applicable.
Just under half of examiners who responded (49%)
were teachers in schools/colleges. 9% were teaching in higher
education. 22% described themselves as self-employed and 20% said
that they were retired.
EXAMINER RECRUITMENT
The most commonly cited requirement was appropriate
subject expertise and 3 years' teaching experience. 40% of responses
mentioned that they had to demonstrate 3 years or more teaching
experience. 24% said that they were required to have less than
3 years' experience.
No respondents said that entry requirements had become
more stringent. Over half didn't comment, but some (27%) thought
that requirements had become less stringent, while others (20%)
felt that there had been no change.
EXAMINER PERFORMANCE
Most examiners (82%) were positive about the way
their performance was monitored by the exam boards. A few (11%)
noted that feedback varied between exam boards and a minority
(7%) were unhappy with the feedback given.
QUESTION PAPERS
Just over half of respondents (53%) felt that question
paper setting and checking procedures were rigorous and several
observed that there are remarkably few errors given the number
of papers produced each year. One examiner noted that the setting
and checking system relies heavily on one or two people and that,
"ironically, it's the number of checks of a paper that can
lead to an oversight, usually based on the assumption that you
have seen it so many times before."
Just over a third of respondents (36%) were critical
of the question paper setting and checking process, citing the
sheer number of exams, short timescales and pressure to meet deadlines,
along with cost-cutting by exam boards as possible reasons for
errors. The remaining 11% did not comment.
MARKING
Over half of respondents (56%) expressed confidence
overall in the reliability of marking, although as one examiner
noted "there is no room in terms of time for any disasters.
It would only take one senior examiner to go ill/have a family
crisis in July and the reliability of the system would be suspect".
22% were less confident in the reliability of marking. Reasons
cited tended to be related to extended answers and the "benefit
of the doubt" marking culture. 11% said that marking was
variable and 11% did not comment in their response.
Just under half of respondents (47%) felt that online
assessment had been a positive development overall, albeit with
some reservations. However, 31% were negative about online marking.
Most commonly cited objections related to the marking of extended
essays. Several examiners did not like the fact that online marking
means that they no longer see whole scripts and cannot go back
over an answer. 22% did not comment on online assessment.
Just over half of respondents (51%) complained about
online standardisation, echoing criticisms made at the examiners'
seminar. One examiner stated that "I found that online standardisation
did not prepare me in the same way as the traditional meeting
did. I did not feel as secure in the standard or implementation
of the mark scheme". Senior examiners expressed similar reservations
and also complained that online standardisation "deprives
senior examiners of a valuable opportunity to mentor junior examiners
and point out potential team leaders of the future".
GRADING
Examiners' views on standards over time varied. Most
respondents thought either that standards had declined over time
(38%) or felt that it was difficult to say with certainty if standards
had changed (42%). As one examiner put it, "today's candidates
do not demonstrate a significantly higher level of ability and
yet pass rates and higher grades are achieved by many more".
Both groups attributed this to a range of factors, relating to
changes in the examination system, such as more accessible and
predictable questions, changes in content, modular assessment
and increased opportunities for re-sits. Many felt that students
were better prepared, due to greater transparency about the system
and the effect of performance measures. One history examiner noted
that it is "very difficult to quantify what has happened
to standards. The average student is certainly achieving a higher
standard than was the case in the past, but teachers have more
help in achieving this." A minority (7%) felt that examinations
had got harder and the remaining 13% did not comment on standards
over time.
Examiners varied in their views on comparability
between exam boards. 28 mentioned this in their responses. Of
these, 43% felt that standards were comparable between exam boards.
One examiner suggested that this had improved in his subject as
in the past "there was wild inconsistency and certainly no
golden age". In contrast, 57% felt that standards were not
comparable between exam boards, which is why, as one examiner
put it, schools "go board fishing".
Three respondents pointed to issues of comparability
between subjects, with one examiner noting that "at present,
over 50% of entries in Latin obtain an A, over 40% in Mathematics,
over 30% in Chemistry and under 20% in Psychology". The examiner
called for more research into why the % of grade As varies so
much between subjects, as he is not "convinced that this
difference is totally justified".
COMMERCIAL ACTIVITIES
Just under half of respondents (49%) said that they
had contributed to textbooks.
Examiners were divided in their views on endorsed
textbooks. 56% expressed concern about endorsements, with several
suggesting that it was a conflict of interest between two functions
of exam boards. Many felt that endorsed textbooks had a negative
impact on teaching and learning, by encouraging teaching to the
exam/book. As one examiner stated "as general texts these
books are woeful; as guides to passing the exam they are very
useful".
44% of respondents were positive about endorsed textbooks,
suggesting that they were a useful resource for teachers. As one
examiner stated "I am not sure why this should be deemed
a conflict of interests. We want our students to do well, therefore
having insight into what the exam board are going to give them,
is helpful". One examiner expressed concern that if textbooks
moved away from exam boards, then this would increase the risk
of others "producing inaccurate material, charging the earth
and confusing the heck out of teachers".
Just over a fifth of respondents (22%) suggested
that Ofqual should do more to regulate in this area, although
a few examiners (7%) felt that Ofqual should not regulate "as
other less scrupulous groups could move in".
ONE "THING TO CHANGE"
Responses varied considerably. The change called
for most often (by 18% of respondents) was to replace online standardisation
with face-to-face meetings, echoing the message from the examiners'
seminar. 16% of respondents called for various changes to improve
examiners' terms and conditions, such as a longer marking period
or improved pay. One examiner called for a "jury service"
for markers to improve marking reliability.
A third of respondents called for a change linked
to organisation of the system. However, suggested changes varied
from a single exam board (9%), limiting competition between exam
boards (4%) and less Government interference (4%) to keeping multiple
exam boards (9%) keeping publishing separate from examining (2%)
and controlling the commercial activities of exam boards (2%).
One examiner called for franchising by subject.
| 
