The administration of examinations for 15-19 year olds in England - Education Committee Contents


Annex 2: Examiners' questionnaire responses


The Committee received over 200 expressions of interest in response to the advert in the Times Educational Supplement (TES) and online for the examiners' seminar held on 14 December 2011. A copy of the questionnaire was sent to all those who were not selected or who were unable to attend on 14 December.

The questionnaire covered areas similar to those discussed at the seminar. These were: examiner recruitment and performance, question papers, marking, grading and the commercial activities of the exam boards. In common with the seminar, examiners were asked to identify one thing which they would change about the current system.

The Committee received 45 completed questionnaires. It should be noted that this is a self-selecting sample with very small numbers, so the findings must be treated with caution. Some examiners did not respond to all questions or sub-questions. The responses capture a "flavour" of examiners' views about the system, in much the same way as the seminar. Some points raised in questionnaire responses echo those made by examiners at the seminar and this is noted below where applicable.

Just under half of examiners who responded (49%) were teachers in schools/colleges. 9% were teaching in higher education. 22% described themselves as self-employed and 20% said that they were retired.

EXAMINER RECRUITMENT

The most commonly cited requirement was appropriate subject expertise and 3 years' teaching experience. 40% of responses mentioned that they had to demonstrate 3 years or more teaching experience. 24% said that they were required to have less than 3 years' experience.

No respondents said that entry requirements had become more stringent. Over half didn't comment, but some (27%) thought that requirements had become less stringent, while others (20%) felt that there had been no change.

EXAMINER PERFORMANCE

Most examiners (82%) were positive about the way their performance was monitored by the exam boards. A few (11%) noted that feedback varied between exam boards and a minority (7%) were unhappy with the feedback given.

QUESTION PAPERS

Just over half of respondents (53%) felt that question paper setting and checking procedures were rigorous and several observed that there are remarkably few errors given the number of papers produced each year. One examiner noted that the setting and checking system relies heavily on one or two people and that, "ironically, it's the number of checks of a paper that can lead to an oversight, usually based on the assumption that you have seen it so many times before."

Just over a third of respondents (36%) were critical of the question paper setting and checking process, citing the sheer number of exams, short timescales and pressure to meet deadlines, along with cost-cutting by exam boards as possible reasons for errors. The remaining 11% did not comment.

MARKING

Over half of respondents (56%) expressed confidence overall in the reliability of marking, although as one examiner noted "there is no room in terms of time for any disasters. It would only take one senior examiner to go ill/have a family crisis in July and the reliability of the system would be suspect". 22% were less confident in the reliability of marking. Reasons cited tended to be related to extended answers and the "benefit of the doubt" marking culture. 11% said that marking was variable and 11% did not comment in their response.

Just under half of respondents (47%) felt that online assessment had been a positive development overall, albeit with some reservations. However, 31% were negative about online marking. Most commonly cited objections related to the marking of extended essays. Several examiners did not like the fact that online marking means that they no longer see whole scripts and cannot go back over an answer. 22% did not comment on online assessment.

Just over half of respondents (51%) complained about online standardisation, echoing criticisms made at the examiners' seminar. One examiner stated that "I found that online standardisation did not prepare me in the same way as the traditional meeting did. I did not feel as secure in the standard or implementation of the mark scheme". Senior examiners expressed similar reservations and also complained that online standardisation "deprives senior examiners of a valuable opportunity to mentor junior examiners and point out potential team leaders of the future".

GRADING

Examiners' views on standards over time varied. Most respondents thought either that standards had declined over time (38%) or felt that it was difficult to say with certainty if standards had changed (42%). As one examiner put it, "today's candidates do not demonstrate a significantly higher level of ability and yet pass rates and higher grades are achieved by many more". Both groups attributed this to a range of factors, relating to changes in the examination system, such as more accessible and predictable questions, changes in content, modular assessment and increased opportunities for re-sits. Many felt that students were better prepared, due to greater transparency about the system and the effect of performance measures. One history examiner noted that it is "very difficult to quantify what has happened to standards. The average student is certainly achieving a higher standard than was the case in the past, but teachers have more help in achieving this." A minority (7%) felt that examinations had got harder and the remaining 13% did not comment on standards over time.

Examiners varied in their views on comparability between exam boards. 28 mentioned this in their responses. Of these, 43% felt that standards were comparable between exam boards. One examiner suggested that this had improved in his subject as in the past "there was wild inconsistency and certainly no golden age". In contrast, 57% felt that standards were not comparable between exam boards, which is why, as one examiner put it, schools "go board fishing".

Three respondents pointed to issues of comparability between subjects, with one examiner noting that "at present, over 50% of entries in Latin obtain an A, over 40% in Mathematics, over 30% in Chemistry and under 20% in Psychology". The examiner called for more research into why the % of grade As varies so much between subjects, as he is not "convinced that this difference is totally justified".

COMMERCIAL ACTIVITIES

Just under half of respondents (49%) said that they had contributed to textbooks.

Examiners were divided in their views on endorsed textbooks. 56% expressed concern about endorsements, with several suggesting that it was a conflict of interest between two functions of exam boards. Many felt that endorsed textbooks had a negative impact on teaching and learning, by encouraging teaching to the exam/book. As one examiner stated "as general texts these books are woeful; as guides to passing the exam they are very useful".

44% of respondents were positive about endorsed textbooks, suggesting that they were a useful resource for teachers. As one examiner stated "I am not sure why this should be deemed a conflict of interests. We want our students to do well, therefore having insight into what the exam board are going to give them, is helpful". One examiner expressed concern that if textbooks moved away from exam boards, then this would increase the risk of others "producing inaccurate material, charging the earth and confusing the heck out of teachers".

Just over a fifth of respondents (22%) suggested that Ofqual should do more to regulate in this area, although a few examiners (7%) felt that Ofqual should not regulate "as other less scrupulous groups could move in".

ONE "THING TO CHANGE"

Responses varied considerably. The change called for most often (by 18% of respondents) was to replace online standardisation with face-to-face meetings, echoing the message from the examiners' seminar. 16% of respondents called for various changes to improve examiners' terms and conditions, such as a longer marking period or improved pay. One examiner called for a "jury service" for markers to improve marking reliability.

A third of respondents called for a change linked to organisation of the system. However, suggested changes varied from a single exam board (9%), limiting competition between exam boards (4%) and less Government interference (4%) to keeping multiple exam boards (9%) keeping publishing separate from examining (2%) and controlling the commercial activities of exam boards (2%). One examiner called for franchising by subject.






 
previous page contents next page


© Parliamentary copyright 2012
Prepared 3 July 2012