APPENDIX 1
Supplementary information given by OCR
on the level grade setting process (QCA 34)
The team of examiners who mark each paper attend
a "standardisation meeting". They are trained to mark
the paper as required by the Principal Examiner—to ensure
that each student is marked in exactly the same way regardless
of which school or college they are from or which examiner happens
to mark their paper.
After the meeting, examiners have approximately
three weeks to mark their allocation of scripts. Examiners are
not permitted to mark papers from their own school or college.
During marking, examiners send sample marked
scripts to their Team Leader. The Team Leader checks to see that
they are continuing to mark scripts as required. If problems are
identified, they are stopped from marking and their scripts given
to another examiner.
At the end of marking, all marks and scripts
are returned to the Board by post.
Shortly after the marking is complete, the Principal
Examiner for each paper will suggest to the Exam Board's Subject
Officer a range of marks within which they think the key judgemental
grade boundaries fall (NB and EJU at A-Level). They make their
suggestions on the basis of the scripts they have seen for the
question paper they have marked; they will not have any direct
experience of the other papers that make up the overall A or AS-Level.
When the Subject Officer has the suggested ranges
for each of the question papers (and for Coursework), he or she
carries out a "pre-award" review with the Chair of Examiners
using the statistical information available to check that they
appear to cover appropriate ranges of marks where the boundaries
might be expected to fall. The Subject Officer then arranges for
scripts at each mark point to be available at the grade awarding
meeting. When there is a new syllabus, a less experienced Principal
Examiner or a significant change to the student cohort, the Subject
Officer tends to err on the side of caution and ensures that scripts
are available above and below the suggested range, should the
Principal Examiner's judgement not be accepted by the other awarding
committee members.
The grade-awarding meeting usually takes place
about two working days after the end of marking. Those attending
are the Chief Examiner, Principal Examiners for each paper, the
Principal Moderator for Coursework unit(s), and a suitably qualified
examiner from a cognate subject at the same level.
The timing of the meeting is important: it may
be held before all the marks have been received at the Board.
Normal practice is that awarding meetings proceed if 80% of the
marks are entered onto the computer system.
The Code of Practice states the process to be
followed in awarding meetings. For each judgemental boundary,
the committee will look at the scripts within the range suggested
by each Principal Examiner. By looking at the bottom of the range
and working upwards and then from the top downwards, they identify
a "zone of uncertainty" within which they think the
boundary should fall.
That may prove to be at one end or the other
of any individual Principal Examiner's suggested range, or indeed
go outside the Principal's suggested range if the committee feels,
on the basis of its experience and evidence of the scripts, that
the boundary should be set at a higher or lower mark than had
been initially suggested by the Principal Examiner.
Key evidence are the archive scripts (if available):
these indicate the minimum level of work required for the award
of a grade A and E in a previous examination. The awarders use
that to ensure that their recommendations maintain the standard
year-on-year.
This process is followed for each paper (and
for coursework) until the committee have agreed recommendations
for all the unit boundaries at NB and EIU. As the recommended
unit boundaries are agreed, the Subject Officer enters these onto
the computer to see what the unit results would look like.
Once all the recommended boundaries are available
and entered, the Subject Officer can see what the outcomes for
the overall qualification look like. This is the first time that
the overall distribution of grades would be evident
Having seen the overall distribution, the Chair
indicates to the awarding committee whether or not the outcomes
appear to be in line with expectations given the nature of the
examination, the cohort of students taking it and evidence from
all of the examiners as to how the students this year compare
to last. If the outcomes do not align with those expectations,
the Chair would lead the meeting to reconsider the initial recommendations.
(The process is then repeated.)
When the Chair and awarding committee have unit
and overall outcomes in which they have confidence and believe
can be recommended to the Accountable Officer, the awarding committee
has completed its role.
The Chair of Examiners and the Subject Officer
then present the recommendations from the awarding committee to
the "Grade Endorsement Meeting" (the GEM), usually one
or two days after the awarding committee has finished. At the
GEM, the Accountable Officer or his representatives (due to the
number of meetings involved) questions the Chair about the recommendations,
unit by unit, using as reference points the statistical evidence
from this and previous years as well as the comments from the
awarding committee in relation on how student performance compares
to archive material.
Because the GEM happens at the end of the process,
the vast majority of marks will now be on the computer system
and the unit and overall qualification distributions will be very
accurate. The GEM team therefore considers the recommendations
in the light of the full statistical evidence available. The GEM
also has access to the outcomes for similar awards in related
subjects. This provides valuable points of comparison that ensure
consistency of standards across subject areas.
If the GEM team consider that the recommended
boundaries are not appropriate given the evidence (comments about
students' performance, what the statistical evidence is suggesting
about students' performance) they request further adjustments
to boundaries and task the Subject Officer with inputting changes
to see what the impact is on the unit and overall qualification
outcomes.
At the end of the GEM, the boundaries are "endorsed"
as being acceptable.
The final stage of the process is an overall
review of all of the outcomes from each subject area by the Accountable
Officer. This stage was introduced because the Accountable Officer
is not able to attend all of the GEMs but, as the person ultimately
responsible for "signing off" all of the awards for
OCR, considers it appropriate to see all of the recommended boundaries
before results processing occurs. Any adjustments will be made
in the light of the final statistical evidence available within
and between subject areas.
At the end of this process the boundaries are
"frozen" on the computer system and the students results
processed. Bulk production of the results electronically then
occurs, in readiness for distribution to schools and colleges
on the due date.
November 2002
| 
