Memorandum submitted by Roger Porkess
(QCA 32)
1. PERSONAL POSITION
1.1 The argument "Let sleeping dogs lie"
1.1.1 It can be argued that what happened
to A Levels this summer should be kept in perspective. Lots of
students don't get the examination results they are hoping for
or even perhaps deserve. Examining is not an exact science. Most
of those involved are now at University and it is much more important
for them to be looking forward and getting on with their new courses
than harking back to what might have been. So we should draw a
line under the whole episode and forget about it.
1.1.2 The danger with that argument is that
it allows precedent to be established on two key principles.
1.2 Adjusting module thresholds to influence
qualification outcomes
1.2.1 The grading problems occurred because
certain modules were marked down in order to reduce the numbers
of candidates getting particular A Level grades. This is a fundamentally
wrong thing to do in a modular syllabus.
(i) It breaks faith with the candidates,
in effect secretly moving the goal posts.
(ii) It is unsound examining practice since
it causes the candidates to be ranked incorrectly.
1.2.2 Modular A Levels have been around
for some 10 years, but, to my knowledge, never before have module
thresholds been adjusted to influence qualification outcomes.
1.3 Fairness to candidates
1.3.1 Until now it has always been a principle
of our examinations that the candidates' interests are paramount.
1.3.2 To the extent that it is humanly possible,
every effort has been made to ensure that each candidate receives
the correct grade.
1.3.3 This summer tens of thousands of candidates
have received incorrect grades but nothing is being done about
it, even though their grades could easily be set right.
1.4 The integrity of A Levels
1.4.1 The future integrity of A Levels can
only be guaranteed if these two key principles are re-established,
and that in turn depends on re-grading this summer's candidates.
2. BACKGROUND
2.1 Curriculum 2000
2.1.1 Work began on Curriculum 2000 in the
middle of 1998 and I was keen to do all I could to help make it
a success. During the later part of that year there were a number
of seminars on particular issues, most of which I attended. In
several cases I followed these up by writing discussion papers
to help QCA take matters forward.
2.1.2 One of these related to the question
of how to aggregate AS and A2 marks without causing grade inflation,
a matter which was causing concern to those designing Curriculum
2000. In that paper I suggested a mechanism that had worked well
with our MEI syllabus for the previous eight years. However, in
the event QCA neither accepted my suggestion nor any other but
let the curriculum go through with this flaw built in.
2.1.3 It became clear to me at that time
that those involved were uncomfortable with the mathematical nature
of the problem. At meetings eyes would glaze over. I suspect that
this had a lot to do with its never being resolved.
2.2 Syllabus submissions (1999-2000): specimen
papers
(This information is included in the light of
some of the questions to earlier witnesses)
2.2.1 Syllabuses (renamed "specifications")
were submitted in 1999 and most were approved towards the end
of that year although some dragged on into 2000. These submissions
including specimen examination papers and mark-schemes. In the
case of the MEI syllabus, the design thresholds are also written
into the approved syllabus.
2.2.2 It is thus untrue to say that the
AS and A2 standards were undefined. QCA put a great deal of effort
into looking at the specimen papers, and held up approval for
substantial periods of time until they were satisfied.
2.2.3 As it happens, in mathematics QCA
got the AS standard badly wrong, contributing to the very high
failure rate (29.1%) in June 2001.
2.2.4 I have seen no evidence of any attempt
by QCA to ensure comparability of standards across subjects. It
remains the case that pass rates are much higher in the arts subjects
than in the science; mathematics remains firmly at the bottom
of the list.
3. A CASE STUDY
FROM THIS
SUMMER
3.1 Rationale for this section
3.1.1 At this point I would like, as a case
study, to describe the events surrounding the award of one particular
syllabus. For reasons of confidentiality this is presented as
a separate Appendix.
4. MODELLING
THE SITUATION
4.1 Description
4.1.1 The rest of this submission is a report
that was issued on 15 November 2002.
4.1.2 Most of this describes the calculations
that led me to the conclusion that tens of thousands of candidates
have received a lower grade than they should have done.
4.2 Calculations
4.2.1 While the actual calculations are
correct, their validity depends upon assumptions about data which
are held by the examinations boards and are not in the public
domain.
4.2.2 Publication of full data would allow
a more accurate estimate of the number of candidates affected
to be made. In the absence of such data, these figures stand.
4.2.3 An exact answer to the question "How
many candidates?" can only be obtained by re-grading all
syllabuses.
Appendix to Section 4 Report: The effect
of moving grade thresholds
SUMMARY
The Tomlinson Inquiry restricted its scope to
the most extreme movements of grade thresholds.
Consequently many of this summer's candidate's
have lost an A Level grade.
As a mathematician I estimate the number affected
to be over 20,000.
BACKGROUND
During this summer's A Level award, there were
many cases where the thresholds set by Awarding Committees were
subsequently made substantially more severe by the examination
boards.
The reason for this was to ensure that the numbers
of students getting high grades were in line with those in 2001,
before Curriculum 2000 was introduced.
Because Curriculum 2000 is modular, where action
was taken it involved particular modules take in June 2002.
Adjusting the results on particular modules
to influence the overall outcome is an intrinsically unsound practice;
it introduces inconsistency in standards across modules and discriminates
against candidates who took certain modules.
When the Tomlinson Inquiry was announced many
of us expected that in all cases where grade thresholds had been
moved the original thresholds would be restored, and candidates
re-graded. This did not happen.
Instead a cut-off was decided upon. Only those
modules with threshold shifts of six marks or more were considered
for re-grading (and two others where there had been many complaints).
The application of the cut-off will inevitably
have left some candidates with a lower A Level grade than would
have been the case if all thresholds had been restored to those
set by the Awarding Committees.
The Tomlinson Inquiry did not address the question
of how many students lost a grade in this way.
There is also evidence that where re-grading
did occur, the original thresholds were not fully restored.
A STATISTICAL ESTIMATE
The Tomlinson Inquiry set a cut-off point of
a threshold adjustment of six marks on a module examination. A
natural group to consider are those just below this cut off point.
Here is a question.
"Thresholds on AS/A modules are increased
five marks at one sitting. What percentage of candidates lose
an A Level grade as a consequence?"
Until now this question would seem not to have
been answered.
Perhaps the reason is because there is no single
neat mathematical answer. It depends on the mark distribution
for the particular A Level syllabus this summer, on how tightly
the thresholds are packed together for the modules in question
and on how many modules a candidate took at that sitting.
To deal with such a problem you need to make
realistic assumptions, in this case about the mark distribution,
the spacing of the thresholds and the number of modules taken,
and then work through the consequences.
— The mark distribution is assumed
to be that in the attached graph.
— The module thresholds are taken to
be five marks apart, so that the Tomlinson cut off point corresponds
to one module grade.
— The threshold adjustment is made
only at the grade A boundary.
— Candidates take two examination modules.
All of these are reasonable assumptions, and
they lead to the conclusion that 16.1% of candidates of that syllabus
would have lost an overall A Level grade.
The next stage is to vary the assumptions and
so obtain a range of values.
— Looking at other mark distributions
gave a range of outcomes: 16.8%, 14.8% and for a very bottom-heavy
distribution, 12.4%. A realistic "average" figure would
seem to be about 15%.
— In coursework modules the thresholds
are usually much closer together and so the final outcome would
be higher, over 20% if one of the modules is coursework.
— In some subjects threshold adjustments
were made to both A and E boundaries (and so to all those in between).
In that case the final outcome would be about twice as large and
so could be over 30%.
— Very few candidates would have taken
only one module in June, but quite a lot took three as some schools
had forbidden January entries. The final outcome for those who
took three would have been one and a half times as great, so over
20%.
In conclusion, a low estimate of the percentage
of candidates losing an A Level grade in such a "cut-off"
syllabus is 15%, and it could be quite a lot higher.
The Tomlinson Inquiry identified syllabuses
from all three examination boards which had had threshold adjustments
above the cut-off level, but by far the majority of them were
in one board, and in that one board it would seem that the average
threshold adjustment was about five marks per module.
So an estimate of the number of candidates who
lost an A Level grade from that one board alone is 15% of the
total A Level entry
or about 35,000 candidates.
There will be some more from the other two boards
but the available data are rather restricted, making it hard to
do more than guess at the number. I prefer to stay with 35,000
than to guess higher.
The assumptions that underlie that figure have
been deliberately on the cautions side. As a further act of caution,
I will allow a large margin for error, and conclude that the evidence
suggest a figure in excess of 20,000.
December 2002
APPENDIX
AN EXAMPLE
When an examination paper is marked it is given
a raw mark. This is then converted into a uniform mark, which
is independent of the difficulty of the paper. In the conversion,
the value of one raw mark varies according to the grade band width.
A grade band width of five raw marks is quite
common and this converts into 10 uniform marks. In that case one
raw mark is worth two uniform marks. In most cases this is close
to reality and so provides a helpful rule-of-thumb.
However, in coursework modules band widths may
be as small as two raw marks, and in that case one raw mark is
worth five uniform marks.
The effect of any change in candidates' uniform
marks on their grading is illustrated for a typical distribution
on the attached graph. In this example the cut-off for two modules
is taken to correspond to 20 uniform marks.
— The black vertical lines are drawn
at the aggregated A Level thresholds of 240 (E), 300 (D), . .
., 480 (A).
— The red vertical lines illustrate
the effect of a change of 20 uniform marks at the A thresholds,
with proportional effects at B, C and D.
— The grey shaded regions represent
those candidates losing a grade. In this example, 16.1% of the
candidates fall into this category.
Roger Porkess
— Roger Porkess is a mathematician
and a Fellow of the Royal Statistical Society.
— He is Project leader for Mathematics
in Education and Industry (MEI), a long established independent
curriculum development body and is responsible for one of the
largest Mathematics A Level syllabuses (MEI Structured Mathematics).
— He was responsible for the development
of the first modular A Level in any subject; this established
the principles for the assessment of such courses and is the model
upon which Curriculum 2000 is based.
— He has long experience of examining,
as a setter, revisor, awarder and marker.
— He is author, co-author or series
editor of over 50 books, mostly on mathematics, and contributed
numerous articles to various journals.
— In his earlier career he taught mathematics
in a variety of schools in the UK and third world countries.
— Being employed by an independent
body, allows him a freedom of speech and association on professional
matters that few others in the examination world enjoy.
| 
