Supplementary memorandum submitted by
Ofqual
On the further points required:
Timetable for the marker training and the impact
that the face-to-face rather than online had on the timetable
Marker training has had a fairly fixed cascade
for several years, culminating in the majority of markers being
trained during the first two weekends in May. For 2008, the earliest
formal schedule of which Ofqual had sight of is from mid-November
2007. It clearly indicates that the final marker training meetings
were scheduled for 10 and 17 May 2008.
Whether switching to a new online training model,
or remaining with the established face-to-face model ETS would
have been aware of the need to meet the dates outlined in this
schedule, and indeed in the Proof of Concept Pilot—Design
document (May 2007), section ix details the ETS contingency
"in the event the pilot proves it does not validate the acceptance
criteria" would be a series of face-to-face training sessions.
How long did it take to conduct the face-to face
training of all markers?
As in previous years the majority of face-to-face
training sessions take place on a single day, for 2008 either
the 10th May (for key stage 3 markers) or the 17th May (for key
stage 2 markers). The vast majority of markers (around 5,000 per
key stage) are trained on these days at a variety of training
locations around the country (around 50 different venues in total).
This is the culmination of a five month cascade where by the Marking
Programme Leader develops the training materials with her deputies
(meeting 1, 2 and 3), who then train the senior markers (meeting
4 and 5), who then train team leaders (meeting 6 and 7) who then
train markers (meeting 8).
Supervising markers (seniors and team leaders)
get two days of training. One day of training on the mark scheme,
and the second day of training for their role as supervisors.
All training days usually run from 9.30 until 16.30/17.00.
Did the NAA reported to us that ETS had said that
the change from online to face-to-face training would lead to
a long delay?
At the meeting with NAA on 19 Feb 2008, the
regulator was given reassurances to the opposite at the meeting
with NAA to discuss the results of the proof of concept pilot,
and as early as January, NAA had assured us that venues were already
being sourced as a contingency in line with the steps detailed
in section xi of the Proof of Concept—Design document.
At what point were markers trained in the handling
of what we considered to be over-ambitious software? Was that
done in early May as well?
Markers were supposed to have completed an online
training module before they arrived for their face-to-face training
in May. As this was to be conducted from home, this wasn't something
we could observe, though we have heard reports that markers had
considerable problems with this system. There also seemed to be
no effective method of confirming exactly who had and who hadn't
completed this module before markers attended their face-to-face
training.
The training module did not go live in time
for supervising markers to have experienced the system, meaning
they were unable to answer queries on the system at the training
meetings in May. Any queries on the system had to be answered
by ETS representatives, who weren't always present at the training
meetings. This left the majority of our observers unconvinced
that markers had been given enough training on the new ETS software
systems.
Also for clarification
Markers entered "question level data"
rather than "item level data" onto the OMC system.
Q206—number of interventions
The Regulation & Standards Division of QCA
had no formal involvement in the process of awarding the test
operations contract for 2008-12, although Ofqual's Acting Chief
Executive, Isabel Nisbet, was a member of the QCA Executive at
the time the contract was awarded. From the summer of 2007, the
National Curriculum Assessments monitoring team ("NCA monitoring")
was invited to comment on planned changes which set out the ways
in which markers were to be supervised and quality assured. NCA
monitoring raised some concerns at that early stage with NAA.
On 2 November 2007, NCA monitoring provided
feedback to NAA on marker training for the marking pilot which
said that communications in the broadest sense gave rise for concern.
For example, how contacts with markers were managed; ease of access
for venues.
ETS carried out a "proof of concept pilot",
examining the effect on the quality of marking of four new approaches
to on-line standardisation and quality assurance. In February
2008, NCA monitoring, together with the Director of Regulation
& Standards, Isabel Nisbet, attended two presentations on
the outcomes of the pilot. On 21 February, Isabel Nisbet wrote
to the Director of NAA (David Gee) expressing the regulator's
support for the four approaches which had been trialled, but conveying
two "significant concerns". The first was a systems—based
concern that the volume of marker use at key points in the marking
process might compromise the ability of ETS to operate effectively.
This was to be tested through load testing and NCA Monitoring
asked to be kept informed of the outcomes. The second concern
was that the delays experienced in delivering the pilot and its
report could indicate that the resources being assigned by ETS
[were] insufficient to meet the required deadlines and standards
of quality.
David Gee replied on 6 March, agreeing to provide
updates on load testing and sharing the regulator's concerns about
the management of the pilot, but commenting that it was managed
by a different team in ETS than would be involved in the 2008
operational cycle. On 21 April, in response to a further request
from the NCA monitoring team for an update on load testing, David
Gee wrote that significant work was done by ETS over the previous
six weeks, and concluded that the data would provide assurances
that preparation is well in hand.
In April 2008 Ofqual was set up within QCA.
On 28 April Kathleen Tattersall, newly-appointed Chair of Ofqual,
had an introductory meeting with Ed Balls, Secretary of State
for Children, Schools and Families. In a general overview of the
issues facing Ofqual over the summer period, Kathleen Tattersall
mentioned, among other things, the heightened risk of there being
a new contractor responsible for the delivery of the National
Curriculum tests.
The NCA monitoring team observed a sample of
the ETS marker training events and, following a training session
on 10 May 2008, reported concerns to NAA about attendance of markers
and the rigour of their selection. During May, conversations with
markers alerted the team to delays in the delivery of scripts
for marking and to problems in communications with ETS.
On 3 June 2008, Isabel Nisbet wrote to David
Gee to seek reassurance from NAA that marking of National Curriculum
tests will be completed and the scripts returned to schools by
the deadline of Tuesday 8 July 2008. Specific reassurances were
sought about aspects of marking, communications and administration
on marker training. The letter also informed NAA that Ofqual would
hold a formal accountability meeting, chaired by Kathleen Tattersall,
with NAA on 3 July. The decision that Ofqual's chair should chair
the meeting with NAA was made on the basis of risk.
David Gee replied on 11 June addressing each
of the specific questions raised in Ofqual's letter of 3 June
His letter stated that the NAA was heavily focused on ensuring
the delivery of results to schools by 8 July. Despite a number
of challenges created by inadequacies in the ETS delivery process
system David Gee had extensive reassurances that this would be
achieved. However, in order to reduce any risks further he dedicated
significant additional NAA resource to support ETS in meeting
its contractual obligations.
On 11 June, the NCA monitoring team took up
a long-standing invitation to visit the distribution centre in
Dewsbury. Following that meeting the NCA team raised concerns
with NAA about the apparent absence of control processes in the
management of scripts.
In mid-June, DCSF asked Ofqual for a short briefing
note to show to Jim Knight, Minster of State for Schools and Learners,
before a Ministerial meeting with NAA about test delivery. Ofqual's
note stated that it was our clear impression at that time that
there is a high risk that all schools will not receive their results
by 8 July as a result of marking not being completed and problems
with the ETS marking and distribution systems. The note went on
to report problems with preparations for the following week's
level setting meetings because of insufficient marks on the ETS
data system.
On 2 July 2008, David Gee wrote to Isabel Nisbet
notifying Ofqual that NAA would be in breach of one requirement
of the Code of Practice because some schools would not receive
all their data by the published date.
The accountability meeting took place on 3 July.
Following the meeting, Kathleen Tattersall decided that Ofqual
should set up an independent inquiry into the regulation and delivery
of the 2008 tests, and on 4 July she wrote to the Secretary of
State informing him of that decision. Her letter also gave some
reassurance about the quality of the marking of the tests:
"While results will be delayed and I cannot
predict the volume of reviews that schools will request this year,
from the processes we have observed, the quality of marking is
at least as good as previous years and justifies issuing the results."
On 4 July, NCA monitoring requested from NAA
details of the marking panels that ETS needed to set up to complete
the marking process and also details of the quality assurance
arrangements for these panels. Visits were made to marking panels
on 16 July.
Starting on 25 July 2008, Ofqual has held weekly
regulatory meetings with NAA (observed by DCSF) to monitor progress
in identifying and marking unmarked scripts and in the review
processes. Those meetings continue. A problem, particularly in
the earlier meetings, was the difficulty of obtaining a precise
estimate of the total number of unmarked scripts and assurance
that they had all been identified and were being dealt with.
On 28 July, NAA announced that it would take
over from ETS the management of the review process in 2008. Ofqual
publicly welcomed that announcement.
On 30 July, Isabel Nisbet wrote to David Gee
expressing concerns about the adequacy of ETS's resources and
quality assurance for marking the remaining unmarked scripts and
asking NAA about their intentions regarding deadlines and charges
for reviews. David Gee replied on all these points on 8 August.
Concerns about data on the system were particularly
relevant to the publication of Key Stages 2 and 3 data. These
must be produced to the standards set out in the Office of National
Statistics Code of Practice and free from any political interference.
DCSF's Head of Profession for Statistics (Malcolm Britton) twice
wrote to Ofqual seeking advice for him to consider when deciding
whether to publish results for Key Stages 2 and 3 at national
and local authority level.
Ofqual's replies focused largely on what evidence
was available on the quality of marking and the confidence that
could be placed in the outcomes. The advice sent on 4 August (about
the Key Stage 3 results) included:
"Ofqual recognises that the confidence of
teachers, parents, pupils and the wider public has been damaged
by the problems in delivering this year's national curriculum
tests|[T]he credibility of results published at this time will
no doubt be challenged. However, in providing advice to inform
your decision on publication, Ofqual needs to consider whether
there is hard evidence to call into question the quality of the
figures which you are considering publishing this year, compared
with previous years, to the extent that would justify a decision
not to publish or require publication with significant reservations.
| [F]rom the processes that we have observed to date, there is
no evidence of widespread problems with the quality of the marks
at Key Stage 3 that would justify withholding publication of the
provisional results at national level."
Similar advice had been sent out on 28 July
in relation to the Key Stage 2 results.
September 2008
| 
