1 The inquiries into the 2008 test
delivery failure
1. As a result of a procurement process which started
in 2006, the Qualifications and Curriculum Authority (QCA) appointed
ETS Global BV (ETS) as the new supplier of test operation services
from 2008. Over the five-year contract term, ETS would be responsible
for the administration of all national testing in England at Key
Stages 1, 2 and 3 and Year 7 progress tests. This would include
responsibility for the external marking of the Key Stage 2 and
3 tests and the Year 7 progress tests.[1]
2. The parent company of the ETS division to which
this contract was awarded is a US non-profit organisation with
long experience in test administration and which was, in 2008,
responsible for managing 50 million examinations in 180 different
countries.[2] Despite
this vast experience, the 2008 National Curriculum testing cycle
was not delivered in its entirety on time. A significant minority
of test papers at Key Stage 2 and 3 were not marked and returned
to schools in time for the deadline of 8 July and some schools
and pupils were still awaiting their test results in the following
Autumn Term.[3] Members
of this Committee are even aware of some cases where results were
still outstanding in the spring term of 2009.[4]
3. The inevitability of ETS's failure to deliver
on time did not, it seems, become apparent to the QCA until the
week beginning 23 June 2008. By the beginning of the following
week, senior officials and Ministers at DCSF had been notified
and, on 4 July, the QCA issued a press release advising of the
delay in publication of Key Stage 2 and 3 results.[5]
On that same day, both Ofqual, the regulator with responsibility
for overseeing National Curriculum tests, and DCSF declared their
intention to set up an independent inquiry into what went wrong.
On 7 July, it was announced that Lord Sutherland of Houndwood
would chair the inquiry and report to both Ofqual and DCSF.[6]
4. Well before the announcement of the Sutherland
Inquiry, members of this Committee had been aware that there were
significant concerns about the 2008 testing cycle and were already
alert to the serious problems with markers.[7]
We were, therefore, able to start taking oral evidence on the
delivery failures on 7 July, when the then Minister for Schools
and Learners, the Rt Hon Jim Knight MP appeared before the Committee;
and we started to receive written evidence from concerned individuals
and schools from mid-July.[8]
5. The very next week, on 14 and 16 July respectively,
we took oral evidence from both Dr Ken Boston, then Chief Executive
of the QCA, and the Secretary of State, the Rt Hon Ed Balls MP.[9]
As the summer break for both Parliament and schools approached,
it became apparent that a significant number of schools and children
would remain without their test results. Unusually, we took the
decision to sit during recess in order to take evidence from both
ETS and Ofqual on 10 September. This would be the only formal,
public occasion on which ETS gave evidence in relation to what
went wrong with test delivery in 2008, and Lord Sutherland has
acknowledged the timeliness and usefulness of our work in this
respect when he thanked us:
…for focusing on the report and on the issues
so promptly in July, as you did. I think you were the first above
the line, making very important questions plain and putting them
into the public arena. I found that a good starting point for
me. I recognise that the Committee's work has been integral throughout
the sad period since June or July, when things started to go wrong.
The sessions you held before the school holidays were very important.
You picked out a number of major issues, and I was able to build
on that in the evidence I asked for and in some of the cross-examinations
that I carried out.[10]
6. We then suspended our inquiries pending Lord Sutherland's
investigation into the details of the 2008 testing cycle. He published
his Report on 16 December 2008 and it amounted to a very thorough
investigation of the sequence of events leading up to the missed
deadline on 8 July 2008. In his Report, Lord Sutherland identified
a large number of factors which contributed to the failure in
2008 and he cited this complexity when he gave evidence to us
on 26 January 2009.[11]
It has not been the role of this Committee's inquiries to examine
the day-to-day management of test delivery at the level of detail
considered by Lord Sutherland. We
endorse the work carried out by Lord Sutherland in exposing the
detail of the many factors contributing to the failures in the
delivery of the 2008 National Curriculum tests.
7. Nevertheless, it is unfortunate that Lord Sutherland
was eventually unable to take formal evidence from ETS. Because
the contractor was a US organisation and likely to withdraw personnel
and resources from the UK, time was of the essence in commencing
formal inquiries. Lord Sutherland stated in the Introduction to
his Report:
In conducting my Inquiry, I launched a call for written
evidence and interviewed individuals from key organisations. I
should say at the outset of this report that ETS declined to submit
documentary evidence to the Inquiry. It has nonetheless been possible
to do a thorough investigation drawing particularly on extensive
materials submitted by QCA.[12]
From the point of view of demonstrable fairness,
it was regrettable that ETS had withdrawn its resources from this
country and chose not to engage with Lord Sutherland's inquiry,
although we understand that there was an exchange of correspondence
between the inquiry and ETS in December shortly before publication
of the Report.[13] It
was clear from Lord Sutherland's report that he had relied heavily
on the evidence this Committee took from ETS in September 2008,
since ETS submitted neither written nor oral evidence to his inquiry.[14]
Nevertheless, Lord Sutherland considered that he was able to
produce a fair report given the access he had to QCA documentation
and exchanges of correspondence between the parties.[15]
He told us that:
I have no doubt that there was adequate evidence
available to me to make the judgments that I believe are definitive
about the role of ETS in this report.
[ETS] believed at one point, in view of something
that it said to your Committee … that there was an embargo
on it in some way. I spoke to the QCA chief executive, and he
wrote to ETS to make it plain that there was no such embargo.
That is point one. Point two: we extended the period available
to it to submit evidence. Point three: it chose not to do so.
Point four: the evidence was available to us, through the QCA
documentation—we had all the exchanges of correspondence
and contracts and so on, so I believe I had adequate evidence.[16]
8. This inquiry provides a good
illustration of how a select committee can use its authority and
powers in a timely manner to investigate problematic incidents
as soon as they arise. Our swift action in this matter has enabled
us to put on the public record both written and oral evidence
from a party to the incident, ETS, which was not available by
the time the official inquiry was fully operational. The Sutherland
Inquiry was then able to rely on the evidence we had secured when
it became clear that ETS would not engage with the process.
9. In carrying out his inquiries, Lord Sutherland
was given separate, but related, terms of reference from both
Ofqual and DCSF.[17]
Dr Boston expressed concern that, by issuing parallel terms of
reference to Lord Sutherland, DCSF effectively closed down avenues
of inquiry which should legitimately have been within Ofqual's
terms of reference to his Inquiry, namely what he saw as DCSF's
active role in the 2008 test delivery failure.[18]
Dr Ken Boston has argued that, in issuing its own terms of reference,
the Department acted to prevent Lord Sutherland inquiring into
the way it set up the framework of policies and other decisions
within which QCA was required to work.[19]
This framework included the terms of the annual remit letter from
DCSF, which is accepted by QCA and which sets out what the Government
expects QCA to achieve each year.
10. It is indeed the case that the terms of reference
issued to Lord Sutherland by DCSF mention DCSF's active role in
the delivery failure only in terms of the appropriateness of its
arrangements to monitor the QCA's delivery against its remit.
The terms of reference went on to specify that Lord Sutherland
could interview "appropriate people" from QCA, ETS Europe,
Ofqual, DCSF and others.[20]
It is certainly not obvious from the terms of reference that Lord
Sutherland was mandated to inquire into QCA's remit letter and
related instructions from DCSF. Nevertheless, the Secretary of
State told us at the time the Inquiry was set up that:
Nothing is out of bounds for Lord Sutherland. He
can look at the whole process, and I am very happy for him to
look at whether the original remit was properly specified.[21]
In addition, Lord Sutherland has said that his terms
of reference gave him:
…ample scope and every opportunity to examine
the processes, roles and responsibilities of all of the key players:
not only QCA and ETS, but also Ministers and officials at the
DCSF.[22]
11. It is not clear to us that the
DCSF having issued parallel terms of reference prevented Ofqual
asking Lord Sutherland to inquire into the DCSF's role, had Ofqual
considered that appropriate. Lord Sutherland chose not to consider
the Department's role beyond its oversight of the particular circumstances
of the delivery of the National Curriculum tests in 2008. However,
this Committee is able to go further than he did and comment in
more general terms on the line between policy and execution.
12. Specifically, we will consider whether there
was anything in the way the Department set up the framework within
which QCA was required to operate which led to an increased risk
of delivery failure. We will also consider the role of departmental
observers and the implications for QCA decision-making and the
independence of the new regulator, Ofqual. We develop these themes
in more detail below.
1 http://www.qca.org.uk/qca_9788.aspx Back
2
Ev 8 Back
3
Report of the Sutherland Inquiry, para 1.4 Back
4
Q 237 Back
5
Q 26; Q 104; Q 144; Qq 346-349; Qq 379-380; http://testsandexams.qca.org.uk/18339.aspx
Back
6
http://www.ofqual.gov.uk/1717.aspx; http://www.dcsf.gov.uk/pns/DisplayPN.cgi?pn_id=2008_0142
Back
7
Q 192-193 Back
8
Oral evidence of Rt Hon Jim Knight MP given during an oral evidence
session on a separate inquiry into the National Curriculum, published
as HC 651-iii. The full transcript is printed as part of the
Committee's Fourth Report of Session 2008-09, HC 344-II. Back
9
Oral evidence of the Secretary of State given during an oral
evidence session on a separate inquiry into Public Expenditure,
published as HC 835-ii. The full transcript is printed as part
of the Committee's First Report of Session 2008-09, HC 46. Back
10
Q 237 Back
11
Q 237 Back
12
Report of the Sutherland Inquiry, p2 Back
13
Letter to the Inquiry from Dr Philip Tabbiner (ETS), 1 December
2008, referenced in the Report of the Sutherland Inquiry Back
14
Report of the Sutherland Inquiry, paras 3.43, 3.68, 3.74, 4.101,
5.20, 5.34, 5.89, 5.126, 5.142, 5.173 Back
15
Q 244 Back
16
Q 244 Back
17
Report of the Sutherland Inquiry, Annex C and D Back
18
Q 311 Back
19
Q 311 Back
20
Report of the Sutherland Inquiry, Annex D Back
21
Q 139 Back
22
Q 360, Secretary of State quoting a statement of Lord Sutherland
given to the Press Association Back
| 
