Examination of Witnesses (Questions 260-279)
LORD SUTHERLAND
OF HOUNDWOOD
KT
26 JANUARY 2009
Q260 Chairman: So who should have
googled ETS? Should it have been PricewaterhouseCoopers or should
it have been QCA? Because, as someone said to us, you only have
to google to get all that press comment that was already there.
Lord Sutherland: PricewaterhouseCoopers
did what they were asked to do. They performed according to contract,
which everyone had. So I would not lay the blame there. So I think
it should have been QCA, and I did put it to their board.
Q261 Mr Chaytor: On the role of
PricewaterhouseCoopers, your report says that they were asked
to do the following tests. They produced the Dun & Bradstreet
rating for each company, they examined the last three years' published
financial statements, and then they did a review of press information
over the past 12 months. That is the key flaw, because, had they
carried out a review of press information over the last three
years, the same period for which they published the financial
statements, that would have highlighted a number of press reports
in serious publications in the United States and, I think, in
the UK, that would have given a different picture. Is it conceivable
that a large multinational management consultancy such as PWC
would not have known of the activities of ETS in the United States,
and its track record prior to the 12 months for which it did the
review of press information? I would find that remarkable.
Lord Sutherland: I have two things
to say on that. PWC is a professional body—it had a job
to do and it did it. There are two constraints on how much testing
you carry out and how far back you look. One constraint is the
12-month period and that was given to them, and the other is that
you look at contracts worth more than £20 million a year,
and that was also given to them.[1]
Within those constraints, PWC did what it was asked to do and
so I would say that it delivered on its contract. The question
is why, if we were dealing with such an important company, was
more not known about it in the system, including in QCA and by
those who would be working with it. Indeed, I would have thought—this
is the point of your question—that they would have had some
international contacts whom they could have asked, but they did
not, and that is a severe failure. When you go back to where blame
lies it is not simply ETS, it is the checking.
Q262 Mr Chaytor: Is it absolutely
clear that not a single person in the Department for Children,
Schools and Families, QCA, National Assessment Agency or Ofqual
went on to their computer and googled ETS?
Lord Sutherland: I cannot vouch
for what several thousand people did or did not do.
Q263 Mr Chaytor: There is no evidence
that anybody did that?
Lord Sutherland: No, and that
is my criticism: not that nobody did it but that there were people
responsible who should have done it.
Q264 Mr Chaytor: In your findings
and recommendations, some of your criticisms are specific to QCA
or the DCSF observers and so on, and others are expressed passively,
with more reluctance to attribute responsibility: "Such-and-such
a process should have been carried out", for example. Is
that conscious or just how the report is written? Sometimes you
are very clear as to who was at fault and at other times less
so, as if you do not wish to point the finger of blame.
Lord Sutherland: The main driver
of the report was to produce an analysis of what went wrong. Sometimes
it was very clear from that analysis who should have done what,
and sometimes it was a question of, "What was the arrangement
operating between NAA and QCA?" That is an uncertain area,
and interestingly, given that you suggest that perhaps my report
has not been tough enough, NAA no longer exists. That is recognition
that the structure there was wrong or uncertain. That is why sometimes
you could not say that it was person X in NAA or person X within
the broader context of QCA. Do not forget that Ofqual was not
a separate body most of the time, and was only a quasi-separate
body from 1 April 2008. I would have to see specific examples,
but that is probably why occasionally the criticism is passive.
Q265 Mr Chaytor: Regarding the
problems that were identified in the early stage of the procurement
process, could you remind the Committee what your recommendations
for change are, and whose responsibility it should be to flag
up problems? There were capacity problems: ETS said that it could
manage with 60 staff, which was a vast underestimate.
Lord Sutherland: You have to go
a bit down the line that says who is awarding the contract. The
contract and its award is recommended by QCA. QCA is on the front
line in deciding who gets the contract. The question then becomes,
what tests might it or should it have carried out that it did
not? And is there anything else that should have been done further
up or down the system? I have made some specific recommendations
on that. One of those recommendations is that I believe that Ofqual
should in future be consulted. It is not its job to award the
contract, but it should be consulted on the nature and the details
of it. The reason for that is that Ofqual needs to have certain
kinds of information if it is to carry out its quality assurance
processes, and it was evident that it was not receiving that information.
Unless that is built into the contract from the start, there will
be difficulties. That is an example of why Ofqual should be consulted
when the details are being prepared. I do not believe that the
presence of DCSF observers on various committees is as clearly
based as it should be. Perhaps there is an assumption that if
a DCSF observer is there and does not say stop, everything must
be all right. We will be told that that is not the role of DCSF
observers. Experience probably varies from one committee to another
because there are a lot of them around. However, if somebody from
the DCSF or the Office of Government Commerce procurement agency
is present, their role must be made clear.
Q266 Mr Chaytor: Are you confident
that the structural changes to the QCA and the establishment of
Ofqual as a separate body will resolve such procedural difficulties
in future contracts?
Lord Sutherland: It is the right
direction, but it is a tight run this year. In the longer term,
I hope that the structural changes will have an impact on future
contracts. I hope that how scrutiny should be allocated is much
clearer in the future.
Q267 Mr Chaytor: Finally, in terms
of reputation and the track record of checks, you do not recommend
that procedure should change to require a longer period for the
press review: it remains as a review over the last 12 months.
Would it not be a sensible change to the procurement procedure
to make it a press review over the previous three years?
Lord Sutherland: It certainly
would be a sensible procedure. It is the experience of my working
life that when you lay something out very clearly and put it under
the nose of those responsible, you are able to scrutinise whether
they have taken good account of it. They claim that they are going
to.
Q268 Chairman: But Lord Sutherland,
on the procurement question, there was gossip going around when
this all started that the QCA had taken the lowest tender and
that it was cheaper to take on this new company. From what I have
heard in this Chair, I do not believe that. I think that it was
based more on a desire to not be too reliant on one supplier.
There are not many competitors in this area. You might have picked
up on this in the report and I might have missed it, but I got
the feeling that it was not about the price. There were very few
players in the area and the QCA wanted a diversity of supply.
As you know, there has recently been controversy over education
maintenance allowances. A different company is having trouble
delivering those on time. What is it called? Liberata. Sorry,
I could not remember the name.
Lord Sutherland: That happens
to me sometimes.
Chairman: Is it not interesting that,
again, there is a great reliance on the company? In an inquiry
on individual learning accounts that we conducted many years ago
when I was first Chairman, we learned lessons relevant to all
procurement and all contracts. I may have missed this in your
report, but what is the general advice that you give? In this
case it seems that there is a lot of reliance on the company.
If you want a diversity of contracts, you cannot afford for one
of the small number of contractors that offer the service to not
be in business.
Lord Sutherland: There are real
issues that this case has helped to illuminate. You and I will
remember the Student Loans Company, which went disastrously wrong
a number of years ago. Its headquarters was in Glasgow. There
are any number of examples, such as the Child Support Agency.
This issue comes back to Mr Stuart's question over the way in
which contracts are awarded and whether this is a good principle
to follow. I have no doubt that you need the reassurance that
there is a good market with customers who will compete with each
other. There must be choice in case a company proves not to be
adequate. To reaffirm the point that you make about, I have looked
in detail at the process, and all the evidence I have—and
it is strong evidence—suggests that the procurement process
marked the companies on abilities and skills first, before opening
the envelope containing the bid cost. ETS came out No. 1 on all
the indicators and was also the cheapest—I am convinced
that that was not a factor, I agree with your judgement on that.
Q269 Mr Heppell: I am fascinated
by the fact that the difference seems to be so great between what
the contractor and the QCA believed to be happening. ETS were
saying that they needed 60 staff, yet at one stage were employing
about 400 staff. A lot of the pilot tests went wrong, but there
did not seem to be any communication that that had happened. As
you said, end-to-end testing of the system seemed to disappear
altogether. I think that ETS told us that the original contract
was for online training, but the QCA say that that was not quite
what it had said. That makes me wonder—it does not reflect
well on the QCA whatever way we look at it. They drew the contract
very badly in the first place—with so many ambiguities and
holes in it that it was not going to stand up—or was it
the case that they changed their mind about it as well?. I wonder
about the online testing—going through it and identifying
that it had not worked very well, suggesting that they do something
else instead—because, obviously, enormous disruption and
extra costs are caused for the contractor when this happens during
a contract. Am I getting the sense of this right?
Lord Sutherland: I think there
was a misunderstanding. Whether you use the word "genuine"
or not—whatever that may mean—about online training
of markers, ETS believed that that was the form that the training
would take. The QCA rightly say that the contract did not say
that that was the form it would take—it said that they would
be interested in looking at a proposal that was properly piloted
and showed that online training of markers would work. There was
a pilot and it was not judged to be adequate. There was a disagreement—not
only did ETS say that they thought there would be online training
of markers, they did not have a reserve plan either. They could
not have deduced from the contract that there certainly would
be online training of markers. They did not have a plan B and
they rushed around setting up eye-to-eye, face-to-face training
sessions, which went badly wrong. If you look at the evidence
from markers, they were appalled at how they were being treated.
People turning up in Manchester were told that their training
was in Birmingham; people were not given the right date or adequate
notice of when the training session would take place. ETS would
say that there was an ambiguity, but the glass is running against
them considering their other delivery failures. Whether or not
there were other ambiguities, there is a very strong case for
looking at online marking. That is a different ball game and I
am persuaded that it would produce a wide range of benefits—we
might come back to that. There was a specific ambiguity—they
said that they looked at it in detail, but I do not believe that
that is sustainable.
Q270 Mr Heppell: You mentioned the
fact that there was an assessment of each of the bidders before
the bids were actually opened. It would surprise me—as we
told them that—if we had had the facts in front of us. Is
it not the case that the bids should have been reassessed as soon
as they were opened? They must have thought to themselves that
the low cost did not match with the services they were saying
they could provide. Did that not ring some warning bells?
Lord Sutherland: You made the
point that the original estimate by ETS was that 60 staff would
be allocated—they ended up needing hundreds in addition
to that. That came out of the QCA and NAA's pockets; they both
had to supply the support in many cases.[2].
Certainly, that is one of the probings that did not take place
and, at that stage—you are right—it should have been
a warning signal, leading to the question, if that is the cost,
have they really got their numbers right?
Q271 Mr Heppell: I am a little
intrigued as to what reputational issues there actually are, because
you seem to be saying that they did an assessment of all the people
and so on. What other things would they have been looking for
if they went online and, perhaps, searched for ETS? Are we in
danger of talking about gossip? That is my worry.
Lord Sutherland: That is where
it takes a bit of probing. It could just be gossip, but if they
have publicly failed to deliver on contract in previous years—I
am not talking 1920 here, but earlier, within the current period,
of five, six or seven years—that is one of the questions
that you would then ask, if you had done that probing, to improve
their delivery and performance. Some scathing comments were published
in reputable journals and newspapers in the States that may or
may not have been justified, but again those were sufficient to
make you want to ask questions. I do not know of any major company
that would not have done that kind of probing if they were bringing
on board a relatively early, completely new player into the game.
Q272 Mr Heppell: But you are saying
that we do that with every Government contract. Am I not right?
Did you not just say that that is the normal procedure and PricewaterhouseCoopers
did not go beyond that because it did not need to?
Lord Sutherland: That is right:
12 months and £20 million.[3]
Q273 Mr Heppell: Every other company
outside, you are saying, would go further, but the Government's
procuring policy does not?
Lord Sutherland: I know of companies
that do that kind of probing, whether by telephone or googling.
If you google first, you will get the press cuttings and then
you say, "Ah, yeah, that's so and so—no, no."
But that is a serious report. Perhaps we need to inquire further.
That is what was not done.
Q274 Mr Heppell: And you have
a recommendation that that should be changed and become part of
Government—
Lord Sutherland: I have laid out
the facts and sensible people like yourself can hang on to them
and push them on this.
Q275 Mr Heppell: But could that
not have been one of your recommendations? If this is the fact
and all government procurement works under the same system, should
it not be changed?
Lord Sutherland: If you are asking
whether we recommended that it should be lower than £20 million
and before 12 months, there is a general recommendation that they
should probe further into the companies that they are dealing
with. [4]
Chairman: Thank you, John.
Q276 Annette Brooke: Referring
back to your earlier comment, Lord Sutherland, I think that one
of our Committee's advisers described this as an accident or a
disaster waiting to happen, given the problems in 2005, which
made it difficult to meet the deadlines for 2006-07. Why on earth,
when changing the contractor, did we decide to make it even harder
and to change the goalposts so significantly in terms of the timetable?
Lord Sutherland: That is a perfectly
fair, appropriate and penetrating question, and it is the right
one to ask. Why so many changes and why was the company not probed?
If you have complete confidence in the company, perhaps you would
do that.
Q277 Annette Brooke: Right. It
also seemed that there was not enough time allocated. Is that
QCA or is it ETS not demanding enough time?
Lord Sutherland: For?
Annette Brooke: For picking up from the
previous contractor and then picking up this year.
Lord Sutherland: I do not believe
that that is so. I believe that there was a long run-in period.
The process of preparing to award the contract was taking place
in autumn 2006, which means that the companies bidding for it
were thinking about these things in 2006. The contract was awarded
at the beginning of 2007. That gave 19 months before the actual
delivery. That should be adequate.
Q278 Annette Brooke: Right. So
the time is there, but presumably things did not happen. So what
sort of things did not happen during that period?
Lord Sutherland: I think this
is one of the difficulties. There were, clearly, late deliveries
on the pilots for training markers, for example. There were requests
to ETS for an outline of the details of the project and whether
testing had been carried out. Again, that was not delivered on
time, or in good time, as it should have been. There was a whole
series of events like that. ETS was being questioned, and indeed
money was held back for late delivery of some of the details in
June 2007, a year before the disaster happened. All those were
warning bells, which is why I am allocating significant responsibility
to those administering the contract.
Q279 Annette Brooke: Did the DCSF
know that there were problems in 2007?
Lord Sutherland: Those responsible—the
National Assessment Authority of the QCA, which was responsible
for withholding cash, for example, and did—certainly knew.
In part, they had a responsibility, if they thought the issue
to be serious, to refer it upstairs—or escalate it: a piece
of civil service jargon that I have learned to say. If they think
it is going to cause major problems, they have a responsibility
to refer it upstairs. The essence of much of the difficulty was
that each problem in itself seemed to be manageable, but not when
you strung them all together—that, and that, and that, was
a problem. If you are late in delivering here, there will be a
consequence further down the line, and that was just not taken
on board. So, the management of the contract in the QCA fell down.
1 See Ev 47 Back
2
See Ev 47 Back
3
See Ev 47 Back
4
See Ev 47 Back
| 
