Examination of Witnesses (Questions 60-79)
DR PHILIP
TABBINER AND
ANDREW LATHAM
10 SEPTEMBER 2008
Q60 Chairman: Okay. Let us drill
down into what you have said. Are you saying, Dr Tabbiner, that
you had glitches with the technology? From the very full letter
that you sent to the Committee, it seems that there were technical
problems that you did not foresee.
Dr Tabbiner: I would call them
operational more than technical, just to put the point on it.
It was more the nature of operational technology, as well as the
interactions with the NAA, that created knock-on effects, or operational
circumstances, and that created issues for us. For example, we
had significant delays in terms of the response from the NAA as
to whether we could conduct online training, which is what we
had originally contracted for, and we were required to do face-to-face
training. That created operational issues around venues and having
training materials for all the trainers done in a very compressed
period.
Q61 Chairman: So you are saying
that the NAA and the QCA substantially changed the contract after
you had signed it.
Dr Tabbiner: That is correct.
Q62 Chairman: With your agreement?
Dr Tabbiner: One of the issues
as I looked at our side of the operation was that we were quite
amenable to changes. In a number of instances, we put forward
change orders, which was the required process under the contract.
But we also made a number of changes on the fly, as we might have
been asked to do in the course of a day's or a week's work under
the process, for which we did not go through the rigorous process
of seeking a change order.
Q63 Chairman: Standing on the
sidelines, as members of the Committee felt we were, we got the
impression—we all know about your formidable reputation
in other parts of the world, including North America and beyond—that
when you arrived in this country, the people who took you on thought
that you would bring your whole operation from the United States
and transplant a bit of it over here. The word that seems to have
come out is that you did not do that and that you started a brand-new
company here. You therefore ran into a lot of difficulties because
you were almost a start-up company. Is that true?
Dr Tabbiner: It is important to
recognise what has happened in the creation of this agreement.
As you often have when you get a five-year agreement, you then
begin to pursue the course of that agreement. This agreement was,
in very large measure, a supply agreement, and that is the part
we fulfilled—fulfilling a standardised test that was already
created and using methodologies that were, in general, already
accepted and dictated. We were then filling in certain supply
elements—certain systems, delivery logistics and the accumulation
and implementation of quality control—rather than fully
developing the standardised methods that would be in the test.
In that light, we used some people whom we already had, as well
as building up staff locally. We also applied a significant amount
of resources during the heightened period of the May, June and
July time frame, when we ran into significant delays.
Q64 Chairman: Was there any element
of that where you were working in a rather different culture from
the one you are used to working in and where some local knowledge
was missing on issues such as how many qualified markers there
might be in a pool to do your marking at a time when there was
an awful lot of marking to be done?
Dr Tabbiner: We have the good
fortune to be working in 180 countries with many different cultures.
We do standardised testing in Serbia, Egypt and Korea, so we have
experience of many different cultures. The other thing that I
would say in the context of how we approached our agreement with
the QCA and the NAA relates to the issue of complete transparency.
We looked to them to give us guidance and direction on some of
the issues that might be encountered. In many instances, when
we would implement efforts to resolve an issue, it was only later
that the NAA would point out what had traditionally happened.
We found that some of the useful information or data that would
have helped to inform early decisions was not shared with us.
Q65 Chairman: Dr Tabbiner, looking
at your date of arrival, it sounds a little bit like you came
in when things started going wrong. You came over when things
were in trouble.
Dr Tabbiner: I arrived in mid-May.
Q66 Chairman: May I switch to
Andrew Latham who has been involved in the operation rather longer?
When did you first think that things were going wrong?
Andrew Latham: We knew that we
had challenges as far back as December, when we were pilot testing
the marking solution. From that time on, we worked very closely
with the NAA to try to address the issues that we saw arising.
Q67 Chairman: Was that before
the NAA asked for any changes in the contract?
Andrew Latham: No. We talked about
changes to the contract. The example that Dr Tabbiner used was
moving to a face-to-face training model, but many other changes
were asked for, such as the marking pilot test itself. In the
contract that had been signed, we had said that we would do a
small-scale pilot with focus groups of about 25 people or so just
to demonstrate that the model worked. In fact, the NAA insisted
on a very complex pilot with more than 800 different markers.
Chairman: We will drill down into a couple
of those issues in a moment, but let us carry on looking at what
happened.
Q68 Mr Chaytor: Dr Tabbiner, is
this the first time that ETS has had to withdraw from a contract
before its completion?
Dr Tabbiner: This is the first
time that we have asked to withdraw from a standardised achievement
contract, yes.
Q69 Mr Chaytor: Have you been
asked to withdraw from a contract? Is this the first time that
you have asked to withdraw? Have you previously been asked to
withdraw?
Dr Tabbiner: No, we have not.
Q70 Mr Chaytor: There has been
no previous experience such as this, in which the whole project
has got into such difficulties that the contract was terminated?
Dr Tabbiner: That is correct.
Q71 Mr Chaytor: In your written
statement to the Committee, you repeat the acceptance that there
were technical and operational difficulties. Just now, you said
that the difficulties were more operational than technical. You
give two examples of work from schools being allocated to two
different markers, or some markers not being able to access online
the work of their students. Could you list all the operational
difficulties of which you are aware, for which you or your company
accepts responsibility?
Dr Tabbiner: One of the things
that I would identify is that as you look back over a contract
and you examine some of the issues that arose from the delays,
we can say that if we had moved on into cycle 2, we would have
made modifications to try to deal with some of the technical issues.
For example, we had set up a process for the marker allocations.
We did not bundle up the materials that were to go to the markers
until such time as each mark was standardised, then we submitted
the materials. In retrospect, that cost us some time. In a future
year, we would pre-bundle the materials and they would be ready
for standardisation. That is just an illustration. Another fact
is that we had designed the systems in a way in which the schools
had to fill in the student register. If a school did not fill
in the register, the system was designed without a default for
it, and there was an absence or a block that was left empty. When
teachers went to enter the marks and they ran into that block,
they had to call the helpdesk to deal with a registry issue rather
than an online issue. In retrospect, that is something that we
would have made adjustments for, knowing that schools, in some
instances, would not necessarily fill in all the data points.
The next point concerns the return of materials. Markers were
asked to put the marked scripts on the top of the box, which they
did. We took them to be 100% marked scripts and returned the boxes,
as appropriate, to schools. Later, we found that, in some instances,
there were unmarked scripts in those boxes, which schools later
returned to us. In a future circumstance, we would do a 100% visual
inspection of the scripts.
Q72 Mr Chaytor: Should those operational
difficulties not have been identified in the pilot scheme that
was carried out last December?
Dr Tabbiner: We intended to do
an end-to-end pilot run, but we were refused the opportunity to
do that. All we were allowed to do was a small pilot on logistics
with the schools in only 25 schools.
Q73 Mr Chaytor: Andrew's response
just a moment ago was that one of the NAA's decisions was to extend
the scale of the pilot, not to reduce it. Can you clarify that?
Andrew Latham: Sure; I was talking
about the marking pilot in which we were supposed to demonstrate
that the elements of our solution that were new this year were
effective, such as having the markers actually enter their marks
online or having the markers be trained online, which was totally
separate from the logistics pilot.
Q74 Mr Chaytor: Okay. So the marking
pilot took place last December and, if I can put this to Andrew,
from your point of view was that completed successfully?
Andrew Latham: The marking pilot—yes.
Q75 Mr Chaytor: But when was the
logistics pilot completed?
Andrew Latham: We never conducted
a full scale logistics pilot in the end. We did a small scale
logistics pilot where we sent some packages to a couple of dozen
schools and said, "Does this packaging look right; does it
work; are the instructions clear?" It was that level of pilot.
Q76 Mr Chaytor: So in retrospect
should there have been a full scale logistics pilot?
Andrew Latham: Absolutely.
Q77 Mr Chaytor: But that was not
specified in the contract.
Andrew Latham: No.
Dr Tabbiner: But we desired to
do it and were refused the opportunity.
Q78 Mr Chaytor: Right. So you
asked the NAA to do it.
Dr Tabbiner: If we could do it.
Q79 Mr Chaytor: If you could do
it—for permission to do it. They refused for what reason—lack
of time?
Dr Tabbiner: They asked us to
do the small 25-school logistics pilot.
| 
