Examination of Witnesses (Questions 220-239)
MS HELEN
GHOSH, SIR
BRIAN BENDER
AND MR
ANDY LEBRECHT
15 MAY 2006
Q220 Chairman: It is a very good
term for us non-IT technicians.
Ms Ghosh: At the moment when we
pressed the button, it was the bit that translated validation
into payment and at that last stage—rather like a space
rocket, where you use bits of your IT system for a purpose for
which you have not used it before, so you have your launch and
then your booster and whatever it may be—what we were using
was effectively a new piece which had been tested but not tested
in anger. At this stage it became clear that even when you had
what you thought was a fully validated application and you pressed
the button, it nonetheless had to leap through a number of further
checks, which in many cases threw the application out. One of
the first things that Mark and his team did on arrival and which
speeded up full payments even to the level we have got full payments
at now, was to look at all those little bits of the system and
ask whether we needed this check, whether we needed to keep that
check in and to make that process simpler.
Q221 Chairman: Could you explain
one very simple thing to me? Why was the system not tested in
anger with all the bits in it before you started the process going?
Why did you not have some, what I call, full-scale dummy runs
with it?
Ms Ghosh: Again, this is an issue
that you can discuss in more detail and they will be able to give
you more information about the precise testing schedule with Accenture.
Q222 Chairman: You just said it was
not tested in anger with all its parts. Your team, your great
superstructure, is sending out messages of reassurance that this
system is capable of delivering both the Change Programme and
the RPA payments and it is all going to be on time and we all
thought in November we were going to get there. Why was it not
tested with some batch runs of real live forms to see whether
it worked in anger, together and under pressure?
Ms Ghosh: It was tested in order
to see whether it would produce some batch runs and that is what
gave us the confidence and batch runs are, of course, by definition,
smaller than the full scale of what you ultimately test. In one
sense, pressing the button on 20 February was effectively a test
of the batch run, but if I may come back to my earlier comment,
all the time of course we were balancing a number of interests.
The strong pressure from farmers to get payments out, the commitments
that ministers had made and, in one sense, the fact that there
were lots of checks against final, final, final payment was something
to do with the assumption that we also needed to minimise the
risk of disallowance. It became clear that those payments would
not be made as smoothly and as swiftly as we wanted once we started
to ramp up to the full volumes and that is why Mark came in, made
some changes to that and obviously that is one element of the
system we shall be looking at very closely in the future.
Q223 Lynne Jones: Do you know when
the batch runs were tested?
Ms Ghosh: I do not have the precise
details with me, but we shall be happy to give you that information.
Q224 David Taylor: Chairman, I have
worked in IT for 30 years which does not make me an expert in
anything, but what on earth does "tested but not tested in
anger" mean? Testing is about trying to compare the system
which has been acquired, designed, implemented with data that
it is likely to meet in the course of its life's running. What
the hell does "Tested but not tested in anger" mean?
Ms Ghosh: No, I meant tested with
the volumes and the pressure of the volumes that were going to
come through once we started to get payments coming through at
large scale. Clearly, again this is something on which, back to
Lynne Jones' question, we can certainly send you more details
about the testing schedules and I know the Accenture submission
has some of that in. The essential principle, back to this idea
that what they were building was an IT system which came in chunks,
of course was that we were testing and rolling out and then implementing
in the live environment, chunks of the IT system as we went along.
Q225 David Taylor: This is not expensive
Lego for goodness sake. It does not quite operate like that. Would
you not accept that one of the casualties of a Gadarene Government
rush to outsource IT to external bodies and organisations is the
lack of a capacity to embed knowledge within the client staff
who are supposed to be involved and making sure that the system
being delivered will work in normal circumstances? The Americans
used to talk about embedding journalists at the sharp end of the
war. There were not very many Defra or RPA staff with sufficient
knowledge to make the government an intelligent client in all
this, were there?
Ms Ghosh: There was actually a
significant IT team within the RPA which was capable of carrying
out the intelligent client function. We also had input from the
chief information officer's team. Ian Watmore was on the Ownership
Board. The issue that the Committee might equally wish to focus
on is not the end of the process. Having got to the end of the
process, although indeed testing and testing at volumes is a very
important part of the process, there is the issue about trying
to meet our commitment for farmers' payments and so there would
be a limit to how many batch tests we would want to do before
pressing the button in anger. So, to give a very positive example,
in the case of the partial payments that went out the week before
last, we tested that on 1,000 and having tested it on 1,000, we
felt that the commitment to make partial payments was the overriding
one and we pressed it for everybody else. Happily for everybody
else it worked. There may well be issues which are nothing to
do with whether you insource or outsource IT supply, which again
is something that Mark and the RPA team are now looking at, which
is how you construct the basic business process from which you
build your IT. So, for instance, the IT process was built around
a task-based process and had built into it a very significant
element, and perhaps to some extent this is a cultural point,
of risk averseness around disallowance, hence all those checks
that suddenly came into play at the very end of the system. Now
that is not an IT specialism issue, that is a business process
issue and that is the kind of question which may well underlie.
Sir Brian Bender: May I just add
one point which came across, but I should just like to make sure
there is no misunderstanding on the part of the Committee? It
was not as though individual releases of IT were not tested along
the way and I hope that is clear. What happened in Helen's rocket
analogy is that very deliberate decisions were taken from the
earlier stages of this programme to break down the IT from a big
bang into specific releases and along the way conscious decisions
were taken to break it down further so that incremental releases
could be made. Those were tested, those were put into place and
indeed the Office of Government Commerce, in January 2005, praised
the fact the development had been broken down into manageable
steps. The problem then appeared to come at the end, when the
last release was put on and the button was pressed for delivery.
It was that stage which Helen has been explaining, but I did not
want the Committee to feel that there had not been full testing
and indeed breaking down into manageable steps along the way.
Q226 David Taylor: Quoting the Office
of Government Commerce observations is not a necessarily very
convincing part of the argument with the track record that they
have. When the present permanent secretary came into post six
months or so ago, no doubt she would have been briefed extensively
on a whole host of background matters, including, I hope, this
Select Committee's assessment of the Department and indeed its
critical observation some time ago of the IT strategy or lack
of it within the Department. We were brushed aside by her predecessor
who said it would all come out in the wash. I am paraphrasing
heavily. Did she read this document and did she give any credence
to the comments that were made then which are coming to fruition
now?
Ms Ghosh: I have not read the
document in detail. I presume that the debate was around the question
of outsourcing our IT provision overall to IBM. Since my arrival
and with the support of an excellent chief information officer
within the department, our IT strategy is becoming, by the day,
clearer and better focused. To comment in general on the relationship
with Accenture, which is obviously not one of our key partners,
it is true to say, and forgive me if I quote OGC gateways again,
that we have had a number of issues in terms of managing the process
with Accenture over the way. I know Sir Brian had issues with
them in 2003 over the quality of the people they were putting
in for testing, but it has been a very open relationship; regular
meetings between myself, Sir Brian before me, the European managing
director and the local team. They have progressively put on a
stronger and stronger team from our point of view and, in terms
of key performance, as they say, the fundamental RITA process
is working well. They also did extremely good work on the partial
payments system. What we would regret about Accenture's performance,
and I am sure that they will discuss this next week when they
appear, are issues around delays. It is true to say that in a
number of significant releases there was a longer delay than we
would have hoped for. That lost us some contingency planning time.
It required some of our processes, for example around handling
entitlement trading a couple of months ago, to do manual processes
rather than IT processes. It is inconceivable that we could have
delivered a programme of this scale without an outhouse partner
of the kind that we have in Accenture.
David Taylor: Well that is absolutely
wrong because you have internal staff who could have done a better
job, in the view of others. It is all very well for Sir Brian
Bender to shake his head; I cannot know how he is that certain
that it would be impossible internally. I just do not accept that.
Q227 Mr Rogerson: The experience
of many customers seems to have been that they are on the phone
to people trying to get things across and on the other end was
someone who did not really understand the intricacies of what
they were going to do. The assumption may be that they were temporary
staff and I know temporary staff were drafted in to deal with
the backlog. Could part of the problem with the testing that you
were talking about or the lack of robust testing be that the people
you were using at the other end were people who were not prepared
to use the system and that the RPA had to draft in people from
elsewhere?
Ms Ghosh: In the note we do about
testing, I should quite like to talk about how we took experienced
people out from the business to do testing regimes and then we
back-filled them with some temporary staff. It is absolutely true,
as you say. This is an issue around responding flexibly to demand.
There is indeed a high proportion of casual or non-permanent staff
in the RPA, but it is still only around 50:50 temporary staff
against permanent staff, so there is not a critical mass problem
in terms of not enough people in the organisation who understand
how the system works. The other thing of course is that what staff
at the RPA were operating was not the old system but the new system
and there was a process of learning going on for all staff in
the RPA, whether they happened to be temporary staff or they were
the former permanent staff, to move through the testing and delivery
process.
Q228 Daniel Kawczynski: Again this
one is for Ms Ghosh. Some European Union countries have done a
very good job in doing this exercise and paying their farmers
on time and Sweden is one of the best ones. We talk nowadays about
being in a single market and the European Union and doing things
collectively or in some sort of uniform fashion. What sort of
interaction has your Department had, if any, with other EU countries
and would you be looking to increase that in the future?
Ms Ghosh: Although you kindly
directed the question to me, I shall hand over to Andy who has
much more experience of this. The answer is yes, we have a lot
of both interaction and comparative experience to learn from other
EU Member States, but of course the choices that EU Member States
made about the kind of model that they would implement were very
different ones. Only a few are actually directly comparable with
our own. Andy, would you like to say a little about the international
comparison work we have done?
Mr Lebrecht: Yes, I am very happy
to do so. One of the characteristics of this reform of the CAP
was that it did give a lot of discretion to the Member States
in a lot of respects as to how they implemented the reforms. Virtually
every Member State did something different one way or another.
We worked from time to time closely with a number of countries,
particularly countries like Sweden, like Denmark, like Germany
which like ourselves were interested in going down an implementation
route other than the straight historic route. Everybody's problems
were different one way or another, but we shared our experience,
identified common problems and, where that was appropriate, we
went together to the Commission to try to ensure that the technical
regulations were sensible from our collective point of view. The
answer to your question is yes, we did work with them but obviously
we were all doing something a little bit different, so there were
limits to what we could do.
Q229 Lynne Jones: I want to know,
in this incremental process, what the date was when the last bit
of kit was added on.
Ms Ghosh: Do you mean added on
in the sense of when was it delivered live? It is in the Accenture
document.
Q230 Lynne Jones: When was the end
of the process?
Ms Ghosh: "Release 3a2—the
core batch functions . . . went live on 3 October 2005".
Q231 Lynne Jones: So that is quite
late on in the day.
Ms Ghosh: Yes. That was the bit
which established entitlements and authorised payments.
Q232 Chairman: Can you just explain
something to me? You are almost giving the impression that more
bits were added on to this system in February this year.
Ms Ghosh: No, I said they were
first used. There was the testing that went on between October
and February, the "not in anger" and then they were
first used in February to make the payments. It was the authorising
payments element of that.
Q233 Chairman: Did nobody, with all
the expertise at your disposal, pick up the fact that this system
was not as robust as the "testing", in inverted commas,
had shown? The idea of testing something is to find out whether
it is going to work in practice and the impression I am getting
is that there were a lot of fixes going on in February and that
you finally failed to make it deliver what it was supposed to
do. Yet Accenture told us that the thing was up, running, working
and stable in October 2005. It does beg the question: what was
happening in November, December and January? Did they just sit
back and do nothing or did they carry on testing? How did we find
out? Mr Lebrecht, according to paragraph 34 of Accenture's memorandum
you have been involved in the implementation group for a long
time on this Single Payment Scheme and you are somebody of very
considerable experience in the operation of Defra, the CAP and
the authorship of the Single Farm Payment. Did you not realise
that something was not happening as it should in terms of the
ability of this system to deliver, bearing in mind what we shall
also come on to look at in more detail, all the messages that
were coming from your customers throughout 2005 and a bit before
in 2004 when they were expressing concerns about various practical
aspects of the ability of the system as it evolved to deliver
the final product?
Mr Lebrecht: We were listening
very closely to our customers throughout that period and the programme
management arrangements which were described earlier were very
active in that period. It was a period when the IT system was
in place and was being used by RPA to process the applications.
Of course the process involved addressing all the tasks. It was
a task-based system and these were being addressed on a systematic
basis by the RPA. What we as Defra were doing was discussing with
RPA their progress as it went through October, through November,
through December and into January against a number of criteria,
but of course top of the list was the criterion "Is this
going to deliver against ministerial commitments?". The assurances
we were getting all the way through this, right through up to
and including the major assessment we made in January that Helen
Ghosh referred to, was that the RPA could deliver. In other words,
it could deliver the first payments in February and the bulk of
the remainder by the end of the March. We were checking that assurance
time and time again and we were getting that assurance all the
way through that period.
Q234 Chairman: When you say you were
checking it, how were you doing this? If the system had not been
up and running and been tested in anger, who was checking the
RPA's assurances? Who was actually responsible for saying "I
believe that the information we are getting from the RPA is valid"
and that when you gave advice to ministers, you were going to
drop them in it, because later on they were going to be found
wanting?
Mr Lebrecht: At one level the
governance process involved opportunities for us as officials,
for ministers, to cross-question the RPA on their performance,
because they were using the system during this period, they were
making progress on the tasks, and where this did not seem to be
performing well to check what the reasons were, to discuss ways
of improving performance. We also had external assurance; Karen
Jordan, our quality adviser, who has been referred to, was very
actively involved in this.
Chairman: Why did this person not spot
what was going on? They are an external expert, they are a quality
adviser and what we have had is an example of a non-quality system
because it could not deliver on time. How come this quality problem
was missed?
Q235 David Taylor: What difficulties
did you see, or were you just accepting and hand waving?
Ms Ghosh: What Andy has said about
what we were focusing on in that period is fantastically important.
The main task as RPA saw it and we saw it in governance terms
was to get the validation process through, because we were extremely
conscious that we needed to reach a point where we could make
definitive entitlement statements, send them to customers, get
trading going. At the same time, we were getting assurances both
from Accenture and from the RPA that the normal testing process
was continuing on the other things. Just to come back, the bit
that went wrong in the end was a bit that was nothing about entitlements
or validation, it was about actually from that point when you
said "Here is a validated claim, pay it", it was that
bit of kit that was between that point and the end. You are quite
right. Probably no one of us, because we were focusing very much
on the "Let's get the entitlements out" was going in
a very detailed way and saying "Can we see the test reports
on that bit of kit which comes after the setting of valid entitlements?".
What we shall do for the Committee is give you the testing schedule
for what was going on in that period on that point, which comes
back to the point: how much did we test or not as the case may
be?
Q236 Chairman: Is the bit at the
end of the process something that is beyond this Release 3a2?
Ms Ghosh: It is the bit where
payments are made. It is the bit where the RITA system plugs into
the actual payment system.
Q237 Chairman: What I should like
to know, because I am not an expert on this system, is why Accenture
in their evidence say the system is stable, which I assume means
up and running and tested, from October 2005? Was what you have
described—and if you do not know the answer now we will
have it later—that bit, the bit that enables the payment
to be made, added after October 2005?
Ms Ghosh: No, that was the same
release which was being tested in parallel. We were not focusing
on it because what we were focusing on was the validation in fact.
Q238 Chairman: So just to be very
specific, did the testing quality validation procedures which
we have just been discussing not actually explore whether that
bit, the bit that did the payment, actually worked?
Ms Ghosh: That is the question
on which we shall come back to you. It was not the thing that
we in governance terms were focusing on because that was not the
big challenge. The big challenge, as we were being told by RPA
and by our external spokespersons, was getting the claims validated
and through. There was no suggestion to us, at that stage, that
even when we got them validated and through, there might be an
issue around getting the payments out of the system.
Q239 Chairman: Was that because you
were having so many problems with the mapping system that you
decided you would concentrate on the validation? Sir Brian is
shaking his head, do you want to respond?
Ms Ghosh: No; no.
Sir Brian Bender: I was going
to answer Mr Taylor's question earlier, and you have moved on
slightly but it may be helpful if I give it. His question was:
what data were you getting? Each of the releases added a functionality,
such as claims processing, such as validation, and the Programme
Board and the Executive Review Group were given data about the
performance of that system in terms of technical performance and
the productivity of the users. I remember some discussions around
the middle of last year, particularly about the second, the extent
to which the systems were, as Helen might have put it earlier,
gumming up a bit and therefore the productivity of the users and
RPA was not sufficient. We then discussed how we could increase
that. That was the sort of data that the various governance groups
were getting. Then, coming back to your last question Chairman,
we were not overwhelmed simply by the mapping point: we were looking
at each of the issues that may be in the critical path. Certainly
while I was still there, each of the meetings I chaired over the
last few months I was there looked at the backlog in maps and
what might be done about that, but it did not only look at that.
It also looked at other things that were in the way of the critical
path, because from early 2004 onwards the Rural Payments Agency
had produced a critical path and one of the issues for each meeting
was what was in the way of that, what was at risk of knocking
us off that critical path.
Ms Ghosh: That is the material
you will see when we release it.
| 
