Examination of Witnesses (Questions 40-59)
DEPARTMENT OF
TRANSPORT AND
DVLA
23 APRIL 2008
Q40 Mr Bacon: Conveniently, I do,
and one of the things it says is, in addition to the various recommendations,
"the following areas warrant further investigation: (a) the
sample design for the roadside traffic observation survey and
whether it remained suitable for the purpose of estimating Vehicle
Excise Duty evasion; (b) the methods used to weight the roadside
survey results in estimating evasion in traffic and how the survey
data are aggregated into different subgroups"—presumably
subgroups means different types of vehicles, does it? Cars, lorries,
motorcycles etc—"(c) whether alternative estimates
of Vehicle Excise Duty evasion can be developed from the roadside
survey results and (d) the methods currently used for precision
estimation and whether they can be further improved." Did
you pay the University of Southampton to do this report?
Mr Boucher: Yes, we did.
Q41 Mr Bacon: How much did you pay
them?
Mr Boucher: I do not recall. I
could find out.[1]
Q42 Mr Bacon: If you could, that would
be good. It would be nice to know. Did you read it, having commissioned
it from them?
Mr Boucher: Yes.
Q43 Mr Bacon: When did you read it?
When it was available in August 2007?
Mr Boucher: As soon as it had
been written.
Q44 Mr Bacon: Yes. So, in other words,
you did have reason to doubt because they had put it out there
in black and white for you: four good reasons to doubt and the
fact that further investigation was warranted. Can I just ask
the National Audit Office was this drawn to your attention at
any point before you prepared the brief for us for our October
hearing?
Mr Gale: I was aware that work
had been done. If I may add that the significance of those points
was not emphasised to us.
Q45 Mr Bacon: You were aware that
there was reason to doubt the methodology because this said so.
It said further investigation was warranted.
Mr Boucher: The University of
Southampton report was primarily based around the statistical
method that is used in the middle of the process, which is extremely
complicated, and as part of the recommendations they put in some
areas where they thought it might be useful to look at in future,
but it was not a recommendation to say "We think there is
a problem with the existing survey." Just if you wanted to
do a complete audit, a complete check of the survey process, those
are other areas that we should look at in future.
Dr Hickey: Can I add, Chairman,
firstly, that report was made available to the Committee before
the last hearing, so it was on the table and everyone knew about
it. Secondly, as I understand it, the recommendations of the University
of Southampton do not particularly impact the change which actually
did happen, which was the cause of the change from 2006 to 2007.
The points they were making there are not the points about the
nature of the data collection and the accuracy of the original
sightings. They are primarily around the sample size. Clearly,
the sample size is much bigger now, but they are also fundamentally
about the very arcane statistical methodology, as to how you get
from the sightings on the road to the estimates of in stock. If
you remember from the last time round, that was a really big shift.
Q46 Mr Bacon: Mr Boucher, there is,
as Dr Hickey describes it, an arcane methodology and you have
to make a whole series of assumptions, do you not?
Mr Boucher: Yes.
Q47 Mr Bacon: Is it not correct that
you then have to test those assumptions?
Mr Boucher: Yes.
Q48 Mr Bacon: And is it correct that,
in testing those assumptions, you base them on tests that were
done 20 years ago using lorries?
Mr Boucher: The tests that were
done on the lorries—
Q49 Mr Bacon: That is right? That
was the basis of it though, was it not?
Mr Boucher: The tests that were
done on lorries were to show that the workings of the statistical
method actually make sense and works. You could have done it with
any vehicle.
Q50 Mr Bacon: Had the assumptions
been tested more recently than 20 years ago?
Mr Boucher: In each survey year
we look at what is called the relative mileage of the different
vehicles, and the University of Southampton looked critically
at that part of the process and gave—
Q51 Mr Bacon: Did you actually go
out and test the assumptions anew?
Mr Boucher: No.
Q52 Mr Bacon: So they were quite
old assumptions in terms of whether they were still valid.
Mr Boucher: But the University
of Southampton looked at those assumptions and said that they
are robust and fine.
Q53 Mr Bacon: I must say, the way
you use the words "University of Southampton" sounds
like a comfort blanket to me. I have to say I am not that impressed.
I think you had available to you reasons to be doubtful that you
could have shared with the Committee earlier than you did.
Mr Boucher: If I could just respond
to that point, please. In October, at the time of the hearing,
in the same week as the hearing, was the first time that we received
what we call matched data, which is the raw data from the roadside
survey which has been matched with the DVLA database. So it is
the first time we had the raw data and that was the week before
your hearing.
Q54 Mr Bacon: The first time you
had the raw data? I thought you had the raw data in June?
Mr Boucher: Sorry. We get the
raw data from the contractor in August. We then check it to make
sure that—
Q55 Mr Bacon: I thought you did the
survey in June.
Mr Boucher: The survey was conducted
in June. It takes time for the contractor to process the data.
Q56 Mr Bacon: Surely that depends
on what deadline you have given him in the contract.
Mr Boucher: Yes, we gave him a
deadline of August, because we wanted to match against the DVLA
database—
Q57 Mr Bacon: Hang on. You used the
phrase "raw data", not me. I repeated it but the whole
point about raw data is it is unprocessed, so you can get that
straight away, can you not, in June?
Mr Boucher: My mistake. I apologise.
The first time that we had matched data was in October, at the
time of the hearing. At that point was when we started checking
the data and running our own quality assurance checks to see what
the data looked like, whether it was making sense. The first time
we had provisional estimates, the first time I had seen any figures
based on the 2007 survey, was in early January. On seeing those
results, the provisional results, it looked so different from
anything that had gone before that I ordered an immediate double-check
of every single stage in the survey process because at that stage
I could not be sure whether what I was looking at was believable
or whether it was simply the result of a processing error. It
takes at least three weeks to run the survey systems from beginning
to end to get to the final estimates. So on 8 January I had the
first provisional figures, that looked so different to anything
that had gone before, ordered a complete re-check of every stage
of the survey process, and that took us up until the end of January
and the finalised figures in early February.
Mr Bacon: It verges on the incredible
that this processing took so long before anything was flagged
up that was a cause for concern or surprise. I have no further
questions, Chairman.
Q58 Chairman: It is a strange coincidence
that at the end of January we publish our Report, there is a lot
of publicity around it, the motorcycle community erupts with anger
at 40% of them being accused of evading paying tax and, hey presto,
two weeks later the Department completely revises the estimate
down from 40% to 9% and apparently you, as statisticians, were
not aware of any problem right up to January.
Mr Boucher: No, I was not aware
of any problem right up to the end of January.
Q59 Chairman: That is what you tell
us and we must assume that is correct then.
Mr Boucher: We pre-announce the
publication several months in advance.
1 Ev 10 Back
| 
