ANNEX
2007 ROADSIDE SURVEY
Survey methodology and data collection
The 2007 Roadside Survey involved observations
of around 1.7 million vehicles. In order to ensure the greatest
possible accuracy, it involved two particular changes to the data
collection and validation methodology.
First, the use of cameras (as distinct from
purely human observation) was extended as far as possible to all
sightings (apart from motorcycles, where camera technology is
less effective and a few survey sites where there were operational
difficulties with the cameras). In the 2006 survey cameras had
been used in just under half the observations.
Second, this meant that it was possible for
the video picture to be manually checked for all the ANPR results.
This enabled number plate errors to be identified and corrected
far more comprehensively than in previous surveys.
The effect of these changes on the eventual
results proved greater than expected. The reason is that the DVLA
database contains many millions of number plate records relating
to old vehicles, including those which have been exported or scrapped,
as well as more recent vehicles. If the survey identifies a vehicle
number which matches one on the DVLA database, but which is not
currently taxed (or SORNed), it is assumed to be an evading vehicle.
The much greater checking and validation during the 2007 survey
substantially reduced this cause of error. What has become clear
is that the impact of the greater data accuracy was "asymmetric"
and not "neutral" in its effect, impacting the results
in one direction; and that the impact was greater than expected.
(It is important to bear in mind, of course, that—since
the vast majority of vehicles are properly taxed—the sample
size of untaxed vehicles is relatively limited, so changes in
the characteristics of this sub-sample may have relatively pronounced
effects on the results).
The methodological changes above were changes
in data collection and validation, not in the method of estimating
evasion. The methodology on which the survey analysis is based
was reviewed in2007 by the University of Southampton. Their conclusion—which
was published and reported to the NAO and Committee—was
that it was largely fit for purpose. They did make a few suggestions
for improvement, but these were minor and had no significant impact
on the result. The impact on the results stemmed from the change
in the way the data were collected, the comprehensive use of cameras,
and the improved checking which this in turn allowed.
Survey timetable
The raw data for the survey was collected by
the contractor in June 2007. Statisticians received the raw data
in August and checked that the database met the specification.
That database was sent to DVLA and DVLNI in September for matching
against the vehicle licensing databases. The outputs from the
matches were returned to the statisticians in October. By the
time of the PAC hearing on 17 October the statisticians had only
just completed their checks that the data files had arrived in
the correct format and that the rules for identifying evading
vehicles had been applied correctly.
Over the following months the statisticians
ran their detailed quality assurance programme, completing in
December. The survey contractor had already manually checked all
the ANPR readouts, but the statistics team took sub-samples of
the observations to double-check against the video images, and
additionally manually re-checked 18,000 video images of suspected
evading vehicles. The database was finalised in December, and
used to produce the first, provisional, estimates in January.
Statisticians immediately noticed that these
provisional results were substantially different to previous years
and so ran a series of further checks. A first draft of the publication
tables were produced, along with an account of the detailed quality
assurance, for peer review on 5 February. One official from DVLA
and one from DfT centre were involved in this review, but for
quality assurance purposes only and on the condition that the
results would not be shared with anyone else, either within the
Department or externally.
In accordance with the National Statistics Code
of Practice, the results were not shared with senior officials
or Ministers until shortly before publication. A very restricted
set of named officials were shown the results on 8 February. Ministers
were briefed on 12 February and the results published on 14 February.
| 
