LSE PUBLIC POLICY GROUP
Review of National Audit Office Report
HC 730 Session 2001-02
Public Private Partnerships: Airwave
PUBLIC PRIVATE
PARTNERSHIPS: AIRWAVE
1. In February 2000, the Police IT Organisation
(a non-departmental public body, hereafter PITO) signed a deal
with British Telecommunications plc (now the separate company
O2) for provision and management of a national radio communications
system across all police forces in the UK. The project is still
in its early pilot stages and is expected to cost nearly £1.5
billion over 19 years' use. This report examines value for money
of the project to date.
Administrative and management context
2. There is a clear overall explanation
of the Airwave public-private partnership. The project involves
a diverse range of public and private stakeholders at various
levels of jurisdiction from local police forces up to Home Office.
A good explanation is given of the TETRA technology, the three-stage
service model, and the PFI charging mechanism and coverage pay-off.
Issues relating to the potential problems of sharing the new network
with other emergency services are effectively set out. Deficiencies
with the current mobile communications systems used by police
forces are effectively described. The powers of the Home Office
to secure central procurement of a national replacement for current
systems, which are currently operated on a force-by-force basis,
are made clear, as is the strong Home Office preference for centralisation.
Structure and presentation
3. The report is well structured and clear.
The three main Parts follow on logically through the procurement
and implementation process to date. The Executive Summary gives
a reasonably concise précis of the main points. However,
a couple of points seemed to be more clearly communicated in the
Executive Summary than in the main body of the report. For example,
a substantial portion of the income stream flows to O2 at an early
stage, thus reducing the incentive for them to take action to
meet outstanding contractual conditions. Using narrative headings
for each of the three Parts is helpful for readers, and they are
not too evaluative. The Part 1 heading is: "The procurement
strategy was well thought through but was not delivered in full".
It rather vaguely relates to the actual contents of Part 1. The
Part 2 heading is: "In difficult circumstances the procurement
was well managed by the project team". This sounds rather
melodramatic, and seems to imply a rather loose definition of
"well managed" given the actual contents of Part 2.
Presentation was reasonably good with a few outstanding points,
such as Figure 2 and its pullout section. In general, this is
a focused and well-written report. It deals with a complicated
and topical area of PFI and new technology in an accessible way.
Graphics and statistics
4. The limited use of charts and other forms
of statistical presentation reflect the limited use of data per
se in the report. Two useful graphs "breakdown of police
spending" (Figure 3) and "time spent in the police station"
(Figure 17). There is a table setting out PITO advisers' costs,
tabulating the total of £1.9 million over the procurement
period (Figure 7). A very basic table shows the public sector
comparator costs against the Airwave should-cost model (Figure
11). Surprisingly, there are no Appendices giving more detailed
breakdown of the two predictions or elucidating the difficulties
of constructing both cost models. Figure 19 on the benefits of
Airwave also seems vague.
Scope
5. The report focuses narrowly on a single
procurement, but also draws out potentially wider implications
for other procurement processes involving new or uncertain technologies.
Important issues relating to PFI procurement are addressed, such
as charging mechanisms, the problems of specifying services requirements
by contract in advance, contractor service delivery standards,
public sector resource and expertise atrophy, single bidder competitions,
and risk transfer. The scope seems to be defined slightly differently
at two points. In Appendix 2, the report's focus is defined as
whether Home Office and PITO "achieved objectives in procuring
and implementing" Airwave. In the Executive Summary paragraph
4, however, the scope is defined as "[examining] the prospects
for achieving value for money from negotiation and early implementation
of the core service". By specifying scope as future prospects
for VFM rather than actual performance to date, the study team
acquire an exit clause which lets them skirt over any hard-edged
critical comments about the procurement of the relationship with
O2. For example, PITO's lack of success in any gain for the police
if potential sharers take-up Airwave passes without evaluative
comment (paragraph 11). By contrast previous NAO studies have
highlighted the importance of using innovative mechanisms to guarantee
that public agencies can share in mutual benefit incentives (for
example, HC 21, The Radio-communications Agency's Joint Venture
with CMG).
Methods
6. The methodology used is set out in Appendix
2 and combines interviews and documentation review. However, there
is very little data collection and "number crunching"
analysis on either the "Should-cost" model or on the
public sector comparator. The study team point out that information
was limited, even for PITO. Nevertheless it would have been good
if the study team had set out to deconstruct independently the
data available, perhaps commissioning some accounting analysis
from specialists in this field. The study team did address the
business model analysis which PITO constructed, but only in a
limited way (paragraphs 3.26 to 3.28). for example, PITO estimated
a reduction of at least 10 per cent in time spent in the police
station and efficiency gains of up 30 per cent. Assuming that
police officers are likely to generate more paperwork by being
outside the police station an extra 10 per cent of the time, these
reductions seem rather optimistic and under-analysed. We were
unclear how much data systems would allow functions to be fully
automated, for example, creating an automatic log of police officers'
movements and timings which they would not then need to redo in
reports? Also, the report suggests that increases in the cost
of services is likely to be compensated by a reduction in staff-costs
and/or services. It is not clear whether these are net estimates
which include increased cost factors deriving from the new contracted
services. Elsewhere in the report the potential for savings or
service enhancements arising out of new police working methods
were given less prominence and there appears to be no overall
assessment of such projections, even though these are relevant
to assessing the value for money of Airwave. Although Figure 2
shows police officers receiving photographs and text on handheld
computers, for the rest of the time the report is incredibly unspecific
on what "data services" will mean. For example, will
police radios give continuous GPS (positioning) information to
officers on where they are, and to their managers? Will police
services get GPS services such as mapping and direction-giving?
Assuming that these presumably high priority services are indeed
included in the contract, where do they sit between the core,
the "market exclusive" services which are only O2 can
supply, and the "market competitive" services which
other companies can supply? These issues should be crystal clear
for readers, not left foggy.
Conclusions and recommendations
7. The major aspects of PFI procurement
are covered well, given the obvious limitations in data available
and in co-operation from O2 as a private sector company. The "lessons
learned" section on pages 6 and 7 is relevant to PFI in general
as well as to this particular project. Some interesting incentive
issues are highlighted which have a considerable impact on national
procurement schemesfor example, the imperative that all
police authorities sign up to Airwave regardless of increased
cost and resulting potential reduction in local services; and
the incentive for O2 to increase the scale of the project (extending
it to other services) despite potential adverse impacts on service
quality. On other occasions the study team seem to not dig too
deeply into what seem to be obvious barriers to effective implementation,
and bland-out phrasing are used to cover up problems: "lessons
learnt from the pilot and earlier cohorts will b disseminated
in order to make migration of later forces smoother" (paragraph
3.6). It is clear from the report that the roll-out schedule is
very tight given the 2004 spectrum expiry and events such as the
2002 Commonwealth Games. It does not yet seem clear that police
authorities will have secure and reliable means of radio communications
by then, given that the first pilot is only just underway. And
the question of contingency plans if systems are not ready on
time is not covered much. The problem of coverage specification
dealt with in paragraph 3.7 was not clearly enough explained,
despite its critical nature in relation to the success of the
system. How can it be that areas of "incidental coverage"
have emerged? How can the study team be so sure that the "procurement
was well managed by the project team" (as the Part 2 heading
says) if they have left contractual room for "incidental
coverage"? Occasionally the report also seems to reiterate
familiar hype around PFI without saying specifically what the
main advantages of PFI would be for this procurement. Paragraph
1.1, for example, is quite peculiar in its rather abstract, specifically
"context-less" assessment of PFI. The study team write
that:
"the PFI can offer the prospect of better
value for money than conventional procurement because it adds
a wider range of private sector capabilities to those previously
available to the public sector [...]. This means that, in specifying
what they want from PFI deals, procuring departments need to avoid
imposing unnecessary constraints on how the private sector carries
out projects".
The validity of these two sweeping statements,
especially the latter, are by no means borne out by the issues
raised in the report. It seems odd that they should appear in
the opening paragraph as a kind of "universal truth"
without more specific reference to the advantages of PFI in this
particular case.
Overall
8. This is a valuable report, at the higher
end of the three scores, which deals with some important issues
of PFI applied to new technology. There are some data weaknesses
and our readers were concerned that at times the report is a bit
vague and the study team withholds analysis from some potentially
disconcerting issues.
Minor Points
The key to Figure 2 contained some interesting
points on the "payment-to-coverage" pay-off, one of
the more interesting issues of the report. Unfortunately, the
rather toy-town portrayal of Airway in the pullout section was
far too simple to add any real value to the analysis. We assume
that officers featured would have to be well within designated
coverage areas, portrayed rather confusingly in Figure 16. The
text states that "Figure 16 shows an example of areas where
coverage is guaranteed as well as areas where coverage is obtained
but not paid for and therefore not guaranteed". In fact all
that the Figure is to shade two areas, point arrows towards them
designating both types of coverage, and then add towns in one
area type or the other.
LSE Public Policy Group Pro Forma Scores
for HC 730: Public Private PartnershipsAirwave
1. To what extent does the report clearly
describe the administrative and managerial context within which
the area examined is carried out? = 4
2. To what extent is the report well-structured
and well-written, and does it include an effective executive summary?
= 3
3. To what extent were graphics and statistics
appropriately used, and how well presented were any that appeared?
= 3
4. To what extent is the rationale for the
report's scope clearly set out (within NAO's remit, which excludes
questioning the merits of policy objectives)? = 3
5. How far is the methodology clearly set
out? And does it include an appropriate range of evaluative criteria
and techniques to answer the question posed in the specification?
= 3
6. To what extent did the conclusions provide
a balanced view of successes and shortfalls in performance, the
underlying causes, and make reasoned and cost-effective recommendations
for remedies? = 3
7. To what extent has the study been successful
in meeting its objectives, and maintaining or enhancing NAO's
reputation? = 3
|