Examination of Witnesses (Questions 320-339)
11 NOVEMBER 2003
MR MARTIN
EVANS AND
MS MOLLIE
BICKERSTAFF
Q320 Mr Betts: Surely you can spot
a piece of nonsense when you see it, can you not?
Ms Bickerstaff: I would hope so.
Chairman: It is difficult for me to get
a smile on the record.
Q321 Mr Betts: Generally you feel
that a statistical measure is not really of any use if it is not
demonstrating the efficiency of the service. It is being collected
because it has always been collected like that. Is it not your
job to say, "Hold on a minute. We are collecting nonsense
here and we ought to change the basis of the statistics we are
working on."
Ms Bickerstaff: You have said
that before. You said it earlier in this session, when talking
about collecting information and efficiency. I think it is more
the effectiveness of the Fire Service that we are concerned about.
We certainly will express our views when we have had a decent
opportunity to make them and give them.
Q322 Mr Clelland: I would like you
to say a bit more about the research and statistics that you are
using to assess the current efficiency of the Fire Service and
in particular what key indicators will you look for to see if
the Service has become more efficient?
Ms Bickerstaff: I do not know
whether my colleague would like to come in on that. We certainly
have a long history of looking at efficiency. We have been auditors
of the Fire Service for a very long time and there is a good deal
that can be done on efficiency although one has to, as always,
be careful with indicators. For example, we worked in Merseyside
and you can say that it is the cheapest in that its cost per fire
is very low but you can also say that it is the most expensive
because it spends a great deal of money because it is a fully
whole-time service.
Q323 Chairman: You are saying it
spends a lot of money because it is a whole-time service. When
you compare it to Manchester they are both exactly the same in
that respect and yet it spends almost a third more than Greater
Manchester. Can you explain that?
Ms Bickerstaff: I am learning
here but I believe that Merseyside—apart from London—is
the only fire service that is whole-time staffed throughout. I
understand Greater Manchester uses retained fire fighters and
that would make a very big difference to their salary bill.
Q324 Chairman: I thought that in
Greater Manchester there were very few retained fire officers.
Ms Bickerstaff: I think the comment
that you make is exactly supportive of the point I am struggling
towards here which is that there are many measures of efficiency—the
Audit Commission has a great deal of experience on that—and
we try to establish a balanced view based on what can be complicated
information. On effectiveness my point is only that the underlying
data has a long way to go yet for the Fire Service before it can
help assess the effectiveness of prevention and protection work.
Mr Evans: If I could add to my
colleague's comments, I think efficiency is a notoriously difficult
concept to capture in the public sector. It is relatively easy
to measure inputs but it is harder to measure outputs. We can
measure inputs through costing data and through benchmarking one
service against another, but it is particularly hard to measure
the outputs and with the increased focus on fire prevention rather
than simply firefighting it is going to be harder, arguably, in
the future. In the short term I think we are going to have to
continue to focus on measuring inputs through costing data,
through comparison against benchmarks, standards, targets and
so forth, but put that in the context of a more rounded assessment
of the performance of the Fire Service and its capacity to improve
through the CPA process.
Q325 Mr Clelland: Which indicators
do you expect to have improved in three years' time? If they have
not, would you regard that as a measure of failure?
Mr Evans: The point about the
performance indicators is that they need to be aligned with the
objectives of the Service and obviously the White Paper is significantly
changing the objectives of the Service and the ODPM is leading
that review at the moment. As my colleague said, we are a consultee
in that process along with other expert stakeholders. In the short
term, through our verification process, we are going to look to
assess progress by fire authorities in making the changes set
out in the National Pay Agreement and in the White Paper. In the
longer term we will be looking to pick up progress through the
CPA process.
Q326 Mr Clelland: It all seems a
bit woolly. In three years' time are we going to be able to actually
measure whether improvements have come about?
Ms Bickerstaff: In three years'
time we will—through all the work that we do with the fire
authorities—look at how they decide to measure themselves.
As Martin has said, different people will have different targets.
We will expect to see improvements over that period. The easiest
one to pick as an example for you is fire deaths. There are certain
national targets on fire deaths. Individual fire authorities often
set themselves even more demanding targets on bringing down fire
deaths. Everyone, including ourselves, will look at performance
against that and the other key indicators: indicators on arson,
indicators on injury. You do not need me to tell you that the
small numbers that are involved in fire deaths can sometimes produce
blips on a local basis, but nationally it remains a very fine
indicator.
Mr Evans: The first Comprehensive
Performance Assessment of fire authorities which we are planning
to undertake in 2005 will set a baseline against which progress
and improvement can be measured over time. Part of that Comprehensive
Performance Assessment will be giving credit to—or accounting
for—the current performance against BVPIs that are set by
ODPM which comprise what we call corporate health indicators and
specific Fire Service delivery indicators. We will be tracking
performance over time and the outcome would be the overall assessment
of that authority through the CPA process.
Q327 Chairman: I am worried about
this question of measuring performance just on deaths. As you
have just pointed out, they are statistically subject to blips.
You could argue that if you are going to look at deaths then you
have to be able to attribute the cause of the deaths. Do you think
that in three years' time you will be able to identify how many
were the cause of poor fire cover and how many as a result of
poor building regulations?
Ms Bickerstaff: I did not wish
to give the impression that anyone was being judged purely on
fire deaths. I quoted that as the most well-known example. There
is a desperate need for more sophisticated indicators that would
help assess how well community fire safety work is going.
Q328 Chairman: What are those indicators
going to be? Is the Audit Commission going to develop those indicators?
Ms Bickerstaff: If I knew the
answers I would be a popular person at the moment. No, the Audit
Commission is not going to develop them. The Audit Commission
is going to work as a stakeholder with the ODPM to do all we can
to help.
Q329 Chairman: So the ODPM is supposed
to be developing these statistical records.
Ms Bickerstaff: The responsibility
rests there, I believe.
Q330 Chairman: How soon are you going
to have this performance assessment methodology worked out and
how soon is it going to be in place? You are, in a sense, holding
up some of the payments to the fire officers at the moment.
Ms Bickerstaff: There are two
things which we are doing. One is the verification audit. I think
your question is mostly about that.
Q331 Chairman: So when is that going
to be finished?
Mr Evans: The Commission, at its
meeting last Thursday, formally considered the responses it has
had to its statutory consultation with the Local Government Association,
the unions and other stakeholders on its proposals for how it
should carry out this work and how it should be funded, and it
has agreed to proceed on the basis of its original proposals.
We are now aiming to send out to fire authorities in the next
couple of weeks the self-assessment tool which will be the first
part of this verification process. By experience we have found
that asking bodies—in this case fire authorities—to
actually assess the progress they are making themselves is a very
valuable and positive part of the process.
Q332 Chairman: It saves you a bit
of work as well, does it not?
Mr Evans: It does help to bring
together the evidence that the fire authority has internally to
support its assertions about how well it is doing. Following on
from that we are gearing up to train our auditors and staff in
the firms that we appoint to fire authorities with a view to them
starting on-site immediately after Christmas (early January) and
the timetable that we have set out—and I have written formally
to all the stakeholders yesterday—provides for the nationally
moderated results (because the results of local work do have to
be moderated nationally to ensure there is consistency across
the country) to be made available to the Local Government
Association and the unions and other stakeholders in mid-February.
If we can shave one or two days or a week off that timetable we
certainly will; we obviously recognise the urgency.
Q333 Chairman: This question of a
nationally moderated system means that you are going to have to
move at the pace of the slowest, are you not? Is it fair that
those fire authorities who are being fairly dynamic and getting
on with it are going to have to wait so that you can nationally
moderate it, taking into account all the information from all
the 40 authorities?
Mr Evans: My understanding is
that the Employers are intending to make a decision about the
stage two award on the basis of all the results. In other words,
they are not going to distinguish between individual fire authorities,
although that is really a matter for them rather than us. All
our auditors will be working to the same timetable and they are
all acutely conscious of the significance of this piece of work,
the sensitivity that is attached to it, and I can assure you that
they will be cracking on as fast as they can to meet our deadline.
The track record of our auditors in delivering to demanding timetables
on other analogous pieces of work—we have done work on waiting
list times in the NHS—is good and they do deliver. We have
not promised more than we can deliver. The raw data will be made
available to the LGA and the unions in mid-February and then there
will be a national report which will be made public in March.
Q334 Chris Mole: Is there a challenge
process in that if any authority is unhappy with its assessment?
Mr Evans: Auditors locally will
form an assessment. They will discuss that with the fire authority.
They will then refer their initial findings to the Commission.
We will review their findings; we need to be satisfied that they
have applied our methodology on a consistent basis and that different
audit suppliers—whether it is one firm or another or our
own in-house auditors—are applying it in a consistent way.
Within that there will be a process of challenge from fire authorities.
What we envisage doing is, for each of the elements of the verification
audit, giving a traffic light indicator about how well that authority
is doing in making progress in implementing the changes in question.
Obviously green will indicate that everything is going fine; amber,
good progress but some problems; red, more serious problems that
need to be addressed in the short term. We need to be satisfied
and the national stakeholders need to be satisfied that those
assessments are being applied on a consistent basis.
Q335 Mr Betts: One of the criticisms
that has been made to us is that there is probably too much bureaucratic
audit-based regulation and review of public services around where
authorities say what they are going to do and you just go along
and check they have done it, put a tick in the box and that is
considered somehow to be a measure of the efficiency or effectiveness.
Can we be assured that that is not what we are going to end up
with here?
Mr Evans: Yes, I think we can
assure you of that. Certainly at the Commission we recognise that
public services are changing, the public's expectations of the
services are changing. The Government has put a lot of new money
into public services and it and the public are expecting to see
real improvements on the ground. That is why we have developed
a new approach to audit and inspection which we have called Strategic
Regulation, which is a more top down, risk-based approach to audit
and inspection, which is proportionate to the success or effectiveness
of the audited and inspected body and is focussed on the areas
where we can add value and contribute most to improvement. We
think we have a key role not only in our traditional role of providing
assurance on expenditure of public money, the conduct of public
business and value for money, but also acting as a catalyst for
improvement by holding up a mirror to individual bodies so they
can see how their performance is perceived by others; an independent
and objective assessment which can be relied upon as being independent
and objective. It is not about getting a tick in the box. In this
particular case I think you have to recognise that this was a
piece of work we were asked to do by the parties to the national
pay negotiations. Whether we would have done it exactly in this
way if we had not been asked I think is a moot point, but we were
asked.
Q336 Chairman: Why did you not refuse?
Mr Evans: We had that option.
We were not consulted about the reference to the Audit Commission
in the national pay award. The first we knew about it formally
was in June when the deal was concluded. Given where we started
from we have done well to get to the position where we are now.
A lot of preparatory work had to be done in the period. The words
in the pay agreement are rather cryptic about our role. We had
to sit down and work out what they meant and we had to talk to
stakeholders about what they thought they meant, what their expectations
of us were. We then had to agree some broad criteria against which
we could assess progress and having agreed those criteria we then
had to carry out some local testing work to see whether what was
being proposed was actually technically doable on the ground.
All this time we were having to keep in close contact with our
sister audit agencies in Scotland and Northern Ireland who are
responsible for the verification process in Scotland and Northern
Ireland respectively. Then we had to go through a statutory period
of consultation which we are required to do under our primary
legislation, The Audit Commission Act 1998. We have been pulling
out the stops; we have not been dragging our feet and we are confident
that we will deliver to what is a demanding timetable.
Q337 Mr Betts: In the future are
we going to get some real useful measures of effectiveness of
the Fire Service? What we have seen so far is that it is a fairly
rigidly run Service—run a certain way as we have heard—laid
down, prescriptive, but there are very few measures of whether
even that is effective. Now we have a much more complicated risk
management approach, and you are saying yourself that you are
not leading on the drawing up of the indicators—that is
going to be done by the ODPM—are you confident you are actually
going to be able to deliver something at the end of the day which
can verify which services are effective, which are more effective
than others and where changes are needed?
Mr Evans: I think that is where
we move across from the first piece of work that we are responsible
for—verification audit—to the second, which is a longer
term assessment, involving the CPA.
Q338 Mr Betts: Can you respond to
that question?
Ms Bickerstaff: The answer is
yes. The CPA—Comprehensive Performance Assessment—is
a much more interpretive form of looking at performance. To answer
your question, it does not put ticks in boxes, it asks some serious,
high level, top down questions about how authorities are setting
their priorities, determining their vision, achieving quality
improvements in the service they deliver. It examines the balance
between strengths and weaknesses and it sees itself very much
as a step in the process of improvement. It is not a rigorous
checklist with a tick in the box.
Q339 Chairman: Where are you going?
What is improvement?
Ms Bickerstaff: Each fire authority
will have its own aims and issues. Each fire authority will be
setting its own targets. They will be doing that within the overall
national framework and, as you know, they have already started
under the IRMP. They will also be making potentially radical changes
locally. Each fire authority will differ. We will assess with
them how well they are going about their process and we will give
a view.
| 
