Written evidence submitted by Ipsos Mori
SUMMARY
Performance management and independent assessment
and inspection have been important in driving improvement across
the local government sector.
Despite the well documented criticisms of the Audit
Commission, in our view Comprehensive Performance Assessment (CPA)
in particular, especially in its initial iterations (prior to
what appeared to be the sector's own "grade inflation"),
was important in improving visibility of both good and bad performance,
and this, with the threat of central intervention, was more effective
in improving local government performance than relying on the
ballot box alone, or auditors alone.
In considering what may or may not be appropriate
in future, we think that some ability for local people, auditors,
independent experts and commentators to make fair and public comparisons
between the performance of authorities is extremely important.
While the process involved needs to be proportionate, our view
is that being able to make valid comparisons on performance between
authorities, on a public, like for like basis, is vital in driving
up performance and highlighting areas for improvement.
Our work for the Local Government Association (LGA)
as far back as 2003 illustrated the general support behind the
principles of CPA, with the majority of local authority chief
executives seeing the assessment as a fruitful process for their
authority. Most also agreed that it was an effective way of comparing
performance between local authorities and that it challenged their
attitudes and the performance of their authority. Most admitted
that the sector would not have improved as quickly without some
external inspection process. While there were many complaints
about the detail of the Best Value and CPA regimes, one thing
that virtually all chief executives and local politicians agreed
on was that they challenged attitudes and performance.
More recently, our evaluation of the first year of
Comprehensive Area Assessment (CAA) showed that there was widespread
support—from assessed bodies, local partnerships, government
departments and inspectorates—for the principles of CAA.
None of this is to suggest that any of these processes
were perfect: throughout there were requests
for greater transparency over the process, a lighter touch approach,
better engagement from elected members and the public, and a concern
about the level of cost involved. For example, the survey above
saw most authorities agreeing that the cost of inspection outweighed
its benefits - views were clearly mixed (see the following chart).
The public, with no detailed understanding of the
costs or processes involved, support cost effective inspection.
While few members of the public have ever had a detailed understanding
about the ways in which local public services are inspected (17%
say they know at least a fair amount), the vast majority (89%)
feel that independent inspection is important and 80% that it
leads to improvements. The public support the principle of
independent inspection (as long as it provides value for money—although
they themselves find this extremely difficult to judge).
At the same time, our research suggests that the
public will not necessarily actively seek out, or remember, information
about how their council performs (despite information
being available, most people—81%—don't know or can't
remember what the key official judgement of their council was).
This does not mean that inspection should be abandoned—the
public like to think some independent inspection and regulation
is available and expect officers, members and parish councils
and others to use this information to improve performance and
ask questions. They support frequent independent inspection of
old people's homes, for example.
This is important when we consider current moves
to put more performance and spending information into the hands
of the public in the hope of mobilising a new force of "armchair
auditors". This has been an ambition of government since
Mrs Thatcher's time (for example, the mandatory publication of
performance data in local newspapers happened long before the
web was even thought of in this context). But, evidence suggests
that citizens are primarily "passive consumers" of this
kind of information, and the desire for this type of information
is limited to a small minority of informed and engaged citizens—likely
to be older residents, more middle class and owner occupiers.
Moreover, if publication is to be mainly online,
because of the continuing class/ age gradient in internet access
(for example, only 18% of working class pensioners (C2DE) are
currently online) there will be a considerable challenge in
engaging the majority of the population in the scrutiny of public
service performance data. We support the current plans to
publish much more information online—as a minority of people
will find it very useful in holding public services to account—but,
we think expectations of the extent to which online publication
will directly engage the public can be overstated, although the
media and the minority of those active and involved will undoubtedly
use it.
Another key challenge is that many members of the
public do not trust official statistics and other types of performance
information: most think that figures are not reported honestly,
or without interference. The public place a lot of emphasis on
public information being produced by independent organisations
and regulators.
Finally, we would point out that it will be important
in any future system for inspection scores to at least resemble
public perceptions of performance. Without
this, judgements become professionals' views of other professionals'
services. In its first few years CPA did correlate positively
with public perception (although rather fewer of the public would
rate their council as excellent or good than would the inspectors).
Later, as councils' "grades" improved, the correlation
diminished.
However, our view is that public perception data
is a vital part of any inspection/ regulation system, to encourage
services to concentrate on what actually matters to local people,
rather than simply counting outputs, or making professional judgements.
Having like for like measures is vital if this is to be meaningful.
Despite some valid criticisms of the Place Survey (until recently,
the key mandatory mechanism used for collecting citizen perceptions
data), consultation with the sector has found there to be general
support for some kind of standard public perception measurement:
"… the idea of a regular questionnaire to get local
people's views on matters that affect them was beneficial to help
local authorities to address residents' feedback… it was
important to track people's changing perceptions to help determine
whether interventions made in an area resulted in the right outcomes
for local people."[20]
Finally, perceptions data should not be treated in
isolation: we would argue that local circumstances need to be
accounted for in drawing a fair assessment of a council's overall
performance from it. Our own Frontiers of Performance analyses
demonstrate just how difficult it can be for some councils to
achieve high perception measure scores due to the challenging
environment in which they operate (with factors such as deprivation,
ethnicity and even region impacting on how easy or difficult it
is achieve high levels of satisfaction amongst local residents).
It showed different types of authorities excelling at different
things. Any future process of inspection should use like for like,
comparable methodologies, for the measurement of public opinion
to allow valid comparisons between authorities and prevent gaming.
March 2011
THE ROLE
OF INDEPENDENT
INSPECTION AND
ASSESSMENT IS
VALUED BY
THE SECTOR
Despite the end of CAA, and its predecessor CPA,
it is clear that performance management, inspection and assessment
has gone some way to helping strengthen the local government sector,
with overall assessment scores improving over recent years. Ipsos
MORI research with the local government sector itself backs this
up.
We would also argue that CPA provided an important
system for recognising good performance and hard work on the part
of councils (and equally where poor performance or service failure
has needed to be addressed).
Our research into the issues illustrates that local
authority leaders and chief executives generally see the assessment
and inspection process as both a useful and challenging exercise,
and one that helps improvement, despite some of the mechanics
of how the respective regimes have worked. Notwithstanding the
improvements made to the CPA inspection regime since the research
was conducted, a study for the LGA in 2003 found a number of interesting
messages still pertinent to the discussion today:
— The
majority (three-quarters of county and single tier authority chief
executives) saw the CPA assessment as a fruitful process
for their authority.
— Most
also agreed that the CPA assessment framework was an effective
way of comparing performance between local authorities.
— Most
thought that the CPA challenged their attitudes and the
performance of their authority and that the CPA had a fair amount
of impact on their strategic planning.
— The
corporate assessment element was well received. The inclusion
of peer reviewers in the corporate assessment was highly supported
by chief executives, regardless of what CPA assessment score they
had received.
— Self-assessment
was seen as giving councils the opportunity
to spend time looking at how they worked, and giving a fair account
of their own performance and priorities: it was the most highly
regarded component of the CPA process.
— Service
judgements were regarded as the least
useful element when compared to the corporate and self-assessments,
although most chief executives (66%) still found this a useful
process. The weaker support for the service judgement seemed to
arise partly from concerns about the service blocks identified
for assessment and more strongly about the weights applied to
them.
— There
was some concern about the quality of inspection: opinion
was divided on whether the basis for the CPA judgement was clear
or not and many were not only doubtful of the evidence, but of
the quality of inspectors too.
More recently, our evaluation of the first year of
CAA showed that there was widespread support—from assessed
bodies, local partnerships, Government departments and inspectorates—for
the principles of CAA including:
— the
focus on outcomes as opposed to individual services;
— reducing
the burden on assessed bodies;
— ensuring
that local inspection is a more unified experience;
— giving
more emphasis to local context and priorities, and
— taking
a more forward looking approach which focuses on risk rather
than past performance.
This said, experiences of the stakeholders researched
in this study were mixed. For example, there was limited engagement
in the CAA process from elected members, the third sector and
citizens more widely, with the Place Survey providing the main
source of the public's views. The majority of assessed bodies
found both understanding CAA's approach and staff time spent working
on CAA took longer than they expected. Views varied over whether
CAA was more of a burden than previous assessments—although
most did not believe it had reduced the burden of inspection.
Assessed bodies felt the CAA process was fair although
there were concerns over transparency, particularly in terms of
the area assessment and criteria and procedures for awarding flags.
Nevertheless, red flags were seen to help drive improvement, and
there was an interest in learning from areas with green flags.
There were mixed views about the Oneplace website and CAA
reports—some organisations thought they were too bland,
with too little detail to support improvement. There was a general
agreement that CAA had had a stronger focus on inequality and
sustainability than previous assessment frameworks. But, there
were mixed views about whether there had also been a stronger
focus on vulnerable people, and no evidence of CAA having had
a greater focus on value for money.
There were a number of views as to how the inspection
programme could be improved looking ahead that might be important
to consider when establishing any future approach, particularly:
— Having
a strong focus on local priorities.
— Better
engagement from elected members, district
councils, the third sector and the public.
— Improving
the transparency of the process.
— Ensuring
that "underpinning themes", such as inequality,
sustainability, vulnerable people and value for money, are better
incorporated into the process. For example, there was considerable
support for the development of area based assessments of value
for money reflecting the concept of "Total Place".
— Reducing
the burden of inspection through a lighter
touch more proportionate approach through, for example, less frequent,
risk based organisational assessments and/ or area assessments
and whole area Use of Resources (UoR) assessments.
INDEPENDENT INSPECTION
AND ASSESSMENT
IS ALSO
VALUED BY
THE PUBLIC
Our research shows that the public also values the
role of independent assessment and inspection. While few members
of the public know about the ways in which local public services
are inspected (around 17% say they know at least a fair amount),
the vast majority (89%) still feel that independent inspection
is important and 80% that it will lead to improvements. The value
of independent inspection is also something we note from our research
in other sectors, such as our work on policing inspection for
Her Majesty's Inspectorate and Constabulary and education inspection
for the Office for Standards in Education (Ofsted).
But, whilst people find the principle of inspection
important, our research suggests that they will not necessarily
actively seek out, or remember, information about how their council
performs (despite information being available most people—81%—don't
know or can't remember what the key judgement of their council
was).
WE SHOULD
BE REALISTIC
ABOUT MOBILISING
THE PUBLIC
AS "ARMCHAIR
AUDITORS"
The issue of engaging the public with this kind of
performance information is important when we consider the current
argument from government, which is seeking to push more performance
and spending information into the hands of the public in the hope
of mobilising a new force of "armchair auditors".
This of course has its merits, but, as our research
suggests, are the public really all that interested in knowing
what local public services are doing? And do they really trust
the information presented to them?
Whilst our research shows that better and more communication
is important in helping to improve residents' overall satisfaction
levels with their councils, it also suggests that we need to manage
our expectations in terms of just how engaged the public are—and
want to be—in monitoring the performance of the sector and
in turn holding it to account. Recent polling points to a general
lack of awareness amongst the public about how their local public
services are performing. In 2010, around two in five members of
the public said they had seen information about the performance
of their local council. But, only a quarter of people who had
seen performance information about local public service providers
actively sought it out. This suggests that when it comes to accessing
information about how well their local authority is doing, the
majority of people remain primarily "passive consumers".
Taking this further, even fewer know about the ways
in which local public services are inspected, with three-quarters
saying they don't know much.
One consequence of this low awareness is that desire
for this type of public information is limited to a small minority
of informed and engaged citizens. Based on what we know about
those who have accessed performance information in the past, evidence
suggests that these are likely to be older residents, of higher
social grades and owner occupiers. Many citizens, especially from
C2DE social grades, say they are uninterested in tracking the
performance of public services, so long as they feel the services
are doing their job. This poses a considerable challenge in engaging
others in the scrutiny of public service performance data—and
those most likely to be the recipients of the high value services
provided by local authorities (social care, etc).
Our research suggests that public services need to
be realistic about how interested the public will be in becoming
"armchair auditors", beyond the small minority of people
who are already well informed and engaged in civic life. We know
that the majority of people want to be more informed about public
services, but not necessarily fully engaged in their decision-making
processes. While most people do consider citizen empowerment to
be important, few take up the opportunities in practice.
Research for the Audit Commission about how the public
access and use information tells us that that public services
need to raise the profile and potential utility of performance
information—polling evidence suggests that the vast majority
of the public still get this kind of information from the media
(39%).
In order to tailor performance information, it is
important to understand citizens' reasons for accessing it, and
current access may be low due to people having difficulty translating
available data into a picture of performance they can understand.
There is also an issue about the medium through which the information
is provided, with much coming via online sources still impenetrable
to a significant minority of the population. Better tailoring
this information to citizens' requirements may therefore increase
access.
Finally, there is an important point to make about
trust. In short, public trust in public officials, particularly
politicians and government ministers, is lower than any other
profession excluding journalists, and in some cases is declining.
Trust in councils is also lower than it is for other local services
(although Citizenship Survey data suggests that trust levels have
been improving here). While the public place a lot of emphasis
on public information being produced by independent organisations
and regulators, at the same time, they are increasingly sceptical
about the legitimacy and accuracy of official statistics produced
by both government and independent sources.
Our research into the specific area of local government
inspection and performance assessment shows that while the public
are positive in principle about being involved in inspection,
they generally make little use of inspection reports in part because
of an overall scepticism about the usefulness, reliability or
authenticity of them. There is a sense that these sorts of reports
can be totally divorced from real life, especially where assessment
scores do not match their own personal experience. Inspection
reports are also seen to rely too much on the filtered evidence
statutory services choose to show inspectors—leading to
a mistrust of official reports and judgements.
Going forward, there will be clear communications
messages to convey about both the potential usefulness and reliability
of inspection and assessment reports to local people and communities,
and about how robust any judgements will be.
THERE IS
AN IMPORTANT
ROLE FOR
PUBLIC PERCEPTIONS
MEASURES—BUT
LOCAL CIRCUMSTANCES
NEED TO
BE ACCOUNTED
FOR
Despite a lack of awareness of the wide range of
services being delivered by or via local government,[21]
as tax payers and recipients of services it is still important
that their opinions are used to inform any formal judgements made
about local authority performance.
The CAA's focus on such public perceptions data has
been important, as it has encouraged services to concentrate on
what actually matters to local people, rather than counting outputs.
Despite some criticisms of the Place Survey in terms of questionnaire
content and methodological approach,[22]
consultation with the sector has found there to be general support
for some kind of standard public perception measurement:
"There was a reoccurring theme endorsing
the principle of having a questionnaire on place. It was felt
that the idea of a regular questionnaire to get local people's
views on matters that affect them was beneficial to help local
authorities to address residents' feedback. It was thought that
it was important to track people's changing perceptions to help
determine whether interventions made in an area resulted in the
right outcomes for local people."
The New Place Survey: Summary of responses
to consultation, CLG, July 2008
However, Ipsos MORI analyses have demonstrated that
it is also important to consider the wider context. Some of the
criticism targeted at previous performance regimes in the past—from
the sector itself—has been around the process of not being
flexible enough to truly take into account the priorities of local
people and local circumstances. This is reinforced through our
Frontiers of Performance analyses, which show that local
circumstances have a large bearing on how local public services
are deemed to perform by their citizens.
The "peril" in using perception measures
is that they are often determined to a large degree by the nature
of the population a local authority serves, as much as the local
authority itself. For example, our research dating back to our
very first Frontiers report in 2001 shows people's satisfaction
with their area and their council is clearly influenced by local
factors such as the level of deprivation, ethnic diversity and
the region where they live.
Our latest report, Mind the Gap: Frontiers of
Performance in Local Government V (January 2010), attempts
to take this into account, by looking at the extent to which perception
ratings in different local areas are higher or lower than we would
expect given local circumstances (using Place Survey 2008-09 data
as a basis). In the simplest terms, our aim has been to level
the playing field when considering scores on perception measures,
whilst highlighting which councils do best and worst given their
local circumstances.
Our Frontiers V report illustrates that a
very high proportion of our top performers on the "satisfaction
with council" perception measure were also achieving the
top grades in CAA—the vast majority achieved a level 3 or
4 (the top score possible) in the latest organisational assessment.
This is perhaps reassuring, but it is also clear from our analysis
that the highest CAA scores tended to go to those with the highest
absolute level of satisfaction, rather than those that
seem to be most out-performing what we would expect given their
local circumstances.
Clearly much more went into CAA than just these simple
survey measures, and CAA involved expert professional judgement
of services most residents know little about, so we should not
expect complete agreement. However, our analyses do illustrate
that an essential element of any future regime for inspection
and performance management is to recognise that individual areas
are unique and local authorities face different challenges. Ultimately,
any centrally imposed approach needs to find away of accepting
local differences, while at the same time still setting challenging
aspirations for perception—which will differ in different
areas.
REFERENCES
Ipsos MORI, Shared Intelligence and Cardiff Business
School (March 2010): Comprehensive Area Assessment: an evaluation
of year one. Research study conducted for the Audit Commission.
Ipsos MORI (January 2010): Mind the Gap: Frontiers
of Performance in Local Government V. Analyses based on the findings
of the 2008-09 Place Surveys.
Ipsos MORI (June 2010): The future of Local Government
Briefing Pack.
Ipsos MORI (August 2009): People Perceptions and
Place.
Ipsos MORI (August 2009): Telephone survey of adults
in Great Britain on behalf of Vertex.
Ipsos MORI (March 2009): How the public access
and use information. Desk research study conducted for the
Audit Commission.
Ipsos MORI (April 2008): Engaging the public in
the development of Comprehensive Area Assessment. Research
study conducted for the Audit Commission.
Ipsos MORI (May 2008): Telephone survey of Londoners
on behalf of London Councils.
Ipsos MORI (June 2003): An Evaluation of the Comprehensive
Performance Assessment, Leaders' & Chief Executives' views.
Research study conducted for Local Government Association.
Ipsos MORI (2004): The impact and cost of Inspection.
Research study conducted for Local Government Association.
DCLG (2001—2009-10): Citizenship Survey.
DCLG (July 2008): The New Place Survey: Summary
of responses to consultation.
20 The New Place Survey: Summary of responses to consultation,
CLG, July 2008 Back
21
An Ipsos MORI poll amongst Londoners in 2008 showed that while
most people knew their council was responsible for picking up
their rubbish (97%), local authorities were also thought to be
accountable for the local hospital and policing by significant
proportions too (by 45% and 49% respectively). Back
22
There were some concerns about the limitations of using a postal
self-completion method, although we do not aim to discuss the
issue or merits of survey methodology in this note. Back
| 
