Audit and inspection of local authorities - Communities and Local Government Committee Contents


Written evidence submitted by Ipsos Mori

SUMMARY

Performance management and independent assessment and inspection have been important in driving improvement across the local government sector.

Despite the well documented criticisms of the Audit Commission, in our view Comprehensive Performance Assessment (CPA) in particular, especially in its initial iterations (prior to what appeared to be the sector's own "grade inflation"), was important in improving visibility of both good and bad performance, and this, with the threat of central intervention, was more effective in improving local government performance than relying on the ballot box alone, or auditors alone.

In considering what may or may not be appropriate in future, we think that some ability for local people, auditors, independent experts and commentators to make fair and public comparisons between the performance of authorities is extremely important. While the process involved needs to be proportionate, our view is that being able to make valid comparisons on performance between authorities, on a public, like for like basis, is vital in driving up performance and highlighting areas for improvement.

Our work for the Local Government Association (LGA) as far back as 2003 illustrated the general support behind the principles of CPA, with the majority of local authority chief executives seeing the assessment as a fruitful process for their authority. Most also agreed that it was an effective way of comparing performance between local authorities and that it challenged their attitudes and the performance of their authority. Most admitted that the sector would not have improved as quickly without some external inspection process. While there were many complaints about the detail of the Best Value and CPA regimes, one thing that virtually all chief executives and local politicians agreed on was that they challenged attitudes and performance.

More recently, our evaluation of the first year of Comprehensive Area Assessment (CAA) showed that there was widespread support—from assessed bodies, local partnerships, government departments and inspectorates—for the principles of CAA.

None of this is to suggest that any of these processes were perfect: throughout there were requests for greater transparency over the process, a lighter touch approach, better engagement from elected members and the public, and a concern about the level of cost involved. For example, the survey above saw most authorities agreeing that the cost of inspection outweighed its benefits - views were clearly mixed (see the following chart).


The public, with no detailed understanding of the costs or processes involved, support cost effective inspection. While few members of the public have ever had a detailed understanding about the ways in which local public services are inspected (17% say they know at least a fair amount), the vast majority (89%) feel that independent inspection is important and 80% that it leads to improvements. The public support the principle of independent inspection (as long as it provides value for money—although they themselves find this extremely difficult to judge).

At the same time, our research suggests that the public will not necessarily actively seek out, or remember, information about how their council performs (despite information being available, most people—81%—don't know or can't remember what the key official judgement of their council was). This does not mean that inspection should be abandoned—the public like to think some independent inspection and regulation is available and expect officers, members and parish councils and others to use this information to improve performance and ask questions. They support frequent independent inspection of old people's homes, for example.

This is important when we consider current moves to put more performance and spending information into the hands of the public in the hope of mobilising a new force of "armchair auditors". This has been an ambition of government since Mrs Thatcher's time (for example, the mandatory publication of performance data in local newspapers happened long before the web was even thought of in this context). But, evidence suggests that citizens are primarily "passive consumers" of this kind of information, and the desire for this type of information is limited to a small minority of informed and engaged citizens—likely to be older residents, more middle class and owner occupiers.

Moreover, if publication is to be mainly online, because of the continuing class/ age gradient in internet access (for example, only 18% of working class pensioners (C2DE) are currently online) there will be a considerable challenge in engaging the majority of the population in the scrutiny of public service performance data. We support the current plans to publish much more information online—as a minority of people will find it very useful in holding public services to account—but, we think expectations of the extent to which online publication will directly engage the public can be overstated, although the media and the minority of those active and involved will undoubtedly use it.

Another key challenge is that many members of the public do not trust official statistics and other types of performance information: most think that figures are not reported honestly, or without interference. The public place a lot of emphasis on public information being produced by independent organisations and regulators.

Finally, we would point out that it will be important in any future system for inspection scores to at least resemble public perceptions of performance. Without this, judgements become professionals' views of other professionals' services. In its first few years CPA did correlate positively with public perception (although rather fewer of the public would rate their council as excellent or good than would the inspectors). Later, as councils' "grades" improved, the correlation diminished.

However, our view is that public perception data is a vital part of any inspection/ regulation system, to encourage services to concentrate on what actually matters to local people, rather than simply counting outputs, or making professional judgements. Having like for like measures is vital if this is to be meaningful. Despite some valid criticisms of the Place Survey (until recently, the key mandatory mechanism used for collecting citizen perceptions data), consultation with the sector has found there to be general support for some kind of standard public perception measurement: "… the idea of a regular questionnaire to get local people's views on matters that affect them was beneficial to help local authorities to address residents' feedback… it was important to track people's changing perceptions to help determine whether interventions made in an area resulted in the right outcomes for local people."[20]

Finally, perceptions data should not be treated in isolation: we would argue that local circumstances need to be accounted for in drawing a fair assessment of a council's overall performance from it. Our own Frontiers of Performance analyses demonstrate just how difficult it can be for some councils to achieve high perception measure scores due to the challenging environment in which they operate (with factors such as deprivation, ethnicity and even region impacting on how easy or difficult it is achieve high levels of satisfaction amongst local residents). It showed different types of authorities excelling at different things. Any future process of inspection should use like for like, comparable methodologies, for the measurement of public opinion to allow valid comparisons between authorities and prevent gaming.

March 2011

THE ROLE OF INDEPENDENT INSPECTION AND ASSESSMENT IS VALUED BY THE SECTOR

Despite the end of CAA, and its predecessor CPA, it is clear that performance management, inspection and assessment has gone some way to helping strengthen the local government sector, with overall assessment scores improving over recent years. Ipsos MORI research with the local government sector itself backs this up.

We would also argue that CPA provided an important system for recognising good performance and hard work on the part of councils (and equally where poor performance or service failure has needed to be addressed).

Our research into the issues illustrates that local authority leaders and chief executives generally see the assessment and inspection process as both a useful and challenging exercise, and one that helps improvement, despite some of the mechanics of how the respective regimes have worked. Notwithstanding the improvements made to the CPA inspection regime since the research was conducted, a study for the LGA in 2003 found a number of interesting messages still pertinent to the discussion today:

—  The majority (three-quarters of county and single tier authority chief executives) saw the CPA assessment as a fruitful process for their authority.

—  Most also agreed that the CPA assessment framework was an effective way of comparing performance between local authorities.

—  Most thought that the CPA challenged their attitudes and the performance of their authority and that the CPA had a fair amount of impact on their strategic planning.

—  The corporate assessment element was well received. The inclusion of peer reviewers in the corporate assessment was highly supported by chief executives, regardless of what CPA assessment score they had received.

—  Self-assessment was seen as giving councils the opportunity to spend time looking at how they worked, and giving a fair account of their own performance and priorities: it was the most highly regarded component of the CPA process.

—  Service judgements were regarded as the least useful element when compared to the corporate and self-assessments, although most chief executives (66%) still found this a useful process. The weaker support for the service judgement seemed to arise partly from concerns about the service blocks identified for assessment and more strongly about the weights applied to them.

—  There was some concern about the quality of inspection: opinion was divided on whether the basis for the CPA judgement was clear or not and many were not only doubtful of the evidence, but of the quality of inspectors too.

More recently, our evaluation of the first year of CAA showed that there was widespread support—from assessed bodies, local partnerships, Government departments and inspectorates—for the principles of CAA including:

—  the focus on outcomes as opposed to individual services;

—  reducing the burden on assessed bodies;

—  ensuring that local inspection is a more unified experience;

—  giving more emphasis to local context and priorities, and

—  taking a more forward looking approach which focuses on risk rather than past performance.

This said, experiences of the stakeholders researched in this study were mixed. For example, there was limited engagement in the CAA process from elected members, the third sector and citizens more widely, with the Place Survey providing the main source of the public's views. The majority of assessed bodies found both understanding CAA's approach and staff time spent working on CAA took longer than they expected. Views varied over whether CAA was more of a burden than previous assessments—although most did not believe it had reduced the burden of inspection.

Assessed bodies felt the CAA process was fair although there were concerns over transparency, particularly in terms of the area assessment and criteria and procedures for awarding flags. Nevertheless, red flags were seen to help drive improvement, and there was an interest in learning from areas with green flags. There were mixed views about the Oneplace website and CAA reports—some organisations thought they were too bland, with too little detail to support improvement. There was a general agreement that CAA had had a stronger focus on inequality and sustainability than previous assessment frameworks. But, there were mixed views about whether there had also been a stronger focus on vulnerable people, and no evidence of CAA having had a greater focus on value for money.

There were a number of views as to how the inspection programme could be improved looking ahead that might be important to consider when establishing any future approach, particularly:

—  Having a strong focus on local priorities.

—  Better engagement from elected members, district councils, the third sector and the public.

—  Improving the transparency of the process.

—  Ensuring that "underpinning themes", such as inequality, sustainability, vulnerable people and value for money, are better incorporated into the process. For example, there was considerable support for the development of area based assessments of value for money reflecting the concept of "Total Place".

—  Reducing the burden of inspection through a lighter touch more proportionate approach through, for example, less frequent, risk based organisational assessments and/ or area assessments and whole area Use of Resources (UoR) assessments.

INDEPENDENT INSPECTION AND ASSESSMENT IS ALSO VALUED BY THE PUBLIC

Our research shows that the public also values the role of independent assessment and inspection. While few members of the public know about the ways in which local public services are inspected (around 17% say they know at least a fair amount), the vast majority (89%) still feel that independent inspection is important and 80% that it will lead to improvements. The value of independent inspection is also something we note from our research in other sectors, such as our work on policing inspection for Her Majesty's Inspectorate and Constabulary and education inspection for the Office for Standards in Education (Ofsted).



But, whilst people find the principle of inspection important, our research suggests that they will not necessarily actively seek out, or remember, information about how their council performs (despite information being available most people—81%—don't know or can't remember what the key judgement of their council was).


WE SHOULD BE REALISTIC ABOUT MOBILISING THE PUBLIC AS "ARMCHAIR AUDITORS"

The issue of engaging the public with this kind of performance information is important when we consider the current argument from government, which is seeking to push more performance and spending information into the hands of the public in the hope of mobilising a new force of "armchair auditors".

This of course has its merits, but, as our research suggests, are the public really all that interested in knowing what local public services are doing? And do they really trust the information presented to them?

Whilst our research shows that better and more communication is important in helping to improve residents' overall satisfaction levels with their councils, it also suggests that we need to manage our expectations in terms of just how engaged the public are—and want to be—in monitoring the performance of the sector and in turn holding it to account. Recent polling points to a general lack of awareness amongst the public about how their local public services are performing. In 2010, around two in five members of the public said they had seen information about the performance of their local council. But, only a quarter of people who had seen performance information about local public service providers actively sought it out. This suggests that when it comes to accessing information about how well their local authority is doing, the majority of people remain primarily "passive consumers".


Taking this further, even fewer know about the ways in which local public services are inspected, with three-quarters saying they don't know much.


One consequence of this low awareness is that desire for this type of public information is limited to a small minority of informed and engaged citizens. Based on what we know about those who have accessed performance information in the past, evidence suggests that these are likely to be older residents, of higher social grades and owner occupiers. Many citizens, especially from C2DE social grades, say they are uninterested in tracking the performance of public services, so long as they feel the services are doing their job. This poses a considerable challenge in engaging others in the scrutiny of public service performance data—and those most likely to be the recipients of the high value services provided by local authorities (social care, etc).

Our research suggests that public services need to be realistic about how interested the public will be in becoming "armchair auditors", beyond the small minority of people who are already well informed and engaged in civic life. We know that the majority of people want to be more informed about public services, but not necessarily fully engaged in their decision-making processes. While most people do consider citizen empowerment to be important, few take up the opportunities in practice.


Research for the Audit Commission about how the public access and use information tells us that that public services need to raise the profile and potential utility of performance information—polling evidence suggests that the vast majority of the public still get this kind of information from the media (39%).

In order to tailor performance information, it is important to understand citizens' reasons for accessing it, and current access may be low due to people having difficulty translating available data into a picture of performance they can understand. There is also an issue about the medium through which the information is provided, with much coming via online sources still impenetrable to a significant minority of the population. Better tailoring this information to citizens' requirements may therefore increase access.

Finally, there is an important point to make about trust. In short, public trust in public officials, particularly politicians and government ministers, is lower than any other profession excluding journalists, and in some cases is declining. Trust in councils is also lower than it is for other local services (although Citizenship Survey data suggests that trust levels have been improving here). While the public place a lot of emphasis on public information being produced by independent organisations and regulators, at the same time, they are increasingly sceptical about the legitimacy and accuracy of official statistics produced by both government and independent sources.


Our research into the specific area of local government inspection and performance assessment shows that while the public are positive in principle about being involved in inspection, they generally make little use of inspection reports in part because of an overall scepticism about the usefulness, reliability or authenticity of them. There is a sense that these sorts of reports can be totally divorced from real life, especially where assessment scores do not match their own personal experience. Inspection reports are also seen to rely too much on the filtered evidence statutory services choose to show inspectors—leading to a mistrust of official reports and judgements.

Going forward, there will be clear communications messages to convey about both the potential usefulness and reliability of inspection and assessment reports to local people and communities, and about how robust any judgements will be.

THERE IS AN IMPORTANT ROLE FOR PUBLIC PERCEPTIONS MEASURESBUT LOCAL CIRCUMSTANCES NEED TO BE ACCOUNTED FOR

Despite a lack of awareness of the wide range of services being delivered by or via local government,[21] as tax payers and recipients of services it is still important that their opinions are used to inform any formal judgements made about local authority performance.

The CAA's focus on such public perceptions data has been important, as it has encouraged services to concentrate on what actually matters to local people, rather than counting outputs. Despite some criticisms of the Place Survey in terms of questionnaire content and methodological approach,[22] consultation with the sector has found there to be general support for some kind of standard public perception measurement:

"There was a reoccurring theme endorsing the principle of having a questionnaire on place. It was felt that the idea of a regular questionnaire to get local people's views on matters that affect them was beneficial to help local authorities to address residents' feedback. It was thought that it was important to track people's changing perceptions to help determine whether interventions made in an area resulted in the right outcomes for local people."
The New Place Survey: Summary of responses to consultation, CLG, July 2008

However, Ipsos MORI analyses have demonstrated that it is also important to consider the wider context. Some of the criticism targeted at previous performance regimes in the past—from the sector itself—has been around the process of not being flexible enough to truly take into account the priorities of local people and local circumstances. This is reinforced through our Frontiers of Performance analyses, which show that local circumstances have a large bearing on how local public services are deemed to perform by their citizens.

The "peril" in using perception measures is that they are often determined to a large degree by the nature of the population a local authority serves, as much as the local authority itself. For example, our research dating back to our very first Frontiers report in 2001 shows people's satisfaction with their area and their council is clearly influenced by local factors such as the level of deprivation, ethnic diversity and the region where they live.

Our latest report, Mind the Gap: Frontiers of Performance in Local Government V (January 2010), attempts to take this into account, by looking at the extent to which perception ratings in different local areas are higher or lower than we would expect given local circumstances (using Place Survey 2008-09 data as a basis). In the simplest terms, our aim has been to level the playing field when considering scores on perception measures, whilst highlighting which councils do best and worst given their local circumstances.

Our Frontiers V report illustrates that a very high proportion of our top performers on the "satisfaction with council" perception measure were also achieving the top grades in CAA—the vast majority achieved a level 3 or 4 (the top score possible) in the latest organisational assessment. This is perhaps reassuring, but it is also clear from our analysis that the highest CAA scores tended to go to those with the highest absolute level of satisfaction, rather than those that seem to be most out-performing what we would expect given their local circumstances.

Clearly much more went into CAA than just these simple survey measures, and CAA involved expert professional judgement of services most residents know little about, so we should not expect complete agreement. However, our analyses do illustrate that an essential element of any future regime for inspection and performance management is to recognise that individual areas are unique and local authorities face different challenges. Ultimately, any centrally imposed approach needs to find away of accepting local differences, while at the same time still setting challenging aspirations for perception—which will differ in different areas.

REFERENCES

Ipsos MORI, Shared Intelligence and Cardiff Business School (March 2010): Comprehensive Area Assessment: an evaluation of year one. Research study conducted for the Audit Commission.

Ipsos MORI (January 2010): Mind the Gap: Frontiers of Performance in Local Government V. Analyses based on the findings of the 2008-09 Place Surveys.

Ipsos MORI (June 2010): The future of Local Government Briefing Pack.

Ipsos MORI (August 2009): People Perceptions and Place.

Ipsos MORI (August 2009): Telephone survey of adults in Great Britain on behalf of Vertex.

Ipsos MORI (March 2009): How the public access and use information. Desk research study conducted for the Audit Commission.

Ipsos MORI (April 2008): Engaging the public in the development of Comprehensive Area Assessment. Research study conducted for the Audit Commission.

Ipsos MORI (May 2008): Telephone survey of Londoners on behalf of London Councils.

Ipsos MORI (June 2003): An Evaluation of the Comprehensive Performance Assessment, Leaders' & Chief Executives' views. Research study conducted for Local Government Association.

Ipsos MORI (2004): The impact and cost of Inspection. Research study conducted for Local Government Association.

DCLG (2001—2009-10): Citizenship Survey.

DCLG (July 2008): The New Place Survey: Summary of responses to consultation.


20   The New Place Survey: Summary of responses to consultation, CLG, July 2008 Back

21   An Ipsos MORI poll amongst Londoners in 2008 showed that while most people knew their council was responsible for picking up their rubbish (97%), local authorities were also thought to be accountable for the local hospital and policing by significant proportions too (by 45% and 49% respectively).  Back

22   There were some concerns about the limitations of using a postal self-completion method, although we do not aim to discuss the issue or merits of survey methodology in this note. Back


 
previous page contents next page


© Parliamentary copyright 2011
Prepared 7 July 2011