House of Commons - Environment, Food and Rural Affairs

Select Committee on Environment, Food and Rural Affairs Eleventh Report

3 Measuring Performance

31. We have already expressed some reservations about the title and intermediate outcomes of Defra's DSO, but our most serious doubts relate to the indicators: the way in which Defra proposes to measure its success in achieving the DSO. When we announced that we would focus our inquiry on the DSO, in February 2008, Defra had only a general idea of the indicators it planned to use:

INTERMEDIATE OUTCOME INDICATORS

The evidenced needs of rural people and communities are addressed through mainstream public policy and delivery This IO has been designed to assess the performance of Government policies in rural areas by comparing a basket of socio-economic outcomes and trends in rural areas to the national picture. The proposed measures are: educational attainment; social capital/quality of life; health and social care; employment and economy; housing affordability; crime and antisocial behaviour

Economic growth is supported in rural areas with the lowest levels of performance This measure focuses on GVA at the sub-regional level: an input measure (workplace based jobs at district level); and an output measure (Gross value added (GVA) at district level)

Source: www.defra.gov.uk/corporate/busplan/spending-review/psa2007.htm

On 7 July 2008, the Minister wrote to us with a detailed list of the indicators Defra had chosen and an assessment of how Defra was currently performing against them. This information was also published on Defra's website.[

56] For each of the two intermediate outcomes, Defra has devised a set of "overviews", which have individual indicators. For example, one of the overviews for the mainstreaming intermediate outcome is "Educational Attainment". This overview has two indicators: GCSE results and full-time entrants to higher education. A full list of the overviews and indicators is included in appendix 2. Defra plans to adopt a "traffic light system" to measure performance. Each individual indicator is rated either green, amber/green, amber/red, or red. The ratings are based on how rural areas perform in comparison with the national average over a number of years.[57] These individual results are then used to produce a traffic light rating for the "overview" and the overviews in turn are used to produce a traffic light rating for the intermediate outcome. In the above example, the "GCSE results" and "Full-time entrants to higher education" indicators are both rated green, meaning that the Educational Attainment Overview is also green. The data for the indicators will be updated annually, at the end of the financial year.[58] Our concerns about the indicators fall under two main headings and are set out belo

The Data

32. From the start, witnesses were concerned about the level at which the indicators would operate. Defra stated that its indicators would mean "that specific problems can be identified and tackled if and when they arise."[59] However, others were sceptical about how much the indicators would reveal. Ms Berry, the Chair of the Wensleydale Business Association, said that she looked at the indicators "with disbelief". She did not see how homogenised statistics for Richmondshire would reveal anything about the small rural community in which she lived. She said: "under the Defra indicators we will appear as a really strong community but we are not."[60] The NFU also commented on the need for the indicators to operate below district level:

The NFU is somewhat perturbed by the Defra decision to use the local authority district as its datum base for measuring progress towards the achievement of its Strong Rural Communities Objective. Most respected commentators and analysts have for long accepted that this level of data collection and analysis is just too coarse-grained to provide an accurate and effective measure of change in rural areas.[61]

The Local Government Association told us that the indicators "give you a top level view of what is happening and that is helpful and is a way of summarising the condition of a particular area overall but if you want to design policy or operational interventions you need a finer grain that these indicators".[62]

33. When Defra published the more detailed information about its DSO in July 2008, it became clear that the level of data used will vary across the indicators. Defra itself draws attention to this point:

For the mainstreaming intermediate outcome, the ONS/Defra Rural Definition has been used […] where possible because the mainstreaming outcome measures the situation, wellbeing and needs of individuals […] For the economic growth outcome […] Defra's Local Authority District Classification […] has been deemed a more sensible approach when examining higher level economic indicators.[63]

The Rural Definition to which this quote refers was developed in conjunction with the Office for National Statistics. It classifies settlements of more than 10,000 people as urban, and separates rural areas into three categories: town and fringe; village; and hamlet and isolated dwellings. It then further divides all four categories according to whether they are "sparse" or "less sparse". Defra's Local Authority District Classification, on the other hand, separates local authorities into six categories: Major Urban, Large Urban, Other Urban, Significant Rural, Rural-50 and Rural-80.[64] While the first system of classification makes it possible to distinguish between data relating to market towns and data relating to smaller, more remote rural communities, the second system, by its very nature, does not.

34. Defra's statement about the level at which its mainstreaming indicators operate fails to make it clear that, in the majority of cases, it is in fact not possible to use data at the more detailed ONS/Defra Rural Definition level. There are 24 indicators for the mainstreaming intermediate outcome: only six use the finer-grain ONS/Defra Rural Definition; 16 use Defra's Local Authority District Classification; and one simply divides the data into "Urban," "Rural", "Rural Sparse" and "Rural Less Sparse".[65] All the health, housing, crime, poverty and unemployment indicators use district level data. Even in the case of the ONS/Defra Rural Definition data, Defra may be pushing a point to say that it is measuring the situation of "individuals", but at least such data provides a meaningful indication of the situation in different types of rural communities. District-level data simply cannot do this.

35. The more detailed information that Defra supplied in July 2008 also makes it clear that the age of the data varies across the indicators. For example, under the "Health Overview", the latest data for the "Life expectancy at birth" indicator is from 2003-05, but under the "Crime Overview", the latest data for the "Violence against the person" indicator is from 2006-07. Moreover, the length of time over which Defra is measuring trends is not consistent. In the case of the "Violence against the person" indicator, a trajectory is plotted using data from 2005-06 to 2006-07—a period of two years. In the case of the "Ratio of lower quartile house prices to lower quartile earnings" indicator, a trajectory is plotted using data from 1997 to 2007—a period of 10 years.

36. Defra told us that its previous rural economy target, PSA 4, "proved to be a difficult target to measure, because of data limitations."[66] Data limitations are clearly still a problem with the rural communities DSO. This seems curious when considered in conjunction with the comments made by Professor Ward of the CRE about "the explosion of data and statistics about rural areas and rural economies".[67] Although there is undoubtedly more information about rural areas than ever before, thanks in part to the CRC's annual State of the Countryside report, there are clearly still gaps—particularly in relation to low level data for some of the indicators Defra has chosen. In part, this is because Defra's traffic light ratings are based on trajectories, and therefore depend on historical data, which may be less complete or less detailed than more recent data. Defra explained that it was reliant on data from the ONS and other departments.[68] It told us that it uses data at the lowest level that is available.[69] The Minister summed up the situation: "Do we have as much [data] as we want in the most uniform way that would be helpful in terms of us proofing? No, we do not. Do we have more than we had in the past? Yes."[70] Mr Robin Mortimer, Defra's Director for Rural, Land Use and Adaptation, added:

I am not sure we are striving for total consistency. There are some indicators where it is meaningful to have data at a very local level so that you can say for a very small community what proportion of A to C GCSE grades there are. That is a meaningful statistic. It is not particularly meaningful to talk about productivity at a very low scale because businesses are complex and will employ people from different places.[71]

37. We are concerned about the level at which the indicators operate. We agree with defra that, for some indicators, it is not meaningful to compare data at a very local level. However, for the majority of indicators, comparing data at a district level will not enable defra or other agencies to identify specific problems and target interventions accordingly. Our experience suggests that remote rural communities and market towns are encountering different problems, but we cannot see how the majority of indicators would pick up on this, or any of the other location-specific problems we encountered. We recognise that defra is dependent on other departments and the office for national statistics for much of its data. We recommend that it identify the indicators for which it would be useful to have more detailed results and set out how it intends to work with its data providers to obtain this level of information in future.

The Missing Pieces

38. One of the most striking things about Defra's Departmental Strategic Objective is that, according to its own assessment, it has already almost achieved it. The table below sets out the overview areas for each intermediate outcome and Defra's current traffic light rating:[72]

INTERMEDIATE OUTCOME NO. 1: THE EVIDENCED NEEDS OF RURAL PEOPLE AND COMMUNITIES ARE ADDRESSED THROUGH MAINSTREAM PUBLIC POLICY AND DELIVERY

Educational Attainment Overview Green

Social Capital and Quality of Life Overview Green

Health Overview Green

Housing Affordability Overview Amber/Red

Crime Overview Green

Poverty and Unemployment Overview Green

INTERMEDIATE OUTCOME NO 2: ECONOMIC GROWTH IS SUPPORTED IN RURAL AREAS WITH THE LOWEST LEVELS OF PERFORMANCE

Productivity Overview Amber/Green

Earnings Overview Amber/Green

Employment Overview Green

Skills Overview Green

Enterprise Overview Green

Investment Overview Amber/Red

Eight out of the 12 overview areas that Defra has chosen for its DSO are already rated green.

39. Our experience suggests that, while Defra thinks that rural areas are already strong in many respects, people living in those areas think differently. This is partly because there are some aspects of rural life—notably transport, communications, planning, and further education—that are not included in the mainstreaming indicators for the DSO, but which are central to the sustainability of rural communities. When the Minister was addressing a different point, he used an interesting example to illustrate his argument: "People in rural areas," he said, "are worried about the bus, not [the number of] civil servants."[73] People in rural areas are indeed worried about the bus. Transport was a recurring issue in our conversations with the people we met on our visit to North Yorkshire, for example: it concerned both the talented apprentice cabinet-maker, who will represent the UK at the World Skills Olympics in 2009 and who spends £120 a month on petrol to get to his workplace in a remote village, and the staff working in the business start-up centre on the edge of a market town. However, transport is not mentioned in Defra's DSO—nor is broadband, another issue that came up again and again during our inquiry. [74] We accept that not every aspect of rural life is measurable. However, this surely does not apply to access to public transport and to the internet. The CRC's State of the Countryside report contains data on both.

40. Other aspects of rural life are harder to measure, but they must not be overlooked for this reason. Planning falls into this category. Broughton Hall Business Park, outside Skipton, is an excellent example of how land and buildings in a rural area can be adapted to new uses and yet remain sensitive to their original setting. The estate buildings, many of which were formerly in agricultural use and which are spread over 3,000 acres of park and woodland, have been restored to provide accommodation for 51 businesses. Broughton Hall was the Yorkshire and Humberside region's winner of the Enterprising Britain competition in 2007. However, the conversion process began in the 1980s, and we were told that a similar project would be unlikely to succeed today. Mr Ian Butter, Head of Planning and Land Use at Rural Solutions, commented that the planning system is increasingly hindering and in some cases preventing beneficial schemes:

Any development that takes place outside a defined settlement boundary is now considered by most planners as prima facie unsustainable. The approach to sustainability is now so rigid that some settlements are effectively blocked from any form of beneficial development—i.e. because they are not sustainable at the outset then development that might help to make them more sustainable is not permitted.[75]

He told us that the poor interaction between Government planning statements "positively discriminates against rural development" and commented that if a building requires more than a simple conversion, it is treated as a new building for planning purposes, meaning that there is a presumption against it in rural areas, just as there is a presumption against new build "irrespective of the suitability or benefits achieved."[76] Similar concerns were voiced repeatedly during our inquiry.

41. Although it is encouraging that rural areas are performing well according to the criteria and measures defra has selected, we see little value in having a departmental strategic objective until 2011 that defra has already largely achieved. We recognise that maintaining success is important, but we question whether this should be the focus of defra's only rural affairs dso. We acknowledge that no target or set of performance indicators can be all-encompassing. However, we are concerned that there are significant gaps in what defra is measuring in its rural communities dso. We would like to see a dso that is sufficiently flexible, and fine-grained, to enable the problems in different communities to be recognised and tackled. We recommend that defra consult the commission for rural communities on whether the indicators for the dso represent the best practicable way of identifying the issues affecting rural communities. Defra should revise its indicators in the light of this advice.

42. If defra does decide to proceed with the current indicators, it should be careful to treat its indicators as no more than that: indicators of success. Defra must recognise that some important aspects of life in rural communities, such as transport, communications, planning and further education, are not covered by its mainstreaming indicators. It should also recognise that there may be problems that are not reflected by a simple comparison with the national average. It should set out how it intends to identify and tackle these problems.

56 www.defra.gov.uk/rural/dso/index.htm Back

57 Defra defines the traffic lights as follows: Green (an indicator where rural areas perform above or equal to the norm, with a trajectory suggesting that it will remain so; or an indicator that is below, but within an acceptable range of the norm, with a trajectory that remains within an acceptable range); Amber/Green (an indicator where rural areas perform below an acceptable range of the norm, but with a trajectory that will converge within an acceptable time frame); Amber/Red (rural areas performing equally to the norm, but trajectory will clearly take rural below the norm; or an indicator where rural performs below the norm, with a trajectory that will converge outside an acceptable time frame); Red (an indicator where rural areas perform below an acceptable range of the norm with either a parallel or diverging trajectory). Back

58 Ev 124 Back

59 Ev 104, para 3.2 Back

60 Q 162 [Ms Berry] Back

61 Ev 196, para 14 Back

62 Q 80 Back

63 http://www.defra.gov.uk/rural/pdfs/dso/dso-doc.pdf (underlining and italics are in the original) Back

64 Significant Rural refers to districts with more than 37,000 people or more than 26% of their population in rural settlements and larger market towns; Rural-50 to districts with at least 50% but less than 80% of their population in rural settlements and larger market towns; and Rural-80 districts with at least 80% of their population in rural settlements and larger market towns. Back

65 There is also one indicator for which no data will be available until Autumn 2008. Back

66 Ev 94, para 18 Back

67 Q 34 Back