56] For each of the two intermediate outcomes,
Defra has devised a set of "overviews", which have individual
indicators. For example, one of the overviews for the mainstreaming
intermediate outcome is "Educational Attainment". This
overview has two indicators: GCSE results and full-time entrants
to higher education. A full list of the overviews and indicators
is included in appendix 2. Defra plans to adopt a "traffic
light system" to measure performance. Each individual indicator
is rated either green, amber/green, amber/red, or red. The ratings
are based on how rural areas perform in comparison with the national
average over a number of years.
These individual results are then used to produce a traffic light
rating for the "overview" and the overviews in turn
are used to produce a traffic light rating for the intermediate
outcome. In the above example, the "GCSE results" and
"Full-time entrants to higher education" indicators
are both rated green, meaning that the Educational Attainment
Overview is also green. The data for the indicators will be updated
annually, at the end of the financial year.
Our concerns about the indicators fall under two main headings
and are set out belo
32. From the start, witnesses were concerned
about the level at which the indicators would operate. Defra stated
that its indicators would mean "that specific problems can
be identified and tackled if and when they arise."
However, others were sceptical about how much the indicators would
reveal. Ms Berry, the Chair of the Wensleydale Business Association,
said that she looked at the indicators "with disbelief".
She did not see how homogenised statistics for Richmondshire would
reveal anything about the small rural community in which she lived.
She said: "under the Defra indicators we will appear as a
really strong community but we are not."
The NFU also commented on the need for the indicators to operate
below district level:
The NFU is somewhat perturbed by the Defra decision
to use the local authority district as its datum base for measuring
progress towards the achievement of its Strong Rural Communities
Objective. Most respected commentators and analysts have for long
accepted that this level of data collection and analysis is just
too coarse-grained to provide an accurate and effective measure
of change in rural areas.
The Local Government Association told us that the
indicators "give you a top level view of what is happening
and that is helpful and is a way of summarising the condition
of a particular area overall but if you want to design policy
or operational interventions you need a finer grain that these
33. When Defra published the more detailed information
about its DSO in July 2008, it became clear that the level of
data used will vary across the indicators. Defra itself draws
attention to this point:
For the mainstreaming intermediate outcome, the ONS/Defra
Rural Definition has been used [
] where possible
because the mainstreaming outcome measures the situation, wellbeing
and needs of individuals [
] For the economic growth
] Defra's Local Authority District Classification
] has been deemed a more sensible approach when examining
higher level economic indicators.
The Rural Definition to which this quote refers was
developed in conjunction with the Office for National Statistics.
It classifies settlements of more than 10,000 people as urban,
and separates rural areas into three categories: town and fringe;
village; and hamlet and isolated dwellings. It then further divides
all four categories according to whether they are "sparse"
or "less sparse". Defra's Local Authority District Classification,
on the other hand, separates local authorities into six categories:
Major Urban, Large Urban, Other Urban, Significant Rural, Rural-50
and Rural-80. While
the first system of classification makes it possible to distinguish
between data relating to market towns and data relating to smaller,
more remote rural communities, the second system, by its very
nature, does not.
34. Defra's statement about the level at which
its mainstreaming indicators operate fails to make it clear that,
in the majority of cases, it is in fact not possible to
use data at the more detailed ONS/Defra Rural Definition level.
There are 24 indicators for the mainstreaming intermediate outcome:
only six use the finer-grain ONS/Defra Rural Definition; 16 use
Defra's Local Authority District Classification; and one simply
divides the data into "Urban," "Rural", "Rural
Sparse" and "Rural Less Sparse".
All the health, housing, crime, poverty and unemployment indicators
use district level data. Even in the case of the ONS/Defra Rural
Definition data, Defra may be pushing a point to say that it is
measuring the situation of "individuals", but at least
such data provides a meaningful indication of the situation in
different types of rural communities. District-level data simply
cannot do this.
35. The more detailed information that Defra
supplied in July 2008 also makes it clear that the age of the
data varies across the indicators. For example, under the "Health
Overview", the latest data for the "Life expectancy
at birth" indicator is from 2003-05, but under the "Crime
Overview", the latest data for the "Violence against
the person" indicator is from 2006-07. Moreover, the length
of time over which Defra is measuring trends is not consistent.
In the case of the "Violence against the person" indicator,
a trajectory is plotted using data from 2005-06 to 2006-07a
period of two years. In the case of the "Ratio of lower quartile
house prices to lower quartile earnings" indicator, a trajectory
is plotted using data from 1997 to 2007a period of 10 years.
36. Defra told us that its previous rural economy
target, PSA 4, "proved to be a difficult target to measure,
because of data limitations."
Data limitations are clearly still a problem with the rural communities
DSO. This seems curious when considered in conjunction with the
comments made by Professor Ward of the CRE about "the explosion
of data and statistics about rural areas and rural economies".
Although there is undoubtedly more information about rural areas
than ever before, thanks in part to the CRC's annual State
of the Countryside report, there are clearly still gapsparticularly
in relation to low level data for some of the indicators Defra
has chosen. In part, this is because Defra's traffic light ratings
are based on trajectories, and therefore depend on historical
data, which may be less complete or less detailed than more recent
data. Defra explained that it was reliant on data from the ONS
and other departments.
It told us that it uses data at the lowest level that is available.
The Minister summed up the situation: "Do we have as much
[data] as we want in the most uniform way that would be helpful
in terms of us proofing? No, we do not. Do we have more than we
had in the past? Yes."
Mr Robin Mortimer, Defra's Director for Rural, Land Use and Adaptation,
I am not sure we are striving for total consistency.
There are some indicators where it is meaningful to have data
at a very local level so that you can say for a very small community
what proportion of A to C GCSE grades there are. That is a meaningful
statistic. It is not particularly meaningful to talk about productivity
at a very low scale because businesses are complex and will employ
people from different places.
37. We are concerned about the
level at which the indicators operate. We agree with defra that,
for some indicators, it is not meaningful to compare data at a
very local level. However, for the majority of indicators, comparing
data at a district level will not enable defra or other agencies
to identify specific problems and target interventions accordingly.
Our experience suggests that remote rural communities and market
towns are encountering different problems, but we cannot see how
the majority of indicators would pick up on this, or any of the
other location-specific problems we encountered. We recognise
that defra is dependent on other departments and the office for
national statistics for much of its data. We recommend that it
identify the indicators for which it would be useful to have more
detailed results and set out how it intends to work with its data
providers to obtain this level of information in future.
The Missing Pieces
38. One of the most striking things about Defra's
Departmental Strategic Objective is that, according to its own
assessment, it has already almost achieved it. The table below
sets out the overview areas for each intermediate outcome and
Defra's current traffic light rating: