APPENDIX 17
Memorandum from the Centre for Crime and
Justice Studies
1. INTRODUCTION
1.1 The Centre for Crime and Justice Studies
(CCJS) at King's College London is an independent charity that
informs and educates about all aspects of crime and criminal justice.
We provide information, produce research and carry out policy
analysis to encourage and facilitate an understanding of the complex
nature of issues concerning crime. We are a membership organisation
working with practitioners, policy makers, academics and students,
the media and voluntary sector, offering a programme of events,
publications and online resources. The Centre also owns the British
Journal of Criminology, the world's leading English language academic
criminology journal outside the United States.
1.2 CCJS publishes a quarterly magazine,
CJM, for its members and separate subscribers, including academic
libraries, voluntary sector organisations, criminal justice practitioners
and the general public. The magazine has a long standing reputation
for providing a unique combination of academic and practitioner
based perspectives on a range of criminal justice and related
issues. It is widely referenced in academic literature and used
as a valuable resource on undergraduate and postgraduate masters
courses.
1.3 The latest issue of CJM, (No 62 Winter
2005-06) is entitled "Uses of Research". Following publication
in February it received widespread media publicity and CCJS was
invited by the clerk of the Science and Technology Committee to
submit a written memorandum setting out the background to the
latest issue and some of the key arguments put forward in CJM.
As the committee is receiving written and oral evidence from the
head of the Home Office research department and Home Office ministers,
CCJS was asked to focus on the criticisms and concerns raised
by contributors to the magazine.
2. USES OF
RESEARCH—THE
BACKGROUND
2.1 When Labour came to power in 1997 it
made a very clear commitment to "evidence-based" policy
making across government and in particular, in criminal justice.
This was summed up by Tony Blair when he declared "what matters
is what works". The clear message sent out to academics and
researchers was that the government wanted to use the application
of knowledge to inform and determine policy making. After many
years of what Tim Hope, Professor of Criminology at the University
of Keele, describes in CJM as "having criminological research
ignored, under-valued and under-funded by the Conservatives"
the change in approach was widely welcomed.
2.2 Nearly ten years on, however, the experience
of many in the academic community has not been what they had originally
hoped. The Centre for Crime and Justice Studies had become acutely
aware of this due to its close links with various academics. Well
respected professors of criminology and criminal justice, supported
by lecturers and researchers, felt that the government's wider
political agenda on criminal justice had influenced the breadth
and depth of criminological research commissioned and so was at
risk of focusing narrowly on certain types of crimes and methodologies.
They were also concerned that evidence was being ignored, simplified
or misrepresented for political ends. Conversely, there were others
who felt that current approaches to evidence based policy research
had been extremely beneficial and were interested in exploring
their merits and further development.
2.3 The role of CJM is to reflect issues
that are topical and relevant within criminal justice and criminology.
It was therefore felt that the magazine should focus on the issue
of research and provide a platform for the debates over evidence-based
policy making in criminal justice to be explored and made accessible
to a wider audience. It is important to emphasise that the views
expressed in the magazine do not necessarily represent those of
CCJS.
3. USES OF
RESEARCH—WHAT
ARE THE
MAIN CRITICISMS
AND CONCERNS?
3.1 The Home Office has commissioned numerous
research studies to evaluate the many schemes, programmes and
projects that have been set up in recent years. But there is concern
that officials and ministers can be selective in their choice
of the evidence used to illustrate success of programmes thus
resulting in the exclusion of some data and the misrepresentation
of others. Tim Hope, Professor of Criminology at the University
of Keele and a former member of the Home research department,
has documented how this was his experience after evaluating the
Home Office's Reducing Burglary Initiative. He shows that the
Home Office highlighted only one of the research's cases studies
in order to demonstrate that the initiative was effectively resulting
in a reduction in burglary. Hope argues that there is an "incompatibility
between the ideology of evidence-based policy and the natural
inclination of the political process to want to secure the best
outcomes" (see Annex for full article).
3.2 A further major concern is that a substantial
increase in funding for Home Office research in recent years has
led largely to a narrow focus on particular types of crime. Dr
Reece Walters, Senior Lecturer in Criminology at the University
of Stirling, and author of "Deviant Knowledge—Criminology,
politics and policy" notes:
"Home Office criminology has a very clear
purpose: to service the `needs' of ministers and members of Parliament.
It is politically driven criminology, one that provides policy
salient information for politically relevant crime and criminal
justice issues. It's research agenda is motivated by outcomes
that are of immediate benefit to existing political demands—it
is embedded criminology."
Walters highlights a study that examined research
published by the Home Office between 1998 and 2003. It showed
that none of the 571 reports looked crimes "committed as
part of legitimate business activities".
3.3 Other academics, including Steve Tombs
of Liverpool John Moores University, have criticised the way government
funding has focused on street and domestic crime, leaving a huge
gap in reliable data on other types of crime, such as corporate
crimes. He argues that the `slim' proportion of government funding
that has been made available in the past to corporate crime research
has been partly due to the "likelihood of the state's generally
pathetic regulatory efforts being exposed".
3.4 Concerns about the Home Office's approach
have not just been about the types of crimes that it has focused
on. There is also unease about a preference for particular types
of methodology based on an apparent hierarchy of valid social
research methods. The Home Office increasingly prefers the use
of Randomised Controlled Trials, large samples, quantitatively
based approaches and quantitatively systematic reviews. Qualitative
methodologies are considered to be less valid and so are less
credible. However, academics including Loraine Gelsthorpe of the
Institute of Criminology at Cambridge University, argue that this
results in a greater focus on strategies to prevent crime and
anti-social behaviour and to "control" rather than "correct"
offending. There is less of a focus on the causes of criminality,
understanding pathways into crime and the contexts of people's
offending. In her article Gelsthorpe quotes Professor Mike Maguire
of Cardiff University who argues that without qualitative research
it is "all too easy for those studying crime to lose their
sense of reality and begin to perceive offenders not as people,
but merely as `problems' or `numbers'".
3.5 There is one area of Home Office policy
that raises particular concerns. Despite a commitment to developing
evidence-based policies academics point out that the Government's
policies on anti-social behaviour (ASB) are not remotely evidence-based.
The vigorous promotion of anti-social behaviour orders (ASBOs)
is not supported by detailed research on their impact or effectiveness.
As Judy Nixon of Sheffield Hallam University notes:
"While there is a diverse and growing literature
on ASBOs the absence of robust empirical research means that much
of what is written is dominated by anecdote, conjecture and rhetoric."
3.6 The embedding of the Home Office's research
department (RDS) into specific policy areas such as anti-social
behaviour and the National Offender Management Service may create
a sole focus on policy led research rather than research that
illuminates policy development. The incompatibility between neutral
evidence construction and as Tim Hope suggests, "the natural
inclination of the political process" may impact on the commissioning
and design of embedded research. Co-incidental is the absence
of even a small Home Office budget for "blue sky" research
that does not have the exigencies of political life to contend
with.
3.7 Disillusionment with the Home Office's
approach should not be under estimated. It has prompted Dr Reece
Walters to call for a boycott. He argues for criminology to become
a "knowledge of resistance":
"This calls for a politics of engagement
that is often prohibited by the proscriptive and regulated culture
of government and corporate-led research, which many academics
are seduced by in the name of income-generation or evidence-based
decision making . . .. Only a criminology of resistance can achieve
an active engagement that upholds the role of critic and conscience
of society as its mandate and seeks to mobilise networks of collective
concern outside the inner and often financially lucrative circles
of government an corporate contracted research."
4. CONCLUSION
4.1 A commitment to a sound and well researched
evidence base is key to understanding social problems and devising
effective policy responses. This commitment is widely supported
by academics and the Home Office would claim to support it too.
However, there is mounting concern about the trajectories of current
research agendas and the associated policies they initiate and
support. CCJS does not necessarily support all these concerns
but feels it is vital that they are seriously considered by the
Science and Technology Committee.
4.2 Finally, a question that stands out
in CJM 62 "Uses of Research" is whom is Home Office
research for? Ministers, Parliament or the public? There appears
to be a great deal of misunderstanding about the answer to this
question. In light of this, one outcome that may help avoid a
future return to the concerns expressed in CJM 62 is a publicly
debated and agreed set of protocols that set out rules for commissioning,
design, publication and evaluation of research.
Annex
Article by Tim Hope in CJM 62, "Uses of
Research"
THINGS CAN
ONLY GET
BETTER
Tim Hope questions the evidence in evidence-based
policy making.
We did not want to enquire too closely when
the Government announced its support for "evidence-based"
policy-making. After the long years of having criminological research
ignored, under-valued and under-funded by the Conservatives, we
were not inclined to be picky. So, like drifting mariners, many
of us succumbed to the siren call of the Home Office for independent
evaluation of its Crime Reduction Programme. I like to think we
had some honourable motives: a desire to support the application
of knowledge to social progress, perhaps? I also like to think
we trusted our Government, whose promises of reform appeared to
merit support. Along with some of my academic colleagues, we have
published our various accounts of our evaluation experiences in
a special issue of the journal Criminal Justice (Volume 4 (3),
2004). For my part, it was with sadness and regret that I saw
our work ill-used and our faith in government's use of evidence
traduced. Yet, though I have been sorely tempted at times, I do
not want to pin the blame entirely on the mendacity of political
culture, or the self-interests of the various coteries who swarm
around politics (Hope, 2004). In many ways, I do not think either
politicians or their advisers could help themselves resist temptation.
Rather, the blame lies with an incompatibility between the ideology
of evidence-based policy and the natural inclination of the political
process to want to secure the best outcomes. Given the power of
politics, it is not rocket science to predict what will happen
when evidence gets in the way of a good policy. Recently, Tony
Bottoms has written that "methodology matters" (Bottoms,
2005). It matters because methodology, complicated and tedious
though it might appear, is the only way in which science can rescue,
defend and indeed empower evidence within the political claim-making
about "what works". And methodology ought to matter,
as it does to scientists, because it is the only way in which
the validity of the evidence itself can be held to public account.
Writing more than thirty-five years ago, at
the crest of another wave of evidence-based policy-making, the
eminent American social scientist Donald T. Campbell (1969) wrote
a famous paper justifying the application of rigorous research
methodology to the evaluation of policy. His chief justification
was to protect the public interest against what he called "trapped
administrators"—politicians in power who become trapped
by their own rhetoric and promises into claiming success for their
policies in advance of the evidence. Trapped politicians are well
disposed to pretend that policies work even in the face of evidence
to the contrary. But Campbell was also aware of certain statistical
artefacts (which have been apparent since the foundation of statistical
reasoning in 19th century) that could be turned to advantage by
the trapped administrator.
The best odds for the trapped administrator
are where you can get away with capitalising on chance: for instance,
the greater chance that if the probability of something is already
declining over time it will continue to do so rather than abruptly
change direction; or the phenomenon of "regression toward
the mean" (RTM)—that if something observed at one time
is extreme, it is more likely the next time to be less rather
than more extreme, and vice versa (Yudkin and Stratton, 1996).
This is especially likely to be so when, truthfully, you have
little understanding of the underlying causes of a problem that
makes its trend go up or down or vary from place to place, and
so you are unable to make an honest prediction of "what works",
especially for whom, and in what circumstances. The best bet,
as Campbell put it, is to pick "the very worst year, and
the very worst social unit . . . there is nowhere to go but up,
for the average case at least" (Campbell, 1978, p. 87).
The coincidence between statistical artefact
and the promises of the trapped administrator is unfortunate:
even if you don't understand (or even care) what causes a crime
rate to vary, let alone understand RTM and other statistical obscurities,
as a politician you are more likely to be tempted to select the
evidence that appears to support your belief than that which contradicts
it. And if you are at pains to protect simple, honest folk from
the black arts of research methodology—after all, practitioners
don't want to be confused by the ifs and buts of research, they
want to get on with job, don't they?—then, conveniently,
neither they nor anybody else is going to be able to contradict
your own desire to present evidence in the best possible light.
Indeed, you may even dupe yourself.
Various evaluation research methodologies have
emerged over the years to overcome or discount effects due merely
to statistical artefact, including experimentation and regression-based
statistical analysis. Yet the risks of erroneous inference due
to selective and artefactual bias inherent in seemingly simpler
research analyses continue to be ignored (see Hope, 2002). By
way of illustration, take the results of two local projects from
my research consortium's own evaluation of part of the Home Office
Reducing Burglary Initiative—Phase 1, contained in Table
1 (for further information see: Hope, 2004; Hope et al,
2004).
Table One
IMPACT OF TWO LOCAL BURGLARY PREVENTION PROJECTS
ON BURGLARY (PERCENTAGE CHANGE) (SEE HOPE, 2004)
| A | B
| C | D
|
| Change in the
target area
| Change due to
project (modelled)
| Change due to
"other things"
(A-B)
| Change in the
rest of BCU |
| Project | -47 | -37
| -10 | -25 |
| Project C7 | 14 | 39
| -25 | 5 |
Recorded burglary offences almost halved during the course
of Project A3, while they increased by 14% in the Project C7 area
(column A). We employed a regression-based, time-series statistical
method to estimate the proportion of change in burglary, over
and above the general trend in each area, that could be attributed
to the impact of the project itself. The same method produced
very different estimates: for Project A3, we estimated that, if
the only thing affecting the trend in burglary in each area had
been the projects themselves, Project A3 would have reduced burglary
by around a third, while Project C7, left to its own devices,
would have actually increased burglary by two-fifths (column B).
As it happened, other (non-project) influences on burglary in
each of the areas served, presumably, to moderate the projects'
effects: rather embarrassingly, the efforts of Project C7 appear
to have off-set an otherwise generally favourable burglary reduction
trend (column C).
Even though our method suggested that two of the other projects
we studied could have produced even greater reductions, the Home
Office selected from our case-studies only project A3 to write-up
for practical benefit (Home Office, 2004a). Presumably, this was
because the area-wide reduction was greater here than elsewhere
(Hope, 2004, Table 1). Even so, the Home Office write-up has a
rather different narrative from our own submitted site-report,
the former conforming to an officially-endorsed descriptive framework
known as the Five-I's (for more of our own details see Hope, 2005).
Publicly, the Home Office has never commented on Project C7, which
clearly remains something of an embarrassment. Not only does it
seem that an official project could have let burglary increase
but, at the outset, a Home Office consultant had described the
project as "straight-forward" in conception. Of all
our projects, this one was focused most specifically on the target-hardening
of individual dwellings to reduce repeat victimisation (ibid)—an
officially-endorsed burglary prevention strategy, carried out
by a police service that had gained a national reputation for
this kind of crime reduction work.
Instead, the Home Office published its own, pre-emptive analysis
of the impact of the projects (Kodz, et al, 2004; Kodz
and Pease, 2003). This used a "simpler" method to estimate
impact, merely comparing the rate of change in the project target
area with that occurring in the remainder of the police Basic
Command Unit (BCU) in which the project was located, and combining
together results of all the projects studied by each of the three
evaluation consortia. Through various manipulations of the data,
the Home Office method does what it can to capitalise on chance,
producing much more favourable findings overall (Hope, 2004).
But for individual projects, the method produces considerable
distortion. Ironically, this method under-estimates the likely
positive impact on burglary of Project A3 (Table 1); and although
for Project C7 we now are presented with smaller numbers, these
cannot disguise the three-times greater increase in the target
area (column A) than in the rest of the BCU (column D).
Lest it be thought that such practices are confined to the
management of a particular political programme, let's take a look
at the current Home Office Public Service Agreements. Number One
on the list is to "reduce crime by 15% and further in high
crime areas, by 2007-08" (HM Treasury, 2004). Going by what
I have just said, this doesn't look like such a bad bet; not only
has crime been going down steadily but RTM would suggest that
we can rely on getting bigger reductions in the high crime areas
too. Indeed, it is in the top 40 (but why 40?) CDRP (Crime and
Disorder Reduction Partnership) areas in 2003-04 that the Home
Office is looking for greater than average reductions over the
period, compared to the remaining 336 CDRP areas (Home Office,
2004b). Of course, by the same reasoning, particularly if you
are not actually doing anything effective, the odds are just as
likely of getting less than average reductions in the lowest crime
areas (but let's not talk about that).
Even so, it might not be as easy as it looks—chance
is fickle after all. Thus, for example, even if we could show
that crime rates this year had reduced in what were the highest
crime areas last year, that would not necessarily mean that the
gap between the highest and lowest areas this year was any less
than it was last year; after all, RTM suggests it is likely that
other areas may have taken their respective places this year.
Reducing the "performance gap" actually means doing
something to affect the distribution of performance as a whole,
across all the partnerships, since the catch in RTM is that reductions
towards the mean are compensated in similar magnitudes by increases.
We shall have to wait and see whether the performance data
released publicly allows us to assess whether any real, rather
than artefactual, reductions in crime have occurred. But when
I say "us" I don't mean the electorate, of course. For
most citizens, everyday life is increasingly resembling a lottery.
By the same token, supporting government policies these days is
like taking a trip to the betting-shop. But if that is the way
we are to be governed, do we not have a right not only to know
how to calculate the odds but also whether to trust the bookmaker?
And are criminologists becoming merely the tipsters of the new
crime reduction sweepstakes?
Tim Hope is writing in his capacity as Professor of Criminology
at Keele University. The views expressed here do not reflect necessarily
other commissions in which he is engaged currently.
March 2006
REFERENCES
Bottoms, A (2005) "Methodology Matters", Safer
Society, Summer, 10-12. London: Nacro.
Campbell, DT (1969, 1978) "Reforms as Experiments",
American Psychologist, vol 24, 409-29, 1969. Reprinted in J Bynner
and KM Stribley (eds.), (1978) Social Research: Principles
and Procedures. London: Longman/Open University.
HM Treasury (2004) SR 2004: Public Service Agreements,
chapter 6. London: Home Office.
12 July 2004 (www.hm-treasury.gov.uk/spending_review/spend_sr04_psaindex.cfm).
Home Office (2004a) Reducing Burglary Initiative Project Summary:
Stoneferry, Hull, Supplement 10 to Findings 204. London: Home
Office.
Home Office (2004b). SR 2004 PSA Targets: Technical Notes.
London: Home Office.
(www.homeoffice.gov.uk/documents/homeoffice_sr04tns.pdf ).
Hope, T (2005) "Sustainability in Crime Prevention: a
Nautical Tale", International Observer, Edition 36, March
2005. Montreal, Canada: International Centre for the Prevention
of Crime
(http://www.crime-prevention-intl.org/io_view.php?io—id=117).
Hope, T (2004) "Pretend it Works: Evidence and Governance
in the Evaluation of the Reducing Burglary Initiative",
Criminal Justice, 4 (3), 287-308.
Hope, T (2002) "The Road Taken: Evaluation, Replication
and Crime Reduction", in G Hughes, E McLaughlin and J
Muncie (eds) Crime Prevention and Community Safety, London: Sage
Publications.
Hope, T, Bryan, J, Crawley, E, Crawley, P, Russell, N and Trickett,
A (2004), Strategic Development Projects in the Yorkshire and
the Humber, East Midlands and Eastern Regions, Home Office
Online Report 41/04. London: Home Office.
Kodz, J, Pease, K, and Stephens, M (2004), Explanatory Note
for Online Reports OLR 40/04 to OLR 43/04, Home Office Online
Report 39/04.
Kodz, J and Pease K (2003), "Reducing Burglary Initiative:
Early Findings On Burglary Reduction", Findings 204.
London: Home Office.
Yudkin, PL and Stratton, IM (1996), "How to Deal with
Regression to the Mean in Intervention Studies", The
Lancet, Vol. 347, 241-243.
|
