Supplementary evidence from Professor
Hazel Kemshall following the evidence session on 8 June 2011 (PB
78)
1. The government says that what works should
be the touchstone for practice. Will payment by results work to
achieve this aspiration?
In principle yes—it should incentivise the
best practice to achieve the most desirable outcomes, in this
case offence reduction. However, there are some caveats, in brief:
— The
danger of "cherry picking" easy, and low risk cases
especially by third sector providers—an issue that has been
much discussed recently and about which you no doubt have other
evidence.
— The
variation and inconsistency across Probation Trusts—there
is not actually a national probation service in that sense. The
best performing areas on work works have had strong management
commitment to it and strong quality assurance, guidance and training
initiatives. It is imperative that the recent national standards
and the recent performance measures reinforce what works—and
the concentration on the four domains should achieve this. However,
any discretion must be exercised within a "what works"
framework not outside of it. In addition, should poorly performing
Probation Trusts on effective practice be penalised—for
example following poor HMIP thematic reports on effective practice?
— PbR
has a number of metric/measurement difficulties. I have found
a recent paper by Collins 2010 (appended) helpful in reviewing
these quickly in an accessible way. I think although the paper
has a "stance", the issues are well reviewed and speak
to a number of the committee's concerns.
2. How should practice be measured or evaluated?
Can you think of any consequences to the Government's proposals
that may not have been sufficiently foreseen or any unwanted consequences
that have not been adequately guarded against?
Professors Pease and Hedderman are likely to assist
you rather better here. I can only add:
— Would
comparisons of the best performing Probation Trusts on re-offending
and effective practice measures to the poorest assist with identifying
the environmental, organisational, cultural, management and practice
differences that may subtly influence practice performance?
— This
may also assist with the complex difficulties in identifying the
causal links between inputs, outputs and outcomes that have proved
so challenging.
— Interventions
are often complex and multi-factorial and delivered in partnership,
thus identifying what has made the difference can be difficult,
perhaps case comparisons over a two to five year time line would
be beneficial. The Collins paper may assist with the second half
of the question.
— Finally,
the potential to "park" high risk offenders, (both risk
of harm and risk of reconviction), in addition to stop start provision
and lack of continuity or sustainability in provision for this
difficult group.
— How
can third sector providers be adequately incentivised to provide
the range of services and respond to the range of offender risks
presented? What will be an acceptable level of risk transfer from
government to the Third sector? Will providers wish to stay within
a comfort zone of well known and easy to deliver services to responsive
and low risk offenders, rather than innovate and develop and services
for more challenging and risky offenders? (see for example Dicker
2011 on this point pp: 18-20, appended).
3. What difficulties could commissioners and
providers encounter in demonstrating the effectiveness of complex
partnership arrangements on reductions in reoffending?
In brief:
— Identifying
and sufficiently evidencing the causal chain of impact- who did
what that made the difference?
— The
cumulative or holistic impact of the "package" of interventions
is likely to prove difficult to trace and evidence over time,
and may need time consuming and costly data collection, and prove
burdensome to suppliers.
— The
potential to over-rely on proxy and intermediate measures (eg
accommodation, employment, skill development), rather than long
term reconviction rates.
— What
has been learnt from the IOM pilots on data collection and evidencing
impact that is transferable?
June 2011
| 
