Supplementary evidence from Professor
Carol Hedderman following the evidence session on 8 June 2011
(PB 79)
RESPONSE TO
SUPPLEMENTARY QUESTIONS
FROM THE
INQUIRY INTO
THE PROBATION
SERVICE
1. The government says that what works should
be the touchstone for practice. Will payment by results work to
achieve this aspiration?
Currently Probation Services might not adhere to
the targeting principle because of other pressures (eg they might
send a sex offender who does not fit the targeting criteria to
the only programme for sex offenders available because the court
has made attendance at a programme part of the order). This has
the effect of making a programme look less effective, as results
will include its impact on those it was always unlikely to benefit.
If a provider is only paid if they reach a certain
threshold of impact, this could encourage better targeting. However,
this carries two risks. First, a PbR provider will only work with
those who are most certain to benefit (thus some of those
who might benefit will be left out and go on to reoffend).
Second, the margin between a profitable outcome and an unprofitable
outcome will be narrow. The current approach could lead to the
delivery of interventions being curtailed when it becomes obvious
that the target for payment is going to be missed. There is public
benefit in reducing reoffending by 8%, but if there is no financial
payment before 10%, a sensible investor might stop delivery and
start doing something else with the money. So if a project is
on track to deliver a 6% impact and, with effort, expects to raise
this to 8% in the payment period but has a target of 10%, it would
make financial sense to stop delivery and put their resources
into something else. This risk could be minimised by having a
graduated relationship of payment to impact so, for example,
achieving 80% of target results in 80% of the payment. This has
other problems of course (eg if the payment for 80% is enough
the provider might just decide to aim for that), but it seems
worth considering.
There are a number of other inter-related risks to
What Works:
— Investors
will only enter the market if rewards were certain enough and
great enough. Effective interventions do not usually bring huge
changes. This is a potential disincentive for investors looking
for a decent return, and for commissioners and evaluators who
want to be sure any change which is observed is the result of
the intervention.
— The
evidence base for what is effective is quite weak which makes
it hard to make carefully calculated investment decisions. PbR
might stifle innovation for this reason as investors would probably
find it safer to invest in refining the tried and tested things
and doing those as well as they can.
— It
would be sensible investment strategy to avoid supporting offenders
at high risk of offending unless the rewards are very high. An
alternative strategy would be to pay according to a target set
for each high risk individual, but there is no reliable way to
calculate a suitable individual target (as predictors only work
at the group level).
— Small
outcomes would require large samples to be significant so again
the small difficult groups would not be attractive (women, sex
offenders) unless other measures of impact than those being used
at Peterbrorough were adopted.
2. How should practice be measured or evaluated?
Can you think of any consequences to the Government's proposals
that may not have been sufficiently foreseen or any unwanted consequences
that have not been adequately guarded against?
Some of the points above are relevant to this. Also,
as I mentioned at the Committee meeting, the impact of sentences
is usually measured comparatively (probation vs prison) not absolutely
(probation vs nothing). The Peterborough experiment is unusual
because short sentence prisoners get no statutory support on release,
so the impact is being measured in absolute terms. As it is assessing
"help" vs "no help", this should mean there
is quite a big impact. If PbR becomes more common the measurement
will either be (i) actual vs expected or (ii) actual for those
receiving X vs actual for those receiving Y. These comparative
rather than absolute differences are likely to be narrower and
both approaches have some serious measurement issues associated
with them.
3. What difficulties could commissioners and
providers encounter in demonstrating the effectiveness of complex
partnership arrangements on reductions in re-offending?
I have jointly authored a paper (attached) on what
happens when "what works" in probation comes up again
"what works" in getting people into employment. Under
PbR there is a possibility that employment specialists will be
reluctant to support those on community supervision or those on
licence. First, they are likely to be difficult to place people
who are known to have severe problems sustaining employment. Second,
the fact that they are under supervision or on licence also brings
risks in a PbR context. Employment might not be sustained because
of enforcement action leading the employment service provider
to miss their target. Alternatively, the employment scheme might
not report risky behaviour on the part of the offender because
of the potential consequences to their target of enforcement action.
4. The MoJ's stance on the reoffending data
published last week which they state shows that prolific offending
has been getting worse over the last 10 years
I think a lot of the explanation for this lies in
the commentary and figures in the publication.
The figures show convictions have increased (which
is presented in the press release as though it is offending).
In my view this is mainly a consequence of the offences brought
to justice targets (it is easier to achieve these by rounding
up the usual suspects than detecting new ones, consequently the
ones you already know about get longer and longer records). This
probably explains why this target was first changed to serious
offences and then abandoned (see page 3). Also (page 6) the increases
in indictable offence convictions are mainly in shoplifting and
cannabis (again easy offences to shift into convictions from warnings/NFA).
Finally, isn't it odd that offenders are getting more prolific
as crime is going down? In my view, the most likely explanation
is that the increase in prolific offenders is an artefact of processing
decisions.
June 2011
| 
