Cutting crime: the case for justice reinvestment - Justice Committee Contents


Examination of Witnesses (Questions 65 - 79)

TUESDAY 17 JUNE 2008

DR CHLOË CHITTY

  Q65  Chairman: Dr Chitty, thank you very much for joining us this afternoon. You have an even more complicated title than I have got, because you are the Assistant Director, Offender Management and Sentencing Analytical Services, of the National Offender Management Service, which is even harder to say, but we particularly welcome you this afternoon because of the work we are doing on justice reinvestment and also we are anxious to know how, within the system as it is at present, the contribution of research aids policy-making. Can you give us a quick thumbnail sketch of how the research units within the Ministry of Justice and the National Offender Management Service actually contribute to the development of policy?

  Dr Chitty: We contribute in a number of ways, very broadly speaking, right from the beginning of the development of policy. So when policy colleagues, ministers are thinking about a new intervention or a new policy, we contribute to thinking about what that might be, what the objectives might be, what the options might be in terms of implementing it, and we also contribute in terms of monitoring and evaluation and feedback. In addition, we do a number of things, particularly, for example, in terms of projecting populations of the prison, in terms of looking at the trends and what is going on and trying to get a sense of what is coming on in terms of the horizon; so really across the piece.

  Q66  Alun Michael: We have heard a couple of comments about the weaknesses in the current evidence base for reducing reoffending. What do you see as the weakness in the evidence base? What are the places where you are put in the situation of having to say, "I am not quite sure because the evidence is not absolutely clear on this issue"?

  Dr Chitty: There are a number of places where we can always develop the evidence base. I think one of the things that is important to say is that, wherever you are within the evidence base, you can still learn something. One of the key things is not to under-interpret the information you have, and, of course, you must not over-interpret it either, but I think there are some areas where we do need to develop our knowledge further. We have got pretty good evidence in certain interventions_things like offending behaviour programmes, things like drug treatment, we have got good associations between things like family ties and the role of family ties in reducing reoffending_so it is promising there, but it is still equivocal. There are a number of ways that we can do that. I think we can increase the quality of our costing and our cost-benefit analysis type work that we are doing and, again, something that we are doing there is looking at developing our appraisal[1], [2] methodology so that we can get a better sense right from the start whether or not interventions are worth implementing in the first place and, relatively speaking, which is likely to be more cost-effective or more effective.

  Q67  Alun Michael: What efforts are you making to identify and improve weaknesses, or, to put it another way round, what one sometimes feels is an enormous amount of evidence but not a great deal of evidence that actually is useful and is used to make sure that more targeted approaches are developed?

  Dr Chitty: We are doing a number of things. First of all, we have spent a fair amount of our time doing things that are called systematic reviews[3], [4] and rapid evidence assessments,[5], [6] trying to draw together the information that we have that is available, not just our own information but nationally and international, and there are a number of techniques. You can do that systematically through a review, you can do that through meta-analysis[7], [8] type techniques, really making better use of the information that is available. Another point, as you say, is to make it more relevant. It is about pulling out the policy implications, pulling out what this really means and spending more time on actually identifying that. One of the things that we have done recently in the work that we are doing is trying to develop what we have described as a "one, three, twenty-five process", so when we do reports, we do not just do an executive summary, we do not just do a report, we actually do a one-pager that sets out the key findings so that people can really get at them, but the final key, I think, is the appraisal side of things, and that is something that we are moving much more towards, which I think is really exciting. What we are really trying to do is bring that all together and say: what is the cost-benefit analysis here? Really what is going to be better to reduce reoffending, or protect the public, or whatever?

(http://en.wikipedia.org/wiki/Systematic-review).

(http://www.gsr.gov.uk/professional_guidance/rea_toolkit/what_is_an_rea/methods_for_reviewing_evidence/index.asp).

(http://www.idea.gov.uk/idk/core/page.do?pageId=8169638).

(http://www.gsr.gov.uk/downloads/magenta_book/complete_Magenta.pdf).

(http://www.jr2.ox.ac.uk/bandolier/booth/glossary/meta.html).

  Q68  Alun Michael: What are the implications of the failure to implement C-NOMIS? What is the impact of that particularly on the quality of data?

  Dr Chitty: They are going to implement a fair amount of C-NOMIS, particularly in the Prison Service, and I think that is very important. As I understand it, the probation systems as well are doing some rationalisation to improve the quality there. From our perspective, I would always like more data, but, having said that, I think the implications of C-NOMIS are such that we will still have a lot of good information. We are still developing our information. We have got three cohort studies for juveniles, prisoners and probationers where we are learning what happens when they arrive into the system, following them through the system and following up afterwards. We are not just focusing on things like what C-NOMIS can do, we are also focusing on data quality in terms of things like the prisons, probation data as well; so there are a number of ways that we are looking at data.

  Q69  Alun Michael: From the outside, we are still left with the situation where there seem to be gaps in communication, gaps in information, and, if that is the case, then, obviously, the quality of the data and the research based on data is going to have a weak foundation. Are those perceptions from outside justified?

  Dr Chitty: There are areas in the past, for example, black and minority ethnic data, where there have been gaps, and certainly previously we have not actually published information because we felt it was not good enough quality, but that is an example of an area where we have been developing with our colleagues the data quality issues, and it has got much better now, so we are publishing it. One of the advantages of using our research and analysis is actually to test that out. I think it is fair to say that, where offenders are being managed locally, there is very good information, and sometimes there is a disconnect between the information collected locally and what we get nationally. That does not mean that they are not being managed well; it might mean that there are differences in the information that is available nationally and, from our point of view, there is quite a lot of work going on at the moment, and there has been for years, in actually improving what we know. An example of that is the weekly receptions data that we get from prisons so that we are in a better position to monitor what is happening in prisons. We have been doing all of that all the time, but it must be continuous improvement.

  Q70  Mr Turner: What are the benefits of the new local reoffending measures?

  Dr Chitty: I think they are very interesting, and for us they are a very exciting opportunity to do a number of things. They are still experimental; we are learning what they are about. What we are trying to do with them is provide a much more timely measure to areas locally so that they get a sense---. Of course, in the past, what we had was a measure where you had to wait almost three years, effectively, until you knew whether or not you had actually succeeded; so the idea behind the local measures is actually to produce something that is much more timely, and in doing that what we will need to do, as we develop it, is to understand the implications of taking a quarterly snapshot. We will move towards an annual rolling average, quarterly average, because we need to look at things like seasonality effects and how much it changes over time to understand what it is telling us.

  Q71  Mr Turner: So you get, across the whole country, applications of specific processes or would you provide one in the south and one in the north, for instance, because you are testing them out?

  Dr Chitty: In terms of the local authority data, we have the national data and we are breaking it down to the local authority areas, to the Government office regions and to the probation area, so we have all of those. Each area gets its own version of their outputs, if you like, and they compare it against the national average and there is an actual versus predicted rate, so they can get some sort of sense of the effect that they are having. Obviously, as we do more of that, we will understand how much that actually changes over time and we will be able to do sub-analyses or cross-area analyses to try and look at some of the differences.

  Q72  Mr Turner: So the Isle of Wight would be one and somewhere in Hackney another?

  Dr Chitty: It will be probation area or local authority area based, and so it may not get down to the kind of level that you are talking about, not least because, as you can imagine, the sample sizes would get pretty small, and if you are comparing data over time, especially if you have got small sample sizes, what you get is a lot of variability and, if you have got a lot of variability, you do not really understand what is happening.

  Q73  Mr Turner: At what level is that?

  Dr Chitty: The local authority areas or the probation areas. There are 42 probation areas in the country; so you get a sense of the size from that.

  Q74  Dr Whitehead: When the Cabinet Office conducted a capability review on the Ministry of Justice, they commented that the Ministry is not able to draw on reliable data when taking prioritisation decisions, is not currently able to accept a reasonable price for a given level of services and drive efficiencies by creating incentives to provide either at or below that price. Do you think it is possible to plan effectively without that reliable evidence base given those sorts of comments?

  Dr Chitty: I am sorry, forgive me, I did not quite hear your first comments. Would you mind repeating them?

  Q75  Dr Whitehead: I drew attention to the capacity review that was carried out by the Cabinet Office this year into the Ministry of Justice and their comments, in particular, that the Board of the Ministry of Justice is not able to draw on reliable data when taking prioritisation decisions and the problem of what the Department knew about price for a given level of services, for example. Do you think you have a reliable evidence base on which to undertake those sorts of longer-term planning decisions?

  Dr Chitty: I think the work that we are doing is a very important part of that. There are a number of things that are happening in the Ministry of Justice at the moment. Apart from anything else, we are doing some work to look at the criminal justice system and offender management in my area, for example, where we are trying to understand what the costs and benefits are across the system for the various aims of the criminal justice system, including not just rehabilitation, but we actually will be looking in things like punishment because that is an important aim, the work that we are doing in relation to appraisal and testing the knowledge we have and building that up and also what we are trying to do is give a sense of the quality of that information. At its most basic, what we are trying to do is to say to ministers and to say to our policy and operational colleagues, "We are confident about this bit of information, we are less confident about this information, so this is what we think you can say about it", but also to bring that together and actually to say, "What does that mean?" I think it is fair to say that by the end of this year we will have something that is pretty good. There are some things, even with relatively good data as opposed to excellent data, that you can say and you can learn and you can do things about. At the same time what we need to do is, of course, continue to build that evidence base, and one of the things that the Ministry of Justice is actually doing is appointing a director of research and analysis, and the key reason for that is to bring together the research and analytical effort in the Department but also to raise the capacity of the analytical capability and use of evidence in the Department; and that was one of the other elements of the capability review that you referred to.

  Q76  Dr Whitehead: Bearing that in mind, what sort of forecasting mechanisms are you using, for example, on projected prison and probation case loads and costs, and particularly the question of how the one impacts on the other in terms of what might be the capacity, for example, of the Probation Service to provide alternatives to prison placement?

  Dr Chitty: There are a number of things that we are doing to project the prison population and the probation population. Indeed, you will probably have seen the annual prison population projections that we have done and, indeed, the work that we did to inform the Carter Review. You have probably seen quite a lot of that. Carter himself actually did a fair amount of work on costings. I am not responsible for the costing side of things myself, but certainly the evidence that we have been involved in is fundamentally support in terms of the forecasts of the case loads for both the Prison Service and the Probation Service over the next few years, but also to offer support in a number of ways. For example, where new policies are being developed, not just to give impact assessments on what the effect is going to be on the prison population, but what the effects are going to be on the Probation Service and to pull the two together to make sure that both are thought about at the same time.

  Q77  Dr Whitehead: But it is the case, is it not, indeed just recently the Secretary of State for Justice indicated that the predictions that had been made on prison population_the high, medium and low scenarios_were running even in excess of the highest predictions and that the forecasting appeared to be rather inaccurate.

  Dr Chitty: I think the forecasting of the prison population over the years_. We have done a number of pieces of work to look at the accuracy of our prison population projections. The key thing to remember is that they are scenarios and they are projections, they are not forecasts, and they are dependent, therefore, on a number of scenarios happening. If you look, for example, at the latest set of projections, what they are based on is a number of things. They are based on what we know about trends_for example, sentencing trends_what we know about impacts of new Acts that are coming in. They are based on a whole number of things. The work that we have done that has been looked at externally suggests that actually where we have got those assumptions right, we are actually 99% accurate. The reality, of course, of projections in the world is that life changes and new Acts come in, there are sentencing changes, and so things happen that mean that actually what you do need to do (and we do do this) is keep an eye on our prison population projections, and we, indeed, do internal forecasts so that we actually monitor the prison population and see what is going on so that we can make sure that policy, operational colleagues and ministers, indeed, have the best possible sense of where the population is going.

  Q78  Dr Whitehead: With respect, when you say other things occur, are a number of those things that do occur not to some extent within the overall operational capacities of the Ministry of Justice in the first place and, therefore, some of the events that occur, one might say, might be placed into the projections before they occur rather than worried about after they occur?

  Dr Chitty: Let me give you an example of the kind of thing that I mean. Before the introduction of the Criminal Justice Act 2003, we had_. Normally when we project the prison population we have three projections. When they were published we had 10, because although we had a sense of what was going to happen, one of the things in the Criminal Justice Act, indeed, surrounding the Criminal Justice Act, period, was the potential, as a result of the Sentencing Guidelines Council, actually to reduce sentence lengths by 15%. Of course, we did not know how that was going to work, but that was one of the potential things that was going to happen, and so what we did was we created scenarios where that happened and where that did not happen. You do have a sense, but you do not know definitely, and, of course, you do need to keep monitoring it.

  Q79  Dr Whitehead: Conversely, the National Audit Office, I think, reported that the Probation Service does not know with any certainty how many orders it has got the capacity to deliver, and nor has it determined the full cost of community orders. Does that suggest to you, perhaps, that the range of evidence which might be relevant to deciding, for example, what is the balance between shall we say incarceration and non-incarceration as far as future policy is concerned may be based rather on what you have the information on rather than what is the best balance between cost, information and projections?

  Dr Chitty: I would not say that myself. I think if you look particularly, for example, at the models on which the prison population projections are based, they are enormously accurate, and the Grove MacLeod[9] model of offending behaviour is actually quite extraordinary in terms of the quality of the model. Of course, the key thing that you need to do in terms of projection is actually be able to predict. You do not necessarily need to be able to understand, you need to be able to predict, and the Grove MacLeod model is a very good predictor of offending behaviour. What you need to focus on, from an entirely analytical point of view, is what is the thing that is going to give you by far the best, the most accurate output, and I am pretty confident that we have gone as far as we can. Indeed, when we have had discussions with people nationally and, indeed, internationally, we have had those debates around what can you do, what should you do, and we are pretty much up there. Actually we have been giving advice to other people for that reason. I would not say we were 100% perfect, I would not say we cannot improve, because we always can, but I think the starting point is the right starting point, which is what is the best way to predict and then to look at that? I think it is very much what you are saying.



1   Note by witness: Back

2   Chapter 2, Managing Appraisals and Evaluation. Overview of Appraisals and Evaluations. The Green Book, HM Treasury, 2003. Back

3   Note by witness Glossary. Appraisal: The process of defining objectives, examining options and weighing up the costs benefits, risks and uncertainties of these options before a decision is made. The Green Book, HM Treasury, 2003. Back

4   Note by witness: Section 2.3. A Solution. Background Paper 2. The Green Book, HM Treasury. Updated October 2007. Back

5   Systematic Review: A systematic review is a summary of research that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies to identify the valid and applicable evidence Back

6   Rapid Evidence Assessment: (1) Quick overview of existing research on a (constrained) topic and a synthesis of the evidence provided by these studies to answer the REA question Back

7   (2) A way of reviewing research and evidence on a particular issue. It looks at what has been done in a particular area and records the main outcomes Back

8   Meta-Analysis: (1) "The statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings." Back

9   (2) A statistical technique for combining the findings from independent studies Back


 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2010
Prepared 14 January 2010