Select Committee on European Union Minutes of Evidence

Examination of Witnesses (Questions 162-179)

MONDAY 19 MAY 2003



  162. Good afternoon. Thank you for coming to give evidence this afternoon. We are very grateful, through you, to the Minister for the written evidence we have received. We have read it all carefully and with interest. You kindly sent in your written evidence very early and you can take it as read, rather than repeating points. Is there anything you would like to say at all by way of brief introduction? Otherwise, we will go straight into questions.
  (Mr Rees) We are specialists in the Department of Trade and Industry. We are economists dealing with evaluation issues. That is the area we were told you wished to cover and we have come along on that basis.

  Chairman: I cannot promise that other issues will not arise but if they do and you feel they are better answered by a separate note we are perfectly happy with that.

Lord Howie of Troon

  163. Over the last ten years, there have been a number of initiatives, schemes, programmes and things of that sort emanating from your department. Can you tell me how many of these, if any, have been modified or have even closed down during that period?
  (Mr Rees) I cover the evaluation evidence in the Department and I see the conclusions that come out of the evaluations and the follow up. Most of the evaluations lead to some modifications in the Department schemes and generally the evaluations have acted to bring about changes in cost effectiveness of individual programmes. The most recent example would be the supply chain initiative that the department operates. There was quite a penetrating analysis from an academic at Reading University a few months ago and that is leading to some changes in the scheme. When we talk about evaluation, we are talking about a body of evidence that has built up over time, the evidence base of the department, the appraisal system and the way that operates. Often, that can deter particularly effective schemes from coming through as well as having an ex post effect. Evaluation is not rocket science. Where you can think of schemes that we have modified or in some cases where there have been less frequent closures, usually evaluation is not the only thing that is tied into that. There is usually a new strategy for that Department, where that should go, what other issues come into play. Over time, there have been quite a few modifications and improvements to schemes that have taken place because of evaluation.

  164. Have you what you might call a hit rate for successes?
  (Mr Rees) On the success side, it is very much on the individual basis but one example would be the Teaching Company Scheme which has been operational since the mid-1970s. There was a very positive evaluation of that in the mid-1990s and that contributed to a doubling of spend on that programme. More recently, there was quite a thorough evaluation done of the Smart programme which was a grant to promote innovation in small companies. That has led the Secretary of State to announce that that is one of the most effective schemes and we are looking at ways in which that can be expanded. Take up and penetration could be improved.

  165. When you were talking about the Teaching Company Scheme, did you say there was an increase in the spend after the evaluation?
  (Mr Rees) There was an evaluation in the mid-1990s and a review of the programme that took into account the evaluation. As a result of that, there was then an expansion in the spend on the programme, so quite a big increase.

  166. You are quite sure that you have value for money, are you?
  (Mr Rees) On TCS, it is one of those schemes that has a very long history at the DTI. It dates back to the mid-1970s. Over time, the effectiveness of the scheme has been worked on and I think there have been several evaluations of that scheme. It would be one of those areas where the Department would think it was giving value for money.

  167. Sometimes people think that if they throw more money into a scheme it is therefore improved. Do you think the increase has led to an improvement?
  (Mr Rees) There are always issues about diminishing returns.

  168. Putting money into a black hole?
  (Mr Rees) We need to watch that area. Some schemes may have reached their maximum penetration and adding more money will not improve cost effectiveness but with TCS the thought was that there was scope to expand that. There are constraints with, for example, the numbers of academics who can supervise the TCS associates. There are constraints on the degree to which you can expand the programme.
  (Mr Lambert) It might be worth adding that on things like TCS, where there is a full, quinquennial process, it is supervised by an independent steering body rather than being done by my Department. It is a programme that draws funding from across Whitehall and from most of the research councils. It is managed through the DTI. It is not wholly a DTI scheme. It is not a DTI official's judgment that is reached on that particular programme; it is independently arrived at and the committee that reviewed it in the mid to late 1990s were definitely convinced that the barriers to growth of the scheme had not been reached and it was possible to expand it while maintaining the quality, for example, by trying to draw other sorts of funding bodies and institutions into what is known as the knowledge transfer partner. The independent research organisations were invited to participate, providing the expertise.

Lord Chadlington

  169. I do not know if you were present when Mr van der Horst was talking about the EU situation but he was quoting an example in his written evidence of the work which David Storey has done at Warwick University on evaluation. I wondered whether, when you think about an evaluation or a new scheme comes in, a percentage of the money that is allocated is set aside for the evaluation. If it is, what is the percentage? If not, in general, what percentage of money behind projects goes into the evaluation system, or is that an absolutely impossible question?
  (Mr Rees) There are some Treasury guidelines that about half a per cent of the programme spend should be dedicated to evaluation but that should not be rigidly applied. It depends in particular on the type and scale of the programme and so on. If you look at the DTI's experience, we did quite well with evaluation but I think we spent a little less than that. Maybe last year we spent about 1.5 million on evaluation, of which about a million was consultancy time and about a half a million was staff time, covering a programme spend of about £ ¾ billion.


  170. The Department spends how much on evaluation?
  (Mr Rees) It is about 1.5 million on evaluation of specific schemes.

  171. On a value of programmes of how much?
  (Mr Rees) About three-quarters of a billion. In addition, there is quite a lot of research that has also taken place that would need to be factored in. I do not know what the broad brush figures for that are, but that would be quite substantial.

  Lord Chadlington: This is terribly important to some aspects of the conversations we have been having. Could we possibly have some indication, sent separately to the Clerk, about what kind of quantum is spent on research so that we could take it together with the 1.5 million and look at the three quarters of a billion? That would be very helpful.


  172. Relevant to evaluation.
  (Mr Rees) Yes, okay.

  173. We understand from your evidence and others that the DTI is currently redesigning its enterprise support policy and rationalising its schemes down from over 150 to about 20. Has programme or project evaluation been used in deciding upon the rationalisation?
  (Mr Rees) Yes. Evaluation played a part in the very first stages of the review of business support. The approach that was taken was to initially look at the 20 largest spending business support programmes and look at the evaluation evidence on those. That I think covered about 80 per cent of the programme spend on business support. Following that, they looked at some of the smaller schemes and some of the generic lessons that came through. Some of the big programmes were quite new and therefore evaluations have not yet been forthcoming, but the evaluation evidence did feature quite prominently in decisions that were taken.

  174. 20 of the 150 covered 80 per cent of the spend. Have the evaluations of those 20 been published?
  (Mr Rees) Yes. We have an evaluation website at the DTI and that material is in the public domain.

  175. If you are able to remind us in a note of those 20 schemes and websites to save our time, and take out, say, the top three spends and send us hard copies of those that would be enormously helpful.
  (Mr Rees) Certainly.

  176. What were the top two spending schemes?
  (Mr Rees) Regional Selective Assistance, the Space Programme, Business Links, Launch Investment.

  177. When you are considering schemes to promote entrepreneurship in the consolidated schemes, your fewer than 20, what will be the specific objectives of those schemes? Are they being decided upon? Are objectives and criteria set out for assessing effectiveness in due course? Has that all been done?
  (Mr Rees) a very strong feature is going to be the feature to improve productivity and competitiveness in the economy. The idea is to develop greater harmony in the way that we develop our programmes and the way we evaluate them. That central theme of productivity and competitiveness is central to the design of schemes and their evaluation but we also recognise that schemes may have other objectives. For example, there may be objectives to raise employment or to promote productivity in disadvantaged areas. There are other factors that we need to take into consideration as well. For the business support schemes, productivity is a key issue.
  (Mr Hallett) The overall strategy does follow down from high level, public service agreement targets that the department has which cover a specific productivity objective and wider issues around social inclusion and assistance for deprived areas and communities.

  178. Your public service agreement is with the Treasury, I assume?
  (Mr Hallett) Yes, and we have joint targets with the Treasury for productivity.

  179. In discussing the objective of each of the consolidated schemes, there will be an agreement between yourselves and the Treasury on the purpose of the scheme, the objectives, the ways of assessing success and so on so that there is presumably a period of time after which, every two years or five years, there will be an assessment based upon the objectives and the agreed evaluation technique. Is that the nature of the agreement?
  (Mr Rees) Yes. There is an investment committee that has been set up in the department to review the business support activities of the Department, which has an independent chair. That will be looking into assessing the schemes in relation to productivity and competitiveness. Somebody from the Treasury sits on that committee so there is an agreement about the strategy that is coming through that committee. Also, the criteria on which to assess programmes. There is the Treasury green book that you mentioned; we also have some more detailed guidance on how to apply that specifically to DTI programmes. That has also been agreed by the Treasury.

previous page contents next page

House of Lords home page Parliament home page House of Commons home page search page enquiries index

© Parliamentary copyright 2003