Examination of Witnesses (Questions 162-179)|
MONDAY 19 MAY 2003
162. Good afternoon. Thank you for coming to
give evidence this afternoon. We are very grateful, through you,
to the Minister for the written evidence we have received. We
have read it all carefully and with interest. You kindly sent
in your written evidence very early and you can take it as read,
rather than repeating points. Is there anything you would like
to say at all by way of brief introduction? Otherwise, we will
go straight into questions.
(Mr Rees) We are specialists in the Department of
Trade and Industry. We are economists dealing with evaluation
issues. That is the area we were told you wished to cover and
we have come along on that basis.
Chairman: I cannot promise that other issues
will not arise but if they do and you feel they are better answered
by a separate note we are perfectly happy with that.
Lord Howie of Troon
163. Over the last ten years, there have been
a number of initiatives, schemes, programmes and things of that
sort emanating from your department. Can you tell me how many
of these, if any, have been modified or have even closed down
during that period?
(Mr Rees) I cover the evaluation evidence in the Department
and I see the conclusions that come out of the evaluations and
the follow up. Most of the evaluations lead to some modifications
in the Department schemes and generally the evaluations have acted
to bring about changes in cost effectiveness of individual programmes.
The most recent example would be the supply chain initiative that
the department operates. There was quite a penetrating analysis
from an academic at Reading University a few months ago and that
is leading to some changes in the scheme. When we talk about evaluation,
we are talking about a body of evidence that has built up over
time, the evidence base of the department, the appraisal system
and the way that operates. Often, that can deter particularly
effective schemes from coming through as well as having an ex
post effect. Evaluation is not rocket science. Where you can
think of schemes that we have modified or in some cases where
there have been less frequent closures, usually evaluation is
not the only thing that is tied into that. There is usually a
new strategy for that Department, where that should go, what other
issues come into play. Over time, there have been quite a few
modifications and improvements to schemes that have taken place
because of evaluation.
164. Have you what you might call a hit rate
(Mr Rees) On the success side, it is very much on
the individual basis but one example would be the Teaching Company
Scheme which has been operational since the mid-1970s. There was
a very positive evaluation of that in the mid-1990s and that contributed
to a doubling of spend on that programme. More recently, there
was quite a thorough evaluation done of the Smart programme which
was a grant to promote innovation in small companies. That has
led the Secretary of State to announce that that is one of the
most effective schemes and we are looking at ways in which that
can be expanded. Take up and penetration could be improved.
165. When you were talking about the Teaching
Company Scheme, did you say there was an increase in the spend
after the evaluation?
(Mr Rees) There was an evaluation in the mid-1990s
and a review of the programme that took into account the evaluation.
As a result of that, there was then an expansion in the spend
on the programme, so quite a big increase.
166. You are quite sure that you have value
for money, are you?
(Mr Rees) On TCS, it is one of those schemes that
has a very long history at the DTI. It dates back to the mid-1970s.
Over time, the effectiveness of the scheme has been worked on
and I think there have been several evaluations of that scheme.
It would be one of those areas where the Department would think
it was giving value for money.
167. Sometimes people think that if they throw
more money into a scheme it is therefore improved. Do you think
the increase has led to an improvement?
(Mr Rees) There are always issues about diminishing
168. Putting money into a black hole?
(Mr Rees) We need to watch that area. Some schemes
may have reached their maximum penetration and adding more money
will not improve cost effectiveness but with TCS the thought was
that there was scope to expand that. There are constraints with,
for example, the numbers of academics who can supervise the TCS
associates. There are constraints on the degree to which you can
expand the programme.
(Mr Lambert) It might be worth adding that on things
like TCS, where there is a full, quinquennial process, it is supervised
by an independent steering body rather than being done by my Department.
It is a programme that draws funding from across Whitehall and
from most of the research councils. It is managed through the
DTI. It is not wholly a DTI scheme. It is not a DTI official's
judgment that is reached on that particular programme; it is independently
arrived at and the committee that reviewed it in the mid to late
1990s were definitely convinced that the barriers to growth of
the scheme had not been reached and it was possible to expand
it while maintaining the quality, for example, by trying to draw
other sorts of funding bodies and institutions into what is known
as the knowledge transfer partner. The independent research organisations
were invited to participate, providing the expertise.
169. I do not know if you were present when
Mr van der Horst was talking about the EU situation but he was
quoting an example in his written evidence of the work which David
Storey has done at Warwick University on evaluation. I wondered
whether, when you think about an evaluation or a new scheme comes
in, a percentage of the money that is allocated is set aside for
the evaluation. If it is, what is the percentage? If not, in general,
what percentage of money behind projects goes into the evaluation
system, or is that an absolutely impossible question?
(Mr Rees) There are some Treasury guidelines that
about half a per cent of the programme spend should be
dedicated to evaluation but that should not be rigidly applied.
It depends in particular on the type and scale of the programme
and so on. If you look at the DTI's experience, we did quite well
with evaluation but I think we spent a little less than that.
Maybe last year we spent about 1.5 million on evaluation, of which
about a million was consultancy time and about a half a million
was staff time, covering a programme spend of about £ ¾
170. The Department spends how much on evaluation?
(Mr Rees) It is about 1.5 million on evaluation of
171. On a value of programmes of how much?
(Mr Rees) About three-quarters of a billion. In addition,
there is quite a lot of research that has also taken place that
would need to be factored in. I do not know what the broad brush
figures for that are, but that would be quite substantial.
Lord Chadlington: This is terribly important
to some aspects of the conversations we have been having. Could
we possibly have some indication, sent separately to the Clerk,
about what kind of quantum is spent on research so that we could
take it together with the 1.5 million and look at the three quarters
of a billion? That would be very helpful.
172. Relevant to evaluation.
(Mr Rees) Yes, okay.
173. We understand from your evidence and others
that the DTI is currently redesigning its enterprise support policy
and rationalising its schemes down from over 150 to about 20.
Has programme or project evaluation been used in deciding upon
(Mr Rees) Yes. Evaluation played a part in the very
first stages of the review of business support. The approach that
was taken was to initially look at the 20 largest spending business
support programmes and look at the evaluation evidence on those.
That I think covered about 80 per cent of the programme
spend on business support. Following that, they looked at some
of the smaller schemes and some of the generic lessons that came
through. Some of the big programmes were quite new and therefore
evaluations have not yet been forthcoming, but the evaluation
evidence did feature quite prominently in decisions that were
174. 20 of the 150 covered 80 per cent
of the spend. Have the evaluations of those 20 been published?
(Mr Rees) Yes. We have an evaluation website at the
DTI and that material is in the public domain.
175. If you are able to remind us in a note
of those 20 schemes and websites to save our time, and take out,
say, the top three spends and send us hard copies of those that
would be enormously helpful.
(Mr Rees) Certainly.
176. What were the top two spending schemes?
(Mr Rees) Regional Selective Assistance, the Space
Programme, Business Links, Launch Investment.
177. When you are considering schemes to promote
entrepreneurship in the consolidated schemes, your fewer than
20, what will be the specific objectives of those schemes? Are
they being decided upon? Are objectives and criteria set out for
assessing effectiveness in due course? Has that all been done?
(Mr Rees) a very strong feature is going to be the
feature to improve productivity and competitiveness in the economy.
The idea is to develop greater harmony in the way that we develop
our programmes and the way we evaluate them. That central theme
of productivity and competitiveness is central to the design of
schemes and their evaluation but we also recognise that schemes
may have other objectives. For example, there may be objectives
to raise employment or to promote productivity in disadvantaged
areas. There are other factors that we need to take into consideration
as well. For the business support schemes, productivity is a key
(Mr Hallett) The overall strategy does follow down
from high level, public service agreement targets that the department
has which cover a specific productivity objective and wider issues
around social inclusion and assistance for deprived areas and
178. Your public service agreement is with the
Treasury, I assume?
(Mr Hallett) Yes, and we have joint targets with the
Treasury for productivity.
179. In discussing the objective of each of
the consolidated schemes, there will be an agreement between yourselves
and the Treasury on the purpose of the scheme, the objectives,
the ways of assessing success and so on so that there is presumably
a period of time after which, every two years or five years, there
will be an assessment based upon the objectives and the agreed
evaluation technique. Is that the nature of the agreement?
(Mr Rees) Yes. There is an investment committee that
has been set up in the department to review the business support
activities of the Department, which has an independent chair.
That will be looking into assessing the schemes in relation to
productivity and competitiveness. Somebody from the Treasury sits
on that committee so there is an agreement about the strategy
that is coming through that committee. Also, the criteria on which
to assess programmes. There is the Treasury green book that you
mentioned; we also have some more detailed guidance on how to
apply that specifically to DTI programmes. That has also been
agreed by the Treasury.