The Evidence Ltd Reports
8. In 2002 DTI commissioned the consultants Evidence
Ltd to develop a wide-ranging set of indicators to benchmark the
UK's performance in science and engineering against that of a
comparator group of 25 countries, including all the countries
in the G8. Evidence Ltd's first report was published in October
2003.[6] Using its findings,
DTI was subsequently able to report that the UK was "on course"
to meet its targets. A more conclusive statement was not considered
possible because "it will take some years before the significant
increase in the Science Budget begins to be reflected in a change
in the UK's performance".[7]
Following our evidence session with Ministers on 1 November this
year, we asked when a discernable change in the UK's performance
could be expected. In response, we were informed that "the
outputs of research are often not realised until 3 to 6 years
after the start of funding and the innovation process may take
approximately eight years on average".[8]
This long lag time makes it difficult to draw any firm conclusions
yet about the outcomes of the Government's increased investment
in science and innovation.
9. Following publication of the second Evidence Ltd
report, in October 2004, the DTI again reported that the UK is
"on course" to meet its targets, although this statement
is fairly meaningless given the time lag detailed above.[9]
In its Autumn Performance Report 2004, it reports that
the UK has an 11.9% share of world citations; is second in six
out of the nine broad scientific disciplines; has 5.8 researchers
per 1,000 members of the workforce; and leads the G8 for citations
per unit of expenditure on publicly performed R&D, citations
relative to GDP, and citations per researcher.[10]
Given that there are very few changes in the data between 2003
and 2004, few firm conclusions can be drawn, particularly because
several of the indicators are subject to year-on-year movements
for external reasons, such as changes in GDP. However, the fact
that little has changed since the October 2003 report itself supports
the Committee's argument, made in its 2003 Scrutiny Report, that
a biannual, rather than an annual, study might be sufficient.[11]
10. One of the most noticeable changes charted in
the two Evidence Ltd reports is a decline in the UK's share of
global publications. When we asked Ministers to comment on this
on 1 November, Lord Sainsbury told us that "people tend to
not appreciate that with more and more countries becoming developed
[…] the likelihood is that in terms of the amount of science,
it is likely to be a declining amount simply because more and
more countries are becoming wealthy".[12]
However, the rise of less industrialised nations such as China
and India does not explain why the UK has fallen behind Japan.
In answers to further written questions, the Government told us
that this could be largely explained by the "steady decline
in government funding of R&D as a percentage of GDP that took
place throughout the 1990s in the UK […] while Japan maintained
relatively consistent levels of investment and overtook the UK
in the late 1990s".[13]
Performance indicators inevitably tell us as much about past
decisions as they do about the current situation. We hope that
the areas in which the UK's performance is now suffering due to
previous underinvestment will remind the Government of the importance
of continuing to increase its investment in science, even when
improvements are unlikely be seen in the short term. This is key
to ensuring that the UK does not slip further against the performance
of other countries in future years.
11. In our Scrutiny Report 2003 we expressed concern
that DTI was in the process of agreeing a number of new high-level
measures to assess the UK's performance against PSA target 2,
particularly if the new measures replaced the useful and detailed
measures already being used by Evidence Ltd. The new measures
were published as part of the Science and Innovation Investment
Framework 2004-2014. Some of the Evidence Ltd indicators have
been adopted as targets in the Investment Framework, but many
have not. For example, in measuring outputs, outcomes and productivity,
the Investment Framework focuses on citations, with no measures
for the total number of publications. Whilst the Evidence Ltd
indicators are, on the whole, specific, measurable and comparable
across countries, many of the new measures are unspecific, vague
and more difficult to use in comparisons. For example, the Investment
Framework sets a goal "to improve success rates in [post-16]
SET" with no indication of what would constitute a sufficient
improvement. Similarly, the indicator of progress for public engagement
is set as "evidence of improvement" with no indication
of what evidence would be acceptable.[14]
No explanation is given about the process of selection used by
DTI in the formulation of the new basket of measures, and there
is a danger that some very important detail has been omitted.
For example, the focus on citation rather than publication rates
masks the fact that the UK has been slipping in its global share
of publications in recent years. In addition, citation rates are
not a good measure of research outputs of primarily local importance.
12. The detailed indicators used by Evidence Ltd
in 2003 and 2004 have been extremely useful in assessing the performance
of UK science within an international context. It is not clear
that the new high-level performance measures adopted by the Government
in its Investment Framework will be either as clear or as useful.
We recommend that DTI continues to use the Evidence Ltd indicators
to ensure that it receives, and projects, an accurate message
about the performance of its Science and Engineering Base. We
further recommend that DTI does not change its indicators again
without good reason in order to allow year-on-year comparison.
OST's new performance management
system
13. On 7 December 2004 we held an informal meeting
with Professor Sir Keith O'Nions, the Director General of the
Research Councils, and other officials from OST on the developing
thinking behind the formation of a new performance management
system for the science and engineering Base (SEB). The purpose
of such a system would be to ensure that Research Council investment
maximises the potential of the SEB, particularly in meeting national
economic and public service objectives. We look forward to discussing
the new system with OST more formally once the details have been
announced.
3 HM Treasury, 2004 Spending Review: Public Service
Agreements 2005-08, Cm 6237, July 2004, p 31. Back
4
As above Back
5
DTI, Science Budget 2003-04 to 2005-06 (December 2002), p 9 Back
6
DTI, PSA target metrics for the UK Research Base (October 2003) Back
7
DTI, Autumn Performance Report 2003, Cm 6067, December
2003, para 3.4 Back
8
Ev 54 Back
9
DTI, PSA target metrics for the UK Research Base (October 2004) Back
10
DTI, Autumn Performance Report 2004, Cm 6442, December
2004, p 8 Back
11
HC (2003-04) 316, p 9 Back
12
Q 171 [Lord Sainsbury of Turville] Back
13
Ev 54 Back
14
HM Treasury, DTI and Department for Education and Skills, Science
and Innovation Investment Framework 2004-2014 (July 2004),
pp 165-166 Back
