3 Objectives and performance measures
24. The OSI is responsible for meeting one of the
DTI's Public Service Agreement (PSA) targets as amended in the
Spending Review 2004 settlement. PSA 2 requires the department
Improve the relative international performance
of the UK research base and improve the overall innovation performance
of the UK economy including through effective knowledge transfer
amongst universities, research institutions and business.
The DTI state that as a consequence of the responsibility
of the OSI for "the health and standing of the UK research
base as a whole", the OSI "has a key role in aligning
Science Budget priorities with those of other funders to deliver
two broad outputs that derive directly from this PSA target".
These outputs are:
|Output 1: A healthy UK research base
Aimto maximise the impact (of the Research Councils' and Academies' investment of the Science Budget) on maintaining and improving the UK's Research Base
Output 2: Better exploitation
Aimto increase the contribution made to improving exploitation of the Research Base to meet national economic and public service objectives.
Beneath this are two further levels of outputs aimed at the Science
Budget and then at the individual Research Councils and associated
bodies. Examples of principal outputs to be measured in the case
of output 1 include the number of publications, citations and
PhD students; and for output 2, collaborative research and co-operative
training. The indicators used in connection with these outputs
are derived from the work carried out by Evidence Ltd for the
DTI, which was commended in reports by our predecessor Committee
on OSI scrutiny, who also expressed doubts that the new measures
introduced as part of the Science and Innovation Investment
Framework 2004-2014 would be "as clear or as useful"
(see paragraph 28 below).
The OSI performance management system
25. In the last two years there has been a major change in monitoring
the contribution to the PSA targets made by the Research Councils
and Academies with the development of a new performance management
system. This was introduced by the OSI in order to provide "a
firm, evidence-based foundation" for the long-term planning
demanded by the Ten year Framework for Science and Innovation
and was outlined in the DTI Science Budget Allocations 2005-06
to 2007-08, published in May 2005. It was intended to provide
"a more robust mechanism for translating the strategic priorities
for the research base into specific aims and objectives for the
Research Councils and Academies",
offering the opportunity to "allow the balance of investment
across the Science Budget to be adjusted in response to a more
strategic view of new priorities".
The system involves measuring performance against a number of
indicators derived from the 2004-2014 Framework aim of
making the UK the most attractive place for science and innovation.
It has three elements: Delivery Plans for each Research Council
and for RCUK, setting out the key deliverables in the next three
years; a scorecard recording the deliverables, targets and milestones
to be achieved; and output frameworks which take the outputs and
measure them against "characteristics of interest" (scale,
quality, sustainability, agility, productivity and user focus,
in the case of output 1; scale and quality for output 2). Data
is collected and published on an annual basis.
26. The system has been in operation since 1 April 2005. In September
2006 the OSI published the first Annual Report on Output Frameworks
which aimed "to outline the baseline data, comment on any
particular aspects for future monitoring and summarise the current
Research Council position".
It did this by presenting OSI data in a matrix and by then commenting
on the data provided by the Research Councils. It is notable that
in several cases metrics are still under development or the OSI
has concerns about their usefulness. For examples, metrics are
not yet developed for the rate of change in publication numbers
and in PhDs awarded in relation to identified priorities. In the
case of joint publications between researchers and industry, the
OSI is not collecting the data because research has indicated
that to do so "would involve significant costs and would
not be a reliable indicator of research base-business interactions".
A similar argument is made for indicators for the level of business
confidence in university knowledge transfer activities where the
possibility examined by OSI has left the OSI with "serious
concerns over how meaningful or robust indicators would be"
and with work to do in identifying alternative indicators.
27. The OSI stresses in the introduction to the Annual Report
on Output Frameworks that "the output frameworks are not
static documents, as the metrics are evolving and open to revisions
We accept that it is necessary to adapt the output frameworks
used to measure the performance of the Research Councils as it
comes fully into use in order that it remains a practical and
useful document. We are concerned, however, that the measures
should be firmly established in order to allow comparison from
year to year. There must not be too many changes once the system
has been allowed to bed down. It would be helpful for the OSI
to highlight in the introduction to future output frameworks reports
exactly which metrics have been amended and which are causing
difficulties in terms of data collection or meaningfulness.
The DTI Performance Report
28. In addition to the output frameworks, the OSI's performance
against the PSA target is assessed in the DTI Autumn Performance
Report against 25 indicators across five attributes of science
and innovation: world-class research at the strongest centres
of excellence in the UK; sustainable and financially robust universities
and public research institutes; greater responsiveness of the
research base to the needs of the economy and public services;
increasing business investment in R&D and increased business
engagement; and a more responsive supply of science, technology,
engineering and maths skills to the economy.
A footnote to the Autumn Performance Report 2006 notes that "These
attributes also provide indicators of progress against the measures
set out in the ten-year Science and Innovation Investment Framework".
There is no obvious other indication in either the Performance
Report or the Output Framework document of the relationship between
the two documents. Nor is there a readily-accessible list of targets
and output performance measures on the OSI website. There is
room for improvement in the transparency and presentation of information
about how the performance of the OSI and its associated bodies
is to be assessed across the board and we urge the DTI and OSI
to address this.
29. The overall assessment of progress against SR04 PSA 2 is that
it "remains on course".
Significant challenges, however, are identified in the areas of
"raising business R&D and increasing the supply of science,
engineering and mathematics skills available to the economy".
Both these areas were highlighted in the Science and Innovation
Investment Framework 2004-2014 which established a headline
ambition that public and private investment in R&D should
reach 2.5% of GDP by 2014 and placed great emphasis on measures
to improve the teaching and learning of science, technology, engineering
and mathematics (STEM) subjects at all levels.
30. There are difficulties in measuring both these outputs. For
the indicators used to measure the supply of skills to the economy,
the most recently available data for one of them is for 2003 and
alternative methods of collecting data for the other are being
explored as the dataset stopped in 2002. There is therefore no
effective way of assessing the up-to-date position of the UK in
relation to the supply of skills. The most recent data for business
R&D as a share of GDP is set out in the table below:Table
1: Business R&D as share of GDP
Source: ONS for UK, OECD remainder, Annual Performance
Report 2006, p 15
These figures are worse than expected because they
were revised by the Office of National Statistics "in the
light of new information".
Previous data had suggested that there had been an improvement
in the last two years but the new figures show that business enterprise
research and development expenditure (BERD) has been falling steadily
since 1999, although we note the time lag in these figures of
up to a year which makes it hard to be precise about the current
state of R&D investment and the effect of Government policies.
31. In addition, there are concerns both that the
BERD figures are not capturing the full extent of R&D in the
UK and that they are in any case not sufficient in themselves
to give a rounded picture of innovation. The Minister of Science
suggested that there was too much emphasis in the measurement
on the manufacturing sector and that the R&D target could
be widened to capture R&D conducted in services.
Sir Keith O'Nions argued more generally that BERD was just "a
single target which can only capture a very small part of the
performance across the landscape" and that "there is
a lot more thinking to be done as to what a broader range and
basket of measures should be in relation to productivity and innovation
He elaborated in discussing the effectiveness of knowledge transfer
It is quite easy to measure spin-outs, licences,
start-up companies, the value of start-up companies. It is very
difficult to measure the economic value of knowledge that moves
in trained people and skilled people and mathematicians that go
into the financial sector
This is a much more difficult area
and there is great scope for improvement. The danger is in setting
silly targets which drive behaviour in an adverse way.
32. It is common sense that performance measures
have to be properly focussed on the right attributes in order
that they may show a true picture of the state of innovation in
the UK. The challenge of devising suitable measures can also be
seen in the gaps in the current systems of measurement. There
is clearly no point in having indicators which cannot be measured,
nor outputs which focus on narrow issues to the detriment of significant
ones. Sir Keith O'Nions told us that, whilst there was no formal
Treasury review of output targets, the OSI was trying to address
the recognised difficulties by "looking at a whole number
of possible, sensible performance measures in close discussion
with the Treasury, particularly around that area [of the percentage
of R&D expenditure] whilst reviewing all of them".
He expected this "will most certainly come to a head at the
time of the allocations of the budget, probably next October".
We welcome the work going on to identify meaningful and accessible
output measures for the OSI and the Research Councils. We expect
to receive updates as the work progresses in this area and before
the changes have been finalised.
THE 2.5% TARGET FOR R&D
33. Whatever measures are decided upon to assess
progress, the Government is, so far, not on course to achieve
its 2.5% target for R&D as a percentage of GDP by 2014. This
target is central to the ten year science and innovation strategy
and the Minister for Science told us that "it remains a priority".
Indeed, it is highlighted again in the Budget Report 2007 where
it is described as a "commitment".
While there are other ways in which innovation and productivity
growth might be measured and a single target such as this might
not be sufficient on its own, it would be a highly significant
for the 2.5% target to be missed. In its response to the OSI Scrutiny
2004 report, the Government told our predecessors that raising
R&D to 2.5% "is a challenging target for the UK, but
one which is feasible given the UK's starting point and the strengths
and structures of its science and industrial bases".
Moreover, it was "arrived at following consultation with
business and has been supported by business subsequently as a
helpful guiding ambition to focus public and private investments
in future years."
We should like to be more confident of the likelihood of achieving
this target. We recommend that the OSI, in its response to
this Report, make a clear statement on how it intends to meet
the target of 2.5% as a percentage of GDP within its current strategy.
In addition, although we appreciate the logic of including service
sector R&D within the scope of business R&D, we believe
that the 2.5% target would need to be revisited if the basis of
measurement were changed.
45 Department of Trade and Industry, Science Budget
Allocations 2005-06 to 2007-08, May 2005, p 2 Back
Third Report from the Science and Technology Committee, Session
2004-05, Office of Science and Technology: Scrutiny Report
2004, HC 8, para 12 Back
Office of Science and Innovation, Annual Report on Output Frameworks,
September 2006, p 3 Back
Department of Trade and Industry, Science Budget Allocations
2005-06 to 2007-08, May 2005, ii Back
Office of Science and Innovation, Annual Report on Output Frameworks,
September 2006, p 4 Back
Ibid, p 7 Back
Ibid, p 21 Back
Ibid, p 4 Back
DTI, Autumn Performance Report 2006, December 2006, p 11-18 Back
Ibid, p 11, footnote 11 Back
Ibid, p 11 Back
DTI, Autumn Performance Report 2006, December 2006, p 15 Back
Q 273 Back
Qq 300, 302 Back
Q 293 Back
Q 292 Back
Q 297 Back
HM Treasury, Budget 2007, March 2007, para 1.18 Back
Sixth Special Report from the Science and Technology Committee,
Session 2004-05, Office of Science and Technology: Scrutiny
Report 2004: Government Response to the Committee's Third Report
of Session 2004-05, HC 453, p 2 Back