Performance measurement
54. If the Research Councils are to be successful
in supporting knowledge transfer, then it is important that regular
performance evaluation is undertaken to determine areas of accomplishment
and for improvement. The publication, by Research Councils, of
assessment metrics frameworks[121]
alongside their Delivery Plans is, therefore, strongly welcomed
by the Committee. The assessment metrics are developed from the
"Outputs Framework" contained within the Public Service
Agreement Targets set down by the DTI alongside the Science Budget.[122]
The Research Council assessment metrics are a series of targets
and milestones arising from the activities set out in each Councils
Delivery Plan. According to RCUK, it is intended that data relating
to the Outputs Framework will be published at the end of each
financial year, commencing with 2005/06 (the first reports should
be made available in June 2006) and that progress against each
Research Councils' Scorecard will be updated quarterly.[123]
The assessment framework consists of Output 1 in which metrics
for development of 'A healthy UK science and engineering base'
are detailed and Output 2 which details factors required for 'Better
exploitation'.[124]
The objectives listed within the Output 2 framework vary between
the Councils since, Professor Diamond told us, it was "agreed
that there should be a matrix of metrics which reflected different
aspects of the knowledge transfer agenda and that each Research
Council would have individual metrics within that matrix which
reflected their own activities and that recognised the breadth
of activity".[125]
We were particularly interested to determine how the Councils
intended to use data from the metrics and were encouraged to hear
from Professor Diamond that "we will be using this to monitor
the directions that we are going in a set of activities, we
will be using that to fine-tune and change some of our funding
areas in terms of knowledge transfer".[126]
55. Whilst we welcome the publication of measures
to evaluate success in knowledge transfer, we are concerned about
the appropriateness of the actual metrics used. Momenta told us
that "what is being measured is largely activity rather
than outcomes"[127]
and we were also heard from Professor Snowden that he does "not
think the metrics today are transparent enough to us all, as a
starting point" and that they seem to be "relatively
short term".[128]
We were also concerned to receive views from QinetiQ that "those
[metrics] considered, such as numbers of industry collaborations,
patents and start-up companies, do not reflect accurately the
value of interactions to users and may distort adversely the behavior
of some research teams".[129]
In response to apprehension regarding the effectiveness of the
metrics for evaluating performance in knowledge transfer, Professor
O'Reilly acknowledged that development of Output 2 metrics had
been difficult since it "is new to try and get metrics on
this".[130] Professor
O' Reilly also indicated that the Councils had sought to "get
things that were measurable in the qualitative as well as the
quantitative sense".[131]
There is always a danger that metrics affect behaviour in unpredictable
ways and that they may become a driver of activity rather than
a measurement of success. This has been a recognised criticism
of the Research Assessment Exercise, which has not adequately
considered industrial collaboration and hence discouraged knowledge
transfer.[132] We
welcome the publication of Research Council performance assessment
metrics but consider that refinement is required. We are particularly
concerned that the Output 2 metrics, as they stand at present,
measure activity rather than output and that they may influence
the activities of the research community. We recommend that the
Research Councils and RCUK regularly review the assessment metrics
and the impact they are having, reporting back periodically.
Cross-Council co-ordination
56. Within their Delivery Plan, RCUK state that they
will "co-ordinate and harmonise increased engagement in innovation
and knowledge transfer by the Research Councils".[133]
RCUK also told us that "whilst having varied academic and
user bases, the eight Research Councils work together where appropriate
across a range of knowledge transfer activities, through the RCUK
Knowledge Transfer Group (KTG)" an assembly which "provides
a focal point for sharing information and good practice on knowledge
transfer and a hub for collective dialogue with external organisations".[134]
57. Despite interaction of the Research Councils
through the KTG, there has been much criticism with respect to
how well the Councils co-ordinate their knowledge transfer activities,
for example by the 1994 Group who told us "Knowledge transfer
is an area of increasing interest and activity on the part of
the Research Councils, but there is considerable variation between
them in effort, approach and success".[135]
We also heard calls for a stronger role for RCUK in this area,
for example, from the CBI who stated that "there is a very
strong need for RCUK to identify and disseminate best practice
across the Research Councils to ensure that they are operating
with optimum efficiency and having the greatest possible impact,
collectively and individually".[136]
We also heard that, currently, there appears to be little added
value from RCUK. For example, AstraZeneca told us that "whilst
the mission statement of RCUK is commendable, it is not clear
to us what additional value RCUK has delivered to the effectiveness
of the research, training and KT activities of the eight RCs [Research
Councils]"[137]
and QinetiQ commented that there is "an obvious need for
co-ordination amongst the various funding bodies but [that] the
role of the RCUK umbrella body is not particularly visible".[138]
58. The Research Councils were unwilling to accept
that there is a need for increased effort toward sharing of best
practice and general co-ordination of their knowledge transfer
activities. Professor Diamond said that this "is precisely
what the knowledge transfer group does" and that it is on
an "upward trajectory".[139]
However, with the exception of forming the BBSRC Business Plan
Competition into a cross-Council scheme, we have found little
evidence for activity by the KTG and we were disappointed that
the Research Councils seemed unable to provide us with examples
of benefits gained through either the KTG or RCUK in respect of
co-ordination for knowledge transfer.[140]
59. Whilst the RCUK view that different schemes
are required to benefit individual Councils' respective communities
may be valid,[141]
there are clearly overlapping areas and advantage could be gained
from more effective sharing of best practice and co-ordination
where appropriate. For example, perhaps a cross-Council approach
toward the employment of external brokers to help identify and
develop potential knowledge transfer opportunities could be taken.
We have found little evidence
of Research Council co-ordination or sharing of best practice
in the context of their knowledge transfer activities and we have
not been persuaded that the Knowledge Transfer Group has achieved
much in the two years since its formation. Also, despite their
clear remit to co-ordinate and harmonise, we have not seen any
added value from RCUK in this area. We urge the Research Councils
and RCUK to take the necessary steps to enhance the effectiveness
of their co-ordination in knowledge transfer.
63 Ev 121 Back
64
Ev 111 Back
65
Q 1 Back
66
Ev 55 Back
67
Q 201 Back
68
Q 201 Back
69
HM Treasury, Department of Trade and Industry and Department
for Education and Skills, Science and innovation investment
framework 2004-2014, July 2005, chapter 5, para 5.35 Back
70
Richard Lambert, Lambert Review of Business-University Collaboration,
December 2003, chapter 1, para 1,24 Back
71
Ev 130 Back
72
Ev 125 Back
73
External Challenge Panel, Independent External Challenge Report
to Research Councils UK, "Knowledge Transfer in the Eight
Research Councils", April 2006, pp 2-3 (overarching finding) Back
74
Q 272 Back
75
Research Councils UK: Delivery Plan, www.rcuk.ac.uk/press/20050526deliveryplan.asp Back
76
As above Back
77
EPSRC, EPSRC Delivery Plan, www.epsrc.ac.uk/CMSWeb/Downloads/Publications/Corporate/DeliveryPlan.doc
Back
78
External Challenge Panel, Independent External Challenge Report
to Research Councils UK, "Knowledge Transfer in the Eight
Research Councils", April 2006, para 4.2 Back
79
Q 109 Back
80
Ev 166 Back
81
Q 212 Back
82
Ev 57 Back
83
Richard Lambert, Lambert Review of Business-University Collaboration,
December 2003, chapter 6, para 6.33 Back
84
Ev 130 Back
85
Ev 135 Back
86
Ev 114 Back
87
External Challenge Panel, Independent External Challenge Report
to Research Councils UK, "Knowledge Transfer in the Eight
Research Councils", April 2006 p 3 (key findings) Back
88
Ev 166 Back
89
Q 272 Back
90
Q 271 Back
91
Ev 127 Back
92
Q 177 Back
93
Q 138 Back
94
Q 273. Back
95
HM Treasury, Department of Trade and Industry and Department
for Education and Skills, Department of Health, Science and
innovation investment framework 2004-2014: next steps, March
2006, chapter 3, box 3.1. Back
96
Ev 127 Back
97
Ev 122 Back
98
Ev 105 Back
99
Q 277 Back
100
Q 217 Back
101
Ev 100 Back
102
Ev 108 Back
103
Ev 114 Back
104
Q 67 Back
105
Ev 136 Back
106
Q 223 Back
107
As above Back
108
Ev 132 Back
109
Ev 100 Back
110
As above Back
111
Ev 135 Back
112
Ev 127 Back
113
Q 226 Back
114
Q 19 Back
115
Q 181 Back
116
Q 182 Back
117
Ev 135 Back
118
Q 226 Back
119
As above Back
120
As above Back
121
RCUK, Research Council Delivery Plans and Scorecards,
RCUK, www.rcuk.ac.uk/deliveryplan.asp Back
122
DTI, Science Budget Allocations 2005-06 to 2007-08,p 2 Back
123
RCUK, Research Councils Publish Delivery Plans, press
release, 26 May 2005, www.rcuk.ac.uk/press/20050526deliveryplan.asp Back
124
RCUK, Research Council Delivery Plans and Scorecards,,
www.rcuk.ac.uk/deliveryplan.asp Back
125
Q 262 Back
126
Q 267 Back
127
Ev 124 Back
128
Q 80 Back
129
Ev 137 Back
130
Q 264 Back
131
Q 265 Back
132
Q 84 Back
133
RCUK, RCUK Delivery Plan, www.rcuk.ac.uk/documents/deliveryplan.pdf,
p 19 Back
134
Ev 62 Back
135
Ev 128 Back
136
Ev 135 Back
137
Ev 104 Back
138
Ev 138 Back
139
Q 253 Back
140
Qq 252-54 Back
141
Ev 55 Back