4 Research funding
36. In the Budget in March, the Chancellor made the
unexpected announcement that the Research Assessment Exercise
for evaluating the quality of research undertaken at universities,
the results of which are used by the Higher Education Funding
Councils and by Government to decide on the allocation of funding
for research, would be ended and replaced with a system based
on the use of existing data, described as "metrics".
A consultation document was published on 13 June which, rather
than discussing the merits of such a change, deals principally
with technical issues to do with the introduction and implementation
of metrics.
37. In very broad terms, the proposals are to replace
the expert peer review system, based on panels of experts assessing
the quality of research, with a system that uses metrics. Metrics
could be used in a number of different ways:
a) metrics as quantitative indicators of volume,
value or quality;
b) metrics-based funding: a funding system in
which metrics directly determine the distribution of funds; and
c) metrics-based quality assessment: a system
for identifying high quality research based wholly or primarily
on metrics.
38. The Government has not been clear about which
of these it is seeking to achieve. What is clear is that it has
decided that it wants to move to some sort of "metrics"
system. This can be seen from the fact that the paper is not consulting
on whether or not it is appropriate to move to a metrics system;
it is taken as a given in the consultation document that there
will be a move away from the use of bespoke peer review towards
a system which makes more use of quantitative indicators.
39. This must mean more than just asking RAE panels
to look at metrics: panels already consider those relating to
income and volume. The most significant difference between current
practice and what is being proposed is that, at present, the panels
rely on peer review to determine the quality of output and give
that the greatest weight.
40. There are, essentially, three options:
i. metrics based exclusively upon levels of external
research income achieved by universities (at the moment, no-one
is proposing using just research council income: charities and
other funding would also count though income may be weighted);
ii. a basket of indicators including output metrics
(almost certainly based on citations or the impact of journal
articles), volume metrics (number of PhDs, number of staff etc.)
and funding metrics; and
iii. some combination of the above with an expert
review element (either just as a check on the results, or possibly
something more substantial).
41. We took evidence on this subject from Sir Alan
Wilson, then Director General, Higher Education at the DfES and
Professor David Eastwood, Vice-Chancellor of the University of
East Anglia and Chief Executive designate of the Higher Education
Funding Council for England, who jointly chaired the group that
drafted the consultation paper published in June. On the question
of the speed with which the change appeared to be being made,
Sir Alan said:
"[...] the notion that it was all very fast
[…] we would say was not the case, partly because the document
that was eventually produced is a follow-up to a document that
was published in 2004, the original 10-year framework for investment
in science. The policies that were further developed in the budget
science paper were really a continuation of the policies that
were announced in July 2004. From our perspective it has been
continuing work. In terms of the Next Steps paper that
was published with the budget—and in a sense this almost
answers the question, 'why the Chancellor?[made the announcement]'—it
goes back to the 10-year science framework. I think the Chancellor
is anxious, as part of the budget, as I understand it, to have
a comprehensive review of progress since the 2004 paper, and research
was part of that. From our point of view it is an ongoing process,
and we have worked with HEFCE all the way through that period
in terms of looking at possible metrics and performance indicators."[27]
42. Professor Eastwood added:
"After the RAE 2001 the funding councils jointly
set up a review of the RAE methodology under Sir Gareth Roberts;
and on the basis of the Roberts recommendations, the funding councils
agreed substantial changes to the methodology for 2008, including
a substantial reliance on metrics in the 2008 exercise. At nearly
the same time the decision was taken alongside the RAE in 2008
to run a shadow metrics exercise; that is to say to test in real
time an alternative lighter touch methodology for research assessment.
Indeed, work was in hand within the funding council, and between
the funding council and other bodies, to build that alternative
model. So there was a direction of travel here towards a robust
RAE in 2008, on the basis of what we might broadly call the Roberts
methodology; but alongside that to test and chart a new future
for research assessment in the world beyond 2008. To that extent,
what was announced at the budget and the announcements around
the budget were consistent with that direction of travel."[28]
43. Professor Eastwood accepted that in consultation
on the Roberts review the vast majority of respondents expressed
approval of the RAE in general and peer review in particular,
but he argued that "alongside that there was the move towards
a greater reliance on metrics, and a sense too that the available
metrics would continue to develop both in terms of range and in
terms of reliability as time moved on."[29]
44. Despite complaints over time about the onerous
burden that the RAE places on university departments, the suggested
move to metrics has not been widely welcomed. For example, a paper
from the Higher Education Policy Institute (HEPI) says of the
consultation paper that "it contains no analysis of the problems
associated with the RAE or the metrics alternatives, and provides
no basis for policy decisions. It asks which of 5 metrics-based
models is preferred but […] all suffer from similar flaws
and there is no basis in the document for making a judgement between
metrics and peer review."[30]
45. A working
group set up by the Russell Group of research intensive universities
has concluded that the Russell Group "could not support
any of the possible models set out in the Consultation Document."
It strongly recommends:
"[…] that consideration to devise
an approach based more on metrics begins afresh, based more on
the approach set out in Annex 2 of the Consultation Document as
adjusted to reflect the broad direction set out in this response.
We recognise and indeed would wish to emphasise the considerable
work that will be necessary to develop such a framework and its
subject group variations. There are difficulties of definition
and of measurement, of avoiding perverse incentives and undesirable
outcomes and of ensuring auditability, probity and rigour. There
are also issues of devising new and better means within such modelling
of making initial QR allocations across the broad subject groupings
we propose, which no doubt will need to be based in part on some
appropriate proxy for volume and in part on the better cost information
that should become available from TRAC data sources."[31]
46. Concern has also been expressed that it will
not be possible to assess research across all disciplines in the
same way. Metrics-systems are much more appropriate for science,
technology, engineering, and maths (STEM)-based subjects than
they are for the arts and humanities. As the consultation paper
itself acknowledges, arts and humanities subjects will need a
different methodology.
47. When we discussed these issues with the Secretary
of State he told us that he was a "fan" of metrics and
that he had been "amazed" when he had been Minister
for Higher Education "that we spent all this money and took
up all this time—something like 82 different panels and
committees—on something that could be done much more quickly."[32]
He pointed out that the proposal was to run a system of metrics
alongside the 2008 RAE:
"I would want to see these two systems running
side by side. I would want to see the correlation and the outcome.
That is the best way to do this rather than to leap from RAE
to a metrics based system, to see how they operate".[33]
He also denied that the intention was to concentrate
research funding on a few research intensive universities.[34]
48. This last point is in many ways the vital issue.
Whatever the method used to assess research quality, the crucial
decision is the extent to which funding is provided for different
levels of quality. For example, under the current system, there
is no HEFCE funding for Departments graded below level 4 (5* is
the top grade), and funding for departments graded at level 4
has been reduced since 2003. A key question that needs to be answered,
therefore, is what does the Government want to achieve through
its research funding? As Sir Howard Newby, who until recently
was Chief Executive of HEFCE, has said, the order of any discussion
must be to clarify policy goals first, then consider the RAE,
and then consider metrics—not to begin with a consultation
on metrics which will impact on the RAE which will impact on research
policy.[35]
49. The Chancellor's announcement and the publication
of the consultation paper have given rise to a great deal of controversy.
Our main concern is that no significant decision should be taken
without a full debate on the substantive issues of what we want
the research funding system to achieve and whether it is appropriate
to replace the RAE with metrics (and if so, what type), or without
a proper evaluation of the evidence from the dual running of the
2008 RAE alongside a system of metrics. Any replacement system
will only be relevant for funding decisions after the 2008 exercise
has been completed. There is therefore sufficient time to weigh
the evidence before a new system is put in place. We
are planning a wide-ranging inquiry into a number of issues concerning
higher education in the next parliamentary session, and research
funding is one of the subjects that we shall be investigating.
We expect the Government not to take any irrevocable decisions
on the next steps until we have reported our findings.
27 Uncorrected transcript of oral evidence taken before
the Education and Skills Committee on 10 July 2006, HC (2005-06)
1405-i, Q 2. Back
28
Ibid, Q 3 Back
29
Ibid, Q 4 Back
30
Higher Education Policy Institute, Using metrics to allocate
research funds: initial response to the Government's consultation
proposals, June 2006. Back
31
Russell Group response
to the consultation document Reform of Higher Education Research
Assessment and Funding, September 2006, paras 6.1 and 6.3,
http://www.russellgroup.ac.uk/2006/research_reform_response.htm Back
32
Q 190 Back
33
Q 191 Back
34
Qq 196-199 Back
35
Professor Sir Howard Newby, Vice Chancellor of the University
of the West of England and former Chief Executive of HEFCE speaking
at HEPI's RAE Conference on 21 June 2006. Back
| 
