Examination of Witnesses (Questions 1-19)
SIR ALAN
WILSON AND
PROFESSOR DAVID
EASTWOOD
10 JULY 2006
Q1 Chairman: I think you are both aware
of the great interest there has been since the Chancellor of the
Exchequer made a reference to the winding up of the RAE exercise
and its replacement. There was some surprise that the Chancellor
had made that announcement rather than anybody else, but we will
come to that later. Looking at this Committee's work on the funding
of higher education and who should pay for it two years ago, it
did play some part in the resolution of our discussions over variable
fees and all that, which many people think was a very important
milestone in the development of higher education in our country.
Equally, if we do not get the research side and the funding of
research right in this country, again that has serious implications
for our university system. All this seems to be a bit of a rush.
Somebody said to me at a recent conference that I chaired at the
Royal Society—we seem to have most of the Vice Chancellors
of the country at it—that this all stems from prior to the
Chancellor's statement that three leading vice-chancellors went
to see either the Chancellor or other people in the Treasury and
seemed to persuade them that we needed some changes fast. Sir
Alan, I cannot believe that that was the case, but you know how
these stories emerge. Sir Alan, why is there haste about all this?
It seems all to be in a bit of a hurry.
Sir Alan Wilson: In terms of stories
about three vice-chancellors, that is something I know nothing
of, if it ever took place, so I start from that position.
Q2 Chairman: So the Vice-Chancellors
of Imperial, University College London, and Bristol did not go
and see the Chancellor!
Sir Alan Wilson: Well, nothing
is impossible, Chairman, but I have no knowledge of it. That is
all I am saying. In terms of haste, the notion that it was all
very fast for that kind of reason we would say was not the case,
partly because the document that was eventually produced is a
follow-up to a document that was published in 2004, the original
10-year framework for investment in science. The policies that
were further developed in the budget science paper were really
a continuation of the policies that were announced in July 2004.
From our perspective it has been continuing work. In terms of
the Next Steps paper that was published with the budget—and
in a sense this almost answers the question, "why the Chancellor?"—it
goes back to the 10-year science framework. I think the Chancellor
is anxious, as part of the budget, as I understand it, to have
a comprehensive review of progress since the 2004 paper, and research
was part of that. From our point of view it is an ongoing process,
and we have worked with HEFCE all the way through that period
in terms of looking at possible metrics and performance indicators.
We have talked to Treasury officials and DTI officials. Much of
what was presented in the press about the rush, and certainly
the story about when DfES officials told them are simply not true.
Q3 Chairman: So it is not true that
you were surprised in the DfES! Professor Eastwood, would you
know whether this was greeted with surprise in HEFCE?
Professor Eastwood: I think there
is a parallel story to the one that Sir Alan has just sketched.
After the RAE 2001 the funding councils jointly set up a review
of the RAE methodology under Sir Gareth Roberts; and on the basis
of the Roberts recommendations, the funding councils agreed substantial
changes to the methodology for 2008, including a substantial reliance
on metrics in the 2008 exercise. At nearly the same time the decision
was taken alongside the RAE in 2008 to run a shadow metrics exercise;
that is to say to test in real time an alternative lighter touch
methodology for research assessment. Indeed, work was in hand
within the funding council, and between the funding council and
other bodies, to build that alternative model. So there was a
direction of travel here towards a robust RAE in 2008, on the
basis of what we might broadly call the Roberts methodology; but
alongside that to test and chart a new future for research assessment
in the world beyond 2008. To that extent, what was announced at
the budget and the announcements around the budget were consistent
with that direction of travel.
Q4 Chairman: Professor Eastwood,
was it not the case that at the time of the Gareth Roberts report
something like 80% of institutions expressed approval and satisfaction
with the peer review and the RAE exercise generally. Was that
not the case at the time?
Professor Eastwood: It is certainly
the case that in response to the consultation around Roberts there
was a strong preference within the sector to retain a significant
element of peer review, and that Roberts did; but alongside that
there was the move towards a greater reliance on metrics, and
a sense too that the available metrics would continue to develop
both in terms of range and in terms of reliability as time moved
on. I do not think that the position we are now in is anything
more than an evolution—a substantial evolution perhaps but
nevertheless an evolution of the position we were in around 2002-2003.
That is reflected in the kinds of responses that are beginning
to emerge from the sector. Of course, there was a flurry of excitement
when the consultation document was published, and one would expect
that; but I think there was a serious engagement with the issues
raised within the consultation document, and something close to
a settled view in the sector that the RAE 2008 is very important—it
is very important that we get it right for a whole series of reasons—but
that this would be the last RAE "in the current form".
Q5 Chairman: To be honest about this,
it is all about who gets the money, is it not? Whatever system
you use, it is about who gets the money to conduct the research.
Is it not the fact that the sensitivity is that if you change
the rules you may be taking money away from one set of institutions
or departments and giving them to others? At the heart of this
is there someone in HM Treasury or someone in the higher education
world or someone in the Department for Education and Skills saying,
"the money is going to the wrong people"? Are they saying
that? Have we got to change it?
Professor Eastwood: From the perspective
of the Funding Council, a research assessment does three things.
First, it identifies and assesses research quality, which is central
to the Funding Council's commitment to fund excellence where it
finds it. On the basis of that, and importantly, it has constituted
a very important benchmark for the quality of research within
UK HE. That matters not just in terms of research and research
performance internationally, but also in relation to the branding
of UK higher education. Thirdly, as you say, Chairman, it is an
exercise which underpins the funding allocations that the Funding
Council makes to institutions. We could have a long discussion
about what might constitute the right kind of distribution funding,
but I have heard nothing in the current debate that suggests that
the broad allocation of funding is inappropriate. It needs to
be dynamic and it will shift over time. There are a number of
rather important debates around the funding of interdisciplinary
and multidisciplinary research for example, and also about the
resourcing of applied research. There are some areas where some
concerns have been expressed, and expressed with some force. It
is a large step from that to saying that there is profound dissatisfaction
at the funding allocations that are emerging; on the contrary,
the Next Step document was rather presuming that the funding
allocations would remain broadly as they were.
Q6 Chairman: Sir Alan, is it not
the fact that if there was a discontent about the way research
funding was being allocated—we have seen a dramatic change
over the years in the number of 5-star and 5 departments; the
number has increased very rapidly, and that may because research
is so much better or because academics have learnt to play the
game rather better—or more academics have learnt to play
the game rather better. What is at the heart of this haste? Everybody
knew that there was a change in the system going on. Everybody
knew that in Australia they were moving from a metrics system
and planning to move to a peer-reviewed RAE system—Hong
Kong similarly. There seems to be a movement in the other direction.
We seem to be going into this metrics area against the flow—is
that right?
Sir Alan Wilson: I think it is
against the flow in the sense that we have had twenty years' experience
of the research assessment exercise.
Q7 Chairman: Which everybody thinks
is wonderful, and they are copying us!
Sir Alan Wilson: They think that
what has been achieved in this country since 1986 is very impressive.
It may be that they have to proceed from the equivalent of a time
base that we may have achieved 10 years ago or something of that
kind. Not uncontroversially—and in that sense you must be
right—there has been a reasonable assumption that a new
method of both assessment and allocated funds would be appropriate.
In terms of your original question, there has not been a position
in the DfES or as far as I know anywhere else saying that certain
kinds of universities should get more money and some should get
less, because at the government level it has always been about
policy. Professor Eastwood has indicated some of the current issues
about funding in applied research, funding of interdisciplinary
research; but the principle that the best research will be funded
wherever it is, is something that underlines all of this. The
scale of the exercise—which is why we look for a system
that is less bureaucratic—is considerable. It is not just
simply that the cost is measured—which is substantial but
reasonable in relation to what has been allocated—it is
the time and, in a sense, the way that it dominates the development
of policies in particular institutions. If there is a simpler
way of doing it, there is the possibility, as we said in the consultation
paper, that we might be able to move from what is a fantastic
platform that has been established to being ever more ambitious
in the future.
Q8 Chairman: So you do not think
there is any truth in the assertion that the Chancellor and HM
Treasury might have been saying, "we want more applied research,
more technology transfer, more team working across departments
and across universities; we want to see research much more shaped
towards what increases the wealth of the United Kingdom and much
more practical outcomes". Do you think that has not been
a Treasury view?
Sir Alan Wilson: It may be a Treasury
view in the sense that those concerns have been shared right across
the sector and right across different funding agencies. Even the
research councils, which you might say are primarily there to
support basic research, have had an increasing concern with the
mechanisms with which the results of that basic research are applied.
I think that many would argue that the distance in time in terms
of what used to be called a linear model between basic research
and becoming useful is shrinking. I think it is a policy question
not just for HM Treasury but for all of us—how come the
funding needs of research in the economy as well as blue-skies
research and research in all kinds of public interests are balanced.
At the end of the day it may go back to funding in another sense;
that generous though the research budget is in this country relatively,
there is never enough to sustain what everybody would like to
sustain.
Q9 Mr Wilson: This is not about a
new way of handling research; this is about the Chancellor saving
money, is it not? This is about HM Treasury making a grab for
£45 million in savings.
Sir Alan Wilson: I think that
all the evidence in terms of the Chancellor's support of science
is that he has been committed to increasing budgets rather than
saving money on research. In fact in the last two spending reviews
he has actually ring-fenced greater than average increases for
research funding, and certainly there is no evidence in any discussions
I have had with officials that that situation is changing.
Q10 Mr Wilson: In 2008 we are all
going to see tighter times ahead in education. HM Treasury needs
to save money. They are looking forward to where they are going
to save it, and this is just one of a number of areas that they
are targeting, is it not?
Sir Alan Wilson: I do not want
to anticipate discussions on the comprehensive spending review
that will take place in all departments, and certainly within
the DfES; and it is a matter of political judgment for our ministers
at the end of the day to decide on these relative priorities.
I would say again, Chairman, that I have no evidence that in any
of the government departments that are party to Next Steps,
DfES, HM Treasury, Health and DTI, that anybody wants to do anything
but sustain a strong research base and provide the funding for
it.
Q11 Mr Wilson: As the Chairman earlier
indicated, there are high levels of satisfaction with the current
system, so why does it need to be replaced?
Sir Alan Wilson: In a sense, as
I think I indicated earlier, the big savings for the community
are less in terms of money, wherever the money savings are channelled—and
they could be channelled into further research—it is the
particular way in which it has dominated the time of many academics,
and there is a good possibility, I would judge—and I am
perhaps making a personal comment, Chairman—that it could
increase research productivity.
Professor Eastwood: The direct
costs of RAE 2008 will be of the order of £8 million, the
direct costs to the Funding Council. The £45 million is a
calculation of indirect costs incurred in institutions in preparing
for the RAE. Some of those costs are constant costs, costs associated
with research management and performance management and so forth.
I think there are almost certainly savings to be made here. The
system has, as Sir Alan said, matured over twenty years, and some
of those embedded costs can probably be stripped out. I would
echo what Sir Alan said: that would be a saving that I see being
redirected into the research effort, rather a saving that was
stripped out of the HE budget.
Q12 Mr Wilson: The consultation paper
assumes that the RAE should be discontinued, and it seems to be
on the basis of widely held views, or what people say. Where is
the substance? You yourself said it is time taken by academics.
Where is the substance? Where is the evidence for the supposition
that you are making?
Sir Alan Wilson: There is evidence
that it is possible in principle to run a simpler system because
of the correlations between the data reported in the consultation
paper.
Q13 Mr Wilson: Where do I actually
see that evidence?
Sir Alan Wilson: There are the
models that are on the website, where the reference was given
in the consultation paper. Indeed, in the Next Steps paper
there are two graphs on pages 20 and 21 of chapter 4, which show
in aggregate levels certainly some very high levels of correlation.
In terms of the feasibility of using something like research income
and then other indicators as measures of quality that can then
be used for funding allocations, the evidence is there, and quite
a number of people have believed for a long time that this would
produce a simpler method that would free people for research,
rather than run the process as it has been run. As we are all
agreeing, none of this is without controversy. Any proposals will
be controversial, and in any consultation there will be people
who say, "Keep the RAE; it has worked very well"; and
there will be others who will say, equally strongly, "the
RAE is now in a diminishing returns phase; there is a simpler
way of doing it; please let us do that".
Q14 Mr Wilson: You are right that it
is extremely controversial, and for that reason do you not think
the Government should have made the case for change a lot more
strongly than it has?
Professor Eastwood: Some of the
case was made in Next Steps chapter 4. There is an analysis
there, and the consultation document rather presumed on that and
did not wish to replicate it. The other thing I would say about
the document that was published at the beginning of June is that
it is genuinely a consultation document. It offers some illustrations
in terms of types of models and in terms of funding outcomes of
those models. It is a consultation with the sector, asking the
sector to engage with the issues and with the analysis. I think
that those involved in this process are fully expecting that the
sector will come back, obviously with the critical engagement
of the kind that you are suggesting; but also will come back with
proposals that will take us forward in maintaining the capacity
to assess research quality, to have a sensible framework for distribution
of research funding, but to do so in a lighter-touch way than
has been possible in the last few years.
Q15 Mr Wilson: But it is not a consultation
that keeps the RAE system on the table; it is a consultation on
a metrics basis, is it not? Therefore it is not a consultation
at all.
Professor Eastwood: It is a consultation
that asks a certain set of questions around the STEM subjects
on the one hand, and the arts and humanities and cognate disciplines
on the other; so it does recognise that methodologies for different
disciplines may vary. To that extent it recognises that there
is a varied disciplinary landscape with which research assessment
has to engage. It is also asking whether or not we have the right
sorts of metrics, or whether the sorts of methodologies we are
envisaging will enable us to make appropriate assessments. You
are right that it is not saying, "Do you wish to maintain
the RAE in its current form?" However, it is I think inviting
serious engagement, and it is also inviting respondents to think
about proposals which might further elaborate the sorts of methodologies
that are implicit in that document. A number of institutions,
not least the research-intensive institutions, are beginning to
engage with the consultation in precisely that spirit.
Q16 Mr Wilson: What are you actually
trying to achieve? What is the end game for this metrics-based
system? What is it that you want it to do that the system is not
doing already?
Sir Alan Wilson: It will be less
bureaucratic; it will cost less; it will take less staff time,
but still support excellence; and the correlations that I have
talked about suggest that it can still support excellence. It
encourages ambition, or facilitates ambition, by taking away people
from the formalities of the RAE. It responds to what is becoming
an increasingly important interdisciplinary research agenda. It
connects to the subject that Professor Eastwood talked about,
which is the relative amount of support for applied research or
user-led research. That is very important for all of us. The extent
to which metrics relate to that should, at the end of the day,
be part of the funding formula. That is a policy decision for
our ministers at the end of the day. This approach will facilitate
the examination of alternatives.
Q17 Mr Wilson: I understand it is
less bureaucratic, it costs less and will support quality, but,
as the Chairman said, in Australia that is not what it achieved.
Lots of research papers have been produced, but the overall quality
diminished, which is why they are switching systems.
Professor Eastwood: We have made
very clear in the documents that we need to have a methodology
that is capable of robust assessment in the manner of the RAE.
We also commit ourselves in the consultation document to model
the likely effects of any change in the system; and indeed that
is one of the things we are consulting on as well. We are aware
that changes in the assessment methodology and changes in the
funding methodology will lead to changes in behaviour. Some of
those we might wish to drive; some of them we might wish to inhibit.
We are aware of the issues that you raise. In modelling the shadow
exercise for 2008, and then in evaluating its likely effects,
those kinds of questions will be to the fore.
Q18 Mr Wilson: I have two questions
in one, coming back to the evidence: what evidence do you have
that the metrics basis is more efficient than the RAE system;
and what evidence do you have that the metrics basis is less expensive?
Professor Eastwood: The evidence
for the latter, that it is less expensive, is that it will enable
us to run the RAE without the current elaborate infrastructure
of panels. We think it will also mean that the amount of preparation
that institutions do for an RAE will be diminished. It will not
be eliminated but—
Q19 Mr Wilson: Do you have any estimates
of the savings?
Professor Eastwood: No, we have
not, but then until we have devised the model we will not be in
a position to—
| 
