Examination of Witnesses (Questions 20-39)
SIR ALAN
WILSON AND
PROFESSOR DAVID
EASTWOOD
10 JULY 2006
Q20 Mr Wilson: It is purely speculation
that it will be less expensive because you have cut some areas!
Sir Alan Wilson: Intuition takes
us quite a long way. I agree with Professor Eastwood that calculations
will eventually be done, but the present RAE in place involves
something like 82 different panels or committees, with quite substantial
numbers of members on each, and in a sense there are their own
administrative demands to go with it. That, alone, is taking quite
a lot of academic staff time because it is the academics that
are staffing these 82 panels. We are confident that there are
gains. What I still feel is just as important as any monetary
gains which might be reinvested are the gains in time. The academics
that are on these panels are often at the height of their research
careers, and it will simply free up some of their time to go back
to the fundamentals of research. It is very difficult to measure
what the value of that would be.
Q21 Mr Marsden: Sir Alan, you know
very well that there have been searching criticisms of the perverse
incentives that the RAE has produced, not least by our predecessor
committee in 2001 and by the Science and Technology Committee,
and indeed across the system. Given that this is the case, why
have you not taken the opportunity of this revaluation to do a
thorough analysis of the existing behavioural impact of the RAE
to inform any successor system?
Sir Alan Wilson: I think the behavioural
impacts have been reasonably well understood. The analysis of
this kind of thing is never perfect, but it is about the transfer
market.
Q22 Mr Marsden: Would you like to
tell us what they are?
Sir Alan Wilson: The main one
which is cited is, as it were, the transfer market in academics;
the possibility that researchers who might have put in a certain
amount of time in teaching are actually negotiating jobs for themselves
that are wholly research jobs; and that may or may not be a good
thing. It is very difficult to judge in general whether some of
the consequences of the existing system are good or bad. You might
argue that increased mobility between universities is a good thing
in many ways. I think the general feeling is that overall, whatever
the positive sides, the transfer market is too intense at the
early stages of an RAE period. To my mind, one of the most important
things, which, again, is terribly difficult to measure, is that
the RAEs almost have to be organised in terms of traditional disciplines.
The Funding Council has put an enormous amount of effort in trying
to add a structure which deals with interdisciplinary research;
but we are almost certainly moving into an era where in science
subjects a large proportion of the research undertaken will be
interdisciplinary.
Q23 Mr Marsden: I would agree, certainly
from my own soundings of academics and my own knowledge of the
area, with all the criticisms of the existing RAE that you have
enunciated. The point about the transfer issue is particularly
relevant in humanities and particularly with younger academics.
However, none of that answers the question as to why that case
is not made systematically here, and how a metrics system will
improve on that. The criticisms that have so far surfaced have
not been so much that the RAE is absolutely wonderful and should
not be touched; it is that the RAE has significant flaws, which
you yourself have underlined today; but there is nothing in what
is on the table at the moment from you to suggest that those significant
flaws will be addressed by a metrics system, is there?
Sir Alan Wilson: What we have
said in the consultation paper is that we are certainly aware
of the consequence of the RAE as it exists in terms of perverse
incentives, and any system—because as the Chairman said
at the outset, it is connected to funding—is bound to have
incentives relating to funding. We have identified in our own
minds some of the dangers of the new system, an obvious one being
that if Research Council income plays a major part in the funding
formula, then research councils could be deluged with a much larger
number of applications for research.
Q24 Mr Marsden: The fact still remains
that what is on the table at the moment—which I accept is
consultation—is a leap from a system which has fundamental
flaws, and criticisms have been made of it, to a system that has
not been tested, where we have no evidence—this is all intuition
or assumption—that it will not make matters worse.
Sir Alan Wilson: In terms of the
untested nature of it, quite a lot of work has been done on correlations
and so on, as I mentioned earlier. One of the reasons why it is
very important, and something that has been a plank of policy
for two years or more—there should be a shadow exercise
alongside the 2008 RAE—planned in the light of a consultation
which is now on the table, will produce the tests, in a sense.
Q25 Mr Marsden: Are you saying it
is a "suck it and see" exercise?
Sir Alan Wilson: I hope the consultation
and the models that have been published with it make clear that
while there is a reasonably general view among many people that
a metrics system can replace the existing RAE, the tests that
have been carried out so far show that it is a non-trivial exercise.
There are a lot of these correlations and a lot of finer points
that have to be dealt with. What we are trying to do in the consultation
paper is to expose these so that the community can help us make
progress. Part of the argument about haste, which understandably
we keep coming back to, Chairman, is that to get this apparatus
in place following a consultation, to have a good shadow exercise,
which will be Mr Marsden's test in 2008, means that we are already
on a relatively tight timetable. I would be reasonably confident
that the consultation would certainly throw up something that
we have not thought of at this stage, but will get us in a position
by around the turn of the year where the Funding Council and Professor
Eastwood can start to plan for the shadow exercise in 2008.
Professor Eastwood: If we had
been proceeding in a different sort of way, then I think it would
have been a "suck it and see" approach, and of course
that would have been wholly inappropriate; but to elaborate what
Sir Alan has been saying, 2008 will run. It will run as a robust
exercise, and it will run on the basis of what I describe as the
Roberts revision, and it will be taken forward by the funding
councils. Alongside that we will then run the shadow exercise.
We will evaluate that shadow exercise in the context of RAE 2008.
That will give us the opportunity, on the basis of that real-time
test of the alternative, to make some modifications and do some
re-engineering, if that is appropriate. That is before it is run
in earnest. That is an orderly process. At the moment, to construct
the model that will be run in shadow form, then to run that shadow
form to obtain an appropriate evaluation; then to make the amendments
which seem appropriate prior to running the new model—as
Sir Alan says, that process, while it might look as if we are
moving swiftly now, is a process that we were embarked upon already
in the ways I was suggesting earlier on, with work going on behind
the scenes. It is coming into the full glare of publicity, but
it was actually going on behind the scenes.
Q26 Mr Marsden: Given all of that,
how is this going to address the issue, which is often raised,
that the RAE exercise, and for that matter the metrics exercise,
will merely entrench rewarding research in universities that have
excellence in them at the moment, and not encourage it in others;
that you are going to create a two-tier university research system?
Professor Eastwood: That is where
the capacity of the existing methodology and perhaps an enhanced
capacity in the future—a methodology to identify excellence—is
important. The distribution of QR, the form of funding that comes
from the RAE, is broader across the sector and broader across
institutions than the distribution of any other form of research
funding; and there is nothing in what we are proposing which would
disturb that.
Stephen Williams: We have heard the Roberts
review mentioned a few times. That was in 2003, and all universities
are currently, so they have told me, in advanced stages of preparation
for the 2008 RAE. Sir Alan, why has the Government decided that
now is the right time to drop the existing RAE system?
Q27 Chairman: Stephen is a Bristol
Member!
Sir Alan Wilson: The Vice-Chancellor
of Bristol University has had well-known views on the RAE, much
quoted! Having said that, I have now lost track of the question!
Q28 Stephen Williams: Why now?
Sir Alan Wilson: I was going to
say that, as I indicated earlier, it is not so much now. It goes
back to the 2004 10-year science investment paper that was published
under the logos of three government departments, and it was assumed
then that there was a strong presumption that something like a
metrics basis would take over after the 2008 RAE. It is not that
this has just been thought of in a very short space of time; it
goes back at least two years. In terms of earlier reviews, they
are ideas that have been around for a long time. The particular
issue now is really what I indicated earlier about trying to get
a good shadow exercise in place by the time of the 2008 RAE.
Q29 Stephen Williams: So when all
these reports that we would have read after the budget speech
back in March, after the Chancellor's announcement about people
expressing surprise—all those people should have seen it
coming in 2004.
Sir Alan Wilson: I am sure they
did see it coming in 2004, Chairman; it was very clearly there.
I think inevitably when there is a publication it brings it to
the top of people's minds and potentially affects funding, and
people start to think about their future strategies. I think that
that is very understandable.
Q30 Stephen Williams: Sir Alan, did
your Department request this review, or was it acceded to a request
from another department?
Sir Alan Wilson: I do not think
anybody has been acceded to anybody else; it has been a process
that has been in place since the 2004 10-year framework. We jointly
publish annual reviews of that. We published one, and I think
the second one is about to be published. I think it was convenient,
and important indeed to use the framework in the budget to draw
a number of threads from that 10-year framework together in 2006,
and research clearly had to be part of that.
Q31 Stephen Williams: Chairman, there
is some suspicion, as both witnesses will have heard from earlier
questions, about which part of the Government is driving the process;
whether it is HM Treasury, which has the primary role, or the
DfES. If you look at the timetable, it is all built in from the
stop point, which appears to be the Chancellor's pre-budget report,
which traditionally is in November in this country, feeding back
to the end of consultation at the end of October. The consultation
has just started. Has this whole process not been truncated just
so that the Chancellor can have the answer he wants by the time
he comes to his pre-budget report in November?
Sir Alan Wilson: I do not think
it is about the Chancellor's timing, Chairman. The fact that Professor
Eastwood and I are sitting here before you now as co-chairs of
the group that produced the consultation paper is a demonstration
that DfES—and in its relationship with the Funding Council—is
one of the lead partners in this exercise. We have certainly had
representatives of HM Treasury and the DTI on our consultation
group, and indeed the other funding councils. What has come out
has been agreed by all parties.
Q32 Stephen Williams: Is it the DfES's
consultation rather than jointly with the Higher Education Funding
Council?
Sir Alan Wilson: It is essentially
joint. Everyone has an interest in it. All the Ministers of the
relevant departments have an interest in the policy developments.
We have indicated areas like applied research and so on, which
will be major policy questions in years to come. The implementation
of the outcome of this consultation, the shadow exercise, will
be the responsibility of the Funding Council. I think that is
why certainly from the DfES point of view it is very important
that Professor Eastwood is closely associated with this.
Q33 Stephen Williams: Professor Eastwood,
your predecessor from your new forthcoming role, Sir Howard Newby,
has said that policy goals should come first rather than the detailed
consultation on the types of metrics; that we need a review looking
at what the UK needs from higher education research. He said that
running a metrics review is no substitute for a higher education
policy review. Do you agree with that?
Professor Eastwood: I think the
expectations of higher education in this sphere are well known
and well articulated. The 10-year framework is one articulation
of that; there was also an articulation of that in the HE White
Paper back in 2003. I think that higher education is well aware
of its responsibility to maintain the UK's position as number
two in the international league table for research performance.
I think the emphasis on translation of research is one that universities
are well aware of, and indeed sympathetic to. The importance of
maintaining funding for blue-skies research is right at the heart
of the dual-support system. In that sense, the key policy drivers
are very clearly articulated.
Q34 Chairman: But are they, Professor
Eastwood? We have still been trying to get to the heart of the
policy drivers here. Is it saving money; is it better allocation
of research resources across our country; is it international
competitiveness? I am still not sure what is at the heart of the
policy drivers.
Professor Eastwood: I think it
is all of those, and it is also having an assessment methodology
and a funding methodology that is fit for purpose for the next
10 or 15 years.
Q35 Chairman: It takes senior academics
out of the loop!
Professor Eastwood: I do not think,
with respect, that that is what is being proposed.
Q36 Chairman: Sir Alan was looking
quite pleased; that all these academics are going to be released
to do more research and not going to be involved in the loop.
Professor Eastwood: Sir Alan can
speak for himself of course, but there is a distinction between
diminishing the weight of peer review, which is there in the current
methodology, which is what Sir Alan was talking about I think,
and maintaining the confidence of the sector in the system of
assessment of funding that we evolve. That confidence will be
there partly because the methodology will have been demonstrated
to be fit for purpose, and partly because peer review does sit
behind a number of the income metrics that we will be using. Research
councils use very heavy peer-review for example. There is a suggestion
in the consultation document that there be an overall assessment
of institutions' research plans and research directions. I think
that in different ways there will be those kinds of inputs. The
issue around metrics is not about a move towards an utterly desiccated
system, but a move towards using some proxy indicators for quality.
Q37 Stephen Williams: Amongst Professor
Eastwood's list of reasons for doing it was to maintain the UK's
competitive position as number two in the world after the US in
world research rankings. Is not one of the advantages of the current
system that a vice-chancellor, whether from UEA or Bristol, can
go to China or somewhere else in the world and say, "I have
got this 5-star department" in whatever it is, and everywhere
in the world people will recognise that? They will also understand
the system by which you have arrived at that assessment. Are you
not at risk of jeopardising that world-wide confidence in what
we have at the moment?
Professor Eastwood: I absolutely
share with you a sense of the importance of that ability to badge
quality in the UK. There is no doubt about that, and I tried to
refer to that in some of my earlier remarks. I would just make
two supplementary points. One is that RAE 2008 will articulate
quality in a different way, and so will not have 5 and 5-stars;
it will have a graded profile. That was one of the Roberts responses
to a previous criticism of the RAE and in particular the funding
that flowed from it. That is to say it has a kind of cliff edge,
and if you were 5, 5-star and then 4, there was a big fall; so
the new methodology is designed to smooth that. There will be
a different language of excellence emerging from RAE 2008. I do
think it is vital, and the consultation believes it is vital that
the new system should be able to identify, to describe and to
badge research quality, for precisely the reasons you have given.
Sir Alan Wilson: It is interesting
in the context of this discussion that the analysis that establish
this country as number two in the league table and second only
to the United States is essentially a bibliometric analysis, an
analysis of papers published in different kinds of journals, and
then the number of citations—in other words, the extent
to which this research has actually been used. That has been one
of the main measures that has established the league table in
which the communities have some confidence. The problem with metrics
is that they do not cover the full range of subjects; but it is
a field that is developing very rapidly, and this kind of international
benchmarking—I think we are beginning to be reasonably confident—can
be addressed through the use of those kinds of metrics. We are
not just talking about research income; we are talking about available
measures on a global basis of the value of research, as perceived
through citations of published work.
Chairman: Do you want to drill down on
this metrics system—David?
Q38 Mr Chaytor: Why are the Australians
abandoning the system that we are now moving towards adopting?
Sir Alan Wilson: In trying to
answer the question, I cannot claim familiarity with the Australian
metrics system—perhaps I ought to be able to! My guess is
that the position that we have reached in this country has needed
20 years of the research assessment exercise to produce a foundation
and move the university system in this country from where it was
in 1986 in research terms to where it is now. I have no doubt
that in that sense it has been very valuable. However, if I can
make another personal statement, I believe that diminishing returns
have set in, and there is a danger that particularly the failure
to address the interdisciplinary agenda and to take on the biggest
challenges because there is risk—and if you want to make
sure you have your four publications for an RAE panel, you perhaps
take less risk than at least we would want some of the community
to take on.
Q39 Mr Chaytor: What happens in the
United States, or is the system so different that comparisons
cannot be drawn?
Sir Alan Wilson: It is essentially
very different. You have a variety of funders. You have a different
system for contributing overheads from government funding in particular.
By that I mean the research funding through National Science Foundation
and National Institute of Health and so on. It is not easy to
draw comparisons.
| 
