Examination of Witnesses (Questions 40-59)
SIR ALAN
WILSON AND
PROFESSOR DAVID
EASTWOOD
10 JULY 2006
Q40 Mr Chaytor: On the arguments
in favour of the metrics approach, you have cited costs, saving
of time and diminishing returns and the changing nature of research
and the growth of interdisciplinary research—but how exactly
will some of the metrics criteria listed in the consultation document
respond to the interdisciplinary issue more effectively? For example,
how will the bibliometric approach reward interdisciplinary studies
more effectively?
Sir Alan Wilson: Can I add to
that research income of various kinds, and take the two kinds
of indicators? In relation to research income, research councils
are increasingly funding interdisciplinary projects and those
defined on an interdisciplinary basis rather than on a traditional
subject basis. That will be reflected in research income so that
universities that are more successful in winning funding for interdisciplinary
projects from research councils—
Q41 Mr Chaytor: Why can that not
be adequately reflected by a process of peer review?
Sir Alan Wilson: My own instinct
about this is that there is a massive combinatorial problem. There
are too many combinations. If you look within the sciences particularly
at the combinations of disciplines that produce an interdisciplinary
research project, if you wanted to find a panel for something
like bioinformatics, which should be a combination of chemistry,
biology and computer science, you would need a very large number
of those panels; whereas experts in a field within a research
council can find ways of doing that. If we look at the aggregate
of all those—
Q42 Mr Chaytor: At the moment presumably
there are a large number of single discipline panels, but how
many panels are there at the moment?
Sir Alan Wilson: There are 67
subject panels and 15 so-called super panels.
Q43 Mr Chaytor: Is it not a question
of reducing the number of 67 down to a smaller number of broader
interdisciplinary panels? Why would that not work?
Sir Alan Wilson: I think what
happens when you try to do that is that you always have a good
number of cross-boundary flows, however you try to define that
smaller number of panels. What we also have to bear in mind, Chairman,
is that in the consultation document we have kept open the question
of the possibility of having exactly those kinds of broader panels
to take an overview of the metrics system that is developed, because
obviously something which is entirely mechanism certainly in its
early days could have dangers. I have not responded to Mr Chaytor's
question, Chairman, on bibliometrics, which is another side of
the metrics analysis. It is very interesting on a global basis:
there are an increasing number of interdisciplinary journals,
so if you are looking at successive citations from journals, the
publishing world has responded to the interdisciplinary agenda
very rapidly, and so it gives us a base which reflects the research
frontier rather more accurately than traditional subject panels
would.
Q44 Mr Chaytor: Would that argument
apply equally to the other criteria that are listed on the consultation
documents, simply the raw numbers of research active staff or
the number of some PhD students? You are arguing that increasingly
research active staff would be engaged on interdisciplinary work,
the numbers of research students would be engaged on interdisciplinary
work and—
Sir Alan Wilson: I think, Chairman,
it is Mr Chaytor's question about panels turned into the way the
metrics are used in a formula, how broad are the groupings and
that is something that has to be explored, so in the models that
we presented and put on the web site for illustrations we have
actually looked at different kinds of groupings, but I cannot
say at this stage that we know what the final recommended answer
will be and that is why we have actually put alternatives on the
website to inform the consultation.
Q45 Mr Chaytor: In terms of the broader
issue between the role of metrics and the role of peer review,
do you think it is absolutely an either/or choice to be made?
Sir Alan Wilson: It is not an
either/or, firstly in the sense that Professor Eastwood referred
to in the context of another question and that is lying behind
metrics, whether it is bibliometrics or research council income,
there are different peer review processes, so it is not as though
these numbers are not, as it were, products of peer review, but
it is a different kind of peer review, so in that sense we are
still connected and, as I said earlier, it is possible that certainly
in some areas particularly arts and humanities and perhaps in
social sciences we still need panels to work with the metrics
base.
Q46 Mr Chaytor: What would be some
of the disadvantages of moving into largely a bibliometric based
system or a largely research income based system, what would you
identify as the biggest problems?
Sir Alan Wilson: I think at the
moment I do not see huge problems. I mean I think it has to be
put to the test in 2008 and I think we will learn a lot from that
exercise. I mean all the signs are, because the correlations are
so high and if I am right about the fact that many of these metrics
are closer to the research frontier than traditional subject panels
might be, then I think on balance it is more likely to be beneficial
than not, but I think it is very important, for instance in terms
of perverse incentives, we have recommended, or we have put in
a question since we are not convinced that we will have seen at
this stage all the possible perverse incentives that this has
to be monitored very carefully. We know what has happened in the
past, if something goes wrong, as it were, with this one in terms
of incentives—
Q47 Mr Chaytor: In terms of incentivising
behaviour and rewarding behaviour or discouraging behaviour, there
must be some underlying assumption about the kind of behaviour
the Government wants to encourage and discourage, so what do you
imagine the outcome will be if post-2008 there is a shift to a
system that is significantly based on metrics and what kind of
behaviours will be encouraged and discouraged?
Sir Alan Wilson: I mean in one
sense I do not think there will be a huge change in that, I mean
it is not to say that it will not be possible for new entrants
to emerge more easily than it may be the case in the present system,
but what we have succeeded in doing in this country is generating
a community of top class researchers and I think those top class
researchers will still present themselves through whatever kind
of metrics are used. Now many of those are now working in interdisciplinary
teams, but I suspect if you went back 10 years they were still
working, they were beginning to work in interdisciplinary teams,
but they were presenting themselves to subject panels and I am
sure subject panels have done their best in assessing the quality
of interdisciplinary work, but I think those are the kinds of
shifts that we will find.
Q48 Mr Chaytor: In terms of the distribution
of research funding this must inevitably lead to a further concentration?
Sir Alan Wilson: I think that
is actually certainly not the case in terms of the word "inevitably".
Q49 Mr Chaytor: Is it likely to lead
to further concentration?
Sir Alan Wilson: I think if you
actually had the patience, and it probably does need patience
to look at the large numbers of tables on the models that we have
used for illustrations on the web site—
Q50 Mr Chaytor: I think I am actually
going to give that a miss.
Sir Alan Wilson: Probably more
of them have moved away from increased concentration and in fact
there has probably been slightly less concentration, if my memory
serves me correctly, so it is certainly not inevitable.
Q51 Mr Chaytor: Slightly less concentration,
because we still have the RAE system and a peer review system.
What I am trying to get at, is there not a correlation between
those who are in favour of greater emphasis on a metric based
system and those who are in favour of a greater concentration
of research funding?
Sir Alan Wilson: Is there a correlation?
Q52 Mr Chaytor: Is there a correlation
between the two?
Sir Alan Wilson: Yes, I mean I
am sure there is a correlation between them but, at the end of
the day, it is a policy question for Professor Eastwood and his
council colleagues because it is the weightings and the funding
formula as much as the assessment and the metrics that will actually
determine the degree of concentration. Whether it is metrics or
RAE neither of the systems determines the degree of concentration,
it is the weightings that go into the funding formula.
Q53 Chairman: Sir Alan, you know
a famous scientist, Joe Schumpeter said "politics about
who gets what, when and how" and two and a half years
ago when we looked at higher education one of the things that
really worried us was a very well known vice chancellor came into
this room and said he believed that a much greater concentration
of research in a handful of research, which at university is a
handful, and I said, "Do you actually mean five only?"
and he said, "Yes, five", and our report said that would
mean that most of the research in research funding universities
would be in London and the South East, it would cut out Leeds,
your old university, it would cut out your university, the University
of East Anglia and we would have that concentration and then what
we said is if there was not a research rich-science base in each
of the regions of our country, it would be a very retrograde step.
Are you telling me that there is not something going on here,
whether you call it metrics or the changing system, that actually
is not the agenda of a small group who still want—they are
very articulate, they are not secret about this, Sir Alan—who
want research based to be based on a handful of universities;
come on?
Sir Alan Wilson: I fully understand
the question, Chairman, and I would say very clearly that there
is no hidden agenda of that kind.
Q54 Chairman: Not from you, no.
Sir Alan Wilson: The policy of
the Government is to fund the best research wherever it is found.
Q55 Chairman: Even if they ended
up in five universities?
Sir Alan Wilson: I think that
the system from where we now stand, you know, it would take a
very different kind of policy change to concentrate research funding
in five universities. No RAE or metrics based system of any kind
that we have thought about would bring about that kind of concentration,
it would need a very different kind of policy directive and that,
to the best of my knowledge, is not under consideration by any
of our ministers.
Q56 Chairman: It is worth us bearing
in mind what major players are saying?
Sir Alan Wilson: Perhaps I could
add, Chairman, that what I am arguing is in part connected to
what has been a core principle for a very long time which is the
best research should be funded wherever it is. You can then apply
different levels of some selectivity, as it were, and still maintain
that principle, but I think any level of selectivity, and in a
sense I am repeating myself, you know would not generate research
in five universities, it would need somebody to say, you know,
"we will just fund research in five universities". I
think the other thing, Chairman, is what you have said yourself
about the regional dimension and as a government department speaking
as someone from DfES I mean we are very well aware of PSA targets
for economic development across the country and we are very well
aware of the contribution that universities make to that agenda
and so we actually have discussions, whether it is with HM Treasury
or with CLG or DTI about how HEs contribute to that and particularly
the research agenda, so we would want to support, I am sure Chairman,
research across the country.
Professor Eastwood: If I could
just add that the concentration of the kind that you were sketching
would run counter to the considered position of the funding council,
would run counter to the view, I think, of the research councils,
both of whom see distributed excellence, if I can use that term,
as fundamental to maintain the supply of outstanding researchers
as well as meeting the regional case that you have articulated.
I think it is also worth noting that alongside the recurrent investments
we have been talking about when we have been talking about QR
or the research councils, there has been a big commitment to invest
in the research infrastructure in general and the science research
infrastructure in particular and if you look at the pattern of
that investment through initiatives such as SRIF, that investment
too in pretty heavy kit, again is distributed over a substantial
number of institutions, so it seems to me that there is, as it
were, a broad consensus around the way in which we are investing
both in capital terms and in recurrent terms in the research base
which reflects the funding of excellence, selectivity, but appropriate
distribution.
Q57 Chairman: I still doubt whether
out of this Committee and I have got a restricted group of people
in this country, they really understand; a lot of people out there
understand the research assessment exercise, peer reviewed the
system committees, however many there are, and they understand
that system works. Now this system called metrics people out there
I do not think, and I bounced this question over the weekend,
"What do you mean, what are you measuring? Mathematicians—what
are they going to measure then and how do they know those measurements
are fair? What is this metrics?".
Professor Eastwood: It is certainly
not a good dinner party discussion topic.
Q58 Chairman: No it is not. Ordinary
people in this country ought to be able to understand, it is assumably
camouflaged by some sort of nomenclature that is impenetrable.
What are you measuring?
Professor Eastwood: What we are
measuring is the investment that is going into research through
income measures, we are measuring the outputs in terms of the
growing use of bibliometrics and the impact that they have and
we are measuring volume in terms of the number of researchers,
the number of PhD students and so forth. A number of those things
have been there in the RAE before, they are not new, and a number
of the peer review panels, particularly in the subject areas which
were the first to develop bibliometrics, used those, and the economists
would be a good example, used those as an aid to make their judgments
about the quality of published outputs in 2001 and, to some extent
even in 1996, and will do so again with the Funding Council's
blessing in 2008, so metrics have been there and I think if you
look at some of the indicators, notably bibliometrics, what they
are telling us is now much more sophisticated than it was even
five years ago, they are telling us very interesting things about
the nature of impact of published research, when it has an impact,
where it has an impact and it seems to me that what we are offering
in 2008 is the possibility of a very serious evaluation of the
sophistication of those sorts of metrics.
Q59 Chairman: So all this, Professor
Eastwood, much more sophisticated measurement, all these measures
go into, and who makes the decision about allocation, because
that is not neutral, is it? Does it go into a wonderful computer
that says, "This is the distribution of resources for research",
or does it go to human beings in committees?
Professor Eastwood: Ultimately
the responsibility for the distribution of research funding to
institutions is statutorily the responsibility of the Funding
Council. In coming to that judgment the Funding Council obviously
is aware of the policy framework and is responsive to the policy
framework and, as Sir Alan was saying earlier, and it is clear
in the published consultation, that at an appropriate level there
will be a number of panels to advise on the appropriateness of
the metrics.
| 
