Examination of Witnesses (Questions 540
- 559)
WEDNESDAY 21 MARCH 2007
PROFESSOR ALAN
GILBERT AND
PROFESSOR MICHAEL
WORTON
Q540 Stephen Williams: Would you,
Professor Gilbert, wish to add to that? Before you do, we are
going to go into STEM subjects and humanities in the next section,
so do not dwell on that.
Professor Gilbert: I would say
that you cannot answer the question about whether metrics is a
good thing or a bad thing or how effectively metrics will replace
or augment the RAE until we know what metrics are being proposed.
I think we have to suspend judgment to see what the elaboration
of the Chancellor's proposal means. I have been pleased in the
last year that there seems to be quite a lot of common understanding
about the fact that metrics will change behaviour in the way that
the RAE changed behaviour. Metrics thought through in a fairly
superficial way can have a devastating effect on performance.
If, for example—and I will just give you one example but
I could give you a whole list—you decide to count enrolments
as the measure in relation to research training performance, what
you will get is a very large leap in enrolments; if you decide
you will count completions, you will get a completely different
set of behaviours, and you really need to know how institutions
will respond to different metrics. The setting of metrics is an
extraordinarily difficult, sophisticated task, and most of us
in universities will only be satisfied that the proposed change
of direction is a good one and likely to be efficacious when we
see the small print.
Q541 Stephen Williams: Chairman,
perhaps I can stick with this as you have led me into the behavioural
aspects of any Research Assessment Exercise as universities are
going through the 2008 RAE at the moment. Do we know enough about
how assessing research distorts behaviour, both in academic recruitment
policies and then how you deploy those academics in between research
and teaching, for instance?
Professor Gilbert: My view is
we will never know enough, but we already know a great deal. I
think universities could give quite a lot of advice about how
particular metrics will work, about the lags, for example, between
the performance and the reward, which is another issue in relation
to metrics. I think a serious, consistent, ongoing conversation
is what is needed here. There is a lot of advice that the system
can give, not least that many metrics applied in isolation of
peer review, rather than in conjunction with it, are likely to
be downright dangerous.
Professor Worton: One of the worries
in the sector is that there is an automatic leap from the word
"metrics" to the notion either of bibliometrics or of
research-income metrics and so one needs to be looking at a series
or portfolio of these. Some of the work we did in the HEFCE/AHRC
group was to model a variety of metrics against the information
we had from 2001. There would need to be serious modelling of
any system against both 2001 and 2008 outcomes, recognising that
these are of course different exercises, so you cannot even compare
directly 2001 outcomes with 2008, but there are ways of making
comparisons. I think we need to recognise also that things like
bibliometrics, I would argue, are not yet an exact science, but
it is an area in which movement is going forward astonishingly
quickly.
Q542 Chairman: Before we leave that,
Alan, can we nail you down a bit on the Australian experience.
When we were in Australia, people kept saying, "Oh, you are
moving your way to us; we are moving from metrics to research
assessment mode and surely that cannot be good", and what
you seemed to be saying in your first answer to Stephen was that
it is not a question of metrics, it is what kind of metrics. It
is rather like what we all learned in this Committee, it is not
PFI, it is what kind of PFI. Is that a reasonable comparison?
Professor Gilbert: That is right.
I had the misfortune of being involved in action initiated by
universities against the Commonwealth of Australia on the grounds
that the Commonwealth was representing some metrics they introduced
as measures of performance or reputation or excellence and, in
our view, whilst they were measures of something, it certainly
was not that, so we proceeded on grounds of judicial unfairness
against the Commonwealth. It can get as serious as that if what
metrics are purported to be doing bears no resemblance to the
actual impact they have. The Australian experience of metrics
was patchy. Metrics were used, and I think quite rightly, not
just to try to describe the world but to change it, and I imagine
that is certainly something the Chancellor had in mind here. What
I would say is whether you mean to introduce metrics to change
the world or whether you do not, they will initiate change, so
it is very important to understand what their impacts are likely
to be, unintended as well as intended, before you start to drive
policy with them.
Q543 Stephen Williams: Are you satisfied
that the sector is being fully consulted so far on the proposed
changes? The consultation which took place after the Budget—we
took some earlier evidence—seemed to be rather rushed. Do
you think since that consultation you have been involved enough
in the evolving process?
Professor Gilbert: The key words
in your sentence are "so far". We have been consulted
adequately "so far", but we have not been consulted
enough because the detailed development of the metrics is yet,
it seems to me, to occur and the conversation will need to continue
to be quite an intimate one through that phase. But so far I think
the voices of universities, CIHE and other bodies have been listened
to.
Q544 Stephen Williams: Do you think
the timetable allows for that further consultation?
Professor Gilbert: I do not know.
Professor Worton: Because art
and humanities were singled out as being this "rump"
of the body politic of research in the UK, which some of us considered
somewhat unfortunate, it gave us the advantage that we lobbied
to have our own group with HEFCE. That was enormously important
for us, but we also had to spend a great deal of time consulting,
not just with institutions but with learned societies, professional
bodies and so on. Having been asked to do this on the basis that
people felt I had good internalised body armour and could deal
with the slings and arrows of outraged academics, it was very
interesting to see the community move from a position where people
were naturally hostile to the notion of metrics by the time we
published our report there was buy-in to what we were arguing
for and people could see it, as long as the issues of perverse
behaviour were addressed and as long as, as Alan was saying, there
continued to be consultation afterwards, and I think that is going
to be a key issue from now on. I will make one last point which
is we know that academics are clever and they will always work
their way round a system. One of the things I think the RAE hitherto
has not addressed is being useful and perceived as useful by all
universities. One of the things we can do in a new system, especially
if it is looking at the research process in a more holistic way,
is to be very useful institutionally; capturing data, in other
words, doing metrics which would have an importance for the national
exercise but also have institutional benefit. There is an awful
lot of stuff that we are not capturing ourselves at the moment
which we could usefully capture.
Q545 Stephen Williams: You say there
is more buy-in now, but is that something which is shared right
across the sector? You both represent particular types of research-intensive
universities but there is great diversity across the sector. Do
you think there is more buy-in by non-Russell Group universities,
for instance?
Professor Worton: Certainly in
the consultation exercises which we did, it was right across the
sector and we were getting as much support from, in fact, teaching-intensive
universities as from the Russell Group. It was interesting also
that after the publication of the report especially I spent most
of my time talking to scientific groups because they liked the
model which we had been arguing for.
Q546 Stephen Williams: Chairman,
it has been mentioned that there is still a need for more consultation
and discussion. Do you think there is still room for negotiation
over the extent that peer review might still play in a future
process?
Professor Gilbert: I would hope
so because I think the hostile reaction to the initial proposal
was motivated by two things, one being the threat of substitution,
that is, that metrics would be substituted for peer review. I
think that would have horrendous unintended consequences, because
one of the most potent forces which has favoured and built the
reputation of UK higher education in the world as a brand, and
clearly the second most potent higher education brand in terms
of national system, has been the rigorous, internationally networked
peer review which has driven the RAE. As a result, no one anywhere
in the world doubts the validity of the findings of the RAE. They
can be understood internationally and they can slot UK academics
and universities into an international ranking. If the UK Government
was seen to be abandoning peer review and substituting metrics
for it, I think the damage to the reputation of the UK higher
education system would be deep and prolonged. Substitution is
not a language which should be used nor a methodology which should
be pursued, in my view. I think that one of the reasons for a
much greater toleration of the proposal is there has been movement
by government away from replacing peer review, but I think there
still is a danger that, if the new policy is based on substitution,
then there will be continuing and increasing serious lack of credibility
in the new system. I think the second issue which caused concern
was the feeling that the metrics were being motivated by government
concern and industry concern that higher education research was
not sufficiently industry-facing, to use the metaphor which has
been used so far. Whilst I think there is a lot of truth in that,
there was a strong feeling that that proposition was not taking
sufficient cognisance of immense changes that have occurred in
the last few years and it was based on a fairly facile understanding
of how universities are in fact linked to industry.
Professor Worton: If I could just
make a point about peer review, I think we need to recognise that
peer review is much more multiple than perhaps we recognise and
it is not just the peer review of the Research Assessment Exercise.
It seems to me that the real debate is really about whether we
want a system which is peer review informed by metrics or a system
which is metrics informed by peer review.
Q547 Stephen Williams: You have both
made a very strong statement that peer review must be part of
the future system. Just to go back to the timetable, we are going
to get the 2008 RAE, we are going to shadow the metrics exercise
alongside it, and then the current proposal is that in 2009 science,
engineering and technology will move straight over to the new
system after only one year. Is that a realistic timetable for
that particular part of our education system?
Professor Worton: Given where
we are at the moment and given where HEFCE is at the moment, it
is going to be enormously tight. Given also that HEFCE has made
an explicit commitment time and time again to consult widely with
the community on the proposals, I would be surprised if that could
be done, but we live in hope.
Professor Gilbert: If I can comment
on that as well, I think it would be a serious mistake to move
totally in one step from RAE-driven QR funding to metric-driven
QR funding. It would seem very strange to have run the 2008 RAE
if it is not going to have an effect on funding for some time,
so I have always understood the notion that there might be an
early introduction of metrics as not being a statement about a
total change, but about a progressive introduction of the new
scheme so that it might be 2011-12 or 2012-13 before the metrics-driven
funding replaced the RAE-related funding. If it was to be done
in one year at the beginning, I think a lot of people would just
ask why we had spent all the money on the RAE.
Professor Worton: That is certainly
what we have all understood by the Chancellor's pre-Budget speech
in December 2006, that it was going to be phased.
Q548 Stephen Williams: I have one
last question specifically about Manchester. Professor Gilbert,
you have said that your mission is to have a "preferred future"
where your new university is going to be in 2015 rather than starting
from your status quo. Do you think this upheaval which
is taking place in the assessment of research is going to help
you with that mission or not?
Professor Gilbert: I would rather
answer that in 2009, I would have thought, than now. My view is
that it has been helpful, but it has also, however, driven a very
adventurous first two and a half years in the university, so we,
like many other Russell Group universities, have deficit-funded
our preparations for the RAE. We have added net 2,800 jobs in
the university, the great bulk of which have clearly been to strengthen
the research profile. It has in a sense been reprofiling by growth
because one of the outcomes of the Project Unity negotiations
was that we were denied, by agreement with the campus-based unions,
the right to address the structural deficit that you always get
in a merger, for at least two years. So we have been both carrying
a structural deficit by not being able to shed some of the redundancy
which you always get when you merge two administrations, and we
have in that period also taken on the deficit-funding of a very
bold attempt to enhance the research profile of the university.
We think that this has been very effective, but also most other
universities in the UK have been doing the same thing, so the
RAE is going to be an interesting test, I think, of the efficacy
of quite complementary strategies which have been used around
the Russell Group in particular, but across the system in general.
Chairman: This is all excellent stuff,
but we have to move on.
Q549 Paul Holmes: This is partly
on the tail end of what Stephen has been asking. I think, Professor
Gilbert, you mentioned earlier that you have got to be acutely
aware of the way in which whatever funding system you have affects
the behaviour of the institution, so under the RAE system there
have been all sorts of allegations that universities will cut
this department, get rid of those staff, poach staff from somebody
else because they have got a good publications list and that sort
of thing. How far will the implications of the new metrics system
steer the way that you achieve your 2015 goal, for example, for
the University of Manchester?
Professor Gilbert: They will either
do good or harm, depending on what they are. If I can just use
one example which I think might be central to the deliberation
of this group today, it relates to a notion that is often favoured
by the Government that universities should be deriving more income
from their links with industry. In a mass higher education system
which is placing pressure on the public purse, universities have
to substitute non-public funds for public funds and, therefore,
universities are often put under pressure to derive a lot of that
from industry, from knowledge and technology transfer. I think
if one of the metrics was to measure income into the university
through industry and technology transfer, that would be a mistake.
In Manchester we think that deriving income from industry and
technology transfer or the commercialisation of intellectual property
ranks fourth or fifth in our hierarchy or priorities. The most
important thing that universities need to do in relation to intellectual
property is to create it. The second most important thing they
need to do is disseminate it, make it available to industry, and
I think the third most important thing they need to do is to have
policies related to IP which help them recruit some of the very
best people in the world. In other words, it is the generosity
of the policies to create IP that is going to enhance the strength
of this system. Universities can derive income and in many cases
they derive lots of income from IP commercialisation, but it should
not be the main driver. This year, for example, just to mention
a point about scale, the University of Manchester has seen one
of its start-up companies sold to a Swiss company, Novartis, for
£308 million. Now, that IP was just an idea in two academics'
heads seven years ago, yet in terms of wealth-creation it has
now generated a £308 million takeover. Another company, Renovo,
was floated in an initial public offering that was the second
biggest on the London Stock Exchange in relation to biotechnology.
The scale is huge, but it would have been much smaller if the
University had been under pressure to derive income flows from
these businesses. Therefore, it is the generosity of an IP policy
which says to creators, "If you think you can commercialise
your IP, the University's greed will not inhibit your being able
to get third-party investment in it". We are less concerned
about the number of companies formed or the number of licences
undertaken, then about the value of third-party investment in
those companies: what third parties making business decisions
are willing to place as a value on initiatives that are happening
within the University.
Q550 Paul Holmes: Are you saying
that, if the wrong types of metrics are applied, then there would
be a bad effect on innovative, blue skies thinking and academic
research and launching of ideas where seven years ago it was two
academics and it is now a £308 million business because universities
will play safe and go for what gets the cash in now?
Professor Gilbert: If the Government
said that universities must derive a third share or a 50% share
of the revenue streams emerging from commercialisation of intellectual
property, I think that the outcome would be similar to a principle
better-known in relation to taxation, that the higher the stream
required by the university, the closer actual revenue will approximate
to zero. What we have said, as a university, is that the university
will never seek more than 15% of the revenue generated by intellectual
property commercialisation or knowledge transfer and in many cases
we may seek even less if, in our view, the commitment of the university
to securing more than that is going to stop third-party investment
in the IP.
Q551 Paul Holmes: Is it alarming,
therefore, the point of what you have said, that 12 months ago
when Gordon Brown announced this shift in thinking, he specifically
said that he wanted to look at developing a metrics-based system
where money is related to the impact of published papers, but
also at how much money it attracts in grants and contracts? Are
you happy that in the 12 months since then, between the joint
committee that you chaired, Michael, and what you have said there,
you have been listened to and that we can avoid that?
Professor Gilbert: Well, it depends
how it is measured and, interestingly, I think that while the
statistics provided earlier about the industry investment in universities
were accurate, but they can be misleading. I visited our School
of Electrical and Electronic Engineering yesterday and they told
me that every EPSRC grant they have secured was what they call
`industry-leveraged'. Not one successful application to the Research
Council for funding was not part of a bigger research enterprise
in which industry was doing a substantial part of the research.
What industry R&D was investing in in that larger research
project under that project umbrella does not show up as industry
investing in universities, but it was being leveraged by EPSRC
funding, and I find it extraordinarily exciting that 100% of that
school's EPSRC funding was leveraging cognate activity in industry.
So you need quite a sophisticated measure, I think, to get a grip
on how universities are reacting to industry when much of it will
not show up in the bottom line of financial indices.
Professor Worton: We also could
give you examples of the relationship between business and industry,
but I think we need to look also at how knowledge transfer is
beyond, that it is not just simply technology transfer rewritten,
and we have got to look at how in fact the knowledge which is
created in the universities is changing policy, how it is changing
behaviour, how it is changing conduct. These are things which
are actually impossible at the moment to measure in terms of any
known metric, yet they are enormously important. I think one of
the most significant shifts over the last nine months certainly
in terms of HEFCE, who in a sense are going to be running the
system post-2008, is the recognition that we need to be looking
seriously at issues around significance which is not the same
thing as, if you like, the 1960s and 1970s debates about relevance,
but it is something, I think, much larger, and we must get that
right before going in blindly to say that income is somehow the
all-important metric.
Q552 Paul Holmes: How do we allow
for arts and humanities in this, and again your joint committee
is looking at this, but it is easy to see how in science, technology,
engineering and maths, the STEM areas, you can apply these measures,
however crudely, but how on earth do you do it without destroying
arts and humanities because they cannot measure up in terms of
pulling in grants and contracts from industry in the same way?
Professor Worton: That is not
strictly true, with respect, and I will give you an example of
one of our professors of philosophy who is working very closely
with Railtrack on risk. Why is it that there are more people killed
on the roads than ever the railways and yet every time there is
a rail accident there are headlines all over the nation? Therefore,
can you take a philosophical attitude towards risk? How do you
deal with the ethics of biotechnology and so on? There are many
examples which one could give, but if you wanted to take, let
us say, your card-carrying English literature graduate, as it
were—
Q553 Paul Holmes: Or a historian.
Professor Worton: Or a historian,
absolutely. Well, historians are essential in the sense that,
there is the issue of how the present cannot exist outside of
the past and outside of the desire, the tension towards the future,
so in a sense, had we listened to the historians a little bit
more, we might not be in certain situations worldwide in which
we find ourselves at the moment. How do you measure quality of
life? Now, that really is one of the biggest issues that we have
and, quite frankly, there is nobody even pretending that they
are anywhere near a metric on this. What you can, however, chart
is how behaviours are being shaped, not necessarily that they
are being shaped directly by the work that is being done by the
humanities researchers, but what they are doing is that they are
shaping policy decision-making and policy-shaping which is then
ultimately itself having an impact on the communities. Now, that,
I think, is something which is enormously important and the whole
issue of how policy, governmental policy, whatever, is left out
of the debate is, I think, one of the most worrying things about
the assessment system.
Q554 Paul Holmes: You are saying
that you can measure this in sophisticated ways, but, on the other
hand, going back to some of the earlier questions, in theory we
are going to start doing this by 2008-09, so how are we going
to get all these sophisticated measures in which save the arts
and humanities from being squeezed out of the system in that timescale?
Professor Worton: For the arts
and humanities, we have a much longer time-frame. The panels will
be set up in 2013-14 in order for a more informed situation to
be brought in in 2014. What we are doing at the moment is working
within the communities and, interestingly, it is the humanities
community which is working hardest on the metrics and I think
that is enormously positive because you have got the people who
may ultimately be judged by it testing them at the moment and
it is not just simply being run by civil servants in the Treasury
or DfES, but I think it has also got to do with the whole nature
of how our communities are changing. We, for instance, at UCL
have published a document for all of our staff called Excellence
in a Shared Community which I can send to you, Chairman.
Q555 Chairman: Can we have a copy?
Professor Worton: Yes, will do.
Basically what we are saying explicitly to all of our staff is,
"This is what we expect of you in research, in teaching,
in knowledge transfer and in enabling", which is our word
for academic citizenship, administration and so on, and we say,
"We expect you all to do all four of these things, not necessarily
all at the same level every year of your career", but we
also then have a section which says, "This is what we, as
the university, will give you. As we are expecting this from you,
you can expect this from us, a well-funded laboratory, proper
libraries and so on".
Q556 Chairman: Alan, have you got
a similar thing?
Professor Gilbert: The answer
to that is I think we would expect those values within the university
to be transferred through our strategic plan "Towards Manchester
2015", but I would certainly think that a contemporary university
needs to be changing the culture in the community in precisely
those ways.
Professor Worton: What this is
meaning, to go back to your point, Paul, about the arts and humanities,
is that they are now getting very excited about knowledge transfer.
They had felt excluded by technology transfer and now they are
becoming part of this and they are looking at not perhaps the
SMEs that we talk about so much, but the micro-enterprises and
that is something which is not captured. I am chairing a research
project for London Higher looking at excellence in the whole of
London and we know that, if we put Imperial and UCL together,
we have a crushing example of knowledge transfer, business investment
and so on, but in the London Higher project we are precisely looking
at all of the universities in London and trying to seek excellence
and actually where it goes, and what is coming out of the work
that we are doing. There is the importance of the micro-enterprises
and we need to capture these data which certainly the LDA is not
capturing at the moment.
Q557 Paul Holmes: Is one way of protecting
the arts and humanities, if you like, to maintain the role of
expert peer review and not just to switch purely to metrics?
Professor Worton: Peer review
as long as the peer review includes expert judgment as well, and
I think that is a very important point that came out of our work,
and the community certainly is totally with it. Defining experts
in different ways as users, but also in other ways; it is about
ensuring that it is not just being assessed by academics, but
there is a relationship which goes beyond the academic world,
and that, I think, is an enormously important step forward.
Q558 Chairman: You describe a very
cosy world, but one of the shocking things when all of this started
being discussed was when I read, and I did not know this, how
few university research publications were ever cited anywhere,
just a huge amount. What is going on? Is this a lot of public
money into navel-gazing that no one else is interested in? There
is a fair criticism there, is there not?
Professor Gilbert: I would like
to answer that, if I may.
Professor Worton: I would like
to come in as well!
Professor Gilbert: I think it
would be wonderful if we could eliminate the 50% of research outcomes
that are not cited. But if you think you can identify them in
advance, then we, as university managers, would like to know how.
It does seem to me that we are in a winnowing process which is
actually quite efficient. Fifty per cent might sound very high,
but, if you think of analogous enterprises, preparation for the
Olympic Games and the public money being invested in the training
of athletes who may succeed to be absolutely world class, it will
be much higher than 50% of those currently having resources invested
in them who will fail to make the required level. In all of these
enterprises, I think universities are actually quite efficient,
and 50% sounds negative, but I think it is a measure of considerable
efficiency because there is no way of knowing, when you are nurturing
the talents of an early career researcher, at what stage the breaks
will occur when a series of unquoted articles suddenly is followed
by an article in Cell or Nature or Science
which is hugely cited. We try to run universities through performance-monitoring
and we certainly try to monitor the performance of our researchers,
but it is very difficult to do it in the way that you rightly
hope we might.
Professor Worton: It is very good
to have a historian in the room because what we do know is that
there has been research published which has been ignored, not
just for a year or two, but for decades which then suddenly becomes
seminal to future developments.
Chairman: I think you are talking about
John Clare, are you not! They hate it when I introduce that!
Q559 Stephen Williams: John Clare
and Huddersfield!
Professor Worton: The other point
I would make is that I think there has been a systemic problem
in the world of research which is that access to research findings
has been enormously expensive and it also has tended to be rather
Anglocentric, and the move towards open access is changing radically
the way that we are actually seeing citations. I will just give
you one example and I know it is a slightly unfair one, but it
is one which makes me smile with joy in that one of my postdoc
fellows came from the University of East Anglia, giving up a full-time
job to come to a two-year postdoc fellowship, working in Danish
film (we have the only Department of Danish in the country). She
decided, with a bit of encouragement, that she would put her Edinburgh
PhD on Danish literature online in our e-repository, our open-access,
electronic repository, but how many people would actually read
her PhD in Edinburgh? It has been borrowed, I think, several times.
I have a report run every term in a spot week on top citations.
Salvador Moncada, the most cited scientist in Europe at UCL, he
is usually up there. This young woman, Claire Thomson, had been
cited in one week, been downloaded and then people were using
her work 250 times. This term I again ran a check and there she
is up in the top 10 again, 187, way above all of the scientists,
the electronic engineers, the computer scientists. What we have
now is a very different mode of dissemination experience, so now
the citations you have of Claire Thomson are suddenly getting
into the realm of the scientific citation levels because we now
have means of access. There is another issue, I think, which is
very important, that science is essentially international, but
it is also essentially Anglophone, whereas in many other areas,
notably the social sciences and the humanities, we publish in
a variety of languages. This is not being captured, if you like,
by the commercial indices, like ISI/Thompson and so on, so there
is a European Science Foundation project on a European reference
index in the humanities in which I am involved which will actually
make available, explain and bring greater rigour and peer review
to European journals. This, I think, will transform the way that
European research is perceived internationally.
Chairman: That is absolutely fascinating.
|
