Examination of Witnesses (Questions 700
- 719)
WEDNESDAY 16 MAY 2007
PROFESSOR IAN
DIAMOND AND
PROFESSOR OLE
H PETERSEN
Q700 Fiona Mactaggart: Professor
Petersen, you said that you can name two advantages of the present
system—and I am sure you can think of others—stability
and transparency. It struck me that a more stable, more transparent
and much cheaper system would be not to have Research Councils.
Looking at the paper by the Higher Education Policy Institute,
which put this thesis—just imagine that instead of using
the Research Councils' peer review system one simply calculated
the share of the total investment in research which each university
had to come and they just issued that over the five year period
between 2000 and 2005. They then worked out how the different
Research Councils' work changed it and they calculated that over
that four year period operating peer review meant that 79 universities
shared an extra £158 million redistributed from 61 universities
who between them lost the same amount. If it costs £196 million
a year to do peer review, that means that Research Councils UK
spent £784 million on redistributing £158 million. Okay,
it is a theoretical calculation but actually there is a point
to it, is there not?
Professor Petersen: There may
be a point to it but to my way of thinking it is somehow not quite
addressing the central point. I think the central point is to
create a system where people really think hard about what they
want to do and where they are exposed to peer review criticism.
You are focusing on the cost of it, but the process itself has
actually an enormous amount of—
Q701 Fiona Mactaggart: But the taxpayer
is paying for it.
Professor Petersen: It is not
the waste that many people think, and this is why I make the point
that peer review is an intrinsic part of the whole academic process,
you simply cannot think it away, it has to be there, and if it
is not there in relation to competition for Research Council funding
it is there in relation to competition for getting papers published
in the best journals, so you have to think through these things.
These figures are very soft figures, quite frankly, because they
are just based on what people report on the number of hours they
do and as we know, quite frankly, we all have to do these time
sheets in universities where you write down whatever you feel
like because it really is almost meaningless. If you look at the
overall figures[10]
you might say this is a stupid way of doing it, but when you look
into the details it is actually quite an intelligent way of doing
it. People are focusing very, very clearly on a particular task
that they would like to solve and they are forced to present that
in a form that can be understood by other people. It is exposed
to criticism by the community, it is being refined and finally
you do the work. It is a process that actually works extremely
well. If you just distributed money without this kind of peer
review process we would see over the next 25 to 30 years a dramatic
decline in the standards of science done in this country. That
would be my judgment.
Professor Diamond: Could I also
just take the opportunity to remind you of the point that I did
make, and that is that the cost to the Research Councils of the
peer review system is under £10 million per annum and that
if you wished to do the same holistic costing of the RAE process
you would find a very, very large sum because the sorts of things
that would have to be included would be the sorts of things that
Professor Petersen has already described in terms of the peer
review of publications, the time that is actually taken to get
those publications together, so we really do need to be absolutely
careful in comparing like with like if we are actually trying
to make the sort of comparisons that were being made in the paper
you described.
Q702 Fiona Mactaggart: What Professor
Petersen has said is that the data about how much time academics
are spending doing things and so on is very soft, almost meaningless,
and they write what they feel like. That actually rang all sorts
of alarm bells in my head as someone whose responsibility is public
accountability, to make sure that the taxpayer is being told the
truth, does understand what is going on. It sounds to me as though
you are saying we have this private little arrangement in universities
where we make it up.
Professor Petersen: It is almost
the other way around I think. What people do not realise is that
there is intrinsic stupidity in the system. For example, we are
only allowed, when we are making up our time, to work 37½
hours per week and most academics I know work 70 hours per week.
You simply are not allowed. When we have to send back to our university
how many hours we spend on different things, the system simply
does not allow you to say what is true, that you will spend 70
hours per week, so the whole system is not working properly but
in terms of accountability I would say that you get fantastic
value for money from academics actually because people work like
mad for very, very poor salaries, so I would not worry too much
about public accountability. People are excited about what they
do and they work quite in excess. Maybe the only other category
of people who work as hard are politicians actually, there is
no other category.
Q703 Fiona Mactaggart: Do not misunderstand
me, I was not trying to say that academics are lazy, what I was
trying to say is that the public does not know the truth about
what they do.
Professor Petersen: That is true
at many different levels and the reason is simply that the systems
are designed in such a way that they cannot be used properly.
I gave you just one example, that you can only work so many hours
when in fact most people work almost twice that many hours. The
other thing that makes it genuinely difficult is, as I said, the
overlapping activities. You can put down one particular activity
under one heading but you could also have put it down under anther
heading, so in a sense this sort of idea which I can understand
to make things transparent and accountable comes into some kind
of difficulty. Also, the exact separation of teaching and research
is not straightforward. You talk to PhD students and you discuss
results with them; is it teaching or is it research? They are
overlapping activities. There are many, many intrinsic problems
in these things and I understand that one would like to be able
to get a very clear-cut answer to these various things, but in
reality it is much, much more difficult. Trust is an important
element of our system and one of the problems in the ordinary
culture that we live in at the moment is that nobody is supposed
to trust anybody else, and that is intrinsically a very, very
big problem actually. Most academics work extremely hard and they
do their very best under sometimes challenging circumstances,
and these kinds of orders that are imposed upon them as to how
many hours on this activity and that activity, it is, in some
cases very, very difficult to separate one activity from the other
one; they are overlapping to a very large extent.
Q704 Fiona Mactaggart: Your point
about trust brings me back to the first suggestion that a simpler
mechanism for allocating resources might be cheaper and more effective,
where you trusted the institutions on the basis of their record.
Professor Diamond: There is a
huge degree of trust and that is the QR element, but one thing
that you absolutely have to enable is the very best research to
be funded wherever it is funded. That does not mean that you have
a system whereby only one place is an institute for, shall we
say, a particular type of science; therefore, if you are within
an individual institution, trying to make decisions about whether
to fund individual X who looks quite interesting but is a junior
colleague working on an exciting new area against individual Y
working in a completely different area, you probably do not have
the skills in that institution to judge that. On the other hand,
by it being possible for those people to go to competitive peer
review, where they will get international quality peer reviewing,
from both the best people in this country and the best people
overseas, where their work will be tensioned against similar types
of work from other institutions, then we have a really strong
and extremely high class peer review system which enables us to
fund the very best science in a competitive way. It is a competitive
world given the budget constraints that we have, which enables
the UK to continue to be at the very height of global science,
and that is an essential element. I absolutely agree with what
Professor Petersen has said that if you did not have that the
system would carry on for five years. In 20 to 25 years—and
I have no evidence for the statement I am about to make, clearly—I
suspect that you would see a great reduction in the quality of
UK science.
Chairman: Let us move on. Thank you for
that and thank you for your kind remarks, Professor Petersen,
about hard-working politicians. Helen.
Q705 Helen Jones: You are both scientists
used to proceeding on the basis of evidence and proper research
and you have told us what this change to a metrics-based system
is supposed to do, but where is the evidence that it will actually
do what it is setting out to do?
Professor Petersen: I do not think
there can be evidence that it will do what it is setting out to
do. One problem that scientists particularly are very familiar
with is what you may call the paradox that if you want to measure
something you are inevitably also influencing what you are measuring,
and the metrics system is an extremely good example of that. At
the moment it is undeniable that there is, broadly speaking, a
certain correlation between what is perceived by peer review to
be good quality research and citation rates—in a very crude
way there is a certain degree of correlation of course and the
people who are very highly cited are generally the people who
also the community will regard as very good scientists. However,
if we now suppose that we are creating a new system for assessment
of research quality, which will influence QR distribution based
on the plans the Government has been proposing, based on quantitative
indicators that are in the public domain, then of course for example
citation rates which have been mentioned as one important parameter
will be signposted to everybody as an important parameter that
people have to think about. What many people who do not work in
the field may not realise is that for most people, for the vast
majority, even a lot of good people, the citation rates are not
very high, so the absolute numbers are quite small. The young
scientist who has been in the field for about five or six years
may have something like 20 to 30 citations per year, very small
numbers in absolute terms, some people of course who have been
in the system for a long time and are recognised will have quite
different numbers, but a large part of the system will have absolute
small numbers. They can be very easily manipulated; suppose 10
scientists in different institutions decide that they will create
a little mutual citation group and that they will cite each other's
work more, then these small numbers can be changed quite dramatically,
so numbers that perhaps at the moment would appear with many,
many teaspoons of salt to be useful, could become completely useless
in a couple of years because this system has been manipulated.
This is the danger of taking a secondary parameter rather than
having the peer review at the moment that is actually looking
at trying to assess what is the quality of the work that has actually
been produced. The other point—and I made that point in
an opinion piece I was asked to write for The Times Higher
Education Supplement which came out just before Christmas—is
that citation rates have almost no predictive value, so if you
are going to try to think about what will be the future it takes
many, many years for citations to accumulate. I gave a specific
example there of a close colleague of mine, Bert Sakmann from
the Max Planck Institute in Germany, who is a Nobel Laureate;
he made his breakthrough in 1980/81 and peer review instantly
recognised that this was really big progress, but if you look
at the citation rates for several years after that they were modest,
respectable, nothing more than that; 15 years later they were
spectacular but then he got a Nobel Prize and everybody could
see anyway that he was great. This citation game, therefore, is
something that will turn out to be quite frankly useless, and
it is very, very dangerous that the Government has somehow, it
seems to me, hitched itself to that wagon.
Professor Diamond: There are a
number of issues that need to be made here. The first point the
Research Councils would make is that there are very many different
metrics that should be taken into account and one needs firstly,
in a system, to identify exactly what it is that one is trying
to reward and to recognise, and that was a point that I made earlier.
Where I would agree with Professor Petersen is that simple citation
rates will not, on their own, answer many of the questions that
we would as Research Councils wish to reward; having said that
they remain for some areas, certainly for the stem subjects, an
area which is recognised as being useful—not alone but useful.
Q706 Helen Jones: Can I just stop
you there because Professor Petersen says that they can be so
easily manipulated, so how can you have something which is useful
in assessing research if, as has just been quoted to us, someone
who eventually got the Nobel Prize did not have a very high citation
rate.
Professor Diamond: There is an
enormous amount of data on, for example, time to citation rates
or the speed of citation rates over time. There is some excellent
work that has been done at the Science Policy Research Unit at
the University of Sussex, for example, which looks at the time
by different disciplines from publication to citation rates, and
you see an increase and then it flattens out, and certainly there
will be examples where I am sure they subsequently pick up. I
do not know the extent to which a small group could get together
and massively influence citation rates for a particular journal,
but certainly I do know that the Higher Education Funding Council
for England in taking forward this review of the research assessment
exercise is working with some of the very best researchers on
bibliometry in the world—I know that their consultants are
those from the University of Leiden who are absolutely excellent—who
will be advising them, I am sure, on all those issues, but I stress
that I do not see citation rates as being the only index. Certainly,
Research Councils' view is that citation rates would not be the
only metric that we would recommend because they will not reward
and recognise all the areas, and that is why one will need a basket
of metrics, and one of the things you have to do is identify what
it is you are trying to reward and to recognise and to provide
a set of metrics which are universally accepted and which will
do that. Let me also say that I have said very clearly right across
the piece that the Research Councils believe that in addition
to this basket of metrics which should inform any measure of quality,
there will be the need for an element of peer review, light touch
peer review, and certainly we believe that that is likely to be
different in different disciplines. If I just give you an example,
currently the citation rates which are widely used are based on
one large database. In the 2001 Research Assessment Exercise,
for the very great majority of subjects such as chemistry or physics—well
over 90% I understand—of the outputs which were submitted
to that exercise were included in the database. In other areas,
particularly in the social sciences, the arts and humanities,
of the outputs that were submitted a very much lower proportion
were included in that international database because of the greater
incidence of the use of things like monographs, which are not
in the database as a research output, inter-disciplinary research
publications which may not be in those outputs, and the fact that
the database does not use some major UK journals. So there are
very good reasons why one would not want to move solely to a measure
of citation, but where you need to use the very best bibliometricians
in the world to advise you on what the appropriate use of bibliometrics
might be to inform the basis, but I stress that simple bibliometrics
are not the only metric which should be used, one needs a basket
of metrics based on first deciding what it is you are trying to
reward.
Q707 Helen Jones: Can you just tell
me what you mean by light touch peer review, where you think it
might be useful and where it would not be useful?
Professor Diamond: Sure. Let me
give you an example of the area of policy. You may wish to try
to find a mechanism which rewards research which has really had
an impact on, shall we say, policy in some areas. It is actually
quite difficult to say has this research impacted on a new Government
bill or has it impacted on some kind of policy in a big way? That
is the sort of thing that you might like some research experts
to look at and take a view on. You may also want to recognise
the fact that some journals have much higher impact factors, if
that were to be used (and I do not know that it would) than other
journals, so you might want to have people just overseeing the
results that would come from a quantitative analysis of metrics,
fine-tuning them in a transparent way. We could look at many,
many examples of that. If one metric was to be research income,
you would not want to disadvantage a group of entirely theoreticians,
if you like, in the area of physics, who may not need as much
research income to undertake their research. That is when you
need the peer review to come in on top. What the Research Councils
would argue is that you can simplify the bureaucracy of a quality
assessment by including and informing your quality assessment
through a basket of metrics but we do not believe you can get
away with it completely.
Q708 Helen Jones: Can I ask Professor
Petersen if he agrees with that argument, because it seems to
me, Professor Diamond, you talked about transparency there but
some of the things you cited—for instance, where the research
had influenced policy—are very difficult to review transparently.
We know from working here how many things go into making a parliamentary
Bill, for instance. Is it possible to do those sorts of things?
Professor Petersen: It seems to
me that Professor Diamond has given very good arguments for retention
of the subject based panels and for peer review. We agree in the
Royal Society that these quantitative indicators can be used to
a certain extent by knowledgeable people in the areas where you
can make sure that you compare like with like. The formula based
approach which one senses is what the Government would like to
produce, has enormous dangers and would be totally unable to deal
with any of the problems that Professor Diamond quite rightly
put forward. The problem is in terms of bibliometrics, which a
lot of the discussion has focused on, because it is one of the
few areas where you can get quantitative measures. For many years
to come, it will be in a precarious state. Professor Diamond has
already alluded to databases not being complete in many areas.
Other things are in a very difficult state. For example, there
are two categories of research articles. There are original articles
that give new research findings and there are review articles
which generally speaking have much higher citation rates than
original articles. If one bases any assessment on citation, that
would be a temptation on behalf of individual scientists. They
would write more review articles rather than do the original articles
based on the grants that they have. In principle, the databases
allow you to distinguish between review articles and original
articles but all of us who have tried it out realise it does not
work. It counts certain things in one category and it does not
work in another category. There is no way a person who is not
knowledgeable in that field will be able to make that distinction.
The idea that somebody somewhere in a public database will do
it does not work. People could be influenced by aggregates if
a number of people decided to spend more time, for example, writing
review articles rather than original articles. There are a lot
of areas where we can go into the detail. I am not even sure that
the bibliometrics people are the people who would necessarily
know about it. The people who know about it are the scientists
who work in that field who use these databases. I use them a great
deal myself because I have to do a lot of assessments. I am very
much aware of the really serious problems that underlie this approach.
I would be somewhat critical of thinking that people who work
in these various science policy units really know about these
things. My feeling is that they may not know a lot about the deficiencies
in the system.
Professor Diamond: You asked what
evidence would be used and I think it is entirely right that what
the Funding Councils have done is to take evidence from the very
best bibliometricians on what can be done. That has to be tensioned
by a real analysis across the stakeholders as to how that is appropriate.
You have spoken to Professor Eastwood and I am sure he gave a
much more lucid answer than I did. I do not think it is the intention
that people, for example, bibliometricians, would simply do the
allocation of funding. It is simply one piece of evidence that
should be used in putting a basket of metrics together which would
enable us to simplify the bureaucracy somewhat and get what we
really want from a quality research base.
Q709 Helen Jones: I understand that
although I am not convinced at the moment that any of this is
simplifying bureaucracy. We have heard about problems with the
present system, for instance, that it has bias against applied
research; it does not reward interdisciplinary work and so on.
Can you give the Committee your thoughts on where this new system
will deal with those problems?
Professor Petersen: There is absolutely
no reason why the new system should be better at dealing with
this than the present system. There are a lot of arguments that
it would be exactly the opposite. The present peer review system
allows flexibility. In a sense, it is just a question of signposting
to the peer reviewers what you would like them to look at and
it can be done. If you have a formula based approach, you are
in a much more difficult situation. It is intrinsic in that system
that you cannot deal with a lot of areas that clearly are not
so well defined and cannot be defined at the moment. That is the
crux of the matter and I would urge the Committee to think very
carefully about that. By dispensing with the subject based panels,
you are entering into something very dangerous.
Professor Diamond: It is very
difficult until the Funding Councils have concluded the results
of a review, which is trying to identify the way forward and a
basket of metrics which will enable us to simplify things. Then
we can have a serious discussion about whether that has been achieved
and whether we have achieved what we are trying to achieve. An
enormous amount of evidence by the very best people is being taken
to inform that debate. The Funding Council is working very hard
to enable that debate to be taken forward. I personally would
not want to prejudge in any way the results of that. The Research
Councils will look very carefully at the results when they come
out and will engage in a full discussion of the approach. We all
know what we are trying to achieve, which is to reduce the bureaucracy
and to reward a wider range of activities in a way that we believe
has not been done thus far by the Research Assessment Exercise.
Q710 Chairman: It is six months since
we had the written submissions from yourselves. Quite a lot has
changed since then. Professor Diamond, you just mentioned a great
deal of work and expertise. If any of that has changed, we would
like to know about it and we would like to know if you have come
across witnesses that we ought to interview. We live on good information
so we need your help on that. It is six months since we announced
this inquiry and quite a lot has happened since that time.
Professor Diamond: I have looked
at the 23 people that you have met before us and I think you have
covered the bases pretty well.
Professor Petersen: I would agree
with that.
Q711 Stephen Williams: Would it be
fair to say that whatever the system of an RAE, whether it is
the existing system or a slightly different system for 2008, it
distorts the behaviour of university departments and whatever
the system they will work the system to get the best financial
outcome?
Professor Diamond: It has to be
the case that if you have a reward mechanism in place people will
try to maximise the reward they get from it. That is human nature.
That is what we should expect to happen. That is why I believe
you need to be very clear about how you will reward and recognise
different elements of a system that you wish to achieve.
Q712 Stephen Williams: Do you think
the assessment system distorts behaviour in staffing deployment?
I was talking to the head of a department in a Russell Group university
recently and he said it was something like the football transfer
market. When you are coming up to an RAE, you take in as many
good staff as you can. You are willing to pay good salaries to
get them. Once you are in the RAE period, you do not want any
new staff. You do not want to lose staff. You just want stability
and you want to be assessed on that. Is that not perverse?
Professor Diamond: That is anecdotal
evidence.
Q713 Stephen Williams: I have heard
it from more than one place.
Professor Diamond: I have heard
it in a number of places. What I have not seen is evidence which
demonstrates the major impact that that is having on the community
as a whole. Where it has been argued that there has been an impact
is where people have not been encouraged to engage in high risk
interdisciplinary research which an take time to develop. You
simply do not get sociologists who sit down with a chemist to
take forward a really exciting, new, cross-disciplinary area and
expect research to happen in five minutes. It takes time and people
have to invest a lot of time in something which may not come off.
Those sorts of behaviours are not encouraged in a system which
rewards simple disciplinary publications. In some areas people
have worked very hard to focus on particular areas of a sub-discipline
because they believe that will be rewarded properly. There is
anecdotal evidence of that too. We have seen behaviour influenced,
which has not been good for the economic development or the quality
of life of the people of this country sometimes. That is why the
Research Councils as a whole have argued that we need a system
which properly rewards applied research, research related to professional
practice, interdisciplinary research and research which really
does have an economic development impact.
Professor Petersen: The dangers
of the sort of transfer market that you refer to have been perhaps
slightly exaggerated. I do not think one should say that it has
necessarily been wholly negative. It is just another part of the
competitive world, I guess. It is not necessarily such a bad thing
that universities are forced to look after their best researchers
and make sure that they do not go elsewhere. There are negative
and positive points about the behaviour. Of course it is massively
influenced by any kind of assessment system. In our submission
we make that point very forcefully. The trick is to encourage
the right kind of behaviour and that is where we have great worries
about the new system that is based on the formula based, metrics
driven approach. The present system, for all its faults, does
signpost the essential thing to the scientists, namely to produce
good science; whereas the new system will look at secondary parameters
rather than primary ones. The change in behaviour that would happen
if one were to switch to the new system would be much for the
worst and I see no arguments anywhere that the kind of problems
that Professor Diamond highlights, that the present RAE system
does not reward, would in any way be helped by a new metrics driven
approach. It is absolutely not the case.
Q714 Stephen Williams: Would it be
fair to say that the evidence is anecdotal on the behavioural
implications of an RAE at the moment because we do not know enough?
Professor Diamond: Exactly. I
do not know of a real study which has looked at the impact that
you describe. The one thing that is very clear is that the costs
of the current RAE are immense and very rarely calculated.
Q715 Stephen Williams: Immense to
the universities themselves?
Professor Diamond: Exactly so.
Very many universities go through one or two dummy runs over a
period of time. I do not know how many universities do that. I
am told very often as I move around universities that we have
just been through a dummy run on our research assessment exercise
and we have brought in consultants to advise us on the position.
If we were to include those in the costs of the RAE, you would
see a very big increase in the overall costs. That is why trying
to reduce the overall costs through a lighter touch approach and
one informed by a basket of metrics is an extremely good and welcome
approach.
Professor Petersen: The costs
of the RAE have been somewhat exaggerated and again it relates
to a point that I made before of overlapping activities. A lot
of the work that goes on in universities in preparing for the
RAE is work that would have to be done in any case because it
is a question of how the university promotes its own research,
how it selects those areas that are valuable or not. A lot of
this is activity that at the moment comes under the RAE heading
and people say it is terribly expensive; but if the RAE did not
exist it would have to occur anyway in the university system.
Secondly—and this is a paradox—everybody agrees that
the present Research Assessment Exercise has considerably improved
the UK's research performance. Professor Diamond himself referred
to that. If we put these things together, it is not absolutely
clear what the intellectual argument is for changing it radically
into something that is totally different and is a secondary rather
than a primary parameter.
Professor Diamond: If you wish
to take out the sorts of things I have just said are the costs
of the RAE, to go back to the points that David and Fiona made
earlier. Therefore you have to say that the cost of peer review
to the Research Councils is under £10 million a year, not
the figure that we calculated, because the figure that we calculated
was the holistic cost which included all the sorts of things that
I mentioned to you, which Professor Petersen suggested we ought
not to include. That is a really good example of the point I was
trying to make. If you wish to compare like with like, you have
to be very clear and careful about what it is that you are including
in the costs.
Q716 Stephen Williams: Professor
Petersen, I was struck by your mutual citation group. That rather
implied to me that you think it would be easier for departments
to distort behaviour, to get a favourable outcome under a metric
system, than under the existing system which does have peer reviews.
Is that a fair summary of what you are saying?
Professor Petersen: Yes, that
is right. In one case you are telling people, "Show us your
best papers. They will be read by your peers and experts and they
will judge whether they are really important, new and true",
the three classical criteria for research excellence. That has
to be the gold standard. If you are replacing that with a lot
of secondary things, plainly you are not going to the heart of
the matter any more. These quality parameters can be distorted.
They will certainly influence behaviour. There is no way they
could not influence behaviour. We do not know in which direction.
We still have to be very careful. The present RAE, for all its
faults—and everybody agrees on this—has substantially
improved the UK's research performance.
Q717 Stephen Williams: I guess that
means that perhaps the Royal Society as well would lament the
loss of peer review on stem subjects. Do you think it is too late
to reargue the case that peer review should be retained?
Professor Petersen: This is an
interesting reflection of how things work in society: when we
made our original submission to the DfES consultation on the future
of the Research Assessment Exercise, we took the strong view that
peer review is essential to the process and most other organisations
agreed with us. Then the Government announced that they were not
going to do it that way. They were going to use a metric based
approach and some organisations immediately shifted and said,
"Fine. Of course this is a good idea." We do not see
any reason why we should change our opinion. This is the truth
as we see it and we have to continue to argue that. I still hope
that there is enough common sense in the system that one will
not throw out the subject based panels which have been the keystone
of assessment and which are the only ones that can assess this
basket of metrics. We all agree that you can use metrics to a
certain extent, to inform peer review, but you cannot just put
it into a formula and expect this to work.
Q718 Chairman: In a slightly throw
away line you said, "the Research Assessment Exercise, for
all its faults . . .". Have you articulated where you would
get rid of all those faults and still retain the essential RAE
package?
Professor Petersen: In our written
submission, we make the point that it has become over-cumbersome
over the years. It started out by being a much more manageable
exercise. Because of complaints about this and that and then you
introduced a new element, gradually it was built up and became
a very complex exercise. There might be something to be said for
going back to the original model which was much simpler. In essence,
what has taken all the time in the university system is to write
the narratives and one could dispense with those narratives to
a very large extent and just look at the best papers because that
is what matters. "What have you produced in that time?"
It is, as Professor Diamond started out by saying, a review of
what has been achieved. That is what we need to look at. This
waste is in the time the universities are spending, on trying
to refine those narratives in that they might marginally improve
your assessment a little bit, although in reality it is the output
that is judged. These are the faults but, at its heart, there
is something intrinsically right about the way it works because
it says—and all academics know this—"Show me
your best papers. I will read your four best papers and I will
make my judgment about what you have done".
Professor Diamond: If you do that,
you would not have the opportunity to have any proper review of
the impact on policy in government, business, local authorities,
education and schools. You would not have any proper assessment
of, for example, the development of interdisciplinary research.
You would not be making that rewarded or recognised, so there
are very many areas which would not be rewarded by a simple look
at four publications. If everything was to be done on that, you
would be rewarding one part of the system but not what the Research
Councils would argue are the wider elements of what you should
be trying to reward, particularly having, where appropriate, research
having an impact on the economic development and quality of life
of the people who pay for it.
Professor Petersen: These are
exactly the areas that are most difficult to deal with on a metrics
based approach.
Q719 Jeff Ennis: One of the pieces
of evidence we have received is a paper from a gentleman called
Tom Sastry entitled A Dangerous Economy: The wider implications
of the proposed reforms to the UK's Research Councils' peer review
system. One of the conclusions that Mr Sastry comes to is:
"If the de facto roles of the Funding Councils and
the Research Councils continue to converge, it will be increasingly
difficult for the Research Councils, as the more expensive arm
of dual support, to justify their role in funding research in
universities. The Research Councils must either find a better
means of reducing costs which does not undermine the distinctiveness
of their role and it is not immediately apparent how they might
do this; or focus upon doing things which the Funding Councils
cannot do. The latter course implies that the Research Councils
should focus upon strategic themes which reflect genuine political
and public priorities, rather than replicate the purpose of the
Funding Councils." Do you concur with that conclusion, Professor
Diamond?
Professor Diamond: I might find
a few areas in the quote that you have just come out with that
I might not quite agree with. If I recollect the overall piece
that you are referring to, it does take as its starting point
that the cost is £190 million. That is a holistic cost. We
need to sit down and compare apples with apples, not apples with
pears if we are going to talk about costs. I might argue that
if we are going to take Professor Petersen's suggestion and take
a lot of things out, the cost of the Research Councils' peer review
is really not that great in terms of direct administrative cost.[11]
The one thing that I would add to that is the absolute, overwhelming
response that we received to our consultation with every higher
education institution and every major stakeholder, that UK peer
review is amongst the best—some people say the best—in
the world and it is the right way to allocate competitive research
funding. I have tried already to give some examples of why you
would want to do that in basic research as well, right across
the piece in other areas. I am happy to revisit some of those
areas but fundamentally you will not have an institute of a particular
type of sociology in one place. You will want to have people from
across the piece and internationally looking at competitive funding
and I believe basic and directed research need to be funded externally,
independently and in a quality way by the Research Councils. I
think we provide an enormous service to the research base in this
country for a very small direct cost. Indeed, it is one that we
work very hard to drive down the whole time. Secondly, I think
you have to recognise what the Research Councils do in funding.
One thing that they do is to fund a response mode, basic curiosity
driven research and that is entirely right and proper. The second
things that the Research Councils do is, where the market is failing
through a really proper analysis of key areas, particularly some
new areas, particularly areas where the Research Councils have
to work together—
10 Note by witness: For time spent on writing
grant proposals Back
11
Note by witness: i.e. less than £10 million Back
|
