APPENDIX 10
Memorandum submitted by Professor Paul
G Hare, School of Management, Heriot-Watt University, Edinburgh
1. In the last decade there have been three
Research Assessment Exercises carried out across the UK university
system, namely in 1992, 1996 and most recently, 2001. The results
of RAE2001 were published in mid-December 2001.
2. In each RAE, Units of Assessment (often,
but not necessarily, corresponding to an individual academic department)
are rated on a 7-point scale: 5*, 5, 4, 3a, 3b, 2, 1 (running
from best to worst). The results of RAE2001 showed significant
increases in the proportions of Units of Assessment (UoAs) awarded
grades of 5*, 5, and 4.
3. This raises three questions which I examine
more fully below: (a) To what extent do the improvements in ratings
represent a real improvement in the quality of research carried
out in UK universities? (b) What are or should be the implications
of the new ratings for the research component of the institutional
grants awarded to universities by the Funding Councils? (c) Is
the RAE system a good way of assessing the research carried out
in UK universities and hence allocating part of the institutional
grant?
Research quality
4. For a given UoA, institutions have always
had a choice between submitting to the RAE a few relatively high
profile researchers in the hope of getting a high rating (accompanied
by a low volume measure), and submitting more people and accepting
a lower rating. Given the funding associated with particular ratings,
this trade off must have encouraged at least some strategic behaviour
by particular UoAs. For instance, both direct funding and staff
recruitment (and possibly even student recruitment) are likely
to be enhanced by a higher research rating, so it can sometimes
be worth an institution's while to submit fewer research-active
staff then they could in order to get an improved rating. How
widespread this practice has been I have no idea, but the incentives
for such behaviour are strong and obvious.
5. On the other hand, across the UK university
system as a whole the number of research active staff included
in the latest RAE has gone up. Overall, therefore, despite the
possible "distortions" resulting from the last paragraph,
the volume of research has gone up (as measured by number of researchers)
as has its average measured quality (as reflected in average RAE
scores). Hence there appears to be strong evidence for an improvement
in research performance by UK universities since 1996. At least
part of this improvement might have been due to the incentives
created by the RAE system (see below, para. 9).
Institutional grants
6. Prior to RAE 2001, institutions knew
how much research funding their RAE1996 research ratings won for
them from the Funding Councils. The relevant formulae were published
and therefore widely known. In advance of RAE2001, there were
hints that ratings of three and under would attract no research
funding at all, and this certainly concentrated minds in institutions
and departments I am aware of, encouraging staff to present their
research achievements in such a way as to maximise their chances
of getting the coveted four or better. Now that the exercise has
been completed, HEFCE has already announced that the unit of resource
for five* departments will be protected, that there will in fact
be "some" funding for three-rated departments, and that
the total available research funding will not be much changed
in real terms. The logical implication must be that departments
with four and five ratings will lose out, as compared to what
they would have received after 1996. Nothing has yet been announced
by SHEFC, but I imagine the picture will look fairly similar up
here.
7. Can anything be said about the most desirable/efficient
relationships between the rewards going to UoAs with different
research ratings? For example, should a four-rated UoA get twice
as much per active researcher from the Funding Councils as a 3a-rated
one? Or should the ratio be five or 10? Or perhaps everything
should go to the five*s and nothing to anyone else? As far as
I am aware, there is no general theory of such matters, though
there is some literature in economics that highlights the relevant
considerations—the need to spread the rewards in order to
attract a lot of departments/institutions to enter the research
"competition" and take it seriously, vs. the need for
high rewards to stimulate maximum efforts on the part of
those who do enter. Finding the right balance is not easy.
The RAE system
8. The basic idea underlying the RAE system
is that it should provide a reasonably objective, periodic measure
of research output and quality by UoA and by institution that
can therefore serve as a basis for allocating the research component
in institutional grants from the Funding Councils. Here I consider
how else one might allocate research resources, and remark on
some of the shortcomings of the RAE itself.
Allocation of research resources
9. Lots of methods could be proposed, and
in every case one has to be aware of the resource costs of operating
whatever method is used. For brevity, I merely list some possible
methods of allocation, with at most short comments on advantages
and disavantages:
— Random, possibly scaled to a simple
measure of institutional size. This is cheap and easy to operate.
Might be perceived as unfair since it makes no attempt to relate
allocation to any measure of research effort or performance.
— Uniform, research grant proportional
to number of academic staff, possibly varied by faculty or broadly
defined subject group. This is also cheap and easy to operate,
related more to research effort, and so likely to be perceived
as fairer than the random allocation.
— Uniform, proportional to value of
research contracts won in a given period. Again, cheap and easy
to operate, rewards success, may be perceived as penalising theoretical
research that often attracts little funding, or research in the
humanities which also often needs little funding. Can also be
criticised for rewarding inputs to research (funding) rather than
outputs (new knowledge)
— RAE-type system. Seeks to measure
research output and assess its quality in a highly disaggregated
manner across institutions. Costly to carry out the assessment.
Only worthwhile if the improvements in resource allocation (as
compared, say, to the second or third item above) more than offset
the periodic costs. I have not seen it convincingly argued or
demonstrated that the costs of the RAE are indeed offset by the
resulting improvements in resource allocation in UK universities.
Shortcomings of the RAE
10. If we insist on measuring and evaluating
the research output and quality in UK universities, then something
like the RAE as currently practised is probably about as good
a system as one could get in terms of fairness, objectivity, etc.
However, it clearly has some significant deficiencies, which I
merely list:
— Incentives, as indicated above (para.
6), to report fewer staff in order to get a higher research rating;
— Incentives to conduct research in
such a way as to generate more, quicker publications than might
otherwise be the case—so that individuals can meet the numbers
requirement of the RAE (publishing at least four items in a given
period);
— Incentives favouring shorter term
research as against more fundamental research with long gestation
periods (there is a difficult and unresolved management issue
here—how to distinguish between brilliant researchers who
need time to develop their next big idea, and those who are simply
lazy and unproductive);
— The fact that the RAE measures number
of publications ensures that individuals and institutions will
respond by delivering numbers. But no one really knows how to
value research from a social point of view;
— The problem that for many UoAs with
relatively few research-active staff, their RAE results are likely
to depend on a few key individuals (given the usual very skewed
distribution of research output across a department). Hence institutions
have incentives to boost their research performance by bidding
for the services of key staff—giving rise to an academic
transfer market. For the UK as a whole, this practice is clearly
inefficient since it merely redistributes research output around
the system, but for an individual institution it can prove highly
rewarding.
11. At best the RAE only rewards institutions
for employing productive individuals, but it does nothing to reward
the individuals themselves. The large relative decline in UK academic
salaries over the past 20 years, and the very limited rewards
for the highest levels of performance, will gradually result in
a deterioration in UK universities' research performance in the
next decade or two (despite the numbers game represented by the
RAE) through two effects: (a) loss of high quality staff to overseas
universities that are better funded than ours; (b) inability to
recruit top quality new people into the lower levels of UK academia.
Both these effects operate slowly, but both will become serious
before very long, with the result that many UK universities will
increasingly become low to medium-grade teaching institutions.
This tendency will be exacerbated by increasingly burdensome systems
of quality assurance/performance indicators.
12. Other OECD countries do not, on the
whole, engage in exercises such as the RAE in order to allocate
public money to their universities. Some countries fund the basic
teaching function of their institutions through a mix of student
fees and block grant (the block grant being sufficient to provide
some basic funding for research), then fund the research project-by-project
on the basis of competitive applications for funding. Others acknowledge
that their higher education system needs to include different
types of institution (a situation that we pretend to deny, following
the mis-conceived 1992 reforms of HE in the UK), some specialising
in teaching and funded accordingly; others with a high research
profile funded more generously, again with the possibility of
raising additional funds through foundations and competitive application
for public funds.
13. Against this background of international
experience, how does our RAE fit into the picture? It is actually
hard to view the exercise as other than a bureaucratic device
for allocating research funds to institutions when everyone knows
that there is really not enough money to do the job properly.
Institutions in the UK are highly constrained in various ways
that prevent them from earning additional money (eg student number
targets that stop successful institutions from recruiting more
students; legal prohibitions on raising fee levels; etc.) to fund
themselves better, but they are still expected to perform at internationally
competitive levels in research. The RAE allocates a small amount
of research money in a highly selective manner, rewarding measured
research success. Since all UK higher education institutions are
seriously under-funded, and subject to far too many constraints
that prevent them from doing better, everyone naturally scrambles
as hard as they can to get a share of the modest "RAE-pie".
But it cannot be claimed that this system is a very clever way
of allocating public money. While it has certainly led to increases
in research output as measured for RAE purposes, there
is little reason to believe that it will have stimulated, over
the past decade, a real increase in research output of significant
social value.
14. I should add here that I strongly reject
the frequent attempts by public bodies in the UK to demand that
research in our universities should contribute directly to UK
competitiveness or economic performance more generally. While
indirectly, good research will eventually do this, the linkages
between research and the economy are necessarily long term, subtle,
often unexpected, and not subject to planning and direction. For
instance, how many people would have predicted that rather arcane
research in number theory would now form the basis for modern
encryption systems of huge economic importance? How many people
anticipated the collapse of communism and the need for studies
of transition economies prior to 1990—not many, but luckily
the UK had kept alive a few centres, and a few individuals like
myself had already worked for decades on these then "unfashionable"
and "unimportant" countries.
15. Moreover, the world of research is indeed
the world. The UK economy would be unwise to expect to draw all
the expertise it needs from UK higher education, and our firms
will naturally cast the net far more widely. By the same token,
UK universities themselves will form partnerships not only with
other UK institutions but with partners in many other countries,
and the results of our research will and should flow wherever
there is a market for our ideas. Research by our universities
can be a major export.
Policy recommendation
16. How should the UK government proceed
with the funding of research in our universities? I see two main
options, one based on a continuation of the current limited research
funding plus continuing restrictions on university activities
(Option A); the other based on a more liberal environment accompanied
by more generous government funding of higher education (Option
B). My personal view is that Option B is strongly to be preferred,
but my judgement is that the government will lack the courage
to be so radical, and will therefore elect to pursue Option A.
Option A—Continuing low funding for research
plus other restrictions
17. In this situation we might as well continue
to allocate research funds selectively through the RAE or some
similar mechanism, despite its shortcomings as noted above. On
this model, it is predictable that UK higher education will continue
its slow decline, with institutions finding it ever harder to
attract and retain high quality staff. Regardless of the formal
policy, institutions will again become increasingly differentiated,
with a small elite of research-intensive institutions, a long
tail of predominantly teaching institutions. This is a viable
model, but within a decade or two it will ensure that the UK is
no longer a leading player in world research in most areas of
fundamental importance.
Option B—A more liberal environment plus
more generous public funding for research
18. This option acknowledges that quality
is costly and that in the long run sustained world class research
performance will only come from recruiting top quality staff and
paying them well. Everything else that we would like to see from
the UK's research establishment will then follow. In terms of
public policy, this approach would entail a number of steps, which
could be pursued in sequence or in a number of different combinations:
— End restrictions on undergraduate
student recruitment (approved funded number targets, etc.);
— End efforts to designate certain
subject as priorities (present practice is no better than old
fashioned manpower planning, and works just as badly—market
signals tell us very well how different skills are rewarded in
our economy, and as soon as engineer salaries double we shall
get a lot more of them);
— Allow universities to charge higher
fees as and when they wish to do so—I expect institutions
can judge their market better than government can;
— Allow universities to depart from
existing national academic pay scales as and when they wish—to
enable them to recruit and retain top quality staff;
— Pay universities a block grant that
funds—more generously than at present (so including an element
for research)—their core teaching function;
— Abandon the RAE system.
8 January 2002
|
