Select Committee on Science and Technology Appendices to the Minutes of Evidence


Memorandum submitted by Professor Paul G Hare, School of Management, Heriot-Watt University, Edinburgh

  1.  In the last decade there have been three Research Assessment Exercises carried out across the UK university system, namely in 1992, 1996 and most recently, 2001. The results of RAE2001 were published in mid-December 2001.

  2.  In each RAE, Units of Assessment (often, but not necessarily, corresponding to an individual academic department) are rated on a 7-point scale: 5*, 5, 4, 3a, 3b, 2, 1 (running from best to worst). The results of RAE2001 showed significant increases in the proportions of Units of Assessment (UoAs) awarded grades of 5*, 5, and 4.

  3.  This raises three questions which I examine more fully below: (a) To what extent do the improvements in ratings represent a real improvement in the quality of research carried out in UK universities? (b) What are or should be the implications of the new ratings for the research component of the institutional grants awarded to universities by the Funding Councils? (c) Is the RAE system a good way of assessing the research carried out in UK universities and hence allocating part of the institutional grant?

Research quality

  4.  For a given UoA, institutions have always had a choice between submitting to the RAE a few relatively high profile researchers in the hope of getting a high rating (accompanied by a low volume measure), and submitting more people and accepting a lower rating. Given the funding associated with particular ratings, this trade off must have encouraged at least some strategic behaviour by particular UoAs. For instance, both direct funding and staff recruitment (and possibly even student recruitment) are likely to be enhanced by a higher research rating, so it can sometimes be worth an institution's while to submit fewer research-active staff then they could in order to get an improved rating. How widespread this practice has been I have no idea, but the incentives for such behaviour are strong and obvious.

  5.  On the other hand, across the UK university system as a whole the number of research active staff included in the latest RAE has gone up. Overall, therefore, despite the possible "distortions" resulting from the last paragraph, the volume of research has gone up (as measured by number of researchers) as has its average measured quality (as reflected in average RAE scores). Hence there appears to be strong evidence for an improvement in research performance by UK universities since 1996. At least part of this improvement might have been due to the incentives created by the RAE system (see below, para. 9).

Institutional grants

  6.  Prior to RAE 2001, institutions knew how much research funding their RAE1996 research ratings won for them from the Funding Councils. The relevant formulae were published and therefore widely known. In advance of RAE2001, there were hints that ratings of three and under would attract no research funding at all, and this certainly concentrated minds in institutions and departments I am aware of, encouraging staff to present their research achievements in such a way as to maximise their chances of getting the coveted four or better. Now that the exercise has been completed, HEFCE has already announced that the unit of resource for five* departments will be protected, that there will in fact be "some" funding for three-rated departments, and that the total available research funding will not be much changed in real terms. The logical implication must be that departments with four and five ratings will lose out, as compared to what they would have received after 1996. Nothing has yet been announced by SHEFC, but I imagine the picture will look fairly similar up here.

  7.  Can anything be said about the most desirable/efficient relationships between the rewards going to UoAs with different research ratings? For example, should a four-rated UoA get twice as much per active researcher from the Funding Councils as a 3a-rated one? Or should the ratio be five or 10? Or perhaps everything should go to the five*s and nothing to anyone else? As far as I am aware, there is no general theory of such matters, though there is some literature in economics that highlights the relevant considerations—the need to spread the rewards in order to attract a lot of departments/institutions to enter the research "competition" and take it seriously, vs. the need for high rewards to stimulate maximum efforts on the part of those who do enter. Finding the right balance is not easy.

The RAE system

  8.  The basic idea underlying the RAE system is that it should provide a reasonably objective, periodic measure of research output and quality by UoA and by institution that can therefore serve as a basis for allocating the research component in institutional grants from the Funding Councils. Here I consider how else one might allocate research resources, and remark on some of the shortcomings of the RAE itself.

Allocation of research resources

  9.  Lots of methods could be proposed, and in every case one has to be aware of the resource costs of operating whatever method is used. For brevity, I merely list some possible methods of allocation, with at most short comments on advantages and disavantages:

    —  Random, possibly scaled to a simple measure of institutional size. This is cheap and easy to operate. Might be perceived as unfair since it makes no attempt to relate allocation to any measure of research effort or performance.

    —  Uniform, research grant proportional to number of academic staff, possibly varied by faculty or broadly defined subject group. This is also cheap and easy to operate, related more to research effort, and so likely to be perceived as fairer than the random allocation.

    —  Uniform, proportional to value of research contracts won in a given period. Again, cheap and easy to operate, rewards success, may be perceived as penalising theoretical research that often attracts little funding, or research in the humanities which also often needs little funding. Can also be criticised for rewarding inputs to research (funding) rather than outputs (new knowledge)

    —  RAE-type system. Seeks to measure research output and assess its quality in a highly disaggregated manner across institutions. Costly to carry out the assessment. Only worthwhile if the improvements in resource allocation (as compared, say, to the second or third item above) more than offset the periodic costs. I have not seen it convincingly argued or demonstrated that the costs of the RAE are indeed offset by the resulting improvements in resource allocation in UK universities.

Shortcomings of the RAE

  10.  If we insist on measuring and evaluating the research output and quality in UK universities, then something like the RAE as currently practised is probably about as good a system as one could get in terms of fairness, objectivity, etc. However, it clearly has some significant deficiencies, which I merely list:

    —  Incentives, as indicated above (para. 6), to report fewer staff in order to get a higher research rating;

    —  Incentives to conduct research in such a way as to generate more, quicker publications than might otherwise be the case—so that individuals can meet the numbers requirement of the RAE (publishing at least four items in a given period);

    —  Incentives favouring shorter term research as against more fundamental research with long gestation periods (there is a difficult and unresolved management issue here—how to distinguish between brilliant researchers who need time to develop their next big idea, and those who are simply lazy and unproductive);

    —  The fact that the RAE measures number of publications ensures that individuals and institutions will respond by delivering numbers. But no one really knows how to value research from a social point of view;

    —  The problem that for many UoAs with relatively few research-active staff, their RAE results are likely to depend on a few key individuals (given the usual very skewed distribution of research output across a department). Hence institutions have incentives to boost their research performance by bidding for the services of key staff—giving rise to an academic transfer market. For the UK as a whole, this practice is clearly inefficient since it merely redistributes research output around the system, but for an individual institution it can prove highly rewarding.

  11.  At best the RAE only rewards institutions for employing productive individuals, but it does nothing to reward the individuals themselves. The large relative decline in UK academic salaries over the past 20 years, and the very limited rewards for the highest levels of performance, will gradually result in a deterioration in UK universities' research performance in the next decade or two (despite the numbers game represented by the RAE) through two effects: (a) loss of high quality staff to overseas universities that are better funded than ours; (b) inability to recruit top quality new people into the lower levels of UK academia. Both these effects operate slowly, but both will become serious before very long, with the result that many UK universities will increasingly become low to medium-grade teaching institutions. This tendency will be exacerbated by increasingly burdensome systems of quality assurance/performance indicators.

  12.  Other OECD countries do not, on the whole, engage in exercises such as the RAE in order to allocate public money to their universities. Some countries fund the basic teaching function of their institutions through a mix of student fees and block grant (the block grant being sufficient to provide some basic funding for research), then fund the research project-by-project on the basis of competitive applications for funding. Others acknowledge that their higher education system needs to include different types of institution (a situation that we pretend to deny, following the mis-conceived 1992 reforms of HE in the UK), some specialising in teaching and funded accordingly; others with a high research profile funded more generously, again with the possibility of raising additional funds through foundations and competitive application for public funds.

  13.  Against this background of international experience, how does our RAE fit into the picture? It is actually hard to view the exercise as other than a bureaucratic device for allocating research funds to institutions when everyone knows that there is really not enough money to do the job properly. Institutions in the UK are highly constrained in various ways that prevent them from earning additional money (eg student number targets that stop successful institutions from recruiting more students; legal prohibitions on raising fee levels; etc.) to fund themselves better, but they are still expected to perform at internationally competitive levels in research. The RAE allocates a small amount of research money in a highly selective manner, rewarding measured research success. Since all UK higher education institutions are seriously under-funded, and subject to far too many constraints that prevent them from doing better, everyone naturally scrambles as hard as they can to get a share of the modest "RAE-pie". But it cannot be claimed that this system is a very clever way of allocating public money. While it has certainly led to increases in research output as measured for RAE purposes, there is little reason to believe that it will have stimulated, over the past decade, a real increase in research output of significant social value.

  14.  I should add here that I strongly reject the frequent attempts by public bodies in the UK to demand that research in our universities should contribute directly to UK competitiveness or economic performance more generally. While indirectly, good research will eventually do this, the linkages between research and the economy are necessarily long term, subtle, often unexpected, and not subject to planning and direction. For instance, how many people would have predicted that rather arcane research in number theory would now form the basis for modern encryption systems of huge economic importance? How many people anticipated the collapse of communism and the need for studies of transition economies prior to 1990—not many, but luckily the UK had kept alive a few centres, and a few individuals like myself had already worked for decades on these then "unfashionable" and "unimportant" countries.

  15.  Moreover, the world of research is indeed the world. The UK economy would be unwise to expect to draw all the expertise it needs from UK higher education, and our firms will naturally cast the net far more widely. By the same token, UK universities themselves will form partnerships not only with other UK institutions but with partners in many other countries, and the results of our research will and should flow wherever there is a market for our ideas. Research by our universities can be a major export.

Policy recommendation

  16.  How should the UK government proceed with the funding of research in our universities? I see two main options, one based on a continuation of the current limited research funding plus continuing restrictions on university activities (Option A); the other based on a more liberal environment accompanied by more generous government funding of higher education (Option B). My personal view is that Option B is strongly to be preferred, but my judgement is that the government will lack the courage to be so radical, and will therefore elect to pursue Option A.

Option A—Continuing low funding for research plus other restrictions

  17.  In this situation we might as well continue to allocate research funds selectively through the RAE or some similar mechanism, despite its shortcomings as noted above. On this model, it is predictable that UK higher education will continue its slow decline, with institutions finding it ever harder to attract and retain high quality staff. Regardless of the formal policy, institutions will again become increasingly differentiated, with a small elite of research-intensive institutions, a long tail of predominantly teaching institutions. This is a viable model, but within a decade or two it will ensure that the UK is no longer a leading player in world research in most areas of fundamental importance.

Option B—A more liberal environment plus more generous public funding for research

  18.  This option acknowledges that quality is costly and that in the long run sustained world class research performance will only come from recruiting top quality staff and paying them well. Everything else that we would like to see from the UK's research establishment will then follow. In terms of public policy, this approach would entail a number of steps, which could be pursued in sequence or in a number of different combinations:

    —  End restrictions on undergraduate student recruitment (approved funded number targets, etc.);

    —  End efforts to designate certain subject as priorities (present practice is no better than old fashioned manpower planning, and works just as badly—market signals tell us very well how different skills are rewarded in our economy, and as soon as engineer salaries double we shall get a lot more of them);

    —  Allow universities to charge higher fees as and when they wish to do so—I expect institutions can judge their market better than government can;

    —  Allow universities to depart from existing national academic pay scales as and when they wish—to enable them to recruit and retain top quality staff;

    —  Pay universities a block grant that funds—more generously than at present (so including an element for research)—their core teaching function;

    —  Abandon the RAE system.

8 January 2002

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2002
Prepared 24 April 2002