House of Commons - Science and Technology

Select Committee on Science and Technology First Report

CHAPTER FOUR: THE RESEARCH ASSESSMENT EXERCISE AND SELECTIVITY

60. We have already discussed the basis on which the Funding Bodies allocate research funds to universities and the use they make of the RAE to determine research quality (see para 14). The RAE was first introduced in 1986 and there have been three subsequent exercises, the most recent in 1996. The RAE assesses the work of groups of researchers across a range of topics which are divided into "units of assessment". All research submitted to the RAE is judged by assessment panels. In the first three RAEs, submissions were ranked according to a five point scale but in the 1996 RAE a seven point scale was used. (The rating scales used in the 1996 RAE are reproduced in an annex.) The 1996 exercise attracted a total of 2,896 submissions from 192 higher education institutions and 60 assessment panels covered 69 areas of assessment. Each assessment panel works to the same broadly-based criteria but each panel is at liberty to choose its own particular methods.

61. The Dearing Report highlighted particular criticisms that have been levelled at the operation of the RAE and the way in which it dictates the behaviour of institutions, including:

that it fails to assess collaborative research well, particularly that undertaken in collaboration with industry;[149]

that it acts as a disincentive to multi-disciplinary research;[150]

that it is not the best way of identifying research excellence as it assesses departments rather than researchers and research teams;[151] and

that it encourages all departments to develop, or maintain, a research capability even when such a capability may not be appropriate or desirable and that, consequently, it wastes resources by encouraging submissions with little or no chance of success and detracts from teaching.[152]

62. Our inquiry has revealed a number of further concerns regarding the operation of the RAE, chiefly:

that it places too great an emphasis on publication as a measure of research quality and not enough on judging the usefulness of research;[153]

that it promotes short-termism in research planning;[154]

that it places too great an administrative burden on researchers and institutions and is not cost effective;[155] and

that it does not apply equal standards across all disciplines.[156]

We also heard complaints both that the RAE was not selective enough in its support for research and, conversely that there was no room for further selectivity without causing long-term damage to the research base.[157] We discuss each of these below.

COLLABORATIVE AND MULTI-DISCIPLINARY RESEARCH

63. The NCIHE acknowledged that many in the research community were concerned that the RAE discouraged collaborative research but did not "come across any convincing evidence that collaboration was genuinely inhibited to a serious extent by the exercise".[158] However the NCIHE did go on to recommend to the HEFCs and the Research Councils that "they review their mainstream teaching and research funding arrangements to ensure these do not discourage collaboration between institutions; and that, where appropriate, they encourage collaboration".[159]

64. A 1995 study by the SPRU found that the number of collaborative research projects was increasing and projected that by 2000, at current growth rates, collaborations between institutions would account for more than 50% of the UK's scientific output.[160] However, as the National Academies Policy Advisory Group noted in their 1996 review of The Research Capability of the University System, there is a real conflict between, on the one hand, the recognition of and the support for collaborative research between institutions and, on the other, the "application of the ... RAE in which universities and their departments compete strongly against each other".[161] Many witnesses to our inquiry agreed. The ABPI told us that, by assigning points to individual departments and individual universities, the RAE does not encourage and "certainly inhibits industry collaborations".[162] SmithKline Beecham went further, telling us that "there are penalties for collaborative work with other departments, other universities and industry",[163] a position supported by the AUT.[164]

65. The NCIHE acknowledged the importance of collaboration-"Collaboration matters. It may, in some cases, make the difference between institutional success and failure"[165]-in terms of institutional management and overall efficiency but failed to recognise its fundamental importance for research. As the Natural Environment Research Council (NERC) pointed out "many of the most important innovative advances in science occur at the interfaces between disciplines and involve collaborative research".[166] Moreover, there are numerous examples of successful collaborative research operations which suggest the potential for similar developments elsewhere. For instance "White Rose Research", a joint initiative between the universities of Leeds, Sheffield and York, secured EPSRC funding to establish a Faraday Centre to conduct research and development in the field of packaging in collaboration with a number of industrial partners; DTI support for a bio-technology consortium; and DfEE support for a collaborative scheme training PhD students. This alliance is also pursuing the possibility of collective equipment procurement, sharing facilities and a joint investment fund for the exploitation of research. It is, therefore, of the utmost importance that the RAE should reward research undertaken in collaboration at least equally with that done by a single department or institution. We are, therefore, disappointed that the Dearing Report made no specific recommendations about funding for collaborative research and that the consultation exercise launched last November by the HEFCs on the operation of 1996 RAE made no specific reference to collaborative research.

66. Similar concerns have been raised about the way in which the RAE assesses inter-disciplinary or, perhaps more appropriately, multi-disciplinary research. The NCIHE gave more attention to this area, recognising that such concerns are widespread and that, despite attempts at improvement in the last RAE, there was little confidence that research spanning the boundaries of traditional disciplines had been adequately recognised. Indeed, Professor May of the Royal Academy of Engineering described the problem vividly:

"If I can quote an area ... that of sensors technology, one is talking about technologies from perhaps some six or seven different subject areas being applied in perhaps a dozen or more. At the moment the Research Assessment Exercise requires that to be reported separately in chemistry, in physics, in electronics, in computer studies and so on. It is virtually impossible to find a mechanism at the moment for presenting the sum total of that research and having it assessed."

Such is the importance of HEFC funding to the financial viability of research activities that the prospect, or indeed even a concern, that multi-disciplinary research will not produce the same level of reward from the RAE as single discipline research of the same standard, must discourage multi-disciplinary projects.

67. The Dearing Report suggested that multi-disciplinary research projects were also discriminated against by the way in which the Research Councils allocated their funding.[167] There is less evidence to support this claim and it was hotly disputed by some of the Research Councils, and other witnesses, in their evidence to us. The BBSRC told us that "the Research Councils' commitment to inter-disciplinarity research is not sufficiently acknowledged" in the Dearing Report and NERC said that the majority of research they supported was multi-disciplinary.[168] Multi-disciplinary research, too, is key to research success and innovation, particularly in the sciences. As the Royal Institution told us "it is a truism to say that the most potent sources of novelty in science lie at the interfaces between mature disciplines".[169] Therefore it is essential that we can be confident that none of the funding mechanisms used to support research discriminates against multi-disciplinary research.

68. The NCIHE's response to these concerns about the funding of inter-disciplinary and multi-disciplinary research was to recommend that the Funding Bodies and the Research Councils commission an external study (the NCIHE suggest by the Royal Society) to evaluate their funding mechanisms.[170] We are concerned that such a study would not produce any worthwhile results. As the MRC said "A major review of interdisciplinarity would be difficult ... the issues are complex and vary between areas, and perceptions of what constitutes interdisciplinarity vary widely".[171] We believe that it would be more appropriate, and more likely to produce workable results, if the HEFCs and the Research Councils together brought forward proposals for improving their assessments of multi-disciplinary research and then consulted widely, thereby by utilising many different sources of expertise. Moreover, any such exercise should have collaborative research specifically within its remit. We recommend that the Funding Bodies and the Research Councils review all their funding procedures, including but not exclusively the RAE and peer review, to identify any areas where collaborative, inter-disciplinary and multi-disciplinary research may be disadvantaged and then put forward, and consult widely on, options for change. Results from the current consultation on the RAE should be used to inform, but not replace, parts of that review.

THE BALANCE BETWEEN TEACHING AND RESEARCH

69. The NCIHE found that

"Although the teaching quality assessments ... carried out by the Funding Bodies, which are designed to measure the effectiveness of teaching, have raised the profile of teaching within institutions, the Research Assessment Exercise ... has been a stronger influence and has deflected attention away from learning and teaching towards research. An analysis of the 1992 RAE in higher education institutions in England suggests that it has devalued teaching because research assessment is closely linked to the allocation of large sums of money, whereas teaching assessment is not. The fact that almost every higher education institution in the country entered the exercise-regardless of whether its primary mission was to research or to teach-indicates the influence of the RAE on institutions' activities".[172]

Witness after witness agreed. For instance the Institution of Electrical Engineers told us that the RAE had resulted in a "dash for research", distorting university priorities and diminishing teaching.[173] The Medical Clinical Academic Committee of the BMA explained to us clearly how this had happened when we met them informally: the RAE rewards departments for good research in terms of increased funding but there is no equivalent reward for good teaching. They also told us that it was almost impossible for staff to gain promotion solely on the basis of good teaching. Departments are, therefore, tempted to concentrate more on their research activities, or take up research activities, and consequently to devote fewer resources and less attention to teaching. The HEFCE pointed out that there was also "a tendency for academics ... to value and esteem research more highly than other activities" which reinforced the financial incentives to concentrate on research.[174] The Medical Clinical Academic Committee were also concerned that, in their discipline at least, there was little or no prospect of promotion on the strength of good teaching which, again, might serve to focus attention on research in preference to teaching. In addition, as the NCIHE acknowledged, some departments are entering the RAE with little or no prospect of success, and thus the resources of both the Funding Bodies and the institutions are being wasted.[175] We agree with the NCIHE that "this situation serves neither research, nor teaching well".[176]

70. The NCIHE made several recommendations designed to enhance the status of teaching which we are not in a position to comment on as they have not formed part of our inquiry. However, the NCIHE also proposed the creation of a second non-competitive, per capita, funding stream from the Funding Bodies which would be open to those engaged in research in support of teaching who opted out of the RAE.[177] The NCIHE's intention, in proposing such a fund, was to make it attractive for departments who felt their main strength was in teaching to opt out of the RAE but, recognising the importance of research to high quality teaching, still receive funding for "private research and scholarship in support of their teaching".[178] The funding for this stream would, the NCIHE envisaged, be released by not funding any of those departments scoring 1, 2 or 3-b in the RAE. This would provide some £30 million per annum, at current rates, for the per capita fund which the NCIHE proposed should be allocated on the basis of at least £500 per eligible staff member. The per capita funding would not be available to departments entering the RAE except, in order not to discourage departments with a developing strength in research, those scoring a high 3-b.

71. While most witnesses welcomed the emphasis that the NCIHE placed on the need to ensure that teaching quality was not undermined, the mechanism proposed found little favour. Fundamental concerns, which we share, were expressed about the implied distinction between research and scholarship undertaken in support of teaching[179] although other witnesses welcomed it.[180] When we questioned Lord Dearing over his distinction he told us:

"research is committed to finding out things that are not at present known whereas scholarship is concerned with the lecturer being able to keep up with the leading edge of knowledge of his or her subject ... looking at it critically and creatively, but not trying to establish new knowledge. It is the foundation of research but also the foundation of excellence in teaching ... it is a continuum".[181]

We see scholarship as an integral part of both teaching and research at every level. Moreover, scholarship in the terms that Lord Dearing has defined it-as underpinning high quality teaching-should be regarded as a normal professional requirement for all university teachers, not just those not involved in leading edge research. We do not wish to become heavily involved in a debate over semantics but we have grave reservations about the principle and practicality of any funding mechanism that relies on a distinction between scholarship and research.

72. Some witnesses also expressed concerns about the proposed method of operation for the per capita fund itself. The Baroness Blackstone had reservations about withdrawing funding from departments scoring 3-b, that is those achieving a rating of national excellence in a majority of sub-areas, in the RAE: "we also have to remember that the decision to allocate a department into a 3a or a 3b is often a very narrow one and it is often based on quite subjective judgements".[182] She also pointed out, as did other witnesses, that funding 3-b departments provided support for emerging research talent which would be lost under the NCIHE's scheme.[183] SBS argued that "the idea that you can allow a member of teaching staff to be aware of developments in their subject at the highest level for £500 a year, which to a garage represents 10 hours of a mechanic's time, is a nonsense".[184] What is perhaps the most convincing argument against the scheme was put by the HEFCW who warned us of a very real, if more insidious, risk: that it could result in a department receiving "scholarship" funding that would have scored 1 in the RAE while another department that scored 3-b would receive no funding at all, thus undermining the concept of rewarding research excellence wherever it is found.[185]

73. Witnesses such as NERC and the UK Life Sciences Committee also questioned whether it was appropriate to fund activities which should be an inherent part of teaching from the research budget.[186] We share all these sentiments; as the Baroness Blackstone told us "we should be funding teaching properly in our universities".[187]

74. We strongly endorse the conclusions of the NCIHE that university departments must make strategic assessments of their research activities and that departments whose strengths lie in teaching should not be inhibited from pursuing teaching excellence; but we cannot support its proposals for a per capita fund which are in principle wrong and in practice would be ineffective. The best way to achieve an appropriate balance between teaching and research is to enhance the status of, and reward for, good teaching rather than to undermine the pursuit of research excellence.

149 The Dearing Report, para 11.71. Back

150 The Dearing Report, para 11.98. Back

151 The Dearing Report, para 11.22. Back

152 The Dearing Report, para 11.63. Back

153 Q. 187. Back

154 Ev.p. 206. Back

155 Ev.p. 17. Back

156 Ev.pp. 174 and 204-5. Back

157 Association of University Teachers, paras 5-7. Ev.pp.86 and 178. Back

158 The Dearing Report, para 11.97. Back

159 The Dearing Report, Recommendation 68, p. 262. Back

160 Science Policy Research Unit, The Changing Shape of British Science, 1995. Back