House of Commons - Science and Technology

Select Committee on Science and Technology Second Report

MERITS OF THE RAE

32. Most of the evidence we have received has suggested that the RAE has had a broadly beneficial effect on research in the UK. We have heard that it has led to better management of research.[42] Many witnesses say that the RAE has led to more "sophisticated institutional strategies" for their research with unproductive research being pruned away and a resulting improvement in research quality.[43] We have been told that the RAE did "stimulate research and research development in those institutions that had previously considered it not anything to do with their business at all".[44] The RAE has forced universities to make more strategic decisions, investing in their strong departments. HEFCE tells us "institutions have actively managed their research by setting specific targets, developing benchmarking systems, and focusing on staff management ... This has enabled an improvement in research capacity in a very real sense".[45]

Cost

33. We have heard of concerns that the RAE has imposed large costs and bureaucracy on universities, hampered teaching, distorted research practice, led to neglect of universities' other activities and severely damaged academics' morale. HEFCE estimates that the RAE costs 0.8% of the available QR funds, which it says "compares extremely favourably with the transaction costs associated with bidding or tendering processes".[46] Estimates of the costs of RAE 1996 ranged between £27 million and £37 million.[47] These figures are disputed by the Association of University Teachers.[48] The Royal Society says "the combination of administrative work associated with the RAE, preparation of grant proposals, coupled with demands of the QAA (Quality Assurance Agency), which assesses teaching quality, are a significant and growing burden on academic staff".[49] Of course, any rigorous assessment system will place burdens on those taking part. The peer-review process employed by the Research Councils is also expensive and time-consuming.[50] An important difference is that a researcher submitting a Research Council grant application will benefit directly from its success, while the same researcher may only see indirect benefit from the RAE, if at all. If HEFCE believes in the value of the RAE, it should establish clearly how much it costs and show why it is worth it. We recommend that, as part of its review of the RAE, HEFCE establish with accuracy the cost of RAE 2001 and publish costings and an explanation of how these were calculated.

Research practice

34. HEFCE points out that "Any assessment process, particularly one as important to its subjects as the RAE, will distort the very thing it intends to measure".[51] There is concern that the RAE distorts the nature of research being undertaken; that it discourages 'blue skies' research and forces researchers to look for short-term goals; and that publication practice is being affected.[52]

35. The BMA says the RAE encourages researchers to stick with safe research or become part of existing projects rather than establish new lines of inquiry.[53] The AUT makes the point that any retrospective funding system favours "tried and tested lines of enquiry over the adventurous and innovative".[54] We are particularly concerned that the Russell Group of universities, where the bulk of world-class research is conducted in the UK, feel that the RAE encourages short-termism in research.[55] We have been told that some departments "have actively instructed staff not to engage in any activity that does not directly lead to the improvement of the department's RAE score".[56] We have heard that institutions discourage their researchers from writing text books and that students increasingly have to rely on books written overseas.

36. HEFCE's review of research published in 2001 concluded that "editors of major international journals report that the RAE has not had a significant distorting effect upon publication behaviour".[57] Mr Bekhradnia of HEFCE insisted that four publications in five years was not onerous.[58] We do not accept this: to produce four publications in 'high impact' journals is certainly demanding. Professor Russ Bowman of the AUT stated that "Every manager in a university is leaning on people to up their publication rate".[59] Natalie Fenton of the AUT argued that "some research that is genuinely innovative and really very much new and at the cutting edge ... does not have an immediate publishable location".[60] Researchers in some disciplines may find it more effective to publish in highly specialised but 'low impact' journals, particularly those resulting from industrial collaboration or aimed at application.[61] Such outputs may not receive equal recognition from RAE panels. Publication in 'high impact' journals does not necessarily equate to scientific progress or research of significant social value.[62]

37. Some of the most outstanding achievements in UK science have resulted from long periods of research with no outputs: Watson and Crick's discovery of the structure and DNA and, more recently, Sir Harry Kroto's work on 'buckyballs' which earned him a Nobel Prize. Today these researchers could be branded as inactive and shunted off to teach first year undergraduates. Some scientific disciplines have a tradition of publishing ideas through books as well as journals, such as evolutionary biology. We recognise that the panels can consider reasons why a researcher may not have submitted four outputs, but we doubt whether 'undertaking long-term speculative research' would cut much ice. We are concerned that the RAE process may discourage long-term research of a highly speculative nature and stifle scientific breakthroughs.

38. The Funding Councils have allowed most arts and humanities UoAs to submit outputs to the RAE from the previous seven years. This change was introduced in 1986 "following representations that research in these subjects generally took longer to come to fruition".[63] While we accept that publication practice is somewhat different in the humanities, we believe that the shorter period of assessment for the sciences discriminates against scientists involved in long-term research. We suggest that seven years would be a more reasonable period of assessment for the sciences as well as the humanities.

Interdisciplinary research and unit boundaries

39. The RAE is criticised for discouraging interdisciplinary research. HEFCE rejects this claim, though it admits to concerns about the quality of judgements made by panels about interdisciplinary research.[64] It cites bibliometric analysis that shows that the UK is responsible for more papers with authors from different disciplines than most other countries and that top-rated departments do not publish fewer interdisciplinary papers.[65] Yet many of those giving evidence to us, including OST and the EPSRC, consider that this is still a problem.[66] We recommend that HEFCE ensure that its quality assessment does not discourage or disadvantage interdisciplinary research. Such research offers some of the most fertile ground for innovation and discovery.

40. The biological science learned societies suggest that some subjects, notably microbiology and environmental science, do not sit easily in existing UoAs.[67] They believe, and their arguments are convincing, that these areas are not adequately assessed. High quality research in these areas is of growing public interest and importance. We recognise that, however boundaries are drawn, some research will not fall neatly within them. But we recommend that HEFCE keep unit boundaries under review to ensure that subjects of increasing importance are fairly assessed.

Careers and morale

41. The RAE can have a very damaging effect on staff morale. Being labelled 'research inactive' for tactical reasons can blight research careers, and even bring them to an end. The AUT's memorandum states that "The exclusion of individuals from the recognition conferred by the status of 'research active' has potentially harmful effects in terms of future promotion, job applications, grant applications, sabbaticals, etc".[68] Natalie Fenton described how researchers may even have difficulty trying to establish whether they are being entered in the RAE or not.[69] She also made the point that if there are redundancies as a result of financial pressures "the people that go are the people not returned in the RAE".[70] HEFCE recognises that the RAE leads to redundancies[71] but it cites early retirement as a positive part of a department's research strategy rather than the unfortunate end to a distinguished career.[72] Personal tragedies may become of more widespread concern if good researchers are lost to the science base. The costs of early retirements have implications for the future operation of pension funds. We have also heard concerns that the RAE may have contributed to the large increase in fixed term research positions.[73] It is clear that the RAE has had a negative effect on university staff morale. Any future research assessment mechanism must be able to give a fair appraisal of the research without tempting universities to continue the divisive and demoralising practice of excluding some academics from the process.

Women and the RAE

42. There is evidence of under-representation of women in the highest-rated departments and that women have been disproportionately excluded from RAE.[74] HEFCE made efforts to address this problem in RAE 2001. Panels were clearly advised that they should take maternity leave into account, but the AUT believes advice given by panels is patchy.[75] There is concern that problems in career advancement for women in higher education are exacerbated by the RAE. Women are already more likely to undertake more teaching and pastoral work at the expense of the research, and many have family commitments. HEFCE is currently developing, with the Equality Challenge Unit, a research programme into the position of women, ethnic minorities and other minorities in research roles in higher education.[76] We welcome HEFCE's imminent research project into women in higher education research and recommend that it should analyse RAE data as part of this study. It is vital that women's research careers are not further impeded.

Departmental closures

43. A serious allegation is that the RAE has played a role in the closure of science departments. In several scientific UoAs, there has been a reduction in the number of university departments entering the RAE. In Chemistry, we understand that the number of cost centres has dropped by 18 since 1992 and that a further 11 are threatened.[77] Mr Bekhradnia claimed that in the Chemistry UoA, while there had been fewer submissions in 2001 than in 1996, the number of active researchers was almost unchanged.[78] In fact the number of researchers submitted fell from 1,388 in 1996 to 1,300 in 2001, representing a 5% reduction, a significant drop in research activity, which will lead to 5% less money going to Chemistry.[79] The situation in Electrical and Electronic Engineering is even worse, with a 28% drop in the number of submitted researchers between 1996 and 2001.[80] The situation is not restricted to science and engineering, but analysis of the figures shows that subjects classified in the RAE as high cost (principally medical, scientific and engineering research)[81] have seen a 1.8% fall in the number of researchers submitted between 1996 and 2001, compared with an increase in low cost subjects (such as English, History, Sociology and the languages) of 3.5% over the same period. In History, there was a 25% increase in the number of researchers submitted in 2001.

44. It is not clear to what extent the reduction in submissions is because of departmental closures. It is likely that some departments, particularly in the new universities, opted out of the RAE process. Some departments may have merged, which can have an adverse effect on postgraduate and undergraduate student access. In some cases, we fear that there have been straightforward departmental closures, because of a decline in demand for undergraduate places and consequent loss of teaching funding. Subjects like chemistry are being hit by a double whammy: less funding for teaching and less funding for research through reduced RAE submissions. This risks a downward spiral in research capacity. The RAE may not be the primary cause of departmental closures but we suspect that it is a contributory factor. Nationally important research that makes a major contribution to the economy must not be destroyed because of trends in student demand.

45. At present there is no policy dimension to HEFCE's QR funding: the system takes no account of the value of an area of research to society or the economy. Yet there are subject areas that are under threat.[82] The DfES suggests further discrimination of 5* departments according to "national importance" and there is some support elsewhere for this.[83] In our view, it is better to address national research priorities through a funding mechanism rather than by tinkering with the assessment process. HEFCE should protect or enhance key research areas by changing the cost weightings for some UoAs or by introducing ringfenced funding.

Other outputs

46. The RAE, and the funding decisions based on it, create incentives for universities that could lead to them neglecting other areas of their functions: teaching; community involvement; commercial activity; and research of local or regional significance. This may have major implications for the nature of UK universities.

TEACHING

47. A particularly serious concern is that university teaching is not being accorded sufficient status and risks being neglected. We have heard that in medicine the "element of personal contact between teacher and student in non-clinical teaching is now greatly reduced".[84] Ms Fenton spoke of the RAE's "negative impacts on the teaching culture" in departments, with students getting less contact time and larger tutorial groups. Extra teaching, she said, was being undertaken by contract staff.[85] We have heard concerns that teaching is increasingly off-loaded onto technicians and postgraduates. The Institute of Physics believes that innovations in teaching have been neglected through departments' concentration on their research output.[86] We are aware that there are those who feel that the RAE disadvantaged universities who are attempting to teach in an interdisciplinary way.[87]

48. If a strong financial incentive is introduced in one area of universities' activities, it cannot fail to have a negative effect elsewhere. It seems likely that the RAE has had this effect on teaching. While there is assessment of teaching quality, no funding is based on it, although HEFCE can withhold future course funding where there is very poor teaching performance. On 20 March 2002, the DfES announced a new system of quality assurance involving a 6 year cycle of institutional audit visits, made up of senior academics. Each university will make available information on quality and standards and will be conduct internal monitoring involving external reviewers.[88] The only financial incentive has been that departments can advertise their teaching rating and attempt to attract more students on this basis. On 23 January 2002, HEFCE announced an end to the cap on a department's student numbers, which may have the effect of encouraging better teaching, but probably only in those subjects that were previously having little trouble in filling places.

49. The issue of teaching assessment is extremely contentious. It is, if anything, even more unpopular with academics than the RAE. There are claims that the paperwork involved is colossal and that teaching quality is difficult to measure. Mrs Hodge hinted that the Government might consider quality-based funding of teaching.[89] Some witnesses supported this view.[90] It is not for the RAE to reward teaching, but there must be a counter-incentive to promote good teaching and encourage good teachers. We believe that there must be financial incentives for improving the quality of teaching but that the burden and the problems of measuring teaching quality are such that funding based on it should be a last resort. HEFCE and universities must work together to provide well-paid and prestigious career positions for academics who are primarily teachers.

50. In her evidence, Mrs Hodge said she was considering 'teaching-only' universities.[91] There might be some advantages in this. Universities could stop pretending that they were all the same and concentrate on their strengths. Also, research capability could be concentrated in a few centres of excellence making it easier to direct funding to the very best. There might not even be any need for a research assessment process; since all the departments would be world class, funding could simply be allocated according to the number of researchers. But there would be a downside. It would mean that university teachers would no longer benefit from a close association with research. University teachers must be familiar with the latest developments in their field. Mr Bekhradnia insisted that while 'scholarship' was important for university teachers, research activity was not.[92] We agree with the Dearing Report that one of the purposes of research is to "inform and enhance teaching".[93] Research is the best incentive for academics to keep themselves at the cutting-edge of their subjects. Students benefit from the exposure to research. Professor Bowman stated that undergraduates only got to grips with their subject during their final year research project.[94]

51. There is a clear tension between the desire to concentrate research excellence and widen participation, though both are commendable in themselves. Since undergraduate teaching is compromised by an absence of a research environment, an elite sector of research universities will condemn the vast majority of students to a second rate university education. 50% participation is an admirable ambition but not if only 5% get an education of the highest standard. The best students, particularly from abroad, are likely to be attracted to the universities with the best RAE scores. We are supportive of high-quality teaching in a high-quality research environment and find it hard to see how this can be reconciled with the concept of a teaching-only university.

52. Learning in a research environment is very important in identifying and nurturing those with an aptitude for a research career. Yet most 18-year-olds will have given little thought to a research career. If only 20-30 universities are undertaking research in a particular discipline, only a small proportion of undergraduates will ever find out if it is something they could excel at. If the best researchers are concentrated in a small number of departments, we risk losing the next generation of scientists. At a time when the Government is concerned about the supply of scientists in the economy, HEFCE should be encouraging high-quality research wherever there is teaching.

COMMUNITY INVOLVEMENT

53. Part of an academic's role is to take part in activities on behalf of the academic community as a whole, outside his or her institution. This includes peer review for journals, external examination, giving public lectures, involvement with learned societies or sitting on governmental advisory committees.[95] It also includes contributing to local community life, as local councillors, perhaps, or as school governors or magistrates. Much attention is being given by the OST and the scientific community to strengthening links with the public. Part of this process must involve universities reaching out into their communities.[96] We are concerned that the pressures placed on academics, not least through the RAE, make community involvement less likely.

KNOWLEDGE TRANSFER

54. There is concern that the RAE discourages knowledge transfer, since industrial collaboration is not adequately considered by RAE.[97] Since 1996 HEFCE has allowed departments to include patents in the list of outputs that panels can consider. We requested information from HEFCE on the number of patents submitted to panels in each UoA. Though not conclusive, these figures suggest that more patents were submitted by departments that received low RAE ratings in 2001.[98] We recommend that, in its review of the RAE, HEFCE consider the impact of the RAE on knowledge transfer activity, and investigate whether panels have accorded due status to industrial research outputs. The Government wishes to encourage industrial collaboration and the commercialisation of research and HEFCE must ensure that the RAE does not undermine this.

REGIONAL PRESENCE

55. The Government wishes to see universities as important parts of their local economies. Yet the RAE concentrates research excellence - as witnessed by the expansion of highly rated science departments and the closure of others - and this threatens the geographical spread in some subjects. HEFCE said it would be concerned if this were to occur, yet there is already some evidence for this: all 5* Civil Engineering departments are in the South of the country.[99] Companies in certain sectors could be deprived of the intellectual input to help them become world-beaters. Departmental, and institutional, mergers can have serious implications for student access. The Government may need to intervene to ensure that research excellence is represented in the regions of the UK, perhaps by encouraging regional networks in important subjects.

56. Not all research, however good, will get published in international peer-reviewed journals and get high RAE ratings. Some has a specific audience and may be more appropriately published in specialised journals read by the user community but which do not score highly in respect of the 'impact factors' as presently defined. These kinds of research build links between a university and its community and local industry for mutual benefit and make contributions to the regional and national economies. We believe that this kind of research can be assessed using a research assessment mechanism but it may need ring-fenced funding or some other system to offset any bias. The HEIF can fund some of this research but the problem remains that RAE ratings, which emphasise publication in high profile journals, do not adequately reflect the value of this type of research. HEFCE should better assess research conducted by universities for local business or policy-makers. We have received evidence that some poorly performing UoAs - social policy and environmental science - may have been discriminated against for this reason.[100] HEFCE accepts this in its recent report on charities.[101] New universities have often specialised in this type of research and so would be hit by any increase in selectivity.[102] Research into matters of local importance can be vital to communities and the economy. If the RAE cannot recognise such work a mechanism needs to be identified that will.

Infrastructure

57. The RAE stands accused of compounding the problem of long-term under-investment in research infrastructure by encouraging diversion of funds away from infrastructure to maintain ratings. According to the Amicus-MSF trade union, "departments have been under considerable pressure to use funds awarded as indirect costs to increase research activity instead of maintaining infrastructure and support services".[103] Any university that made capital investment is unlikely to see the results within the period of a single RAE. To do so would risk dropping a grade with a subsequent loss of income. The DfES acknowledged the need to "ensure that funding mechanisms do not provide perverse incentives", pointing in particular to the importance of encouraging institutions to invest in the physical infrastructure.[104] The introduction of JIF and now SRIF are welcome but are not enough.[105] A recent report found that there is a backlog of investment in universities' research infrastructure of around £2.7 billion.[106] HEFCE should monitor levels of investment in infrastructure carefully and if necessary introduce a recurrent funding stream.

Conclusion

58. The AUT's Natalie Fenton said in her evidence, "It is impossible to overestimate the driving force, in terms of the culture of a department, that the RAE has; so it focuses completely a department's attention on a particular end sight, because that is ... the only multiplier ... they have control over".[107] In contrast, Bahram Bekhradnia of HEFCE said "The RAE is a tool, no more than that".[108] It is a tool that we feel has had a profound impact on much of universities' behaviour. In their evidence to us, HEFCE seemed to believe that any side effects of the RAE were unfortunate and somehow nothing to do with them. If HEFCE has a mechanism for selective research funding then it must take responsibility for any distortions it causes.

59. The RAE has undoubtedly brought benefits but it has also caused collateral damage. It has damaged staff careers and it has distracted universities from their teaching, community and economic development roles. Higher education should encourage excellence in all these areas, not just in research. Universities should be assessed on a balanced score-card.

42 Eg Ev 72, para 5; Ev 89 Back

43 Ev 24, Ev 113 Back

44 Ev 34, para 90 Back

45 Ev 4, para 33 Back

46 Ev 3, para 28, footnote 7 Back

47 Ev 3, para 28; Qq 61-62 Back

48 Ev 24 Back

49 Ev 99, para 14 Back

50 Ev 66, para 5 Back

51 Ev 7, para 60 Back

52 Ev 66, para 4 Back

53 Ev 105 Back