Memorandum submitted by the Association
of University Teachers
INTRODUCTION
The Association of University Teachers (AUT)
represents over 43,500 academic and academic-related staff in
UK universities, colleges and research institutes. We welcome
the chance to respond to the inquiry into the research assessment
exercise (RAE). Higher education has become a powerful element
within the UK research system and of central concern to the economy
and national life. Nearly £1 billion of research funding
is allocated each year using the results of the RAE, and the bulk
of this money is allocated to science and technology. The status,
role and funding of the RAE therefore has major implications for
the UK science base.
The 2001 RAE results show a marked increase
in the proportion of departments whose research is judged to be
of international excellence. The credit is due to our academic
staff—supported by academic related colleagues—for
the world-class standard of the research produced in UK higher
education. While we welcome the government's emphasis on the importance
of world-class research, we regret their decision not to fully
fund the improvements in quantity and quality that have been revealed
in the 2001 RAE results. We are also disappointed that the Higher
Education Funding Council for England (HEFCE) have proposed a
further concentration of research funding in a small number of
5* departments. This short-term palliative will create a longer-term
crisis in UK science as new opportunities to conduct research
become stifled. The science base is too important to the economic,
cultural and social health of the UK to allow this ossification
and stagnation to occur.
1. What are the Problems with the RAE?
The successful distribution of public funding
for university research involves striking a balance between the
need, on the one hand, to ensure accountability and 'value for
money' in the use of public funds, and, on the other, to encourage
creative enquiry and flexibility. We believe that the balance
has gone too far in the direction of audit, assessment and instant
evaluation over the last decade or so. The main expression of
this process has been the concentration of funding council support
through the mechanism of the RAE and the closely connected phenomenon
of the weakening of the dual support system.
We therefore wish to preface our comments by
reiterating very strongly our continuing opposition to the RAE.
As a trade union, we are particularly concerned about the immediate
effects of the 2001 exercise on staff morale and members' jobs,
but we also believe there are inherent problems with the RAE as
a means of allocating research funding. Our members experience
of research selectivity in the context of funding cuts has been
overwhelmingly one of divisiveness, unfairness and demoralisation.
As a result, we believe that fundamental changes are needed to
the way research is currently funded and assessed.
1.1 Too much bureaucracy
The UK RAE is the largest and most comprehensive
evaluation programme of its type in the world. For example, in
the 2001 exercise, sixty assessment panels assessed approximately
200,000 publications submitted by almost 50,000 academics. The
UK research assessment process is also unusual in that it involves
a direct link with the funding allocation mechanism. Only a small
number of other countries, such as Australia, the Slovak Republic
and Poland, carry out a research evaluation nationally and use
the result to decide on the allocation of funding.1 The UK assessment
process is also highly significant in terms of its contribution
to overall research funding. For example, whereas nearly 20 per
cent of UK higher education funding council funds in 2000-01 were
allocated as a result of the RAE, less than 5 per cent were allocated
via the Australian equivalent (the Institutional Grants Scheme).2
As a result, preparing for the RAE has become a colossal exercise
in terms of staff time and resources.
Unfortunately, official attempts to estimate
the total cost of the 1996 exercise have come up with a highly
conservative figure (£27.5 million). This estimation does
not include the indirect costs incurred by the departments in
organising and managing the submission. Other studies have highlighted
the behavioural costs imposed by the higher education "audit
culture". For example, a study by Ian McNay on the impact
of the 1992 RAE revealed that 65 percent of staff surveyed thought
that the pressures from the exercise had increased their stress
levels.3
1.2 Too much selectivity
A key objective of research policy is to strike
a balance between on the one hand supporting research excellence,
and, on the other, encouraging dynamism in the sector, allowing
new subjects and centres to develop. We believe that the RAE has
led to an over-concentration of research funding in a small number
of leading departments and institutions (for example, 75 per cent
of HEFCE research funds are allocated to just 26 higher education
institutions). In the last two decades the degree of selectivity
in UK university research funding has increased significantly.
This contrasts with research funding in the US university system,
where the trend has been in the opposite direction, and which
is now less concentrated than its UK equivalent.4
The RAE funds research on a retrospective basis
only. As a result the exercise has a built-in element of irreversibility
and self-reinforcement: a department that fares badly in one assessment
loses funds and status and will find it extremely hard to re-establish
itself before the next assessment, with the consequence that it
then receives a further reduction in its research capacity and
potential. The RAE also affects the ability of departments to
compete for research support from outside sources, and so the
trend towards concentration of funding is greatly strengthened.
Underlying the old dual support system was a
healthy scepticism about the ability of any funding system to
"pick winners" in the research race. We now have a funding
system which relies almost exclusively on past performance as
a predictor of future research success. A retrospective element
obviously has a part to play in picking winners but it not the
only predictor and it does have peculiar limitations, such as
its inherent tendency to favour the tried and tested lines of
enquiry over the adventurous and innovative. Over-concentration
of research money has deprived many talented researchers, especially
relatively new entrants to the profession, of access to research
support. The fact is that research potential is distributed very
widely throughout our higher education institutions, but it is
increasingly frustrated by a funding system that operates as if
it were to be found only in 15 or so universities.
Our belief that the degree of selectivity in
university research funding was beginning to have a negative impact
on the overall research capacity and potential of our universities
was the main reason why we argued in 1998 for the postponement
of the current RAE and the introduction of a "seedcorn fund"
to stimulate research growth in some of the departments hardest
hit by the selective effects of successive RAEs.5 It gives us
little comfort to say that "we told you so", but if
our advice had been heeded by the funding councils we would not
now be facing the current, entirely predictable funding dilemma
(see 1.3.2). Because of the current debacle, we recommend if not
the abandonment of the RAE in its current form, then the postponement
of the 2006 exercise.
1.3 The 2001 RAE
On many measures UK higher education researchers
are already among the best in the world. For example, the UK ranks
first in the world in terms of the numbers of publications and
citations generated per million dollars spent on research. The
proportion of staff in 5 and 5* rated departments (that is, departments
of international excellence in at least half of their research
activity) in UK higher education institutions however has increased
from 31 per cent in 1996 to 55 per cent in 2001.
1.3.1 Is the improvement genuine?
We believe there has been a genuine increase
in the international excellence of UK research. This improvement
has been ratified by overseas assessors linked to the RAE panels
(in all but three per cent of the cases, international experts
confirmed the panels' judgement). Bibliometric analysis has also
shown that the proportion of UK entries in the annual top 1 per
cent of most highly cited papers in the world increased from 11
per cent in 1995 to 18 per cent in 2000.6 At the same time, we
acknowledge that the improvement in RAE grades is also the result
of more sophisticated institutional strategies. Compared to 1996,
fewer researchers and departments were included in 2001 as institutions
gambled on securing more money for fewer researchers rather than
receiving less funding for a larger number of researchers. Institutional
'game playing' is an inevitable consequence of performance-based
research funding, but it has had a detrimental impact on the increasing
numbers of able staff who are excluded from the exercise (see
section 1.4).
1.3.2 The impact of the funding shortfall
on staff
The increase in highly-rated research, without
a corresponding increase in the level of funding, means that there
will be a recurrent funding shortfall of £200 million for
university research in the UK. At the time of writing (January
2002), HEFCE have indicated that they are only willing to maintain
the unit of resource in 5* departments. The basis for funding
research with a 3a, 3b, 4 or 5 rating is yet to be announced,
but it seems inevitable that many departments will not receive
the funding merited by their ratings. Many staff who have devoted
their efforts and talents to improving the quantity and quality
of their research now fear that their work will go unrewarded
because of the need to fund the outcome of the 2001 RAE even more
selectively than the normal translation of the research ratings
into funding allocations would suggest.
The allocation of funding proposed by the funding
council will have a catastrophic impact on staff morale and motivation
at a time when our institutions are already under intolerable
pressures to dance to a variety of government tunes, not entirely
harmonious. There is widespread cynicism among staff about the
value of the various bureaucratic hoops that they are forced to
jump through these days. In the case of the RAE, the university
community have responded effectively to the government's challenge
and delivered sizeable improvements in research quality. A failure
to reward this success undermines the whole basis of the performance-based
approach to funding. We cannot see how the RAE can possibly survive
the current debacle, at least in anything resembling its present
form.
More seriously, from the point of view of our
immediate concerns, is the potential impact of the RAE on our
members' jobs. Some departments and institutions are likely to
have funding shortfalls that they will find great difficulty in
sustaining. It would be utterly unacceptable if the RAE were to
lead to staff redundancies at a time when the government is demanding
a further massive expansion in student numbers.
Part of the problem lies with the government
and their refusal to fund university research properly. Even with
additional Science Research Investment Fund (SRIF) money, the
continuing success of UK scientific research is threatened by
lack of adequate funding and the poor condition of the basic research
infrastructure and equipment in many universities. However, the
RAE itself is also seriously flawed and we believe the funding
councils have a responsibility to mitigate its impact on institutions
and their staff. It is particularly important to ensure that the
funding allocation protects institutions from sudden, significant
falls in their grant.
1.4 Exclusion and division
In addition to the funding shortfall, the association
believes that there are inherent problems with the RAE. One of
the most detrimental long-term effects of the exercise has been
the creation of a "competitive, adversarial and punitive
spirit" within the profession.7 The RAE promotes an invidious
distinction between those whose work is submitted for assessment
and those whose work is not. The 1996 exercise led to the victimisation
of some staff who were said not to have contributed to the RAE
and this happened even in some highly-rated departments.
Universities were far more selective in the
numbers of researchers (and departments) submitted to the 2001
assessment compared with previous exercises. We have many members
who were told that they were unquestionably eligible to be entered
in the 2001 exercise but would not be 'for strategic reasons'.
These colleagues, often women at the early stages of their careers,
are now officially deemed to be "non-research active".
The exclusion of individuals from the recognition conferred by
the status of "research active" has potentially harmful
effects in terms of future promotion, job applications, grant
applications, sabbaticals, etc.
Other instances of "game playing"
include transferring "research inactive" academic staff
onto "other-related' grades" (in some cases with a teaching-only
contract). This problem arises from the perceived need on the
part of HEIs to decrease the total number of academic staff so
that the proportion of staff submitted as research active is increased.
This perceived need arises in the first place because RAE quality
ratings are published alongside a banded score for the proportion
of academic staff submitted as research active. In fact, the proportion
data has only a marginal effect in terms of RAE ratings, and such
tactics have been questioned by HEFCE. Moreover, this misjudgement
is leading to actions which disadvantage some members of staff
(for example, in relation to eligibility for early retirement).8
Regrettably, in some departments, "non-research
active" colleagues are treated as second class citizens,
despite the fact that it may be precisely their contributions
to teaching, pastoral care, curriculum development and administration,
which enable their colleagues to concentrate on their research
activities. The effect of this, apart from the unfairness and
de-motivation expressed by individuals, is to undermine the professional
collaboration and teamwork among academic and related staff on
which the effective provision of high quality learning, teaching
and research depends.
A related issue concerns the status of contract
research staff (CRS). Ninety-four per cent of "research only"
staff in 1999/00 were on fixed-term contracts. HEFCE research
shows that anatomy, physiology, pharmacology, biochemistry and
veterinary science have particularly high proportions of staff
on fixed-term contracts.9 RAE rules on counting CRS as 0.1 of
a person unnecessarily distorts and devalues their role in the
research productivity of an institution and perpetuates further
their exclusion from the research culture of a department. The
desire for full RAE returnees to achieve international standing
and be seen to bring in research money often results in CRS being
denied conference expenses, deliberately sidelined from being
principal investigators on grant proposals and effectively pushed
out of other means of research career development. The low recognition
given to CRS in the RAE also encourages malpractice in designating
authorship to published works that have been written in research
teams. The RAE in its current form offers justification for the
exploitation of the contractual vulnerability of CRS. AUT recommends
that CRS are treated with parity to their permanent academic colleagues.
1.5 Impact on teaching and other academic
activities
One of the continuing concerns of the science
and technology committee has been the quantity and quality of
university science education. There are clearly specific issues
that pertain to science, such as the poor quality of teaching
labs and shortages in teaching and support staff. However, there
is also a general problem that needs addressing: the relatively
low status of teaching and learning in higher education. We have
for many years argued that contributions to teaching should be
valued equally to contributions to research, and that this should
be reflected in all aspects of career progression, including promotion
criteria. Establishing this change in culture has been an uphill
struggle since the introduction of the RAE which has grossly distorted
the higher education value system, not just in favour of research,
but in favour of research output of particular kinds (ie those
that count towards RAE ratings).
In many institutions performance in the RAE
has become the overwhelming focus of their activities because
of the funding and prestige that it confers. The pressure on staff
to contribute to the RAE has led to a tendency to neglect important
activities which would normally be seen as part of a balanced
university culture: development of courses and teaching materials,
personal tutoring, reporting and monitoring student progress,
training and mentoring teaching assistants, staff development,
and so on.10 The dominance of the RAE has also hindered the development
of contributions to the local, regional and national economies
through applied research, training and consultancy.11
1.6 Gender difference and activity
A key issue facing UK university science is
the need to attract women into the profession and to enable them
to progress onto the highest grades. One of the factors hindering
gender equality in academia has been the disproportionate numbers
of women who have been designated as "research inactive".
Analysis of data from the higher education statistics agency,
for example, suggests a significant under-representation of women
in the 1996 RAE, including those in science departments.12 HEFCE's
own evidence shows that the highest graded departments (5 &
5*) have a higher proportion of men than women. One of the recommendations
of HEFCE's fundamental review of research, published in August
2000, was to consider the reasons for this under-representation
and whether there are other groups who appear not to be realising
their full research potential. We regret that this research has
not yet been published, and now believe that more decisive action
is required (see below).
RAE procedures on maternity leave
One of our specific concerns is the treatment
of women who have taken maternity leave. During preparations for
the previous RAE in 1996 the association received complaints from
a number of women members whose work was deemed "inadequate"
for submission to the assessment. It was clear that in some of
those cases the members' research output had been affected by
periods spent away from research on maternity leave. For the 2001
exercise, the final RAE panel criteria and procedures did acknowledge
the need to take individual staff circumstances into account,
including maternity leave:
"The situation of staff who have taken maternity
leave or other career breaks, who hold part-time contracts, who
are disabled or who have been absent for long periods through
illness (where this is indicated by HEIs) will be taken into account
in reaching overall judgements of quality where it is indicated
in submissions." (para 2.18, p12, RAE 2001: Assessment Panels'
Criteria and Working Methods, RAE 5/99)
Although this was a helpful statement, only
20 per cent of the panels issued guidelines mentioning maternity
leave as a possible factor in research performance. None of the
science-based panels included a specific reference to maternity
leave. In the run up the current exercise we received anecdotal
evidence that heads of department were reluctant to implement
these guidelines (on the grounds that panels were unlikely to
adhere to them). Consequently, there remained a tendency to exclude
maternity leavers, and returners from career breaks, in the 2001
exercise if they had not returned four items of research. We believe
that the funding councils should issue stronger guidelines to
protect the position of women who take maternity leave.
An equal opportunities audit
In the 2001 RAE, HEFCE collected information
on the gender of staff entered in the exercise, which would enable
it to produce information on the performance of men and women
in the RAE. Given that the names of selected staff are due to
be published on the Internet in the spring of 2002, we see no
reason why HEFCE is unable to publish this gender data as soon
as possible. Publishing the raw data will enable analysis of whether
women have suffered from discrimination in the RAE. In addition,
we believe that the time is ripe for a comprehensive equal opportunities
audit of the whole 2001 exercise. This audit should be undertaken
by an independent body, such as the Equal Opportunities Commission,
in conjunction with the newly-formed Equality Challenge Unit.
2. REFORMING
RESEARCH FUNDING
AND ASSESSMENT
To summarise our view: some selectivity in the
distribution of research funds is necessary and given the limits
on the availability of funds, inevitable, but the degree of selectivity
and concentration of research funding practised over the last
decade has now become wasteful of talent and conservative in its
reliance on the "safe bet". We accept that there can
be no return to the old dual support system in its original form
which in essence distributed funding council money for research
according to a headcount of student numbers in each institution.
However, we believe that the case for extra funding to broaden
our university research base is absolutely overwhelming. Our specific
reforms for achieving this are as follows:
2.1 "Seedcorn" funding
Investment in future and potential success is
just as important a part of the process of funding research as
is investment in existing excellence. We believe there should
be a "seedcorn" fund available to university researchers
for starting up new programmes. This should be aimed at counterbalancing
to some extent the concentration effect of research selectivity.
The present RAE ratings are a crude mechanism for evaluating departments.
A low departmental rating can stifle research opportunities for
all its members, even though some of them have great potential
and exciting ideas. We need to be able to throw a lifebelt to
these researchers before they are demotivated or leave the system
in frustration. In the 1994-95 funding allocation, as an interim
measure, HEFCE earmarked £16 million for supporting start-up
research in the former polytechnics. This idea should be built
upon by establishing a new element of funding for Research Development,
but through the provision of additional funding by the government,
rather than by reallocating existing funds. It should be available
on application to any researchers in departments with an RAE rating
of 1 or 2, or in departments which did not enter into the RAE.
It should be available for all areas of research, across all disciplines,
and applications should be assessed by established peer review
procedures. In order to have a real impact on the quality of research
and teaching in our universities, the size of the fund would need
to be at least 10 per cent of the total amount currently provided
by the three funding councils for research.
2.2 Reducing bureaucracy
Alongside the creation of seedcorn funding,
we favour the postponement of the next RAE. However, if there
is to another RAE in 2006, it needs to be in a much more streamlined
form, to reduce the burden of bureaucracy and workload on staff.
One of the ways to achieve this might be to
examine whether the RAE is equally relevant to all areas of research.
It may be that, at least for as long as underfunding persists,
some measure of selectivity is inevitable in certain areas of
experimental science in which very expensive instrumentation and
installations are required. But do the same arguments apply across
all other areas of research, including, for example, theoretical
science or philosophy or literary studies? It may be that a close
examination of these questions would suggest a much more limited
exercise in future, rather than a repeat of the comprehensive
assessments that have characterised previous RAEs. Lessons may
be learnt from the experience of QAA teaching assessments, whereby
universal subject reviews have recently been abandoned in favour
of more limited institutional audits.
2.3 Protecting peer review
Although we are keen to reduce bureaucracy,
peer review remains central to any performance-based approach
to research funding. Peer review is not perfect, but is widely
understood and respected within the academic world. It is also
a cost effective way of making use of existing expertise and of
spreading knowledge of research activity among academic colleagues.
We certainly have not come across a better system.
The other main performance-based alternative is to use quantitative
performance indicators (for example, as in Australia). However,
performance indicators such as the volume of external research
funds and higher degree research student completions provide only
a very rough approximation of research quality. Nor is bibliometric
analysis a real substitute to peer review. This is because citations
cannot always be used as a proxy for quality (eg some of them
are for negative reasons, authors routinely cite their own work,
there is a problem of US bias, and there is a time-lag between
publication and citation). As a result, we do not believe that
citation data is reliable enough for it to become an important
element in determining national funding policy.
Association of University Teachers
January 2002
NOTES
1. Aldo Geuna, Dudi Hidayat and Ben Martin,
Research allocation and research performance: the assessment
of research, SPRU: Brighton, July 1999.
2. Tertiary Education Advisory Commission,
Shaping the Funding Framework: Fourth Report of the Tertiary
Education Advisory Commission, Wellington: New Zealand, November
2001, pp 88-92.
3. Ian McNay (1997) The Impact of the
1992 RAE on Institutional and Individual Behaviour in English
Higher Education: the evidence from a research project, Centre
for Higher Education Management, Anglia Polytechnic University.
4. Higher Education Funding Council for
England, Review of research (00/37), HEFCE: Bristol, 2000,
Table G5, p 62.
5. Association of University Teachers, Research
Assessment Exercise: response to consultation document RAE 2/97.
6. See Times Higher Education Supplement,
"Selectivity raises scores across the board", December
14, 2001, p 3.
7. The phrase belongs to Professor Lewis
Elton, "The UK Research Assessment Exercise: Unintended Consequences",
Higher Education Quarterly, Vol 54, No 3 July, 2000, pp
274-283.
8. Lee Elliot Major, "Goodbye to all
that", Guardian Education, February 20 2001.
9. Higher Education Funding Council for
England, Characteristics of research active staff, HEFCE
Analytical Services Group, March 2000.
10. Alan Jenkins, "The Impact of the
Research Assessment Exercise on Teaching in Selected Geography
Departments in England and Wales', Geography, Vol 80 part
4 (349), 1995, pp 367-374; Ian McNay, "The Paradoxes of Research
Assessment Funding", in Changing relationship between
higher education and the state, ed. Mary Henkel and Brenda
Little, London: Jessica Kingsley Publishers, 1999, pp 191-203.
11. Association of University Teachers and
the Institute of Education, Academic and academic related staff
involvement in the local, regional and national economy, AUT/IoE,
2000.
12. Association of University Teachers,
Gender differences and activity in the 1996 research assessment
exercise (RAE), http://www.aut.org.uk/pandp/documents/RAE/rae96gender98-99.pdf
| 
