APPENDIX 34
Memorandum submitted by the Institute
of Biology and the British Association of Cancer Research, British
Association for Lung Research, British Ecological Society, British
Electrophoresis Society, British Society for Plant Pathology,
British Society for Soil Science, Laboratory Animal Science Association,
Royal Entomological Society, and the Society for Applied Microbiology
1. This response's principal points include:
(i) Generally, the RAE confirms that UK research
is extremely competitive internationally in terms of scientific
publications.
(ii) However it is impossible to tell if there
has been a genuine improvement in research performance but there
has been an increased ability to meet RAE criteria.
(iii) Secondary "scholarship" criteria
do not have sufficient prominence.
(iv) Industrial, applied and policy driven university
research are not recognised and so is disadvantaged by the RAE.
(v) Quality of research does not necessarily
equate with cost of research.
(vi) Though improved compared with the last RAE
there are still concerns over collaborative research and that
the RAE encourages short-term research.
(viii) Those taking maternity career breaks are
currently disadvantaged.
VALUE OF
UK RESEARCH
RAE indicates UK research excellent value in scientific
publications terms
2. The RAE provides an affirmation that
the UK research is of excellent value in terms of scientific publications
and their impacts. As such it provides corroboration of Professor
Sir Robert May's (the previous Chief Scientific Advisor's) analysis
that the UK research is most cost-effective compared to that of
its competitors. UK research provides an excellent return for
the investment made in it and so warrants nurturing.
HAS THERE
BEEN A
GENUINE IMPROVEMENT
IN RESEARCH
ALONG WITH
RAE SCORE?
Difficult to tell but it is almost certain that
departments' RAE strategy has improved
3. It is difficult to tell in a practical
sense whether there has been a real improvement in UK research
quality even though the RAE scores indicate this. Clearly there
have been discoveries and developments, but that it the nature
of research. It is impossible to tell whether the rate and/or
nature of this progress have improved. However a view common to
nearly all who submitted comments to this response was that some
or all of this improved RAE score is undoubtedly due to increased
familiarity with RAE exercises and the ability of university departments
to play the RAE game. This was something that many of the biological
Affiliated Societies warned the HEFC in March 1998.
Our 1996 advice to combat gamesmanship has proven
to be prudent
4. The Institute and a number of the Affiliated
Societies suggested in their 1998 response to the HEFCs' pre-RAE
consultation that greater standardisation across the Units of
Assessment (UOAs) as well as simplification, as a way to combat
gamesmanship. It would appear that, given the 2001 RAE results,
this was prudent advice as, though there was improvement compared
to previous RAEs, these gains in general (but not always) were
across subjects and universities. This meant that nearly everyone
played a better RAE game such that gamesmanship improvements largely
cancelled out. Broadly scores improved equally.
In a strict technical sense it is not possible
to compare scores from different RAE years
5. It should be noted that technically it
is impossible to categorically compare RAEs from previous years
as though broadly they are all based on the assessment of publications,
grant levels, student numbers, etc each has been different in
a number of ways. Consequently only general comparisons can be
made.
Journal numbers have increased so paper publication
would increase anyway
6. Given that journal numbers have increased
since 1986, there are more opportunities to publish so there is
an argument that any increase in scientific paper publication
would have happened anyway.
ARE PREVIOUS
RAE CONCERNS STILL
RELEVANT?
Yes. Secondary scholarship criteria are not considered
with sufficient strength
7. Yes. While it is important that research
is assessed and that quality of research is the prime criterion,
assessment of what are called scholarship activities (attending
symposia, participating in learned society activities, assisting
with public consultations on scientific matters, etc) should be
very much included as secondary criteria. While the inclusion
of activities on external (non-university) committees is included
the perception is that this is peripheral to determining the RAE
score. Furthermore some other scholarship activities are simply
not considered. We have heard on a number of occasions of Departments
that have actively instructed staff not to engage in any activity
that does not directly lead to the improvement of the Department's
RAE score.
Scholarship activities are vitally important
8. Scholarship activities are vitally important
for putting work into context, obtaining synergistic benefit through
discussion with other researchers, by furthering a researcher's
specialism, and in making expertise available to policy-makers
and Government in return for taxpayer support.
Though useful, there still remain concerns as
to the domination of citation in the RAE
9. There are still the concerns that over-reliance,
directly or indirectly, on citation analysis and citation impact
factors is not an accurate reflection of research activity or
excellence. Whether for example citation indices or journal impact
factors are used directly or not, the pressure on individual researchers
is to publish in outlets with a "high rating". How is
this being judged and by what means? This places an over-emphasis
on particular types and forms of output. Only just this month,
to give a contemporary example, of how this may be misleading
even when done "formally", the journal Nature expressed
concern over errors in citation statistics (on which to some extent
the RAE depends). Nature was concerned that the paper by the International
Human Genome Sequencing Consortium was given such a low citation
count that it was absent from the lists of "hot papers"
in biology. The Nature editorial (v415, pp101) concludes that
"researchers, policy-makers, and publishers who depend heavily
on citation statistics should be urged to treat them with greater
caution".
Industrial research is not assessed but industry
refers to the RAE
10. We are still concerned that industrially
funded university research is not assessed. The argument against
the inclusion of industrially funded research in the RAE is that
without publications it is difficult to assess and that the RAE
is used to assess Government funded research. However the reality
on the ground is that the RAE does have a substantial influence
on where industry places research. Furthermore there are options
for some form of industrial assessment. (For instance, as suggested
in 1998, independent financial auditors could affirm how much
industrially funded work a Department had conducted. This at least
would provide a cash indicator.) The quantity and quality of such
research cannot sensibly be ignored if RAE is to have realistic
meaning.
Quality of research does not equal cost of research
11. It should also be remembered that the
quality of research as indicated by judgement of publication in
outlets which are recognised as having a high reputation (high
citation rates and impact factors) from which funding follows,
does not necessarily reflect the cost of research. For example,
in the biosciences work with animals rightly necessitates welfare
costs and is not always cheap. Such work is invariably of scientific
significance (otherwise Home Office licences would not be granted)
but the work—though sometimes of break through importance—may
not always generate an Earth-shattering journal paper capable
of boosting an RAE score or may only do so over an extended period
of time. This is not the type of research that a Head of Department
would encourage colleagues to undertake with a view to RAE scrutiny.
RAE determines infrastructure funding but other
factors need to be included
12. Research infrastructure remains a problem:
in the past funding determined by RAEs results as well as from
other sources failed to meet infrastructure needs and so infrastructure
declined. Currently, the worst of the immediate infrastructure
problem has been greatly eased by the Joint Infrastructure Fund
as well as the Science Research Infrastructure Fund initiatives.
These initiatives are greatly welcome and send an encouraging
message to both the UK scientific community as well as multinational
industry that can choose with which nations research investment
will be placed. However, there remains the longer-term problem.
The RAE may well have a part to play in how infrastructure funds
are distributed but other factors will undoubtedly have to be
included, and of course resources in the longer term need to be
made on an on-going basis. This is an area that warrants further
consideration.
The RAE encourages short-term research
13. There are still concerns that the RAE
encourages research capable of generating high-impact papers quickly.
Departments are therefore discouraged from under-taking longer-term
research despite the possibility that some of such research may
be of greater value to the nation, its economy, environmental
quality and/or human well-being. Concerns over long-term research
were cited in the Affiliated Societies Science Policy Priorities
2001. Similar RAE concerns have been expressed by others including
a British Medical Journal editorial (320:636-637) with regards
to health sector research. (Here bioscientists work on biomedical
research.)
There are still collaboration concerns though
matters were better in the last RAE
14. Despite reassurances there are still
concerns that research arising out of inter-Departmental (especially
between universities) collaboration is not properly recognised
by the RAE, though steps have been taken to improve matters for
the most recent RAE. This concern also covers, of course, highly
prized international collaboration.
ARE THERE
NEW CONCERNS?
Is environmental science anomalous?
15. A recent survey of the biological Affiliated
Societies has questioned whether we currently have the correct
balance between whole-organism and molecular research. Both branches
of biology are important and are especially illuminating if one
is seen in the context of the other. However it may be that there
has in recent years been e more focus on one to the detriment
of the other. In the bioscience spectrum, the areas furthest away
from molecular biology are perhaps the disciplines of ecology
and environmental science. Notwithstanding that environmental
science is a recent area of research its Departments tend not
to have been awarded as high a score as other Units of Assessment
(UOAs). It is not clear as to why this might be. It may be: that
the quality of such research is genuinely anomalous; or that being
both a multi and interdisciplinary research activity that it fell
foul of collaborative Departmental RAE concerns; or that the academic
interests of the UOA panel members did not reflect that of areas
of environmental research; or it might be that much environmental
science research is policy-driven rather than the fundamental
or blue skies which is at the RAE's focus; or a combination of
such factors; or something else. However we believe that this
is worth following up.
THE WAY
FORWARD?
An independent audit of the RAE's utility is required
16. A full and independent audit of the
cost and benefit of the RAE to the nation is required. The audit
needs to be conducted by those unconnected with the UK university
system. Only through such an audit can we determine how the RAE
has been successful, and where it has let researchers down. Indeed
at a cost to the nation of some £40 million (equivalent to
four per cent of the annual Funding Councils' infrastructure grants)
the RAE needs to be seen to be providing genuine value for money.
The RAE does not assess research across the board
and so needs a name change
17. The RAE, if it remains as it stands,
needs a name change. It does not assess UK research per se across
the board. It only assesses university research, and even here
it does not fully account for applied or industrial research (particularly
where confidentiality is a highly imposed condition), nor does
it properly account for policy-driven (Government Departmental)
research. There is a tension between any assessment of "quality"
and the impact factor that the research has in reality, in other
words and its usefulness and application. This is common in the
biosciences where research not only can be blue skies and fundamental
(which is the RAE's prime focus) but applied (humans are living
creatures dependent on living creatures). For example in plant
pathology, to take just one specialist area, basic research is
often required to solve real problems in industry. While the results
may be incredibly useful, they will not necessarily result in
publication in a high impact journal and/not within a modest timeframe.
Indeed it is widely accepted that no (or very few) agricultural
journals have a high impact though we all depend on food in a
World with a growing human population.
Clarity as to what exactly is being assessed is
required
18. It needs to be made far clearer as to
actually what is being assessed by the RAE. We repeat our conviction
that the quality of research should remain the prime criterion
but in clear conjunction with scholarship activities as the secondary
criterion, with the final score being determined by both criteria.
A separate score is required for industrial and
policy-driven research
19. It may be that an assessment or assessments
with separate scores need to be simultaneously made of Departments'
industrial and policy-driven research. The Funding Councils may
not welcome having to do this but it would be far more efficient
to have one all-encompassing assessment delivering two or three
scores that revealed Departments' various strengths in different
types of work.
Take time to devise a fresh assessment mechanism
20. The RAE has provided some benefit but
has arguably had its day in its existing form. It is important
that the next Research Assessment is a clear development of previous
RAEs. Care and time should be taken to formulate it. Clearly given
the general improvement across the board in RAE score, there would
be little to be gained from repeating exactly the same exercise
in five years time. Therefore there would be little harm, and
much benefit, in waiting for a completely re-vamped assessment
in 7 to 10 years time. (This would also enable researchers to
focus in the interim on research and free them of RAE constraints.)
OTHER POINTS
Problems with being declared research inactive
following a maternity break
21. There is general concern as to the state
of women in science. Of the three pure sciences, biology has the
greatest proportion of women in its younger cohorts. However in
common with all of science there is the problem that if someone
takes a career break to start a family and then returns between
RAEs, then they may be declared research inactive. This issue
is surely resolvable and needs to be addressed.
Some subjects may fall between Units of Assessment
22. There are concerns that some subject
areas such as microbiology and environmental science are falling
between Units of Assessment. Their subject areas make them capable
of being assessed by more than one UOA and end up not being properly
assessed by either. It is important that areas of research are
not turned away from being properly assessed.
Consultation deadline concerns
23. There was general dismay at the way
this consultation was only afforded a month (one third of the
Cabinet Office minimum guidelines) and was launched just prior
to the Christmas/New Year break (itself contrary to the guidelines).
This does not facilitate confidence in open and fair consultation
and made it extremely difficult to obtain and approve views from
key constituent representatives of the UK biological community.
22 January 2002
| 
