Government response
1. The Funding Bodies should have looked at the
quality of the arguments set out in the responses to their consultation
and not just the numbers. The move away from the "one size
fits all" approach advocated by Sir Gareth Roberts is an
important principle which should have been adopted. We consider
the Funding Bodies to be unjustifiably conservative in their proposals.
We do not see it as HEFCE's role to protect the sensitivities
of universities. (Paragraph 16)
In undertaking and considering the Roberts' Review,
the Funding Bodies engaged the sector in two rounds of extensive
consultation. The results of these consultation exercises were
analysed both quantitatively and qualitatively. A persuasive case
was made against the multi-track assessment process, which was
thought by many respondents to encourage unhelpful stratification
of the sector. In developing the processes for RAE 2008, the Funding
Bodies have taken into consideration the responses to the consultation
on Sir Gareth Roberts' proposals, and incorporated many of the
key features into the assessment process. Ultimately the Funding
Bodies are mindful that the RAE needs to carry credibility both
internationally and nationally. This could not be achieved without
substantial buy-in from the HE sector to the revised proposals.
2. We believe that the panel/sub-panel structure
is a positive step in improving consistency and improving the
treatment of interdisciplinary research. We recommend that the
Funding Councils seriously consider the establishment of panel
moderators. (Paragraph 26)
We welcome the recognition that the two-tier panel
structure introduced for RAE 2008 is a positive step towards improving
consistency of assessment and the treatment of interdisciplinary
research. Ensuring consistency of assessment is a major function
of the new Main panels chaired by eminent individuals. We believe
that the panel structure—with sub-panel chairs sitting on
the Main Panels, Main Panels working across a number of disciplines,
and Panel Secretaries appointed to work across a Main Panel—meets
the spirit of this recommendation.
3. We welcome proposals to strengthen the use
of overseas panel members. It should form part of a wider exercise
to benchmark UK research. (Paragraph 32)
We agree that benchmarking UK research is an important
task. The Funding Bodies are working with the OST and others to
consider other possible benchmarking tools and approaches.
4. The panels and sub-panels need to be properly
resourced. Overstretching staff and panel members could lead to
panels reviewing individuals selectively and coming up with a
biased or wrong conclusion on quality. Under-resourcing is an
affront to the researchers and institutions who have gone to the
trouble of putting the submissions together. (Paragraph 33)
We agree the need for a properly resourced RAE. The
Funding Bodies have taken this into account in the establishment
of the core RAE Team; through the revisions to the appointment
process for Panel Secretaries, who in 2008 will be seconded full-time
from HEIs; as well as with the recruitment of Assistant Panel
Secretaries to provide appropriate administrative cover for the
exercise. On 28 September 2004, the four UK higher education Funding
Bodies announced the appointment of 15 distinguished academics
to chair the main panels for RAE 2008. The 15 main panels are
part of a new two-tier panel structure for the 2008 exercise.
5. It is clear that the workload of panels is
excessive but it is less clear how it can be reduced while all
higher education institutions continue to enter the RAE. An excellent
opportunity to reduce the burden on panels has been missed by
the Funding Bodies in their decision not to support different
assessment routes. In particular our proposal to exempt top-ranking
departments would reduce the number of submissions that each panel
needed to consider and enable them to give closer consideration
to submissions. (Paragraph 34)
Whilst we acknowledge that the workload for the panel
members is burdensome, the Funding Bodies' consultation with the
sector on the desirability of the "three track" assessment
process had a mixed response. There was a substantial majority
opposed to the idea, which runs counter to the desire for criterion
referencing. Furthermore, in detailed consideration of the next
process, the Funding Bodies were firmly persuaded that a competition
based on subject review cannot exclude entire institutions. The
two-tier panel system and the use of specialist advisers will
significantly assist the assessment process.
6. We conclude that the definition of research
employed in 2001 is broadly adequate. It is important that the
panels give equal weight to pure and applied research and that
higher education institutions perceive this to be the case. HEFCE
should ensure that it is understood by everyone. (Paragraph 41)
We welcome this recommendation and agree that it
is important that the definition is properly applied in the assessment
of all forms of research. The revised framework is designed to
ensure that applied research, joint submissions and interdisciplinary
research are not disadvantaged. Panel membership will include
people with experience of commissioning and using research, including
those in industry, commerce and the public sector. The Funding
Bodies are working with panel members, subject communities and
institutions to ensure that this is understood.
7. It is not acceptable for peer review panels
to rely on the place of publication as a guarantee of quality.
We recommend that HEFCE instruct panels to desist from this practice
for RAE 2008 and ensure that panels are sufficiently large and
well staffed to make informed judgements of the quality of the
submissions. (Paragraph 42)
The Funding Bodies will seek to ensure that the panels
being established for the 2008 RAE are configured in such a way
that they have within them, and through their special advisors,
the competences they need to support informed judgement. Panels
will not be allowed to make purely mechanistic judgments of quality.
8. The RAE should recognise that excellent research
may not be internationally significant but it may transform the
fortunes of a local business or the provision of public services.
We recommend that quality criteria concentrate more on the impact
of research rather than the place where it has been published.
(Paragraph 43)
In determining the grade quality descriptors, the
panels will take account of the fact that research of the highest
quality may not always have an international dimension.
9. We recommend that, once formed, panels publish
rapidly clear guidance on how they plan to use metrics to inform
their appraisals. This should be considered a priority. (Paragraph
46)
There is widespread support in the higher education
sector for an assessment process which is underpinned by peer
review. The current RAE 2008 framework already allows for differential
use of metrics in different disciplines; but this would be a matter
for each subject panel to consider and consult with its subject
community about. Notwithstanding this, the Funding Bodies are
investigating how metrics may be used in future to support or
challenge the decisions of panels. In September HEFCE jointly
sponsored with OST an international workshop on the use of metrics
to examine these issues.
10. The introduction of a quality profile is a
significant step forward and, if associated with an equitable
funding formula, could eliminate many of the iniquities of the
previous grading system. (Paragraph 49)
We welcome this recognition that the quality profile
is a potentially useful measure to reduce games playing and the
difficulties of "cliff edges" in grading structure.
We see this as one of the major reforms enabling the Government
with Funding Bodies to assess and reward areas of research excellence
wherever they might be found.
11. We welcome HEFCE's acceptance that the tactics
employed by universities to improve their RAE grade are not all
part of a legitimate research strategy and recommend that it publishes
analyses of the strategies being employed by institutions and
provide guidelines on what it considers to be acceptable practices.
(Paragraph 52)
The Funding Bodies recognise that the RAE is as susceptible
to "games playing" by those who participate in it as
any other competition, and have taken steps to limit the degree
of advantage which can be gained from such tactical activity.
As in previous exercises, the Panels will consider institutional
research strategies (both retrospective and forward-looking) in
their assessment process. There will be Equal Opportunity training
and guidance for panel members. Submitting institutions will also
have to declare that they have Equal Opportunity policies in place,
which have been operated in relation to the RAE submissions process.
12. It seems likely that the media will defy the
Funding Bodies' best intentions and distil the quality profile
down to a single figure, thus encouraging the exclusion of the
least productive researchers if this is permitted. (Paragraph
54)
The Funding Bodies are committed to publishing the
results as "quality profiles" and will attempt to explain
the procedures for collecting and assembling those profiles. Although
it is true that single "league tables" will oversimplify
this complex data, it is not possible to pre-judge how the media
will interpret the results of the RAE.
13. We think that greater transparency about the
RAE, including the public disclosure of included academics, would
have many benefits. It is important to highlight the valuable
work done by excellent administrators and by lecturers who invest
time and intellect in their teaching. We feel that greater clarity
about the role of these academics could increase the value attached
to their work. We recommend that as a condition of their block
grant, higher education institutions publish an annual staff audit,
describing the contributions of all members of academic staff
to research, teaching, administrative and other functions. (Paragraph
55)
14. In our earlier Report we reported that women
academics were more likely to take on more of the teaching and
pastoral functions within departments. It is our view that the
issue to be resolved here is the status accorded to academics
who take on these non-research but nonetheless essential roles.
(Paragraph 56)
(13,14) The Funding Bodies recognise the important
work undertaken by all those involved in the Higher Education
system, whether researchers, educators or administrators. The
creation of the UK-wide HE Academy to focus specifically on the
learning experience, including standards and development, enhancement,
and research will further support the Funding Bodies' individual
initiatives. The review of research priorities, and the way that
institutions plan, develop and use all their staff are matters
for institutions to determine as part of the management of their
organisation.
However, it is not the place of the RAE to make judgments
on the quality of either teaching or administrative work. Teaching
quality is already assessed and reported by other means. An additional
survey of the non-research contributions of all staff would in
our view be an unwelcome extra reporting burden on the sector,
and would add little value to the judgments formed either through
the RAE or other assessment and monitoring processes.
The Research Assessment Exercise makes no judgment
about the quality of non-research activity undertaken, nor the
status of those who undertake it, whether male or female. HEFCE
has undertaken its own research into the issue of female participation
in RAE 2001, and will be discussing this issue with, amongst others,
the AUT in the near future.
15. HEFCE assures us that panel members, secretaries
and RAE team staff will be bound by a duty of confidentiality.
We anticipate that this will be challenged in the courts in a
bid to reveal publicly the judgements made about the performance
of researchers. We recommend that such a move should be pre-empted
and that the grades awarded to individual researchers should be
made public. This would bring welcome transparency to the process.
(Paragraph 57)
All those involved in the assessment process will
be bound by a confidentiality agreement. Furthermore, it is very
important that all those in the assessment process understand
that the assessment is not an assessment of individuals, but rather
it is an assessment of the quality of research undertaken by an
institution in a submission to a unit of assessment. We will be
discussing this with Panel members to ensure that they do not
operate at the level of grading individual researchers.
16. Although Sir Gareth's proposals for the assessment
of "research competence", if implemented, would place
a heavier burden on the RAE, we believe that the need to promote
good practice is so important that it should be used alongside
other incentives to promote good practice in higher education
research. (Paragraph 61)
As noted in response to recommendation 11 above,
the assessment of research strategies is an important part of
the assessment process in the RAE. Furthermore, the Funding Bodies
are introducing other measures of research competence (eg through
introducing minimum standards for Postgraduate Research Degree
programmes and Human Resource Strategies), which should help to
ensure good practice in research. The RAE process itself explicitly
recognises that institutions should have appropriate equal opportunities
policies in place, paying particular attention to gender, ethnicity,
disability and experience issues.
17. We are disappointed that the Funding Bodies
have rejected any form of mid-point monitoring. The RAE is designed
to fund research excellence selectively and this funding should
therefore reflect a department's current, and not only past, capabilities.
(Paragraph 62)
In the Funding Bodies' consultation on the Robert's
Review, 80% of respondents were against mid-point monitoring.
This was seen to place a much greater burden on institutions already
concerned by the level of work required to submit to the RAE.
Institutions themselves have the opportunity to distribute funds
according to current capabilities and needs, as QR is made as
a block grant to institutions, not to departments.
18. The figures provided by the Funding Councils
of the cost of the RAE to institutions do not appear to be excessive.
The fact remains that this burden is resented by universities.
The Funding Bodies should be sensitive to this feeling when developing
their plans for 2008. (Paragraph 66)
We welcome the acknowledgment that the overall cost
of the RAE is not excessive. The Funding Bodies endeavour to keep
the burden on institutions to the minimum required to produce
robust results upon which to make important funding decisions.
19. The Funding Bodies' proposals have addressed
positively many of our concerns about the RAE mechanism and HEFCE
has adopted a more open-minded and constructive approach to its
reform, which is a welcome change. A more radical approach, employing
a range of metrics to reduce the bureaucratic burden on universities
is still needed. We accept that their application will be a complex
and time-consuming task for RAE and the Funding Bodies but we
believe that the administrative burden should fall here rather
than on the universities. (Paragraph 67)
20. We conclude that a range of measures could
be used to replace the peer review process in some subject areas,
such as the physical sciences. There are strong reasons to believe
that they could be as reliable as the current system while being
more cost effective and imposing less of a burden on institutions
and panel members. We recommend that the Funding Bodies commission
an external study to consider options. (Paragraph 74)
(19, 20) We welcome the Committee's acknowledgement
of the positive steps the Funding Bodies have taken to reform
the RAE mechanism. The Science and Innovation Investment Framework
2004-2014 sets out the Government's commitment to develop metrics
to shadow RAE 2008 to provide robust information to benchmark
the outcomes of the RAE exercise (para 3.48). This will ensure
that the metrics collected as part of the next assessment exercise
will also be available to undertake an exercise shadowing the
2008 RAE itself. The aim will be to provide a benchmark on the
information value of the metrics as compared to the outcomes of
the full peer review process. DfES and OST are working closely
with HEFCE and other partners (including international expertise)
to explore the use of a range of metrics, and the Funding Bodies
and key stakeholders are looking beyond the 2008 RAE to see how
metrics can be most sensibly deployed to reduce the burden on
institutions without compromising the credibility and reliability
of the exercise.
21. We accept that there are practical difficulties
in delaying the next RAE and recommend that the RAE continue as
proposed in 2008 but that the Funding Bodies draw up a clear timetable
for the development of alternative models of research assessment.
(Paragraph 77)
As stated at the beginning of this response, the
Government and the Funding Bodies welcome the Committee's recommendation
that the RAE should go ahead as planned in 2008. All aspects of
the RAE process will be kept under careful review and inform decisions
about research assessment in the longer term.
22. Departments need to know how to play the RAE
game, yet HEFCE is asking them to do it blindfolded. HEFCE should
draw up guidance to universities on how the quality profile will
be used to calculate the funding. We appreciate that there are
a number of variables that cannot be known in advance of the RAE
but HEFCE should have the capability to produce estimates which
would enable it to provide indications about the level of funding
provided to each band of the profile. It should do this without
delay. (Paragraph 79)
It is important to separate the need for transparency
and support for institutions and departments which need to know
how to follow the rules, from unhelpful games playing. The priority
is to fund the best research, but we are mindful of Sir Gareth
Roberts' recommendation about the steep cliff edge. As his report
says, there are too many unknown variables - the gradient of each
quality profile; the numbers of departments submitted; the numbers
of staff submitted and the exact level of funding from Government
- to make it virtually impossible to make any definitive statements
on funding prior to the RAE results being known. The general thrust
of the report welcomes attempts by the Funding Bodies to minimise
games playing (para 80). The Government will work with HEFCE on
funding to ensure that the sector is clear about our overall policy
objectives, and it will consult with HEFCE about funding levels
within our overall policy of excellence.
23. It is not clear to us why HEFCE has deemed
it necessary to further increase the level of selectivity of QR
funding. We regret that it will intensify many of the problems
caused by the RAE and the funding decisions based on it. (Paragraph
81)
The Government's policy is to target resources selectively
on the best quality research to sustain world class status in
the long run. Within the context of finite funds, it is important
that the Funding Bodies should continue to fund the highest quality
research (as judged by the RAE) at a sustainable level. The selectivity
ratio will therefore vary according to both the overall performance
by the sector in the 2001 RAE and the funding made available by
Government. A less selective ratio for funding would have to be
at the expense of reducing funding for 5 and 5* departments.
24. We welcome HEFCE's capability funding as a
means of building research capability and promoting dynamism in
the research base. We are concerned, however, that it is too restrictive.
We believe that all departments should be eligible and grants
should be awarded on the strengths of their research and investment
strategies. (Paragraph 84)
The Government believes that HEFCE's present policy
to support research in emerging areas is appropriate. HEFCE's
research capability fund is a recognition that certain subjects
have a less well established research base. It is targeted on
those subjects and on departments in those subjects that have
demonstrated a reasonably strong base of existing achievement
on which to build. HEFCE has reviewed its capability funding and
has agreed to work with Research Councils and other partners to
identify and jointly fund research in sub-disciplines (for example
chemical engineering or nuclear physics) where the UK's capacity
is threatened.
25. We are pleased that Sir Howard Newby now recognises
that a policy of highly selective research funding, based on the
RAE, has had an effect on the viability of university departments
in core subjects. The RAE does not take place in a vacuum and
further changes are also now taking place in higher education
following the 2003 White Paper and the introduction of variable
tuition fees. It is too early to say what the precise impact of
those changes will be. Concerns expressed so far, however, suggest
that variable fees may also lead to closures of further university
departments, quite possibly in the physical sciences. The operation
of the RAE and variable fees may, therefore be mutually self re-enforcing
and HEFCE should remain vigilant in these respects. (Paragraph
90)
The factors affecting departmental closures are complex.
The principal driver for closures is a lack of student demand
in certain subjects. In some cases, it may be part of the institution's
own financial management and institutional strategies, which may
be accentuated by poor research performance. The ten year investment
framework asks HEFCE to work with a range of stakeholders to co-ordinate
national effort to address these subjects.
26. The provisions for HEFCE to delay closure
or offer funding to struggling departments have been criticised
for threatening the autonomy of universities but this encroachment
on their independence is a price worth paying for the preservation
of core disciplines on a national basis. We accept that these
powers should be used with restraint but this is an important
shift in policy we welcome. (Paragraph 91)
HEFCE has welcomed the Government's request for the
Council to explore the proposal that institutions should give
notice to HEFCE of departmental closures. The issue is not about
stopping autonomous individual institutions making sensible business
decisions. It is about the sum of individual decisions having
a negative impact on access to certain subject provision in regions
and sub-regions.
27. The Government says it will assess the "trajectory"
towards the full economic cost model in 2006 in time for the next
Spending Review. We hope that it will use the opportunity to review
whether the model is a viable one and whether the aim of rebalancing
the dual support system could be achieved by a straightforward
increase in the research funds available to the Funding Bodies.
(Paragraph 94)
The Government has stated its commitment to Dual
Support and to continue to increase funding for both sides of
the Dual Support system. This has been welcomed by the Funding
Bodies and the sector. It is recognised that the Funding Bodies'
side of dual support has not kept pace with the other side. The
Government's intention is not to increase either leg of the Dual
Support system at the expense of the other, but to achieve balance
on both sides of the system against a rising overall investment.
We support the principle that institutions should
understand the costs of their various activities and that they
should aim to operate on a sustainable basis. HEFCE's Financial
Memorandum with institutions requires them to recover the Full
Economic Costs across the full range of their activities. Not
to do so would continue to erode the HE asset base with long term
damage to the higher education system. OST has worked closely
with the sector to ensure the modelling base for Full Economic
Costs (FEC) is based on robust data. The percentage of FEC and
the timetable for reaching 100% FEC are due to be announced in
early December.
28. We would like to see diversity in higher education
research funding but it is hard to see how this can be achieved
while the RAE dominates the funding landscape. We have concluded
that new incentives for all areas of universities' work are needed.
Quality assessment for teaching has proved problematic and unpopular.
The Government should consider more radical solutions, perhaps
awarding teaching funds on the basis of outputs rather than inputs
as has been the case. The "third leg" funds for knowledge
transfer have grown in recent years but it is nor clear whether
they are yet sufficient to act as an adequate counterbalance to
RAE-based funding. We conclude that a greater diversity of funding
streams would act as a counterbalance to the RAE. The proposed
European Research Council could contribute, as would the greater
availability of research funds from other Government Departments.
(Paragraph 96)
The ten year investment framework (in paragraph 3.4)
highlights one of the strengths of the UK university system as
its ability to secure a wide range of current and capital funding
for research from a variety of public, private and other sources,
ie Other Government Departments, UK based charities, UK industry,
EU bodies and others. In the HE White Paper, the Government recognised
the great deal of diversity that exists within the sector. We
believe this needs to be further acknowledged and supported by
a funding regime which enables each institution to choose its
mission and the funding streams necessary to support it; and to
make sure that our system recognises, celebrates and encourages
different missions properly. Knowledge transfer is of particular
importance. As noted in the ten year investment framework, the
Government's aim is to create a funding regime which promotes
and rewards high quality knowledge transfer. As part of the 2004
Spending Review settlement, funding available for the Higher Education
Innovation Fund (HEIF) will increase to £110 m a year by
2007-2008. The Government will also move towards a predictable
funding allocation to HEIs on a national basis for knowledge transfer.
This new allocation process will be introduced for a substantial
part of the next round of HEIF.
November 2004
| 
