Memorandum submitted by the Society for
the Study of Artificial Intelligence and Simulation of Behaviour
(FC 54)
NATURE OF
THIS SUBMISSION
(i) This submission is by the executive
Committee of The Society for the Study of Artificial Intelligence
and Simulation of Behaviour (SSAISB or AISB), which is the
main academic society in the UK for the discipline of Artificial
Intelligence (http://www.aisb.org.uk).
(ii) Founded in 1964, the society has an
international membership drawn from both academia and industry,
and from a wide range of disciplines, such as Philosophy and Psychology,
as well as Computing. It is a member of the European Coordinating
Committee for Artificial Intelligence (ECCAI) (http://www.eccai.org).
(iii) We make references in various places
to the HEFCE proposal / consultation document concerning the proposed
new Research Excellence Framework (REF) at http://www.hefce.ac.uk/pubs/hefce/2009/09_38/09_38.pdf.
(iv) Comments on this submission should
go to the main author and society chair, Professor John Barnden.
EXECUTIVE SUMMARY
1. Some impact measurement may be reasonable
in the REF, but it should not insist on academic departments themselves
converting their own research results into impact.
2. Two different things should be distinguished
and (possibly) assessed:
ability to produce impactful research; and
ability to convert research results (one's own
or of others) into impact.
3. Our points on this matter are based not
just on what appears to us logical, but also on (a) the pragmatics
of assessment, and, most importantly, (b) properly serving the
governmental policy aims that impact assessment is intended to
serve in the first place.
4. There is a grave need for the HEFCE proposal
to be augmented with safeguards concerning conflicts of interest
of research "users" such as companies who are involved
in impact assessment and who could benefit from particular assessments
of impact.
DETAILED COMMENTS
5. In the following we put aside grave concerns
we have about an excessive amount of weight being given to whether
an academic department has already demonstrated the economic/social
impact of (some of) its research. By inevitably making departments
concentrate on research whose impact can be securely demonstrated
in a 10-15 year period, this could constrain research in such
a way as to militate in the long term against the very sort of
impact that the public would understandably like to see from academia.
6. However, in the rest of this document
we assume that impact assessment is in some way to be included
in the REF (Research Excellence Framework).
7. We applaud the careful thought that has
gone into several aspects of the proposed REF scheme, and especially
the avoidance of an expectation that every researcher should be
engaged in impact generation.
8. However, the currently proposed scheme
is in parts inadequate, confused or counterproductive, for the
following reasons.
Confusion of two aspects of impact
9. Paragraph 68 of the HEFCE proposal is
exactly wrong in the view encapsulated in the following segment:
We do not envisage that a unit [eg academic department]
could claim credit for impact which was based on research undertaken
in the unit but which was exploited or applied through the efforts
of others, without a demonstrable contribution by the unit to
that exploitation.
This mixes up two things: having the research
excellence to be able to generate ideas that turn out to have
important and solid impact; and having the ability to convert
research ideas (whether one's own or other people's) into impact.
It seems to us entirely unjustifiable that the originators of
an idea with major impact should fail to get credit just because
they have not themselves done the exploitation. This is especially
important point in view of contributions academics in the Computer
Science area make to open-source software. More generally, the
route to market often passes through a chain of academic and industrial
researchers.
10. But conversely, we feel that conversion
even of others' ideas into impact is worthwhile and should be
credited.
11. Thus, the current scheme is doubly unfair
and illogical. We strongly urge adoption of a scheme that separately
measures both impactfulness— the extent to which a department
or other unit's research has led to impact (however, whenever,
wherever and by whomever achieved)—and the extent to which
the unit is engaged in conversion into impact (wherever the ideas
came from and however old they are).
12. This separation would have several pragmatic
benefits for the REF assessment process, apart from being fairer
and more logical. It would help to some extent with the time-lag
problem of deciding how far back the exploited research ideas
can go, as this issue would only be relevant to assessing impactfulness
(it simply does not matter in the case of conversion how old the
ideas are). It would help by encouraging separate sets of rules,
criteria and indicators for impactfulness and conversion, rather
than mixing them up together.
13. Also, the current mixing-up of the two
aspects is potentially dangerous to the very policy that impact
assessment is meant to serve. Chasing conversion to impact could
inhibit the production of the very sorts of ideas that could have
long-lasting, sustainable, revolutionary,... impact. Equally,
requiring conversion to be based on a unit's own ideas could inhibit
the very sort of conversion activity the government would like
to encourage.
Who is responsible for conversion?
14. We also wish to draw attention to two
matters of general policy where we feel that most discussion about
impact, knowledge transfer etc. is naive, misguided and misdirected.
15. First, whether an attempt at impact
succeeds is often dependent on matters beyond the control of researchers
or even whole universities, making it invidious to compare different
attempts without taking all relevant external factors into account,
such as the politics of the day, the general economic climate,
volatile technological fashions, etc.
16. Secondly, the impact assessment scheme
in the HEFCE proposal falls into the trap that many have fallen
into in the past of imagining that it is primarily the responsibility
of universities to do conversion into impact. Conversion is at
most a responsibility shared with industry and government, and
indeed is arguably more the responsibility of the latter two organs.
17. Rather, it is impactfulness, as opposed
to conversion, that universities might be expected to provide.
User involvement in impact assessment
18. We are worried by the statement in para
96(c) of the HEFCE proposal that it is mainly user members of
panels who would assess impact submissions. Users should certainly
be strongly involved, but academics need also to be strongly involved,
to ensure that the claimed impact is genuinely founded in research
ideas rather than only having a superficial connection to a body
of research. It is very easy for someone who is not an academic
expert in the discipline to have a wrong idea of the nature or
significance of ideas within the discipline. The HEFEC proposal
is naive as to the wisdom and vision of "users".
19. We are also concerned about how problems
of users' conflicts of interest will be adequately dealt with,
given that users will be allied with particular sectors of the
economy, etc. and with particular companies, etc. The REF scheme's
careful attention to such ethics-related matters as the panel
nomination process and to equality and diversity issues in returned-staff
selection needs to be matched by proper safeguards concerning
user conflicts of interest when assessing particular departments,
projects, etc.
Breadth of impactfulness
20. Paragraph 72 of the Hefce REF proposal
says:
The expert panels will assess the evidence against
two criteria for impact:
— their reach (how widely the impacts have
been felt); and
— how significant or transformative the
impacts have been.
They will also consider how far a submitted
unit has successfully achieved impact across the full range of
activities and contexts appropriate to its field of activity;
it should not be possible to achieve the highest score by concentrating
narrowly on only a part of the territory that the unit might have
been expected to cover. Through the pilot exercise we will refine
these criteria.
We believe that this is wrongheaded in penalizing
departments that exert impact only in a "[narrow] part of
the territory that the unit might have been expected to cover".
The unit could well be able to be much more effective this way
than attempting breadth. Surely what is important that the university
system as a whole (and not even a particular whole university,
let alone a single department) cover the breadth of possible impact,
rather than that each department do so. The provision in the paragraph
could actually be detrimental to the success of government policy
in deriving impact form the university system.
Impact weighting and its uniformity
21. Paragraph 84 fo the HEFCE proposal is
wrongheaded in imposing (great) uniformity of weighting of research
outputs, impact and environment across different academic areas.
The importance of impact (especially) could be radically different
across different areas. Decisions on the issue should not be based
just on simplicity of assessment.
22. Also, we question why a university's
decision to place a department in one area for REF purposes versus
another on the basis of weighting differences should be pejoratively
labelled as "tactical". If the research in question
can validly be assessed in two different areas, then it can validly
be subject to different provisions for those areas. Why should
the choice not be viewed as universities acting in a responsible,
strategic way?
23. As regards the actual weightings proposed,
60% for the actual research outputs seems low for a Research Excellence
Framework. We would suggest 70% for research outputs. The remainder
should be divided in a discipline-sensitive way between impactfulness,
conversion to impact, and research environment, with 10% each
as a guide.
Significance component of research-output assessment
24. A further problem in the currently proposed
scheme is that the "significance" component in research-output
assessment allows for economic/social/... impact as well as academic
impact/significance. This seems entirely illogical now that there
is a separate impact arm of assessment. The impact part of a submission
to REF could refer to relevant research outputs. Removing economic/social/...
impact from research-output significance would also have the pragmatic
benefit of easing the task of academics in assessing outputs and
would reduce or eliminate the possible need for complex provisions
for "users" to be involved in research-output assessment
as opposed to impact assessment.
January 2010
| 
