Examination of Witnesses (Questions 242
- 259)
MONDAY 30 MARCH 2009
PROFESSOR MARGARET
PRICE, DR
CHRIS RUST,
PROFESSOR ROGER
GOODMAN AND
PROFESSOR ALAN
RYAN
Chairman: We welcome our second even
more distinguished panel: Professor Margaret Price from Oxford
Brookes University, welcome to you; Dr Chris Rust from Oxford
Brookes University; Professor Roger Goodman from the University
of Oxford and Professor Alan Ryan from the University of Oxford;
welcome to you all. I am going to ask my colleague Graham Stringer
to begin the questioning.
Q242 Graham Stringer: Dr Rust, in
your evidence you have said that there are worrying differences
in assessment of degrees across different universities. Can you
detect that this is because of different methods that are used
in assessing degrees or what reasons would you give for the different
processes that there are in the assessment of degrees?
Dr Rust: There are lots of answers
to that question. The work of both of the groups that I am connected
with and representing here today suggests that you can see a whole
host of reasons why—for example, very simply, the algorithms
that the university uses to compute the marks into some final
classification. The group SACWG has shown that you can have up
to a degree classification difference with the same set of results
from one student, simply by feeding them into a different algorithm
used by a different institution. Beneath that we know there is
other evidence that marks will vary depending on a host of factors.
We know that students do better on coursework assessment compared
with examination, we know that in certain disciplines—maths
for example—you will get higher marks because you can get
full marks for certain types of activity. What then happens of
course is those numbers all get crunched together in I would suggest
quite indefensible ways if you looked at them as a statistician,
and a whole host of results can come out from that. Is that enough?
Do you want more?
Q243 Graham Stringer: That is a start.
Are these different methodologies the main vehicle for fuelling
degree inflation or are there other reasons?
Dr Rust: Like my good friend Mantz
Yorke I am going to try and avoid the notion of grade inflation
on the grounds that that is a pejorative term. There are a host
of potential reasons for why there are now more firsts and 2:1s.
We can hypothesise about a range; it could be that teaching has
got better—we certainly have more postgraduate certificate
courses for new teaching staff and we have made a move towards
professionalising academics as teachers—it could be that
students are working harder, it could be that we are clearer with
courses framed in terms of learning outcomes et cetera,
so students are clear as to what it is they have got to do and
can then perform to the task, or it could be that we have grade
inflation. The fundamental point that I think I would want to
make today is that we do not know. We can have those discussions
but it is at the level of conjecture. As the QAA said in 2006
we have no system that will actually enable us to show whether
it is inflation or not.
Q244 Graham Stringer: Could it be
that different universities are choosing methodologies that give
them better results because they are concerned about their position
in the league table?
Dr Rust: I have never come across
that or have no reason to believe that.
Q245 Graham Stringer: But you could
not rule it out.
Dr Rust: I just do not know.
Q246 Mr Boswell: Could I just interpose
on what Dr Rust has said? The way I read it is first of all, following
what he said about the QAA, there is actually nobody sitting above
this process as in the schools sector there is for example now
with Ofqual who can moderate it or say what is happening—that
is the first point. The second point is that whatever merits there
may be in the external examiner system they are not actually effective
in moderating these conceptual differences. Is that a fair account
of what you are saying?
Dr Rust: I am not sure I want
to encourage the creation of an Ofqual for higher education.
Q247 Mr Boswell: That is a separate
issue and an important one.
Dr Rust: Otherwise, yes, that
is what I am saying. About external examiners, there are many
merits to the external examiners system, it brings many positive
outcomes, but it is certainly not a system that is going to guarantee
your standards.
Q248 Mr Boswell: As we have raised
the hare about having an Ofqual for HE[2]
or whatever is it your view that that would be a less good or
a better intervention than in effect saying you judge your own
standards and you award accordingly either within a degree classification
or without one?
Dr Rust: There is another possibility
as recommendation one in the ASKe submission. There is a way that
we could develop academic communities to take account of comparability
of standards across institutions, and that is the way to go with
a grassroots-up method.
Chairman: I want to return to that a
little bit later so I do not want to pursue that. We are back
to you, Graham.
Q249 Graham Stringer: It is a similar
point really. If we are not going to have a higher education Ofsted
what is the solution to getting comparabilities between different
universities in degree standards, or at least knowing where we
are?
Dr Rust: As I said, I would support
ASKe's recommendation, but you wanted to leave that until later.
Q250 Chairman: You ought to come
in here, Professor Price, because this is your work.
Professor Price: I would very
much resist the idea that there can be a body sitting above to
actually impose standards because standards very much belong and
are created and are maintained within the academic communities.
That does give rise to the issue of how you make comparisons between
different disciplines and how those disciplines operate within
different institutions. I have just come back from Australia so
forgive me, I am not quite on the same time zone as you, but I
had some very interesting discussions in Australia about the nature
of a sort of chicken-wire network where you would group discipline
communities, where there are sort of overlaps with them, and effectively
create a network whereby you could have comparisons with close
disciplines which then cover the whole of the discipline span.
Q251 Mr Boswell: Is that by institution
or by sector or either?
Professor Price: It would be either,
yes.
Q252 Ian Stewart: How do you do the
sector one? I can understand it within the institution but how
do you the sector one if you do not have a body that oversees
the sector?
Professor Price: You would need
to create those networks between institutions and many disciplines
do have external bodies that they feel more affiliated to than
necessarily their institutions, particularly professional bodies,
and they take their standards from those, so you have already
got networks. One of the things that we have also proposed is
that within the UK there are subject centres that may create a
focal point to allow the communication and discussion of standards
right across the nation.
Q253 Ian Stewart: Would that show
up this very elusive difference of experience that Dr Hood was
talking about?
Professor Price: I suppose it
depends on whether you are talking about outcomes or whether you
are talking about the process, and you need to look at those in
slightly different ways. If you are looking at outcomes one of
the best ways of determining standards is to actually look at
examples, so rather than ask people to talk about standards in
the abstract, which is very difficult to do because standards
are held both explicitly and tacitly, in order to create understanding
between people you have to have concrete examples to look at,
so we can look at outcomes. In terms of looking at processes you
have to be careful about not just looking at the input measures
but looking at the output measures from the students' experience
as well, so we would need to gather evidence and data about that
in order to do that. One of the things that will probably come
out of the discussion today is that there is actually not a great
deal of evidence on which we can draw conclusions.
Ian Stewart: That is honest, thank you.
Dr Harris: Before I bring in Professor
Ryan I want to go back to something that was not quite picked
up from what Graham Stringer was asking. Dr Rust—or anyone—we
observe an increase in the number of firsts and 2:1s relatively
speaking, we observe a variety of techniques in your work that
are used to do marking, some of which, for what look like some
quite small or innocent changes, can have significant impacts
on their own, let alone in combination, to change marks—and
they seem to be changing in an upward direction—yet you
tell us that you cannot say there is grade inflation in a pejorative
way, in other words you see no evidence that this is unjustified.
That is my first question. Secondly, when Graham Stringer asked
whether league tables might be an incentive to have directly or
indirectly this impact you said you could not see that that was
necessarily the case. I have seen no example of league tables
where people do not want to be at the top rather than at the bottom
and I have seen no example of league tables introduced by politicians
without the point of blaming the people at the bottom and rewarding
the people at the top by incentivising people. Putting those two
questions together can I ask you to reconsider your response as
an academic?
Q254 Chairman: Could you do it fairly
briefly?
Dr Rust: I am sure that league
tables incentivise people; what I hope I said was I personally
know of no evidence that a university has changed its system or
even a department changed its system in order to artificially
create higher marks.
Q255 Dr Harris: Human nature ends
at universities, does it?
Dr Rust: I just do not know of
any evidence; I have not come across that happening to my knowledge.
The other point is—and in support of the argument that we
just do not know what the reason for more firsts and 2:1s is—in
fact it is not as simple as saying it has just gone up and gone
up. The latest work that Mantz Yorke has done, which is on the
Academy website, looking at 13 years, shows that for different
subject disciplines in fact it has gone down, so within the same
institution and using the same systems you will have had some
places where in fact the grades have gone down rather than up,
so it is just more complicated than to say it is just going up.
Q256 Dr Harris: Professor Ryan, can
you first explain to me why universities are uniquely different
from every hospital when it comes to the impact of league tables
measuring their performance on their behaviour—if you agree
with Dr Rust?
Professor Ryan: I do not, of course
I do not.
Q257 Chairman: That was a leading
question.
Professor Ryan: He is my MP.
Q258 Dr Harris: I said if you agree.
It cannot be leading if I say "if".
Professor Ryan: How would I disagree
with you? To start at the beginning there once was a version of
Ofqual for universities, it was the CNAA. The non-old-fashioned
sector gave CNAA-validated degrees and nobody in the CNAA believed
that there was anything very clever to be said about whether a
CNAA degree in history was more or less demanding than a CNAA
degree in sociology or whatever. What was true was that you could
not put on a degree course without getting it past the CNAA, it
did look at the syllabuses, it looked at your teaching resources
and the external examiners came from the CNAA and what they had
going for them was they would have been deeply humiliated to validate
and approve of courses that other people later thought were not
up to scratch. It is not so to speak, therefore, an impossible
state of affairs; to my mind the CNAA was much more like the right
animal than the QAA. As to league tables it seems to me it just
has to be an incentive. If you grade people on the number of 2:1s
and the like that they get at the end the temptation is bound
to be to smudge the 2:2/2:1 boundary. If you pay them through
HEFCE in a way that penalises them for throwing people off courses
if they are not up to it then there is bound to be a pressure
to hang on to them at all costs. I just do not see how it can
possibly work differently.
Q259 Dr Harris: Does anyone want
to rebut that? Professor Goodman, you have kept quiet.
Professor Goodman: I have kept
quiet, I have been listening with great interest. I do go back—I
know we had an earlier conversation about this and there were
doubts cast on it—to the external examination system, which
is a system that we utilise as much as possible to get the feedback
on our courses, because we are only as good as the world thinks
we are and if we lower our own standards we are going to be the
ones who are going to suffer the results. We use our external
examiners and we call them critical friends—I have to say
from my end they tend to be pretty critical about the things that
they do not like—and then we review our processes in that
light. They are only comparative of course between their own institution
and our institution, there is not this kind of overview.
Dr Harris: But external examiners can
only do what you ask them to do, so if they do not know whether
the tutors told the students what questions are coming up there
is nothing they can do about that. If they are only asked to arbitrate
between two borderline cases they will do that very well no doubt
in an external examiner way, but is not the whole question how
you use external examiners, especially if you depend on them,
and is there a variation in the way they are used between institutions?
2 Higher education Back
| 
