Examination of Witnesses (Question Numbers
20-39)
MR PETER
WILLIAMS, MR
DOUGLAS BLACKSTOCK,
DR NICK
HARRIS AND
DR STEPHEN
JACKSON
17 JULY 2008
Q20 Mr Wilson: A risk-based system,
so that you can target your resources more effectively and ensure
that standards are more robust by doing that.
Dr Jackson: This is an issue we
are currently discussing in the agency. The current cycle of audits
is due to be completed by 2011. We are discussing what methodology
we should adopt after 2011. One of the models we are currently
looking at is a risk-based proportionate model, with scrutiny
of institutions based on pre-existing evidence about their quality
and standards.
Q21 Mr Boswell: Perhaps I could come
back to something about the grades "no confidence",
"limited confidence" and "confidence". I do
not want to say anything about degree classification obviously,
but are you happy that is the right sort of framework? Would it
be that the level of confidence is sufficiently refined to drive
some institutions to do better, rather than run the risk of resting
on their laurels? In an ideal world, would you design it this
way or would you seek to modify it? To put it another way, to
give you an opportunity at a rounded answer: Can we set a threshold
which is adequate and only that, or are we looking to drive the
system upwards?
Mr Williams: We have a classified
system.
Q22 Mr Boswell: That was a rounded
up way of saying that.
Mr Williams: We use a confidence
judgment. I think it is important to recognise precisely what
a confidence judgment is. Most processes of this sort take a snapshot.
They are essentially historic, looking backwards. They are looking
at what is done—essentially, the day before yesterday. That
is not much use unless you can be absolutely sure that what happened
yesterday is going to continue to happen tomorrow.
Q23 Mr Boswell: This is not much
use to your student sample: they want to know when they are there
that it is going to be all right.
Mr Williams: Our judgment of confidence
is essentially a judgment we make as informed peers, or as our
peer reviewers do, of whether this is meeting, at present, the
satisfactory maintenance of standards and quality and whether
we think in our judgment it will continue. Of course that is really
quite difficult to predict, but we want to see that everything
is in place, so that if individuals leave or things change there
will still be a sufficient infrastructure which will be likely
to carry on maintaining the standards and the quality. That is
what we do. Under those circumstances, the danger of having a
graded profile is that the costs would be greater than the benefits.
Q24 Mr Boswell: Have you ever been
judicially reviewed on the assurances that you have given?
Mr Williams: No.
Q25 Mr Boswell: Nobody has ever sought
the legal route?
Mr Williams: No. We have had some
sabres rattled sometimes but nobody has ever come forward.
Q26 Dr Evan Harris: Is there an appeals
process?
Mr Williams: There is an appeal
against no confidence.
Q27 Dr Evan Harris: Who hears the
appeal?
Mr Williams: The appeal is heard
by a subgroup of our board but it does take external evidence
as well. It has independent external advisers to make sure that
the story that our colleagues tell is valid.
Q28 Dr Evan Harris: I was intrigued
by this idea that a student who wants to pass on concerns about
the quality of the institution and wishes to do so in confidence
is referred invariably to the Office of the Independent Adjudicator.
Let me give you the example. If a student contacts you to say,
"Can I tell you in confidence that I have concerns about
widespread plagiarism in this institution and that is not being
picked up" and there are mechanisms to pick it up, you will
deal with that.
Mr Williams: That is our area
of responsibility. If it is a complaint about the treatment of
the student within the institution or their individual experience,
that is something that we recognise as the responsibility of the
Office of the Independent Adjudicator.
Q29 Dr Evan Harris: The example the
Chairman gave was of a student who said he had been treated too
well. I think the Office of the Independent Adjudicator would
deal with harms to students, not unjustified benefits. Surely
you should not be referring a case like that, even if it is an
individual experience, to a complaints procedure when the student
is not making a complaint on their own behalf. It would be curious.
Mr Williams: It is difficult to
answer on, in a sense, bizarre and hypothetical cases. It happened,
presumably. I think I was given a grade above what I should have
been—but many, many years ago—but I am not a judge.
I am not a competent academic judge. I am not an academic.
Q30 Mr Boswell: I take it you do
talk to one another. If the office for student complaints gets
something which looks like being systematic in an institution
rather than just one person who did not get on with their supervisor,
or, conversely, if you think this is really something which has
its background in an individual clash or difficulty, you can presumably
talk to one another and pass them over.
Mr Williams: Yes. Although I think
it is fair to say that it is only recently that our links with
the OIA have become really effective. We now have a very good
working relationship.
Q31 Dr Evan Harris: I am still concerned.
I must say that I thought the OIA deals with complaints once they
have gone through the university's own procedure and only then.
Mr Williams: That is right. It
does.
Q32 Dr Evan Harris: I think the example
the Chairman gave is clearly inappropriate for the OIA. I am concerned
that you would think that you could directly refer to the OIA,
when you can really only do that once you have gone through the
university's complaints procedure. I can understand that you should
say this is something you should raise with the university first.
Mr Williams: Yes.
Q33 Dr Evan Harris: Have you ever
thought of using other techniques to judge quality than the ones
you have? Some people use what are considered powerful techniques
of quality assurance, such as the mystery shopper. Or is that
just too radical for your austere organisation?
Mr Williams: If we wanted to use
the mystery shopper we would have to be very clear about precisely
what we were doing, why we were doing it, how we were doing it,
and how it would validate the outcomes.
Q34 Dr Evan Harris: Yes, assuming
you could do all that—I am not saying you should do something
that is useless—would you consider that? Have you ever considered
it?
Mr Williams: It is something we
have thought about. We have also thought about things like unannounced
visits. You have to be very careful as to how you are going to
use the evidence you get. The mystery shopper is after a particular
piece of information, so it could be quite a useful technique
for looking at specific questions, but it is not going to tell
you very much about the overall way in which the institution is
managed.
Q35 Dr Evan Harris: As you know,
it is the threat of it: if institutions know that they might be
out there. Most of the thrust of it is to raise standards out
of fear. That is the idea.
Mr Williams: Of course the press
do this from time to time. It is an interesting technique but
we would need to be sure how we were using it.
Q36 Dr Evan Harris: My other question
relates to the use of the audit procedure. Your external audit
is essentially a form of peer review.
Mr Williams: Yes.
Q37 Dr Evan Harris: If peer review
is fine and reckoned to be good enough, then that is reassuring
for you. But if there were questions about peer review, that would
lead to concerns about how you did your business. Are you happy
that peer review, as practised in this country, is as good as
it can be?
Mr Williams: I think the answer
to that question is: It is not a perfect system but it is a good
system.
Q38 Dr Evan Harris: Could it be improved?
Mr Williams: Everything can always
be improved.
Q39 Dr Evan Harris: Really.
Mr Williams: We believe in continuous
improvement. To improve is part of our mission.
| 
