Examination of Witnesses (Questions 200
- 219)
MONDAY 21 JANUARY 2008
DR ANDREW
BIRD, MURRAY
BUTCHER, JERRY
JARVIS AND
GREG WATSON
Q200 Fiona Mactaggart: I have heard
that administration is hard, and that we need particular skills
to assess the can-do bits of the coursework of the Diplomas. Do
you think they are going to work? A yes or no answer will be sufficient.
Dr. Bird: Yes.
Murray Butcher: Yes, we are working
very hard to ensure that.
Jerry Jarvis: I would rather come
back with the statement that they have to work. We are making
a huge investment, and we have made a very bold decision. We all
feel exactly the same way. We all suffer from trepidation in a
whole series of different areas, but we have to make this Diploma
work.
Greg Watson: I will say yes, with
one important proviso. I return to an earlier question about whether
employers and universities will really feel that the Diplomas
are worth while. The answer is that they will see the proof with
their own eyes, because they will get to know young people who
have been through these programmes. Whether the Diplomas will
succeed in the early years will be more about how they are taught,
and then about the kind of young people that they produce. Regardless
of all the technicalities of assessment—and there will be
issues that we have to iron out over time—if the Diplomas
are really to succeed in the first two or three years, it will
be all about the teaching. That means that support for teaching
has to be exemplary and that we should be sensible about the number
of students that we want to see on those programmes early on.
As Andrew said, it has been helpful that the Government have put
some criteria in place. If the numbers do not grow too fast, and
we devote an extraordinary effort—not just the Government,
but people like us—to supporting teachers in the first few
years of teaching, Diplomas will succeed.
Chairman: Murray, may I ask you to be
one of the first respondents to Graham?
Q201 Mr. Stuart: Fiona has given
you a pretty good work through. We ended there on a positive note
that, "We are going to make it work because we've got no
choice." City & Guilds has the greatest experience of
vocational training; may I put it to you that there has been insufficient
time for component awarding bodies to research and develop new
assessment approaches? Are Diplomas being introduced to a political,
rather than educational, timetable?
Murray Butcher: Timetable, perhaps.
In terms of trying to draw—
Q202 Mr. Stuart: Is that a yes or
a no? Would any one of you, or anyone whom you know in the educational
establishment, have gone for this timetable if it had not been
dictated by the Government?
Murray Butcher: I think I would
need to collaborate with my colleagues on either side to reflect
on how some of the Curriculum 2000 qualifications were introduced.
That was also an extremely short time scale, but it was achieved.
It is possible that we can achieve this, but some very concentrated
thought is needed. It is, in historical terms, more of a political
timetable than a curriculum development one—that is true.
Q203 Mr. Stuart: Yes; it comes in
in a few months, and you say that there is a lack of clarity over
the purpose of the qualification and its underpinning curriculum.
Murray Butcher: Yes—
Q204 Mr. Stuart: That is pretty damning
is it not?
Murray Butcher: There are considerable
concerns—if I can explain. As a vocational awarding body,
we see this as an opportunity to introduce a hint of vocationalism
into the school and post-school curriculum. We carry considerable
anxieties that, as we go through that process, the pressure on
the award may limit the amount of vocational experience available
and that the award may drift back to the standard, general qualifications—much
as we believe GNVQs did. My anxiety is that if it is successful,
this qualification will also be a challenge for higher education,
because I believe that it will draw in a slightly wider cohort
than has been experienced up to now. There is a lot of pressure
on HE to increase its intake, and it must be borne in mind that
the nature of the intake, as well as the skills and approach of
young people going into higher education, may change. There will
need to be a response to that.
Q205 Mr. Stuart: The advanced Diploma
is supposed to be worth 420 UCAS points, whereas three As at A-level
are worth 360 points. Is that realistic? Will it carry public
confidence?
Murray Butcher: I do not see why
it should not. We are devising a new qualification that is trying
to attract young people with a range of skills. I would not wish
to compare the skills of this young person to that young person.
If they are successful in the award, they deserve the three or
three and a half A-levels that they achieve.
Q206 Mr. Stuart: People have had
theories over the years and they have always come up with conceptually
excellent ideas that, like GNVQs, end up not working. The two
biggest challenges, according to The Guardian, are university
endorsement and parental endorsement. Is not the real challenge
in getting the new qualification going that you have to get enough
good pupils and good teaching to get it off the ground? The tendency
is that a qualification comes in and it is the poorest cohorts
at the poorest establishment who are often the first adopters.
Is that not a danger? How can we prevent it from happening, so
that the Diploma can flourish?
Jerry Jarvis: You are absolutely
right. I made the point earlier that the reputation of this qualification
will be critical to its success. The programme of introduction
is too fast. There is too much involved in this programme, and
too many other parallel changes are going on at the same time.
We would not have done it this way, but we have made a commitment.
We have had our arguments with our regulator, with the Department,
with each other and inside our own organisations, and we have
made a commitment, so there is a point at which we have to say,
"We wouldn't have started from here, but we have to make
this work." So much is at stake. The ultimate success of
the qualification—
Q207 Mr. Stuart: What if we had had
you here when we were talking about GNVQs? We can look at every
previous initiative, because we have always recognised we do not
have the right system for vocational education. If we had had
your equivalents in the past year, you would have all said the
same thing, wouldn't you—about the need to make it work?
And they have not worked. I am trying to tease out just how serious
the problems are with the lack of clarity over the purpose, the
timetable, and the risk of undermining GCSEs and the AQA coming
in.
Greg Watson: I am not sure I necessarily
agree with the statement that GNVQs did not work in the end. Through
a process of iteration over about 10 years, GNVQs became confidently
used in some schools and colleges, but it took 10 years. When
the Education and Skills Committee looked into Diplomas—we
offered evidence about the speed—there was, I think, recognition
in the Committee's report that going at this speed does not help
and putting pressure on numbers early on does not help.
I mentioned OCR Nationals earlier. When we were
developing those, we probably took a year longer than we took
over Diplomas. That gave us a year to triangulate more effectively
between what employers wanted and what schools and colleges thought
they wanted, and to look hard at assessment structures and ensure
that grading arrangements and the like were going to work well.
I think we would have all wished for a little more time. On a
positive note, there has been an excellent dialogue between ourselves,
the Department and the QCA about the first phase of Diplomas and
how that process works. I think everybody has recognised that
it was very loaded towards the employer view and we did not take
on board enough about teaching and pedagogy and did not, early
on, spend enough time on the business of assessment and standards
and ensuring that the outcome of the Diplomas would command confidence.
There is a commitment all round, among all the people I have just
talked about, to go about the second phase in a rather different
way. That in itself shows that we are perhaps recycling the learning
faster than we might have done with the GNVQ.
Chairman: There is a bit of pressure
on time. We have to move on to coursework.
Q208 Annette Brooke: You have already
hinted that there are quite a lot of advantages in using coursework
in the examinations system. Given the rather dramatic moves, particularly
with the GCSE, is this a question of throwing the baby out with
the bathwater?
Dr. Bird: In my early remarks
about coursework and the extended project, I was, I suppose, broadly
hinting in that direction. Very few people in their daily lives
do not discuss with colleagues an issue they are trying to tackle,
or use the Internet or any of those wider research skills. Clearly,
properly constructed coursework questions allow people to demonstrate,
in preparing a document for moderation, those wider study skills
and extended communication skills, which it is very hard to assess
in a quite short, formal examination.
Q209 Annette Brooke: I shall ask
another question and perhaps somebody else can pick it up. Is
it simply that you guys have not been smart enough in setting
the coursework?
Chairman: Who is that question to?
Annette Brooke: I am looking for volunteers.
I though that it would be a bit unkind to pick on somebody.
Chairman: Jerry can go first, followed
by Greg.
Jerry Jarvis: I think that you
have two volunteers here. It is probably a case of throwing the
baby out with the bathwater. It is not that difficult to detect
plagiarism. After all, the same mechanism that allowed the student
to find the material to plagiarise makes it just as easy for the
moderator to find it. We believe that coursework makes a hugely
valuable contribution to the way in which we conduct assessments.
It is a far richer method than an end-of-term examination. Last
year, Edexcel did a huge amount of work on detecting cheating
throughout the UK. We examined all sorts of issues, but detecting
cheating in coursework is easy, actually.
Greg Watson: Yes, I am disappointed
with our direction of travel. Hopefully, some of what we learn
from the extended project within the Diploma will flow back into
coursework. Ironically, the extended project could prove to be
one of the most exciting parts of the Diploma development. It
could be a genuinely personal and independent project that would
build evaluation, research, thinking and self-organisation skills,
which are exactly the sort of skills that coursework—done
well—has the power to develop. Coursework is not right for
every subject, and in some ways we have suffered from having veered
from one side of the road to the other. At one time, we decided
that coursework was a good thing and, through over-mighty regulation,
we were compelled to put coursework into subjects such as maths,
where it does not sit readily. There has been a sudden change
of tack and now we are compelled to take coursework out, even
when, in subjects such as geography or history, it can breed fundamental
skills that, incidentally, carry on very well into higher education,
too.
Murray Butcher: I was just reflecting
on some good, simple assessment reasons for maintaining coursework.
We are trying to get as broad a picture as we can of individuals'
abilities. If you take out coursework, you are focusing on just
one or two other forms of assessment, which gives you a biased
picture of the individual. I share the view that by withdrawing
coursework, we would be going in the wrong direction.
Q210 Annette Brooke: On gender balance,
I have the impression that girls started performing rather well
in some GCSE coursework. Is that statistically true? I suppose
that I should look to the awarding bodies for that information,
but I thought that you might have the gender breakdown of results
with and without coursework.
Jerry Jarvis: We could provide
it but, anecdotally, it is generally the case that girls do better
in coursework, as well as the other subjects, although that varies
enormously in coursework subjects. However, yes, they are better
at coursework.
Q211 Annette Brooke: I find that
interesting. I wonder whether getting rid of coursework would
solve one of the Government's problems by potentially closing
the gender gap—I accept that that is rather cynical. Something
that concerns us greatly—this might be in line with our
questions and other lines of inquiry on social mobility—is
that young people from different backgrounds and with differing
levels of family support clearly will have different opportunities
within their coursework. For instance, I recall that the Prime
Minister's wife could get help with certain coursework, and I
used to take my children to museums and so on. How do you factor
in those things for a true assessment?
Chairman: You cannot, can you? Is it
fair? Is it biased towards middle-class kids? Do less privileged
kids find it harder to achieve?
Greg Watson: That is very hard
to iron out. However, returning to something that a couple of
us have said, the notion of a truly personal piece of learning
could get over that problem, because it would be in the bounds
of the imagination and creativity of the young person and their
teacher. I can think of good examples in GCSE business studies,
in which coursework is open-ended. A student goes to a local business,
works up a project on one aspect of the business, and writes it
up. A teacher who encourages young people to think imaginatively,
or a school that maintains good links with local employers, is
able to do that, regardless of any advantage or disadvantage in
the home background. Part of what creates the difficulty is the
tendency, through regulation, to drive coursework towards being
a standardised task. If anything, those other advantages kick
in to a greater degree when that is the case.
Chairman: David, do you want to come
in on coursework?
Q212 Mr. Chaytor: Yes. Specifically,
what are the most effective means of preventing plagiarism in
the first place? I take the point about making coursework more
personalised, as against standardised, but how do you prevent
someone from accessing the Internet or from getting a disproportionate
advantage from having well-informed parents?
Jerry Jarvis: The view that we
have taken is that the best way to prevent plagiarism is the likelihood
that your efforts will be detected—it is like the idea that
you will probably drive slower if you suspect that there is a
traffic camera around. The best way to do that is to sample students'
work. As I said earlier, moderators and examiners can use the
same methods to access non-original material as the students did.
These days, even with material that is being shared among students,
which might not be generally available, we have deployment technology
and we digitally scan everything that we do, so we have the ability
to have a machine literally looking for similarities that could
not otherwise be detected without the deep reading of work. For
me, the way to discourage plagiarism is through the probability
that you will be caught.
Chairman: Andrew, do you want to come
in on that?
Dr. Bird: I agree with some of
that. I think the real way of inhibiting plagiarism is to make
sure that you have set the right task. If we set a task that is
basically a knowledge task, we are encouraging somebody to go
to the Internet and find out all they know, so we need to set
a task that requires them to acquire some knowledge, from wherever
they get it, and then to do something with it—to process
it and convert it into something that is about them and their
insight into that knowledge. The task needs to be about underlying
ability, rather than a recounting of what they can find in a book
or on an Internet site. Tasks that require people to give empathy
of analysis are the sort of coursework tasks that we ought to
be setting. That, in my view, is the strongest way of avoiding
plagiarism, followed by the sense that there is a speed camera
around the corner.
Q213 Mr. Chaytor: City & Guilds
is particularly keen on the role of coursework, and local assessment
of coursework is more frequent in vocational qualifications than
in more academic qualifications. What are your views on all that?
Do different standards apply to courses that are defined as vocational,
as opposed to those that are defined as academic?
Murray Butcher: There are similarities
and differences. Within a National Vocational Qualification course,
you are dealing with the fact that the curriculum contains certain
criteria. When you come to assess what, in vocational qualifications,
is called the portfolio, which is a collection of evidence, you
are testing that evidence against certain specified criteria and
looking for the supporting evidence within it. That does not mean
to say that there are not occasions when you find that individual
students and trainees come up with extremely similar evidence.
Supported by regulation, we must carry an investigations team
within City & Guilds so that whenever questions occur that
might suggest plagiarism—or not so much plagiarism as the
possibility that the centres providing the documentation are not
encouraging individuals to do things themselves—we need
to explore that and confirm whether something untoward has taken
place. So, in a sense, plagiarism is possible in NVQs, and we
have to make sure that we eliminate it.
Q214 Mr. Chaytor: Broadly, what proportion
of awards are rejected because plagiarism has been identified
by City & Guilds and the other boards?
Murray Butcher: Within City &
Guilds, it is going to be an extremely small percentage—less
than 1%.
Greg Watson: For OCR, 192 candidates
in 2007 had sanctions applied for plagiarism.
Q215 Mr. Chaytor: As a percentage
of the total?
Greg Watson: Minute.
Q216 Mr. Chaytor: Broadly, on the
use of the Internet in coursework and to support learning for
external examinations, is there any evidence that the digital
divide between families is leading to a widening gap in the achievement
of young people? Is access to the Internet a factor that drives
up standards very quickly for the most affluent families, leaving
the children of poorer families behind? Is anyone doing any research
on this, or do you have any thoughts about it?
Greg Watson: That is actually
a very difficult question to answer from where we are sat. We
assess what is put before us. Given that most of it arrives in
a digital format, you could say, prima facie, that there
is no obvious divide, but I cannot gauge—I am not sure that
I even want to—who has used what means to get there. We
simply mark it all to the same standard when we are presented
with it. There is an interesting research subject there, but I
cannot comment from the evidence that we see.
Jerry Jarvis: Following on from
an earlier question about setting an examination to ensure equality
of access, particularly for disadvantaged children, we do quite
a lot in setting all our examination and assessment work on the
more obvious things, such as language and culture. We would not
want to set a paper in geography in which someone who was a non-UK
resident would be disadvantaged, so we think very carefully about
that. We would not require someone to do a field trip to France,
so we can make sure that when we set our assessment instruments,
we do not. However, I am afraid that one or other of the population
will be inadvertently disadvantaged. Geography comes in as well,
with remote villages and so on where access to appropriate research
facilities is limited. We have to be careful but, inevitably,
some students have greater advantages than others.
Q217 Mr. Chaytor: May I move on to
the QCA? All four boards agree that the changes to the QCA are
welcome. That is right, is it not?
Dr. Bird indicated assent.
Greg Watson indicated assent.
Jerry Jarvis indicated assent.
Murray Butcher indicated assent.
Q218 Mr. Chaytor: But what has it
done wrong? Where has it gone wrong?
Chairman: And why were you not lobbying
for the change before?
Greg Watson: I was.
Chairman: Oh, you were. Okay.
Dr. Bird: We were as well. The
QCA has not done an awful lot wrong, but having the two activities
together can generate a conflict of interest, and separating curriculum
development from the regulation of assessment can only be a good
thing. The regulators spent a lot of time thinking about the development
of the new A-levels, but never mentioned what it would cost to
assess those various models of assessment. Sometime later on,
QCA quite properly wanted to explore our pricing strategy for
those A-levels—after we had been in development for some
time and the die had been cast. That is the very worst set of
arrangements, because we are already committed to a style of assessment
in association with the curriculum development side of QCA at
a time when the regulatory side started talking about pricing,
which is influenced by what it costs to deliver the style of assessment.
Separation of the two roles should lead to a more robust dialogue
between the regulatory side and the development side so that we
triangulate the issues more effectively, because the consequence
of development requirements on us changes what we can deliver
and what it might cost to put it into the marketplace. One can
see a more robust triangular relationship across the industry,
which is why we favour it.
Q219 Mr. Chaytor: The example that
you have given could equally be used as an argument to scrap the
curriculum development agency function completely and devolve
it to each of the four boards.
Dr. Bird: It could be—
Mr. Chaytor: There is no guarantee that
a separate development agency will also not overlook the practical
implementation costs of the financial arrangements
Dr. Bird: I agree, and I think
some of us might say that that might be quite nice, from the innovation
perspective. Certainly we would encourage the development side
of the organisation to try and be strategic and output-driven,
rather than detailed and prescriptive, in its future thinking.
We do suffer from the very detailed and prescriptive approach
to what A-level English looks like, which does not help innovation
across this table.
Greg Watson: I think, to go right
back to what I said at the start this afternoon, that managing
a large and high-stakes qualification system means constant trade-offs
between change in order to keep qualifications relevant, to respond
to changes in the economy, and to adapt to the different needs
of higher education versus stability, which is the thing that
builds public confidence and familiarity. I think that a good
regulator would constantly be balancing those two drivers. I think
that QCA, because of the position it has occupied very close to
Government, has tended to find that its role in being a sponsor
of change has far outweighed, over time, its responsibility for
stability. I hope that the one difference that we will see with
the new regulator is a greater weighing of the pros and cons of
change at any given moment, whether it is at a system-wide level
in introducing a whole new qualification, in the form of Diplomas,
or at an individual qualification level, in deciding whether to
have more or less coursework in GCSEs. I think that that would
be a very positive development.
| 
