Examination of Witnesses (Questions 160-179)
JERRY JARVIS,
SIMON LEBUS
AND DR
VIKKI SMITH
22 APRIL 2009
Q160 Annette Brooke: We have
mentioned in the Committee an IT qualification that possibly led
to four GCSEs and did not take up a great number of hours per
week. I do not know whether anybody has any comments on that—a
situation where a multiple number of points, as it were, go into
the performance tables, but they perhaps come from an area that
does not necessarily take up a high proportion of teaching time.
Simon Lebus: In a sense, that
is a good illustration of the absurdity of the whole construct.
IT, and the mastery of IT, is a skill; it is something you either
can or cannot do, and there are various features and bits of that
skill that you acquire. A lot of the other subjects that are tested
in general qualifications are knowledge-based or have to do with
understanding. The difficulty arises when people try to create
these equivalences, which is what distorts the behaviour. A number
of IT qualifications perform valuable functions, but they are
essentially skills-based. Trying to create an artificial parity
with general academic qualifications inevitably leads to those
sorts of distortions. Where incentives are attached to that, it
may well direct the behaviour of institutions that are being judged,
as they are used for accountability purposes.
Q161 Annette Brooke: I have
a general question. Cambridge Assessment makes the point that
it would like Ofqual to look at this. That would seem to follow
on from its recent report on the science GCSE, for example. Is
that a general view? Would Ofqual give you all more credibility
if it was looking genuinely at the equivalence of the different
subjects?
Jerry Jarvis: Absolutely. It is
in our interests for public confidence to be raised. We desperately
need Ofqual to become a respected and effective institution. It
is important that parents and students can turn to Ofqual for
confidence in the examination outcomes and the standards set,
and not on issues such as equivalence and so on. It is in all
of our interests that Ofqual performs that role.
Chairman: Vikki, do you want to come
in on this?
Dr Smith: I am a little curious.
I agree with the separation of the different things that can be
acquired in terms of skills and knowledge. Those need to be demonstrated
differently and will be valued depending on where that takes you.
There is difficulty in drawing equivalence across different types
of acquisition. That said, there is a difficulty—I guess—in
drawing equivalence from the IT agenda to English, to Maths and
to French and so on. However, throughout the discussion, I would
not want to see greater weight given to knowledge as opposed to
skill, because that would actually disadvantage a great number
of individuals in the school system. I do not think that that
is where we are going, but I wanted to put that on record.
Q162 Annette Brooke: Thank
you for that; I think it was a very important comment. Finally,
I would like to ask about the qualifications that are currently
excluded from the performance tables and what impact that is having.
That could apply to vocational qualifications that are excluded,
and obviously at the back of my mind are the IGCSEs. What should
Ofqual and QCDA be doing about those?
Dr Smith: Absolutely, I think
that it will serve to reinforce the divisions that have certainly
existed since I have been working in education. Linking back to
league tables and what counts—we have heard mention of teaching
to the test and so on—there will be a funnelling of students
to the detriment of UK plc, because we will be sending individual
pupils through particular streams of education rather than looking
to everything that is available. I think it will have quite a
drastic impact.
Simon Lebus: The problem at the
moment is that the league tables are owned by the DCSF and while
that is the case, there are inevitably suspicions and unease about
how they are compiled. I think that giving them to Ofqual, as
part of the confidence objective in the legislation that is going
through the House, would be an important and valuable reform.
If one takes things such as the IGCSE and pre-U and the fact that
they are not included in the tables, I think it leads to some
manifestly strange results, inasmuch as high achieving schools
appear outside the league tables or at the bottom of the league
tables, because they are not taking the qualifications that the
tables include, which are clearly directly comparable qualifications,
so there is an issue. Having recently been through the process
of getting IGCSE and now the pre-U approved, I think that there
is also an issue of new qualifications being made to fit design
straitjackets, so that they can easily be slotted into the appropriate
spot in the league table. You have a washback effect in terms
of the design of the qualifications that is unhelpful. I think
that the current system does not work very well: it is vague,
imprecise and gives peculiar results that are not felt to be fair
by a number of schools that are taking part. I think it is time
for it to be reformed.
Jerry Jarvis: I am not 100% in
agreement with everything so far. Let me take us another step
back. I guess that our education system at 14 to 19 is dominated
by progression to higher education. We have a fixation on academic
qualifications. Arguably, that is part of the reason why many
industries believe that kids can come through that formal education
process not fit to work, so they have to acquire the so-called
softer skills that we continue to talk about. If you take IT,
for example, I might argue that you can go through a vocationally
based programme and acquire learning in a different way so that
it can be applied in a different way and not necessarily limited
entirely to skills, although there is certainly a movement in
that direction. There is an issue, of course, about the way that
we produce equivalence in order to have those points scored. There
is no question about it: vocational qualifications contribute
enormously to that. The vast majority of existing vocational qualifications
count. In fact, looking at the BTEC qualifications that are used
at the moment for example, they have such a dominating contributing
effect that if they did not exist, the proportion of five A*s
to C equivalence would drop between something like 8 and 12%,
but whether they have value and worth is a very different argument
from whether they affect the tables. The fact is that the tables
affect us in many ways; they affect house prices, they affect
the entire drive from many schools and learning institutions,
and are the single measure being used. There are some considerable
disadvantages in trying to bring all these different facets of
education together in a single measure of success. It goes far
beyond simply the league table figures.
Chairman: Let us move on to inspection
with Edward.
Q163 Mr Timpson: Recently, I was
at a secondary school in my constituency that had just had its
Ofsted inspection, and the biggest gripe to me was that the process
involved hardly any observation of interaction between teacher
and pupil. Is that something that you think is deficient in the
inspection process as it currently stands? If so, how do we rectify
it? I know that Cambridge Assessment has put forward that proposal.
Simon Lebus: Our sense is that
the current inspection process is extremely bureaucratic and a
lot of it relies on verification of certain processes and arrangements.
So there is very much a tendency to look at processes and evidence
of processes being carried out, and less of a focus on teaching.
I suppose in a sense that part of that is also reflected in the
whole issue of the use of exam results as a mechanism of accountability.
I think that drives some of the emphasis to a school-level focus
rather than an individual teacher-level focus. If one looks at
internationally successful systems, as, for example, in Singapore,
where we are involved very heavily in the exam system, and in
Finland, where we are not, the emphasis is very heavily on teaching—the
quality of teaching and teachers' interaction with pupils. It
is not clear to me—I would not claim huge expertise in this—that
the current inspection system is very good at focusing on expertise
and the quality of pedagogy, and that enough attention is paid
to the observation of that.
Dr Smith: I would agree—more
from a personal perspective as a school governor than with my
City & Guilds hat on.
Jerry Jarvis: I provide a complete
personal analysis of the performance of every student during every
examination, and we can track that during the learning process.
It is not used well in schools. The relationship, however, between
teacher and pupil is absolutely critical, as I said when I opened
my remarks. The relationship between teacher and pupil at a formative
age is absolutely transformational. I believe that we have to
be able to value both of those relationships. We need our teachers
to be tremendous managers, great users of information and inspired
individuals, but also leaders in thought for kids. The one thing
I would say about the issue of Ofsted's inspections is that arguably
there are too many different agencies interested in the accountability,
and that perhaps we should be looking at trying to draw out some
sort of commonality in the way in which we value the learning
that is going on in our country.
Q164 Chairman: Could we push
you a bit on "not used well"? It was a sort of throwaway
comment. In what sense, not well used?
Jerry Jarvis: One of the things
that Ken Boston did when he came to this country was to open the
door and allow technology to be introduced and developed. Every
one of the awarding bodies has gone down that road, to one extent
or another. So for every examination, and between examinations,
I can provide a complete personal breakdown of how every student
is actually doing in their understanding—
Q165 Chairman: Do Simon's
lot do that as well?
Simon Lebus: Yes. We do not do
it quite as extensively, but we have started trialling it in a
couple of subjects at GCSE.
Q166 Chairman: Can the AQA
do it?
Simon Lebus: I cannot comment
on that.
Jerry Jarvis: The AQA does it
in a range of examinations, but again to a lesser extent. I also
go to the extent of offering that analysis to students personally,
but I give schools the opportunity to block that information.
I guess that less than 10% of students get that information directly.
I cannot interfere in the learning process—it would be wrong
for me to do so—but those schools that get it right, which
use the analytical information well, are schools that perform
very well as institutions against the measures that we currently
use. Again, they are being professional managers of analytical
data, but I never want to take away the other part of the thing
that is really important when we value institutions, which is
the personal relationship—we must get closer to what actually
makes a difference in a classroom.
Q167 Mr Timpson: Bearing in
mind what you have told us, what faith do you have in the proposals
for the new inspection regime for this September in terms of addressing
the interaction of the pupil-teacher relationship and the personalised
information on each child and tracking them through the school?
Is the new system going to address those problems?
Simon Lebus: I am not sure that
I am sufficiently expert on the arrangements coming into place
in September, but I think the general thrust is a rather less
bureaucratic approach to inspection, which is a positive thing.
Returning to what Jerry has said, I think that the issue relates
to data on individual learner performance in terms of the technology
that is available. One of the reasons we have been slightly slower
in adopting that technology, which relates to the item-level data
that Dr Boston talked about earlier—the capability to generate
a lot of such data now exists through the use of onscreen marking
technology—is that one can end up in a trap involving overly
mechanical marking schemes that tend to make the learning experience
less enjoyable and fruitful. One has to be very aware of that
hazard. I know that you are going to visit New York to look at
its system of a balanced scorecard, but generating huge quantities
of data can become highly complex, because you end up with a lot
of different measures that are set off against each other. It
is then very difficult to come to a judgment and hold institutions
properly to account because you are looking at too wide a range
of measures. I think that is a hazard that needs to be watched
quite carefully when the new arrangements come into place.
Q168 Chairman: Do you share
those concerns, Jerry?
Jerry Jarvis: No, for a whole
series of reasons. The onscreen assessment regimes are no different
from those on paper; it is just that they are far more accurate
and efficient. Secondly, the availability of information enables
teachers to teach better. Those teachers who use that analytical
data well actually have more time for personal interaction; they
do not batch deliver information to students as a group, but are
able to take students at their own pace and time. We can actually
see that happening. I return to the issue of management. I think
that we are quite often disingenuous to academic institutions
in many ways. Let me use that risky analogy again: if an art gallery
has evolved over time into something quite important that people
love to visit and so on, there is no point in pulling in a load
of management consultants to bring all the artists into a room
and say, "You have to manage the way in which people look
at the pictures better and think of intuitive ways of increasing
the funding for the institution." I am an engineer and I
get excited about making things faster, higher, more efficient
and so on, and I guess that teachers do not start by saying, "My
role is to get as many kids to be able to answer as many questions
as possible." There is a higher ideal here, and I think that
we need to make the appreciation of the management part of an
institution's role much closer to being a core part. We are actually
asking people who set out to be teachers to be something else
as well, but we are not preparing them for it. We can see it in
the fact that there is evidence that the tools are being provided,
but they are not being used well. That is not a slight against
teachers and teacher institutions; we are actually setting expectations
that are not right. If we set up another series of measures on
schools without thinking about what it feels like to be a teacher
and to have those requirements, they are not liable to work.
Chairman: All the sections, as we go
through them, are a little bit truncated today, for the reasons
you know. We are now moving on to school improvement and I am
asking Paul to begin on that.
Q169 Paul Holmes: You've got
a school that Ofsted has inspected and it says "This is a
really good school; great teachers, strong leadership team. They
are doing a really good job." It might be so good that it
could be asked to be a mentor to failing schools; then, come August,
the GCSE results are published and they are below 30% five A to
C grades, so it is now a National Challenge school and has failed,
and is nationally named, shamed and condemned. There seems to
be a dysfunction on a massive scale on how to measure schools.
Do you have any comments on that?
Dr Smith: For me it relates back
to Jerry's earlier answer and whether the schools are actually
using the data that is available to them to look at how they are
performing and what their school improvement might look like,
versus the running of the school. It is the dichotomy between
the teaching and the management of the system itself.
Jerry Jarvis: Because of the way
I collect information on students and pupils I can see two schools
in the same street, with the same catchment areas, with the same
free school meals, that are dramatically different in their performance
in academic qualifications, but also dramatically different in
the well-being and health of the students who are actually at
those schools. I can see it happening. You all have seen so-called
failing schools turned around. It is about the management process.
It is about the fact that we need to be able to give those gifted
teachers—the people who can inspire and who have those personalities—the
framework that they are able to succeed in. They are teaching
the same syllabus and they all have the same degrees; all over
the place in our education system we have some wonderful teachers
and wonderful leaders, but as you said it is not scaling. I suspect
it is not scaling because we are failing to understand the management
and structural issues that underpin the ability of good teachers
to perform.
Q170 Paul Holmes: Edexcel's
submission is very critical of the effect of the raw use of league
tables and the distorting effect it has on schools and the deterrent
effect it has on parents wanting to go to certain schools or staff
wanting to work in certain schools. You say, for example: "The
use of undifferentiated standards like the 30% ... A* to C criterion
of the `National Challenge' has had damaging consequences for
pupil intake, parental engagement, staff recruitment, governor
authority/accountability" and so on. You go on, through the
submission, to be very critical of the whole process. What could
we do instead that allows a better way of assessing whether schools
are succeeding or failing, and holding them accountable?
Simon Lebus: I think there needs
to be a complex measure. One of the things that we have said is
that there is a much greater role for inspection if it is done
properly. I come back to the sort of comments we were making earlier
about observation of teachers and teacher interaction with learners.
That is, if you like, the fundamental building block. I think
part of the problem is that a lot of the system emphasis is at
school level and it does not necessarily capture some of the quality
and complexity of those interactions at classroom and teacher
level. I would like to see a much greater use of a more teacher
and teaching process-centred inspection regime replacing some
of the current focus on the end-of-process outputs represented
by terminal examinations.
Q171 Paul Holmes: Edexcel
says in its submission that "`league tables' reflecting achievement
and attainment scores fail to differentiate between schools according
to their intake, resourcing and value added", and therefore
tend to increase the competitive pressure for kids to go to one
school rather than another; and that if you look at schools as
low performing simply in raw exam terms it "has damaging
consequences for learners, communities and social cohesion."
You talk elsewhere about the problem that setting schools up as
competing units has had a very negative effect over the years.
What does that mean for league tables? If you have a league table
it is going to be used for those purposes, so should we have league
tables of exam results or not?
Jerry Jarvis: Yes, we should.
I guess that I could join the ranks of others who might speculate
one way or another. I think that we should hold our learning institutions
to account for the excellence of the learning that is given. It
is critical to all our futures. There is no question about that.
I would argue from a personal standpoint that we should focus
on those subjects and qualifications for which there is a critical
national interest. However, we must keep league tables in some
form or another. We must ensure that they are measuring those
issues that are important for us. However, it can never be the
single measure against which we hold learning institutions to
account. Unfortunately, that is what they have become. I could
not sit here and give you a trite answer, or start the debate
on how we might set up something. Twenty-five years ago, industry
recognised that commercial companies simply defined by profitability,
particularly short-term profit, were institutions in danger of
not understanding their customers and of losing their way. They
introduced a five-part measure to try to bring into their boards
other measures of achievement. Bonuses were paid on how well we
treated our staff rather than just on how much profit we made.
Those sorts of balanced scorecards have been used in the past.
However, places such as New York, for example, have done that
and the answers are actually too complicated for most people to
understand. All I can suggest is that there is a real need for
us—all of the players—to sit down with our sleeves
up and try to find a way to establish accountability in learning
institutions in a better and more holistic way.
Q172 Paul Holmes: You represent
three of the major awarding bodies in the country, and you obviously
operate partly around the world as well. Have you looked at other
systems? Vikki, you have worked in various countries. You say
that we must have league tables, but many countries do not. In
a number of countries, they are actually illegal, so why must
we have them? What is your experience of other countries?
Dr Smith: My experience elsewhere
has been, as it is now, post-compulsory, where league tables tend
to be less prevalent. I am not sure whether I would want to make
parallels in the school agenda.
Simon Lebus: We have a lot of
experience, for example, in Singapore, where we work actively
with the Singapore exam authority. We are very aware of what goes
on in the school system there. They are very geared around exam
results. Data are made publicly available. That is one of several
measures by which schools are judged. In response to your question,
"Should there be league tables?", I think that public
information about qualification success should be public, but
I do not think that it should necessarily be made public or presented
in the format of league tables. That returns to the point about
who owns league tables. At the moment, they are owned by the DCSF,
so they are designed to meet a certain accountability agenda.
Giving Ofqual responsibility for, and ownership of, those tables
would result in a much greater challenge for some of the equivalences
introduced. The data would be used in a more contextually sensitive
and sensible way and we would not necessarily have the attempt
to conflate all the results. The data should be public. It should
clearly be a matter of public record. People, parents of pupils
and institutions are entitled to know how well they have done.
However, I think that because the different types of qualifications
and learning experience get conflated, it becomes very difficult
to make proper judgement. Also, if you think of dispersed qualifications
like diplomas, where there is multi-institutional responsibility,
how do you make judgements? Who is going to own the diploma result
when it is eventually certificated? You are back to this thing
that the accountability measure is distorting the shape of the
educational experience or the shape of the qualification.
Chairman: We are going on to the school
report card.
Q173 Mr Heppell: I have three
quick questions. I see Edexcel has an awful lot in its written
submission about my first question, so you might want to stand
back on some of this, but the other two people have not mentioned
report cards at all. What do you think should be in the report
card? What should be represented in there? Should it take account
of the specific circumstances in the school—for instance,
should there be value added in a report card?
Dr Smith: The agenda for City
& Guilds is really about how we can risk-manage centres, whether
colleges, training providers and employers. Increasingly we are
engaging with schools as a result of the diploma. I do not know
exactly what needs to be in the school report card yet. My plea
would be for the transparency, the openness and the availability
of that detail, so that awarding bodies can better risk manage
the centres that they are working with and support them on an
improvement journey, where appropriate.
Simon Lebus: Quality of planning,
quality of leadership, quality of teaching, strategic management—there
are a variety of measures that, if one is trying to assess an
institution, need to be taken into account, beyond solely the
outcome of the pupils' exam results.
Q174 Chairman: If you are
operating in 150 countries, have you been to America to see how
they do it?
Simon Lebus: Curiously enough,
North America is one of the few places where we do not operate
very effectively. We do not have many centres in North America.
I think that part of the issue here is that there is so much change
in the system and people are always trying to measure the effect
of change—in a lot of the countries where we operate, there
is not this constant cycling through of change and, as a result,
there is much less preoccupation with end of school exam results,
because they are not looking all the time to observe differences.
What they are interested in is long-term management of improvement
of the school system.
Jerry Jarvis: We did have a great
deal to say. I think that the value-added argument about circumstances
is a very interesting one. For anyone who goes into a psychology
course, one of the first things they learn is a Hawthorne experiment,
where you reduce the lighting and find that people work even faster.
The reason that people are responding to all the changes is that
someone is taking a personal interest in them, so they respond
to the personal interest.
Chairman: I thought you were an engineer.
Jerry Jarvis: I was an engineer—very
astute of you.
One of the dangers is that we make excuses.
If we are going to make some sort of success, we need to think
about the language. Let me go back to what I said to start with.
We could simply make some sort of statement—again, this
is where Ofqual's role can be pivotal—about how well a school
prepares pupils for higher education. How well does it prepare
them to be citizens and to take their place in work? How well
were those kids inspired? How much did they love and enjoy their
time at school and how much fulfilment did they get out from it?
If we could use language that a lay person could absorb and say,
"Yes, that actually makes some sort of sense", I could
separate those three values quite quickly. At the moment we disguise
what is actually going on in a lot of very inaccessible information
and we do not actually think back to how it feels to receive it.
Q175 Mr Heppell: In some respects,
I think that there is a bit of a problem, because you want to
get all the details on the report card, but in your evidence you
talk about the dangers of oversimplification. You are taking objective
and subjective stuff, putting it together and trying to have a
value that is then judged by people and seen to be a just one.
I honestly cannot see how anyone is able to do that and make it
work. Does anyone think they can?
Jerry Jarvis: We shall never make
it work perfectly. It is going to be about the balance, but I
firmly believe that learning institutions and awarding bodies
should be held to account, positively and with real numbers that
have some sort of value. We need to have the method to hold institutions
to account. We shall have to struggle along together to find the
least bad way of doing that.
Q176 Mr Heppell: This is a
very specific question. What about the Government's proposal to
restrict the school report card to 11 to 16-year-olds, rather
than 11 to 19-year-olds? What do people think about that? Is that
a good thing or a bad thing?
Dr Smith: How realistic is that
with the advent of the diploma, which bridges 14 to 19? That would
be my question in return.
Mr Heppell: Sorry, I am not getting that.
Dr Smith: If you are restricting
it to 11 to 16, but the diploma coming into the schools is working
from 14 to 19, how realistic is that proposition?
Simon Lebus: My sense is that
it is likely to be an evolving experiment, to the extent that
it is likely eventually to encompass the whole of the school cohort.
There is merit in looking early on at how to achieve that. As
we have already alluded to, it is highly complex and it needs
quite a long time to get levels of trust established. Therefore,
aiming for the whole school cohort to begin with would probably
be a useful thing to do.
Jerry Jarvis: I would measure
things. In principle, I would go for complete coverage.
Chairman: Last section—diplomas
and 14 to 19 provision. David first, and Graham will come in after.
Q177 Mr Chaytor: Sorry Jerry,
I missed your last comment there.
Chairman: You were speaking very softly.
I do not know if that is a psychological experiment for the Committee,
but your voice is right down. John was close but we were all straining
to hear.
Jerry Jarvis: My very last comment?
Mr Chaytor: It was your very last comment
that I missed, which is relevant.
Jerry Jarvis: I apologise; I have
done my very best to sound English, but my accent is still there
a little bit.
What I said was that, in principle, the idea
of establishing measurements of performance is something that
I would endorse, so I would be in favour of taking a report card
all the way through.
Q178 Mr Chaytor: All the way
through. In your written submission, you refer to a unified 14
to 19 reporting mechanism, so I am interested to hear what each
of the three witnesses understands by that. Do you think that
the introduction of the diplomas inevitably means the end of league
tables as we know them?
Simon Lebus: At the moment, the
estimate is that 12,000 people are taking diplomas in this first
year and, of course, they will not all certificate at the end
of this year. I think that it is far too early to think that the
new level or the new type of working and cross-institutional working
represented by diplomas heralds the end of old-style league tables.
To be honest, I think that if league tables are killed off eventually,
it will not be as a result of the diploma, because take-up of
it will be far too slow, so, no, I do not see it radically challenging
the current league table arrangements.
Q179 Mr Chaytor: By 2014,
or whenever we have the full range of diplomas, you think that
league tables will have changed but not because of the impact
of diplomas.
Simon Lebus: The current big issue
is that people do not feel that league tables are fair and we
have a saying that people have to feel that exam results are fair.
People do not feel that league table results are fair. Every summer,
when the results are published, we have exactly the situation
that Paul Holmes has described, that schools have high contextual
value added but actually they have terrible results. There is
not the trust or confidence in the system, although that will
evolve. As I have already said, there is obviously the opportunity
now to look at the institutional arrangements in relation to DCSF
and Ofqual, but that alone will not deal with it; it is also about
the design and the approach. I think that league tables will change
by 2014, but I do not think that change will be diploma-led, as
it were. I think it will reflect a number of other dissatisfactions
with current arrangements.
Chairman: Jerry is nodding. Vikki, do
you agree?
Dr Smith: I agree.
| 
