You are here: Publications and Records > Commons Publications > Select Committees > Children, Schools and Families > Children, Schools and Families

UNCORRECTED TRANSCRIPT OF ORAL EVIDENCE To be published as HC 353-iii

HOUSE OF COMMONS

MINUTES OF EVIDENCE

TAKEN BEFORE THE

CHILDREN, SCHOOLS AND FAMILIES COMMITTEE

 

SCHOOL ACCOUNTABILITY

WEDNESDAY 22 APRIL 2009

JERRY JARVIS, SIMON LEBUS and DR. VIKKI SMITH

 

Evidence heard in Public

Questions 151 - 189

 

USE OF THE TRANSCRIPT

1.    

This is an uncorrected transcript of evidence taken in public and reported to the House. The transcript has been placed on the internet on the authority of the Committee, and copies have been made available by the Vote Office for the use of Members and others.

 

2.

Any public use of, or reference to, the contents should make clear that neither witnesses nor Members have had the opportunity to correct the record. The transcript is not yet an approved formal record of these proceedings.

 

3.

Members who receive this for the purpose of correcting questions addressed by them to witnesses are asked to send corrections to the Committee Assistant.

 

4.

Prospective witnesses may receive this in preparation for any written or oral evidence they may in due course give to the Committee.

 

 


 

Oral Evidence

Taken before the Children, Schools and Families Committee

on Wednesday 22 April 2009

Members present:

Mr. Barry Sheerman (Chairman)

Annette Brooke

Mr. David Chaytor

Mr. John Heppell

Paul Holmes

Fiona Mactaggart

Andrew Pelling

Mr. Graham Stuart

Mr. Edward Timpson

Derek Twigg

 

Examination of Witnesses

Witnesses: Jerry Jarvis, Managing Director, Edexcel, Simon Lebus, Group Chief Executive, Cambridge Assessment, and Dr. Vikki Smith, Director of Assessment and Quality, City & Guilds, gave evidence.

 

Q151 Chairman: Could we have the next set of witnesses? I am sorry that you have had a slight delay. May I welcome Jerry Jarvis, Simon Lebus and Dr. Vikki Smith to our proceedings? I am sorry that we are going to have a shorter session than we planned. You know exactly why, because you were sitting there listening earlier. We usually give people a chance to say something about accountability and the inspection system and how you view it. You are in a very powerful but privileged position in your organisations. Can I start from the left, Jerry, Simon, Vikki, if you do not mind me using your first names? Do you want to say something to get us started Jerry, or do you want to go straight into questions? It is up to you.

Jerry Jarvis: I have not prepared anything in advance. I am very comfortable to take the questions as they are.

 

Q152 Chairman: Why did you come here?

Jerry Jarvis: I came here, first, because I was invited. I am head of one of the principal examination boards in the country. We have a huge responsibility. We have just gone through a very important set of evidence in the previous session. It is very important that people like me are held to account and make as big a contribution as we can to the well-being of the system. I am here out of duty.

 

Q153 Chairman: Thank you very much, Jerry. I just say to all the witnesses that if you feel that anything asked by members of this Committee touches on a commercially sensitive area we understand that you might not be able to answer. One of you mentioned to me that there are some sensitivities in one particular area. Just make that clear in terms of your response. Simon?

Simon Lebus: I am the group chief executive of Cambridge Assessment, a department of the University of Cambridge, which owns the exam boards OCR, Cambridge International Exams and Cambridge ESOL. We operate in 150 countries throughout the world, as well as in the UK, so have a very good perspective on the situation internationally.

I am here, likewise, because I was asked-inevitably out of interest in your previous session-but also because I think the whole issue of accountability, and the use to which exams are put in terms of their application as an accountability measure, is critical. In terms of the overall system and the wash-back effect on educational exams that arise from their use as a measure of accountability, there are a number of impacts that are of concern and need reflection.

Dr. Smith: I am Vikki Smith, director of assessment and quality at City & Guilds. Again, we were invited to submit evidence, and we were pleased to be invited. For us, it signals a potential blurring of historical boundaries that tend to see a separate vocational qualification. I hope that the issue to be discussed will be how we move to a more holistic picture and better sharing of data that is of use and more accessible. Also, City & Guilds has made a very firm commitment to diplomas. If the market leads how diplomas develop and they become more vocational, as we believe they should, that will be core to us, and we will need to look at different ways of managing that accountability, because the diplomas will demand that. We are very pleased to be here.

Chairman: Good. Let's get into the questions. Graham.

 

Q154 Mr. Stuart: In your view, what aspects of provisions should a school be accountable for, and to whom?

Chairman: Who wants to take that? Jerry?

Jerry Jarvis: I would almost prefer to answer that as someone from the street, if you like, rather than as head of an exam board. I believe that they have a responsibility to prepare students for higher education, but also to prepare for broader issues, such as the ability to take a place in society, to be ready for work and perhaps to develop those characteristics that engender achievement in people-to celebrate and develop things that people are good at. Part of the reason for this today, I guess, is that the achievement of academic qualifications clearly dominates, so I think that it would be advantageous if we could broaden the scope of that accountability away from the narrow focus on academic qualifications.

 

Q155 Mr. Stuart: It is a very broad question. I think when we did our testing and assessment report, there were 23 purposes of examinations and accountability that we came up with. I was trying to get your point of view, less as ordinary citizens, but more as experts in this area. What accountability can examinations provide, and what are the areas where examinations cannot provide that, and it would be better provided using some other method, such as sampling? Would one of you deal with that broad issue?

Simon Lebus: I think, in a sense, the other name for exams is qualifications, and they are about the qualifications that individuals need to succeed in the various routes that they choose to pursue with their career and their life-that is their primary purpose and function. I think that a lot of the difficulty arises when multiple functions are then heaped on top of that. Clearly, parents, teachers, taxpayers and citizens all have an interest in seeing how well schools equip children to be successful in life, and exams have become a form of proxy for that.

That, in itself, is not necessarily damaging. What is damaging is the apparatus that is put in behind that. Once that comes to be done in a systematic and mechanical way, all sorts of distorting factors come into play: various artificial equivalences, a whole philosophy of credentialism and an approach to the design of qualifications, all of which interfere with that primary, educational purpose. I do not think that it is an illegitimate thing for a variety of interested parties to be looking at qualifications and results to evaluate how well an institution is succeeding in its task of equipping learners for their later life. I think the difficulty arises when a whole edifice of construction is built on that using rather elaborate and artificial equivalences and measures.

 

Q156 Mr. Stuart: Is there room for greater teacher assessment in place of the formal examinations that you provide?

Simon Lebus: There is no question that there is room for greater teacher assessment. I think the difficulty, as ever, is the question of public trust. There have been various debates about coursework and the extent to which people are schooled in coursework so that they can do very well in it, and then how that compares to written qualifications. There is nothing educationally wrong with teacher assessment at all. The question is how ready people are to trust that. Also, just thinking from an international perspective, and looking at what has happened in qualifications over the last 10 years, we live in a global economy. People are increasingly mobile. Qualifications are a form of currency and a support for them in their mobility and their careers, and they need to be trusted. I think it is a case that where systems have very large elements of teacher assessment, degrees of trust tend to be slightly reduced.

 

Q157 Mr. Stuart: At primary level, for instance, do you have a view on the fact that there could be more teacher assessment and, in terms of schools accountability, that we would be better using alternative methods such as sampling? If the teachers are not contributing to their own assessment, so to speak, through the assessment of pupils, then that distortion will be removed and there would be less teaching to the test, and hopefully the assessments provided to secondary schools would be more useful than they currently are with a supposedly independent external examination. Do you agree with that?

Jerry Jarvis: Ken repeatedly made the case for onscreen marking. We have virtually 100% onscreen marking running at the moment, and I provide a complete breakdown and analysis of every teacher's own performance in delivering the curriculum that they are required to deliver. It is, however, the case that probably less than 10% of those teachers actually use that analytical information sensibly and sincerely. Part of it is because of the way that we come about it. We expect a great deal of our teachers-we expect them to be the sorts of individuals who can inspire and lead and give us values. Certainly teachers did that for me when I was young. But we also need them to be accomplished managers of processes as well, because we have huge examination processes going on.

The current system, as Simon is alluding to, clearly separates the role of an awarding organisation such as mine so that there is clearly regular separation from the delivery of the process, almost to the exclusion of a teacher unless it is to do with coursework and so on. But I will hark back to the technology again: you can blend the two if you use the technology intelligently. Continuous personalised assessment is a key issue of learning and yet we separate that from the formal process that we engage in, and the technology could actually blend those to great effect.

As I say, we expect a great deal of our teachers. We expect them to be able to do both. Let me risk an analogy. If you were running an art gallery in which the material was hung by artists who were really committed to the purpose of their art and so on, you would not necessarily ask one of them to run the art gallery and take the money at the door. But we do expect our learning institutions to do both of those things that I have talked about, and we separate the way in which we measure those things to a huge extent.

 

Q158 Mr. Stuart: Can I move on to contextual value added? There seems to be more and more criticism that it gives no more accurate an assessment of a school's performance than conventional league tables. What views do you have on CVA?

Simon Lebus: I think the issue to some extent is that it becomes very confusing. The more measures that are introduced the less clear the picture. There is a sense that one set of measures is introduced that does not necessarily give people the information they feel they want, or does not necessarily give the result, so another set of measures is introduced, and then a third set of measures is introduced. If you take something like CVA you can have the peculiarity of a school that performed very well on the CVA but not very well on the five to eight A*-C at GCSE. What conclusion do you draw from that? It is difficult to know what conclusion can be drawn from it as a taxpayer, a member of an LEA, a parent or a teacher. All those different groups will draw different conclusions. I do not think that there is anything wrong with CVA as such, but it is not clear that it adds a lot of value in terms of clarifying the position and enhancing understanding. That is simply a function of the replication of measures, not necessarily that measure itself.

 

Q159 Annette Brooke: I should particularly like to ask Cambridge Assessment about its comments regarding "perverse incentives" for schools to choose easier qualifications as an outcome from the performance tables, and the game playing. First, has this intensified over recent years? I recall that in the past schools played examination boards, but perhaps now we are talking about subjects as well.

Simon Lebus: League tables are a relatively recent phenomenon-they are only 15 years old. With the passage of time, institutions that are judged in performance terms by those league tables become more sophisticated at how they play the game. More and more judgment of school performance is based on performance in league tables, to the extent that schools have become very sophisticated. However, whether they have become more sophisticated over the last two or three years I could not say. Once the incentives are put up and the equivalences are created-so that one GNVQ is equivalent to four GCSEs or whatever, and there is a five-GCSE threshold-we set in train a pattern of behaviour that is bound to arise from using the results for accountability in that way. Whether that has intensified over the last two or three years I am not sure.

 

Q160 Annette Brooke: We have mentioned in the Committee an IT qualification that possibly led to four GCSEs and did not take up a great number of hours per week. I do not know whether anybody has any comments on that-a situation where a multiple number of points, as it were, go into the performance tables, but they perhaps come from an area that does not necessarily take up a high proportion of teaching time.

Simon Lebus: In a sense, that is a good illustration of the absurdity of the whole construct. IT, and the mastery of IT, is a skill; it is something you either can or cannot do, and there are various features and bits of that skill that you acquire. A lot of the other subjects that are tested in general qualifications are knowledge-based or have to do with understanding. The difficulty arises when people try to create these equivalences, which is what distorts the behaviour. A number of IT qualifications perform valuable functions, but they are essentially skills-based. Trying to create an artificial parity with general academic qualifications inevitably leads to those sorts of distortions. Where incentives are attached to that, it may well direct the behaviour of institutions that are being judged, as they are used for accountability purposes.

 

Q161 Annette Brooke: I have a general question. Cambridge Assessment makes the point that it would like Ofqual to look at this. That would seem to follow on from its recent report on the science GCSE, for example. Is that a general view? Would Ofqual give you all more credibility if it was looking genuinely at the equivalence of the different subjects?

Jerry Jarvis: Absolutely. It is in our interests for public confidence to be raised. We desperately need Ofqual to become a respected and effective institution. It is important that parents and students can turn to Ofqual for confidence in the examination outcomes and the standards set, and not on issues such as equivalence and so on. It is in all of our interests that Ofqual performs that role.

Chairman: Vikki, do you want to come in on this?

Dr. Smith: I am a little curious. I agree with the separation of the different things that can be acquired in terms of skills and knowledge. Those need to be demonstrated differently and will be valued depending on where that takes you. There is difficulty in drawing equivalence across different types of acquisition. That said, there is a difficulty-I guess-in drawing equivalence from the IT agenda to English, to Maths and to French and so on. However, throughout the discussion, I would not want to see greater weight given to knowledge as opposed to skill, because that would actually disadvantage a great number of individuals in the school system. I do not think that that is where we are going, but I wanted to put that on record.

 

Q162 Annette Brooke: Thank you for that; I think it was a very important comment. Finally, I would like to ask about the qualifications that are currently excluded from the performance tables and what impact that is having. That could apply to vocational qualifications that are excluded, and obviously at the back of my mind are the IGCSEs. What should Ofqual and QCDA be doing about those?

Dr. Smith: Absolutely, I think that it will serve to reinforce the divisions that have certainly existed since I have been working in education. Linking back to league tables and what counts-we have heard mention of teaching to the test and so on-there will be a funnelling of students to the detriment of UK plc, because we will be sending individual pupils through particular streams of education rather than looking to everything that is available. I think it will have quite a drastic impact.

Simon Lebus: The problem at the moment is that the league tables are owned by the DCSF and while that is the case, there are inevitably suspicions and unease about how they are compiled. I think that giving them to Ofqual, as part of the confidence objective in the legislation that is going through the House, would be an important and valuable reform. If one takes things such as the IGCSE and pre-U and the fact that they are not included in the tables, I think it leads to some manifestly strange results, inasmuch as high achieving schools appear outside the league tables or at the bottom the league tables, because they are not taking the qualifications that the tables include, which are clearly directly comparable qualifications, so there is an issue.

Having recently been through the process of getting IGCSE and now the pre-U approved, I think that there is also an issue of new qualifications being made to fit design straitjackets, so that they can easily be slotted into the appropriate spot in the league table. You have a washback effect in terms of the design of the qualifications that is unhelpful. I think that the current system does not work very well: it is vague, imprecise and gives peculiar results that are not felt to be fair by a number of schools that are taking part. I think it is time for it to be reformed.

Jerry Jarvis: I am not 100% in agreement with everything so far. Let me take us another step back. I guess that our education system at 14 to 19 is dominated by progression to higher education. We have a fixation on academic qualifications. Arguably, that is part of the reason why many industries believe that kids can come through that formal education process not fit to work, so they have to acquire the so-called softer skills that we continue to talk about. If you take IT, for example, I might argue that you can go through a vocationally based programme and acquire learning in a different way so that it can be applied in a different way and not necessarily limited entirely to skills, although there is certainly a movement in that direction.

There is an issue, of course, about the way that we produce equivalence in order to have those points scored. There is no question about it: vocational qualifications contribute enormously to that. The vast majority of existing vocational qualifications count. In fact, looking at the BTEC qualifications that are used at the moment for example, they have such a dominating contributing effect that if they did not exist, the proportion of five A*s to C equivalence would drop between something like 8 and 12%, but whether they have value and worth is a very different argument from whether they affect the tables. The fact is that the tables affect us in many ways; they affect house prices, they affect the entire drive from many schools and learning institutions, and are the single measure being used. There are some considerable disadvantages in trying to bring all these different facets of education together in a single measure of success. It goes far beyond simply the league table figures.

Chairman: Let us move on to inspection with Edward.

 

Q163 Mr. Timpson: Recently, I was at a secondary school in my constituency that had just had its Ofsted inspection, and the biggest gripe to me was that the process involved hardly any observation of interaction between teacher and pupil. Is that something that you think is deficient in the inspection process as it currently stands? If so, how do we rectify it? I know that Cambridge Assessment has put forward that proposal.

Simon Lebus: Our sense is that the current inspection process is extremely bureaucratic and a lot of it relies on verification of certain processes and arrangements. So there is very much a tendency to look at processes and evidence of processes being carried out, and less of a focus on teaching. I suppose in a sense that part of that is also reflected in the whole issue of the use of exam results as a mechanism of accountability. I think that drives some of the emphasis to a school-level focus rather than an individual teacher-level focus. If one looks at internationally successful systems, as, for example, in Singapore, where we are involved very heavily in the exam system, and in Finland, where we are not, the emphasis is very heavily on teaching-the quality of teaching and teachers' interaction with pupils. It is not clear to me-I would not claim huge expertise in this-that the current inspection system is very good at focusing on expertise and the quality of pedagogy, and that enough attention is paid to the observation of that.

Dr. Smith: I would agree-more from a personal perspective as a school governor than with my City & Guilds hat on.

Jerry Jarvis: I provide a complete personal analysis of the performance of every student during every examination, and we can track that during the learning process. It is not used well in schools. The relationship, however, between teacher and pupil is absolutely critical, as I said when I opened my remarks. The relationship between teacher and pupil at a formative age is absolutely transformational. I believe that we have to be able to value both of those relationships. We need our teachers to be tremendous managers, great users of information and inspired individuals, but also leaders in thought for kids. The one thing I would say about the issue of Ofsted's inspections is that arguably there are too many different agencies interested in the accountability, and that perhaps we should be looking at trying to draw out some sort of commonality in the way in which we value the learning that is going on in our country.

 

Q164 Chairman: Could we push you a bit on "not used well"? It was a sort of throwaway comment. In what sense, not well used?

Jerry Jarvis: One of the things that Ken Boston did when he came to this country was to open the door and allow technology to be introduced and developed. Every one of the awarding bodies has gone down that road, to one extent or another. So for every examination, and between examinations, I can provide a complete personal breakdown of how every student is actually doing in their understanding-

 

Q165 Chairman: Do Simon's lot do that as well?

Simon Lebus: Yes. We do not do it quite as extensively, but we have started trialling it in a couple of subjects at GCSE.

 

Q166 Chairman: Can the AQA do it?

Simon Lebus: I cannot comment on that.

Jerry Jarvis: The AQA does it in a range of examinations, but again to a lesser extent.

I also go to the extent of offering that analysis to students personally, but I give schools the opportunity to block that information. I guess that less than 10% of students get that information directly. I cannot interfere in the learning process-it would be wrong for me to do so-but those schools that get it right, which use the analytical information well, are schools that perform very well as institutions against the measures that we currently use. Again, they are being professional managers of analytical data, but I never want to take away the other part of the thing that is really important when we value institutions, which is the personal relationship-we must get closer to what actually makes a difference in a classroom.

 

Q167 Mr. Timpson: Bearing in mind what you have told us, what faith do you have in the proposals for the new inspection regime for this September in terms of addressing the interaction of the pupil-teacher relationship and the personalised information on each child and tracking them through the school? Is the new system going to address those problems?

Simon Lebus: I am not sure that I am sufficiently expert on the arrangements coming into place in September, but I think the general thrust is a rather less bureaucratic approach to inspection, which is a positive thing. Returning to what Jerry has said, I think that the issue relates to data on individual learner performance in terms of the technology that is available. One of the reasons we have been slightly slower in adopting that technology, which relates to the item-level data that Dr. Boston talked about earlier-the capability to generate a lot of such data now exists through the use of onscreen marking technology-is that one can end up in a trap involving overly mechanical marking schemes that tend to make the learning experience less enjoyable and fruitful. One has to be very aware of that hazard.

I know that you are going to visit New York to look at its system of a balanced scorecard, but generating huge quantities of data can become highly complex, because you end up with a lot of different measures that are set off against each other. It is then very difficult to come to a judgment and hold institutions properly to account because you are looking at too wide a range of measures. I think that is a hazard that needs to be watched quite carefully when the new arrangements come into place.

 

Q168 Chairman: Do you share those concerns, Jerry?

Jerry Jarvis: No, for a whole series of reasons. The onscreen assessment regimes are no different from those on paper; it is just that they are far more accurate and efficient. Secondly, the availability of information enables teachers to teach better. Those teachers who use that analytical data well actually have more time for personal interaction; they do not batch deliver information to students as a group, but are able to take students at their own pace and time. We can actually see that happening.

I return to the issue of management. I think that we are quite often disingenuous to academic institutions in many ways. Let me use that risky analogy again: if an art gallery has evolved over time into something quite important that people love to visit and so on, there is no point in pulling in a load of management consultants to bring all the artists into a room and say, "You have to manage the way in which people look at the pictures better and think of intuitive ways of increasing the funding for the institution."

I am an engineer and I get excited about making things faster, higher, more efficient and so on, and I guess that teachers do not start by saying, "My role is to get as many kids to be able to answer as many questions as possible." There is a higher ideal here, and I think that we need to make the appreciation of the management part of an institution's role much closer to being a core part. We are actually asking people who set out to be teachers to be something else as well, but we are not preparing them for it. We can see it in the fact that there is evidence that the tools are being provided, but they are not being used well. That is not a slight against teachers and teacher institutions; we are actually setting expectations that are not right. If we set up another series of measures on schools without thinking about what it feels like to be a teacher and to have those requirements, they are not liable to work.

Chairman: All the sections, as we go through them, are a little bit truncated today, for the reasons you know. We are now moving on to school improvement and I am asking Paul to begin on that.

 

Q169 Paul Holmes: You've got a school that Ofsted has inspected and it says "This is a really good school; great teachers, strong leadership team. They are doing a really good job." It might be so good that it could be asked to be a mentor to failing schools; then, come August, the GCSE results are published and they are below 30 per cent. five A to C grades, so it is now a National Challenge school and has failed, and is nationally named, shamed and condemned. There seems to be a dysfunction on a massive scale on how to measure schools. Do you have any comments on that?

Dr. Smith: For me it relates back to Jerry's earlier answer and whether the schools are actually using the data that is available to them to look at how they are performing and what their school improvement might look like, versus the running of the school. It is the dichotomy between the teaching and the management of the system itself.

Jerry Jarvis: Because of the way I collect information on students and pupils I can see two schools in the same street, with the same catchment areas, with the same free school meals, that are dramatically different in their performance in academic qualifications, but also dramatically different in the well-being and health of the students who are actually at those schools. I can see it happening. You all have seen so-called failing schools turned around. It is about the management process. It is about the fact that we need to be able to give those gifted teachers-the people who can inspire and who have those personalities-the framework that they are able to succeed in. They are teaching the same syllabus and they all have the same degrees; all over the place in our education system we have some wonderful teachers and wonderful leaders, but as you said it is not scaling. I suspect it is not scaling because we are failing to understand the management and structural issues that underpin the ability of good teachers to perform.

 

Q170 Paul Holmes: Edexcel's submission is very critical of the effect of the raw use of league tables and the distorting effect it has on schools and the deterrent effect it has on parents wanting to go to certain schools or staff wanting to work in certain schools. You say, for example: "The use of undifferentiated standards like the 30%...A* to C criterion of the 'National Challenge' has had damaging consequences for pupil intake, parental engagement, staff recruitment, governor authority/accountability" and so on. You go on, through the submission, to be very critical of the whole process. What could we do instead that allows a better way of assessing whether schools are succeeding or failing, and holding them accountable?

Simon Lebus: I think there needs to be a complex measure. One of the things that we have said is that there is a much greater role for inspection if it is done properly. I come back to the sort of comments we were making earlier about observation of teachers and teacher interaction with learners. That is, if you like, the fundamental building block. I think part of the problem is that a lot of the system emphasis is at school level and it does not necessarily capture some of the quality and complexity of those interactions at classroom and teacher level. I would like to see a much greater use of a more teacher and teaching process-centred inspection regime replacing some of the current focus on the end-of-process outputs represented by terminal examinations.

 

Q171 Paul Holmes: Edexcel says in its submission that "'league tables' reflecting achievement and attainment scores fail to differentiate between schools according to their intake, resourcing and value added", and therefore tend to increase the competitive pressure for kids to go to one school rather than another; and that if you look at schools as low performing simply in raw exam terms it "has damaging consequences for learners, communities and social cohesion." You talk elsewhere about the problem that setting schools up as competing units has had a very negative effect over the years. What does that mean for league tables? If you have a league table it is going to be used for those purposes, so should we have league tables of exam results or not?

Jerry Jarvis: Yes, we should. I guess that I could join the ranks of others who might speculate one way or another. I think that we should hold our learning institutions to account for the excellence of the learning that is given. It is critical to all our futures. There is no question about that. I would argue from a personal standpoint that we should focus on those subjects and qualifications for which there is a critical national interest. However, we must keep league tables in some form or another. We must ensure that they are measuring those issues that are important for us. However, it can never be the single measure against which we hold learning institutions to account. Unfortunately, that is what they have become. I could not sit here and give you a trite answer, or start the debate on how we might set up something.

Twenty-five years ago, industry recognised that commercial companies simply defined by profitability, particularly short-term profit, were institutions in danger of not understanding their customers and of losing their way. They introduced a five-part measure to try to bring into their boards other measures of achievement. Bonuses were paid on how well we treated our staff rather than just on how much profit we made. Those sorts of balanced scorecards have been used in the past. However, places such as New York, for example, have done that and the answers are actually too complicated for most people to understand. All I can suggest is that there is a real need for us-all of the players-to sit down with our sleeves up and try to find a way to establish accountability in learning institutions in a better and more holistic way.

 

Q172 Paul Holmes: You represent three of the major awarding bodies in the country, and you obviously operate partly around the world as well. Have you looked at other systems? Vikki, you have worked in various countries. You say that we must have league tables, but many countries do not. In a number of countries, they are actually illegal, so why must we have them? What is your experience of other countries?

Dr. Smith: My experience elsewhere has been, as it is now, post-compulsory, where league tables tend to be less prevalent. I am not sure whether I would want to make parallels in the school agenda.

Simon Lebus: We have a lot of experience, for example, in Singapore, where we work actively with the Singapore exam authority. We are very aware of what goes on in the school system there. They are very geared around exam results. Data are made publicly available. That is one of several measures by which schools are judged. In response to your question, "Should there be league tables?", I think that public information about qualification success should be public, but I do not think that it should necessarily be made public or presented in the format of league tables.

That returns to the point about who owns league tables. At the moment, they are owned by the DCSF, so they are designed to meet a certain accountability agenda. Giving Ofqual responsibility for, and ownership of, those tables would result in a much greater challenge for some of the equivalences introduced. The data would be used in a more contextually sensitive and sensible way and we would not necessarily have the attempt to conflate all the results. The data should be public. It should clearly be a matter of public record. People, parents of pupils and institutions are entitled to know how well they have done. However, I think that because the different types of qualifications and learning experience get conflated, it becomes very difficult to make proper judgment. Also, if you think of dispersed qualifications like diplomas, where there is multi-institutional responsibility, how do you make judgments? Who is going to own the diploma result when it is eventually certificated? You are back to this thing that the accountability measure is distorting the shape of the educational experience or the shape of the qualification.

Chairman: We are going on to the school report card.

 

Q173 Mr. Heppell: I have three quick questions. I see Edexcel has an awful lot in its written submission about my first question, so you might want to stand back on some of this, but the other two people have not mentioned report cards at all. What do you think should be in the report card? What should be represented in there? Should it take account of the specific circumstances in the school-for instance, should there be value added in a report card?

Dr. Smith: The agenda for City & Guilds is really about how we can risk-manage centres, whether colleges, training providers and employers. Increasingly we are engaging with schools as a result of the diploma. I do not know exactly what needs to be in the school report card yet. My plea would be for the transparency, the openness and the availability of that detail, so that awarding bodies can better risk manage the centres that they are working with and support them on an improvement journey, where appropriate.

Simon Lebus: Quality of planning, quality of leadership, quality of teaching, strategic management-there are a variety of measures that, if one is trying to assess an institution, need to be taken into account, beyond solely the outcome of the pupils' exam results.

 

Q174 Chairman: If you are operating in 150 countries, have you been to America to see how they do it?

Simon Lebus: Curiously enough, North America is one of the few places where we do not operate very effectively. We do not have many centres in North America. I think that part of the issue here is that there is so much change in the system and people are always trying to measure the effect of change-in a lot of the countries where we operate, there is not this constant cycling through of change and, as a result, there is much less preoccupation with end of school exam results, because they are not looking all the time to observe differences. What they are interested in is long-term management of improvement of the school system.

Jerry Jarvis: We did have a great deal to say. I think that the value-added argument about circumstances is a very interesting one. For anyone who goes into a psychology course, one of the first things they learn is a Hawthorne experiment, where you reduce the lighting and find that people work even faster. The reason that people are responding to all the changes is that someone is taking a personal interest in them, so they respond to the personal interest.

Chairman: I thought you were an engineer.

Jerry Jarvis: I was an engineer-very astute of you.

One of the dangers is that we make excuses. If we are going to make some sort of success, we need to think about the language. Let me go back to what I said to start with. We could simply make some sort of statement-again, this is where Ofqual's role can be pivotal-about how well a school prepares pupils for higher education. How well does it prepare them to be citizens and to take their place in work? How well were those kids inspired? How much did they love and enjoy their time at school and how much fulfilment did they get out from it? If we could use language that a lay person could absorb and say, "Yes, that actually makes some sort of sense", I could separate those three values quite quickly. At the moment we disguise what is actually going on in a lot of very inaccessible information and we do not actually think back to how it feels to receive it.

 

Q175 Mr. Heppell: In some respects, I think that there is a bit of a problem, because you want to get all the details on the report card, but in your evidence you talk about the dangers of oversimplification. You are taking objective and subjective stuff, putting it together and trying to have a value that is then judged by people and seen to be a just one. I honestly cannot see how anyone is able to do that and make it work. Does anyone think they can?

Jerry Jarvis: We shall never make it work perfectly. It is going to be about the balance, but I firmly believe that learning institutions and awarding bodies should be held to account, positively and with real numbers that have some sort of value. We need to have the method to hold institutions to account. We shall have to struggle along together to find the least bad way of doing that.

 

Q176 Mr. Heppell: This is a very specific question. What about the Government's proposal to restrict the school report card to 11 to 16-year-olds, rather than 11 to 19-year-olds? What do people think about that? Is that a good thing or a bad thing?

Dr. Smith: How realistic is that with the advent of the diploma, which bridges 14 to 19? That would be my question in return.

Mr. Heppell: Sorry, I am not getting that.

Dr. Smith: If you are restricting it to 11 to 16, but the diploma coming into the schools is working from 14 to 19, how realistic is that proposition?

Simon Lebus: My sense is that it is likely to be an evolving experiment, to the extent that it is likely eventually to encompass the whole of the school cohort there is merit in looking early on at how to achieve that. As we have already alluded to, it is highly complex and it needs quite a long time to get levels of trust established. Therefore, aiming for the whole school cohort to begin with would probably be a useful thing to do.

Jerry Jarvis: I would measure things. In principle, I would go for complete coverage.

Chairman: Last section-diplomas and 14 to 19 provision. David first, and Graham will come in after.

 

Q177 Mr. Chaytor: Sorry Jerry, I missed your last comment there.

Chairman: You were speaking very softly. I do not know if that is a psychological experiment for the Committee, but your voice is right down. John was close but we were all straining to hear.

Jerry Jarvis: My very last comment?

Mr. Chaytor: It was your very last comment that I missed, which is relevant.

Jerry Jarvis: I apologise; I have done my very best to sound English, but my accent is still there a little bit.

What I said was that, in principle, the idea of establishing measurements of performance is something that I would endorse, so I would be in favour of taking a report card all the way through.

 

Q178 Mr. Chaytor: All the way through. In your written submission, you refer to a unified 14 to 19 reporting mechanism, so I am interested to hear what each of the three witnesses understands by that. Do you think that the introduction of the diplomas inevitably means the end of league tables as we know them?

Simon Lebus: At the moment, the estimate is that 12,000 people are taking diplomas in this first year and, of course, they will not all certificate at the end of this year. I think that it is far too early to think that the new level or the new type of working and cross-institutional working represented by diplomas heralds the end of old-style league tables. To be honest, I think that if league tables are killed off eventually, it will not be as a result of the diploma, because take-up of it will be far too slow, so, no, I do not see it radically challenging the current league table arrangements.

 

Q179 Mr. Chaytor: By 2014, or whenever we have the full range of diplomas, you think that league tables will have changed but not because of the impact of diplomas.

Simon Lebus: The current big issue is that people do not feel that league tables are fair and we have a saying that people have to feel that exam results are fair. People do not feel that league table results are fair. Every summer, when the results are published, we have exactly the situation that Paul Holmes has described, that schools have high contextual value added but actually they have terrible results.

There is not the trust or confidence in the system, although that will evolve. As I have already said, there is obviously the opportunity now to look at the institutional arrangements in relation to DCSF and Ofqual, but that alone will not deal with it; it is also about the design and the approach. I think that league tables will change by 2014, but I do not think that change will be diploma-led, as it were. I think it will reflect a number of other dissatisfactions with current arrangements.

Chairman: Jerry is nodding. Vikki, do you agree?

Dr. Smith: I agree.

 

Q180 Mr. Chaytor: Can I ask Vikki specifically how can your qualifications be better reflected in the current accountability arrangements? Although the majority of your qualifications will be held by adults, there will be a sizeable minority of 16 to 19-year-olds who are sitting those qualifications.

Dr. Smith: I think that something like 41% of our qualifications are taken by those under 19, so it is a sizeable number. We are in constant dialogue with Ofqual about raising the profile and the issues associated with vocational qualifications, because there is a tendency to focus on general qualifications and traditional schooling. The diploma may well start to impact on that, but only if it can evolve in a manner-as I would hope was its original intent-that offered something more than an academic qualification, which clearly it should not be.

To go back to your earlier question, I do not think that the diploma will be the catalyst to change the league tables; the league tables will need to change if they are to reflect the diploma fully, if you think about the collaboration and all the different parties that will be involved. The current league tables will not reflect that element.

 

Q181 Mr. Chaytor: But is there an easy way to reflect the contribution to the diploma as a whole by individual institutions-individual schools and colleges? Is there a way of doing that?

Simon Lebus: I think that it would be a virtually impossible task and I think that even to try would be misplaced. There are quite enough issues, in terms of the management of the introduction of diplomas, without getting into the creation of accountability mechanisms at this stage.

Jerry Jarvis: Technically, the assessed components of the diploma already have league table points so they will fit in and figure in league table attainment.

 

Q182 Mr. Stuart: You just mentioned the difficulties or challenges with introducing diplomas. Do any of you have concerns about the timing of phase 4 diplomas? In particular, are you happy with the timing for the introduction of the design and production of diplomas in languages, international communication, humanities and social sciences?

Simon Lebus: It was interesting to listen to Dr. Boston talking about change programmes and some of the risks associated with change programmes. That has certainly been borne out by our experience of the development of the first phase of diplomas. The low levels of take-up and the institutional learning that is going on are functions of their having been over-hurried-they happened too fast. With the development of phase 4 diplomas, in a sense we are back to where we were with phase 1 diplomas. The diploma development bodies have been taking a long time to decide what should be in those diplomas. The content of diplomas is not handed over to the awarding bodies for assessment design until that process is finished, whereas our preference was for it all to happen in parallel.

We have written to the Secretary of State to say that we feel that the humanities and languages diplomas need to be delayed in the way that the science diploma has been delayed, for precisely those reasons. We think that the programme is becoming too compressed. At present, I understand that the decision is that only the introduction of the phase 4 science diploma will be delayed. We are uncomfortable about some of the risks associated with what is now becoming a rather compressed timetable.

Jerry Jarvis: Yes, we share that view. We think that phase 4 should go back a year. We have written to that extent. We need to learn from the initial introduction far more steadily before phase 4 diplomas come through. We are being put under quite tremendous pressure to develop those in a short space of time.

 

Q183 Chairman: How happy and confident are you about the development of diplomas at this stage overall?

Simon Lebus: Clearly we hope that the current low levels of take-up will not persist. We hope that the programme will increase. It is having quite a shaky start. To some extent that is a result of the fact that the arrangements were put together very hurriedly. I do not think that it has been given the opportunity to develop organically as a qualification. Our experience in the non-regulated sector and from operating externally overseas is that qualifications take quite a long time to pick up currency. The stated desire that they should be the qualification of choice by 2013 will simply not happen. The sort of pressures that we are under in the development of the phase 4 diplomas in humanities and languages and the decision not to defer their introduction will aggravate that difficulty.

Jerry Jarvis: I think that Simon has articulated the whole thing very well. If you questioned yourselves, when would you see yourself advising your son or granddaughter to take a diploma rather than an A-level with the confidence that it is a better qualification? Qualifications have to develop a brand. They have to get out there in the real world and people have to learn how to value them. The notion that by 2013 everyone will understand and embrace the diploma, perhaps to the exclusion of existing provision, feels very ambitious.

Dr. Smith: I completely agree. The question from a City & Guilds perspective would be about the appropriateness of the actual subjects of phase 4 diplomas and whether those are the right areas. I completely agree with a delay so that we can see how those in existence are operating and help to support them and help them to get traction and value.

 

Q184 Mr. Stuart: This is fairly serious evidence from the three of you. You are suggesting not only that Ministers have betrayed the central vocational purpose of diplomas, but that they are bungling their introduction and ignoring the united advice of examination bodies. Is that what is happening?

Jerry Jarvis: It's one view, isn't it? If only my life was really simple, then I could say, "Yes, it takes 15 weeks to do that and I have 15 weeks, thank you." On balance, we would like more time. We will deliver what we are required to deliver to the best professional extent we can, and there are imperatives for introducing change and not taking for ever to do it. I feel that I could do with more time, and that view is shared across the awarding body community quite strongly, but we are not saying that we are about to fail or throw our cards up in the air. We will do what we have to do, but we just feel that we should be taking this more steadily. We all share the view that the diploma is a critically important qualification. Coming on the back of all the analysis that has been done on world economics, there is a real need to develop technicians and professional people within this country that is second to none. Money is being put on the diploma, so it has to work. I can empathise with the anxiety about bringing in the diploma, but we just feel that it is too fast.

Simon Lebus: Pursuing the metaphor that Dr. Boston used earlier when he talked about programmes being permanently on an amber light, it seems to me that the pace at which this has been driven, despite the advice, means that you do have a programme that is permanently on an amber light. When you are introducing a new qualification, you need to have that programme on a green light most of the time, because otherwise the risk of failure is greatly multiplied. Introducing a new qualification is always a high-risk business, so to the extent that extra risk can be avoided by managing the programme more deliberately, that is highly desirable, but it is not happening at present.

 

Q185 Paul Holmes: I was fascinated listening to Simon and Jerry and by Jerry's comment on whether you would advise your son or daughter to do a diploma at the moment. When I was a head of sixth form at one of the first institutions to introduce advanced GNVQs, exactly the same things were being said. GNVQs are now dead, dying and being replaced by diplomas, so in 10 years' time will we be reinventing the wheel and replacing diplomas with a new qualification of choice?

Jerry Jarvis: It is up to us to make this a success, and we talked endlessly today about bringing all the institutions that have a role to play together in some sort of cohesive way, but it really is important. Ken Boston said that qualifications have to "earn their spurs". We really do have to get this brand. The thing that really keeps me awake at night is that we will go too fast and damage the existing provision, which is world-class in this country. Someone asked why some countries do not have league tables, and I reckon that many countries are very envious of the fact that we have league tables. Vocational provision in this country is absolutely second to none, right through the learning line. We absolutely have to make this work to the very best of our ability. The argument for us is that it will take a lot longer than current expectations suggest. If it is good enough, it will earn its spurs and take its place, and maybe the day will come when we all say to our children, "For God's sake, do not do old-fashioned A-levels, but do this modern qualification that everyone values." But it has got to get there.

Simon Lebus: I would agree with that. It seems to me to be a fantasy to assume that it will be a qualification of choice by 2013. That simply will not happen.

 

Q186 Mr. Stuart: What does failure look like? We know that with the SATs it was late delivery, and you have said that that heightens the risk, but what does failure look like for diplomas?

Simon Lebus: Failure looks exactly like what Paul Holmes has just described-that in 10 years' time no one will have heard of them and someone presenting themselves for a job will say that they have a diploma and the employer will ask what it is.

 

Q187 Mr. Stuart: So there will be a quiet failure of take-up and of building the brand, rather than a spectacular failure, as occurred with the SATs.

Dr. Smith: I completely agree.

Jerry Jarvis: Yes, I absolutely agree. It is also an extremely complicated qualification with multiple inputs, so we will have to avoid the implementation issues as we get into the position where it establishes.

 

Q188 Chairman: Are the two Departments involved in this working harmoniously on the diploma?

Jerry Jarvis: I meet on many occasions with the Department, and all the agencies involved in it. There is a real sense of purpose right now, as we run up to the very first awards, for a full diploma coming through.

 

Q189 Chairman: So the reports of friction between DIUS and the DCSF over diplomas are nonsense?

Jerry Jarvis: I am not in a position to report on that.

Chairman: You have not heard then? Have you heard anything like that, Jerry?

Jerry Jarvis: I have heard lots and lots of things that would be inappropriate here.

Simon Lebus: I have not observed this. Like Jerry, I have heard things, but I have not observed them.

Chairman: What diplomats. Vikki?

Dr. Smith: I am absolutely not close enough to it. So I would not have observed it.

Chairman: I should have confessed that I am a fellow of City & Guilds, should I not? I might have been softer on the questioning with you.

Thank you very much for your attendance. You know, as well as I do-we all do-that you were pushed and squeezed. We value your experience and knowledge greatly in terms of this inquiry. Will you stay with us? Due to the shortened nature of this session, we did not squeeze you enough for information. Can we squeeze you more informally later? Will you remain in contact with us, because we want to make this a very good report? Thank you very much for your attendance.