Select Committee on Children, Schools and Families Minutes of Evidence

Examination of Witnesses (Questions 380 - 399)



  Q380  Lynda Waltho: I did not know what you were saying to us, but I have now had a chance to read it. It answers some of the questions that I was forming.

  Interestingly, you refer to some unexpected patterns in the results. Indeed, Sue Hackman said the same in her letter to schools in January. You go on to say what might have caused those unusual patterns, but you do not say what the patterns are. I wonder whether you would expand on that? You touched on the subject slightly with Fiona, and I wonder whether you would expand a little on what those patterns might have been.

  Jim Knight: As I said at the outset, we will publish a proper evaluation of the December and June tests in the autumn as it can be analysed more fully than in the early stages of a pilot. We should bear in mind that it took four years for the SATs to be piloted. All of these new tests take some time, and you will inevitably have some teething troubles. We do not publish as each of the test results come out. We do not publish them in a drip-drip fashion; we tend to do an annual publication of test results. I do not think that we should do anything particularly different for this, because it might skew things and put undue pressure on those schools that are in the pilot scheme. As I said in the letter, the most significant unusual outcome was variations between Key Stage 2 and Key Stage 3 pupils taking the same test. So, let us say that they were taking a Level 4 writing test. The Key Stage 2 students were doing significantly better when they were taking exactly the same test as the Key Stage 3 students. Now, that was a bit odd. We will have to wait and see why that was the case. It might just be that the sorts of scenarios that they would have been writing about in that test were more engaging for younger children than for older children; I do not know. Maybe there are issues of motivation in Key Stage 3 that are different from Key Stage 2 around taking the test. There were some other patterns around the higher levels and the lower levels, and the expectations were different between the two. However, when we had a first look at the overall patterns that were emerging, we just thought that there were enough oddities that, although they are not out of keeping at all with early pilots, we should ask the National Assessment Agency to run some checks and make sure that the marking was right before we gave the results to the pupils and the schools, which we have now done.

  Q381  Lynda Waltho: There is a perception that part of the problem might have been a higher rate of failure, if you like.

  Jim Knight: No, it certainly was not about the results. It was the patterns of results and the differences between different types of students, particularly that difference between Key Stage 3 and Key Stage 2 students, that we were concerned about. The decisions that we made in November were about how we pitched this test so that the results are more comparable with the current SATs, in terms of the level gradings. We made those decisions before these tests were set or before we had the results. For those sections of the media that think that we have changed the rules because of results, they are misunderstanding at two levels: one, they are misunderstanding if they think that we are unhappy with the overall level of results in these pilots; and two, they are misunderstanding the sequence, because they are interpreting that we have made these changes in response to results.

  Q382  Lynda Waltho: If we could drill down on this issue, basically what I want you to confirm is whether the passing rate was lower than you expected it to be. I think that that is cutting to the chase.

  Jim Knight: Again, it is more complicated than that. In some tests, the results were better than expected and in some tests the results were worse than expected. So, it was not about the pass rate; it was about the pattern.

  Ralph Tabberer: There were good results and there were some weak results, but the anomalies were sufficient to make us appreciate that there were some things that have changed within the tests. As we had set up these new tests, there was a test effect, but we do not know what that effect is yet and we will not know until we run another pilot. There are some things related to teacher behaviours changing, including which children they are putting in for the tests and at what stage. We do not know how much is down to that factor. There are also some questions about the level that we are pitching the tests at. However, it is impossible from this first pilot to separate out which effect is pushing in which direction. You must also remember that these are self-selecting schools, so there is no sense in which they are a national representative sample of the performance across the country. So, we are having to find our way through this process quite carefully. We need another round of results before we know what is going on. We will have a chance to try another set of tests and then we will be in a position to make the review of this process available at the end of the year.

  Q383  Lynda Waltho: If the problem is a higher rate of failure, it might imply that there is perhaps a discrepancy between the tool of assessment and the teacher judgment about whether a child has reached a particular stage. If that is the case, how could we resolve it?

  Ralph Tabberer: First, the only thing we can do is speculate—we are not in a position to know and we cannot answer whether it is down to a change in teacher behaviour. If a child does less well in a test, say, you may deduce that that reflects that the teacher has got the level wrong. However, we do not know whether teachers behave differently or take a different approach to progression tests than they would to an external test at the end of year 6. They might, for example, be pitching for a child to show a level earlier than they normally would in order to take part. However, we will not know enough about how people behave until we review the situation.

  Jim Knight: Equally, a Key Stage 2 maths teacher, for example, might be very familiar with Levels 3, 4 and 5, because they deal with those all the time. However, they might be less familiar with Levels 1 or 7, say. Making the assessment in those very early days—in December, they were only two or three months into the pilot—and making the right judgment on whether pupils were ready to take some of the tests, might have been difficult.

  Q384  Lynda Waltho: The Government have put a lot into single level testing and have stated that it will be rolled out nationally subject to positive evidence from the pilot study. That shows quite a lot of confidence. Why are you backing single level tests so publicly before we have sufficient evidence from the pilots? I know that you are a confident man.

  Jim Knight: Theoretically, the progression pilot, single level testing and testing when ready, accompanied by one-to-one tuition, is compelling. It would be a positive evolution from SATs, for reasons that we have discussed. Naturally, we want such a positive evolution to work. If it does not work, we will not do it.

  Lynda Waltho: That was a very confident answer.

  Chairman: Douglas.

  Q385  Mr. Carswell: I have four questions. The first is general and philosophical. There are lots of examples in society of testing and qualifications being maintained without the oversight of a state agency, such as university degrees, certain medical and legal qualifications, and musical grades. I cannot remember hearing a row about dumbing down Grade 2 piano or an argument about whether the Royal College of Surgeons had lowered a threshold. Therefore, why do we need to have a state agency to oversee testing and assessment in schools? Does the fact that the international baccalaureate has become more popular in certain independent schools suggest that some sort of independent body, which is totally separate from the state and which earns its living by setting rigorous criteria, is necessary?

  Jim Knight: Yes, it is necessary, which is why we are setting up an independent regulator that will be completely independent of Government and directly accountable to Parliament.

  Q386  Mr. Carswell: But it will not earn its living by producing exams that people want to take—it will be funded by the taxpayer.

  Jim Knight: Yes, but the examinations are absolutely crucial to the future of the country and to the future of children in this country—marginally more so, I would argue, than Grade 2 piano—so it is right to have an independent regulator to oversee them in the public interest. However, we should move on from the QCA as it is currently formed, which is to some extent conflicted, because it is both develops and regulates qualifications. Because it undertakes the development of qualifications, it has a vested interest in their success, which is why we thought that it would be sensible to split them. We will legislate in the autumn, but we will set up things in shadow form later this year under current legislation. That means we will have that independence.

  Q387  Mr. Carswell: If the QCA earns its fees by setting competitive examinations in competition with other bodies, I have no doubt that it will be setting good tests.

  Jim Knight: Would I not then appear before the Committee and be asked about the over-marketisation of the education system? People would say that valuable exams are not being properly regulated or set because there is not enough of a market to make that particular speciality commercially viable.

  Q388  Mr. Carswell: Architects and surgeons seem to get on okay.

  Jim Knight: Yes, but there will always be a good market for architects and surgeons, but there may not be for some other important skills.

  Q389  Mr. Carswell: Without wanting to move away from asking the questions, I wonder whether you would deny the claims of those people who suggest that over the past 15 to 20 years, under Governments of both parties, standards have dropped. I will give you some specific instances. In 1989, one needed 48% to get a C grade in GCSE maths. Some 11 years later, one needed only 18%. That is a fact. Successive Governments and Ministers have claimed that exam results get better every year. However, in the real world, employers and universities offer far more remedial courses to bring school leavers up to standard than they did previously. International benchmarks show that UK pupils have fallen behind. Does that suggest that, paradoxically, we have created an education system that is drowning in central targets and assessments, but one that lacks rigour? Central control is having the opposite effect to the one intended.

  Jim Knight: You will be amazed to hear that I completely disagree with you.

  Q390  Mr. Carswell: Which fact do you dispute?

  Jim Knight: Ofsted, an independent inspectorate, inspects the education system and gives us positive feedback on standards. We also have the QCA, which is an independent regulator. Although we are strengthening the independence of the regulation side, the QCA still remains relatively independent. It regulates standards and ensures that the equivalence is there. It says categorically that standards in our exams are as good as they have ever been. Then we have National Statistics, which is also independent of the Government. We commissioned a report led by Barry McGaw from the OECD, which is a perfectly respectable international benchmarking organisation, and he gave A-levels a completely clean bill of health. What has changed is that we are moving to a less elitist system. We are trying to drive up more and more people through the system to participate post-16 and then to participate in higher education. Some people rue the loss of elitism in the system and constantly charge it with dumbing down, and I think that that is a shame.

  Chairman: You did not answer Douglas's point about the particular O-level in percentage terms.

  Mr. Carswell: In 1989, one needed 48% to get grade C GCSE maths. Some 11 years later, one needed 18%. Do you agree or not?

  Ralph Tabberer: I do not agree. The problem with the statistics is that you are comparing two tests of very different sorts.

  Mr. Carswell: Indeed.

  Ralph Tabberer: The tests have a different curriculum, a different lay-out and different groups taking them. We cannot take one percentage and compare it with another and say that they are the same thing. That is why we need to bring in some measure of professional judgment to look at the tests operating different questions at different times. That is why in 1996 we asked the QCA to look at tests over time, and it decided that there were no concerns about the consistency of standards. In 1999, we asked the Rose review to look at the same thing, and it said that the system was very good regarding consistency of standards. In 2003, as you rightly pointed out, we went to international experts to look at the matter. We put those questions to professional judgment, because it is so difficult to look at tests.

  Q391  Mr. Carswell: Quangos and technocrats are doing the assessment. The Minister has mentioned three quangos, so technocrats are assessing performance.

  Jim Knight: Look at the key stage results. Look at Key Stage 2 English, where the results have gone up from 63% to 80% since 1997. In maths, they have gone up from 62% to 77%. In English at Key Stage 3, they have gone up from 57% in 1997 to 85% in 2007. There is consistent evidence of improvement in standards. It should not be a surprise, when you are doubling the amount of money going into the system and increasing by 150,000 the number of adults working in classrooms, that things should steadily improve. The notion that the improvements are because things are dumbed down is utter nonsense. The international comparators are obviously interesting and very important to us. We started from a woeful state in the mid-90s, and we are now in a much better state, but we are still not world class. We know that we have to do better to become world class, and we said so explicitly in the Children's Plan. We also know that if we do not carry on improving, we will be left behind, because the international comparators also show that more countries are entering the league tables and more are taking education seriously and doing well. Globally, the competition is out there, and we must respond.

  Q392  Mr. Carswell: We may not agree on that, but there is one area where I think that we agree, because I agree with what you said earlier about education needing to respond to changing social and economic circumstances. If the centre sets the testing and assessment, it is surely claiming that it knows what is best, or what will be best, and what needs to be judged. If you have central testing, will you not stifle the scope for the education system to be dynamic and to innovate? It is a form of central planning.

  Jim Knight: We do not specify things for the end of Level 4 examinations; we talk about equivalency in terms of higher-level GCSEs, so if people want to take other, equivalent examinations, that is fine. The only things where we specify are SATs, which, as we have discussed, are intended to provide a benchmark so that we can measure pupil performance, school performance and national system performance.

  Q393  Mr. Carswell: I am anxious about the way in which the SATs scoring system works. I was reading a note earlier about the CVA system, which we touched on earlier. If testing is about giving parents a yardstick that they can use to gauge the sort of education that their child is getting, that is a form of accountability, so it needs to be pretty straightforward. Is there not a case for saying that the SATs scoring system and the CVA assessment overcomplicate things by relativising the score, for want of a better word? They adjust the score by taking into account people's circumstances, and I have read a note stating that the QCA takes into account particular characteristics of a pupil. Is that not rather shocking, because it could create an apartheid system in terms of expectations, depending on your background? Should it not be the same for everyone?

  Jim Knight: There is not an individual CVA for each pupil, and I do not know what my child's CVA is—the CVA is aggregated across the school. The measure was brought in because there was concern that the initial value added measure was not sufficiently contextualised, that some were ritually disadvantaged by it and that we needed to bring in a measure to deal with that. On questions from Annette and others, we have discussed whether it is sufficiently transparent to be intelligible enough. I think that the SATs are pretty straightforward. They involve Levels 1, 2, 3, 4, 5, 6 and 7—where are you at? That is straightforward enough. Obviously, you then have 1a, b and c and the intricacies within that.

  Q394  Mr. Carswell: There is all the contextualising and relativising that means that you cannot necessarily compare like with like.

  Ralph Tabberer: There is no difference in the tests that different children sit or in the impact of levels. You could perhaps suggest that the CVA is providing an analysis that includes a dimension of context, but that is different. The tests are constant. With the analyses, we are ensuring that—over the years, through consultations, people have encouraged us to do this—a variety of data is available, so people can see how a child or school is doing from different angles.

  Q395  Mr. Carswell: A final question: living in this emerging post-bureaucratic, Internet age, is it only a matter of time before a progressive school teams up with a university or employer and decides to do its own thing, perhaps based on what is suitable to its area and the local jobs market, by setting up its own testing curriculum? Should we not be looking to encourage and facilitate that, rather than having this 1950s attitude of "We know best"?

  Jim Knight: We are encouraging employers to do their own thing and to become accredited as awarding bodies. We have seen the beginnings of that with the Maccalaureate, the Flybe-level and Network Rail. We have a system of accreditation for qualifications—you might think it bureaucratic, Douglas. We are rationalising it to some extent, but a system of accreditation will remain. After accreditation, there will be a decision for maintained schools on whether we would fund the qualification, and I do not see that as going away. We will publish a qualifications strategy later this year setting out our thinking for the next five years or so. The independent sector might go down that road. All sorts of qualifications pop up every now and then in that area. Any other wisdom, Ralph?

  Ralph Tabberer: I go back to the starting point. Before 1988, we had a system whereby schools could choose their own assessments. We introduced national assessment partly because we did not feel that that system gave us consistent quality of education across the system. I do not know of any teacher or head teacher who would argue against the proposition that education in our schools has got a lot better and more consistent since we introduced national assessment. We have put in place the safeguards on those assessments that you would expect the public to look to in order to ensure that they are valid, reliable and consistent over time.

  Jim Knight: Obviously, with the Diplomas it is a brave new world that has started with employers and with asking the sector skills councils to begin the process of designing the new qualifications. That has taken place at a national level—it is not a local, bottom-up thing, but a national bottom-up thing.

  Chairman: We are in danger of squeezing out the last few questions. I realise that this is a long sitting, but I would like to cover the rest of the territory. Annette.

  Q396  Annette Brooke: We heard a view from a university vice-chancellor that it is possible that pupils from independent schools will account for the majority of the new A* grades at A-level. What is your view on that?

  Jim Knight: I tried to dig out some statistics on the numbers getting three A* grades, which has been mentioned in the discussion—I am told, based on 2006 figures, that it is just 1.2% of those taking A-levels. To some extent that is on the margins, but we have done some research into whether, judged on current performance, those getting A* grades would be from independent or maintained schools, because we were concerned about that. We believe in the importance of adding stretch for those at the very top end of the ability range at A-level, which is why we brought in the A* grade. However, we were conscious of worries that it would be to the advantage of independent-sector pupils over maintained-sector pupils. The evidence that has come back has shown the situation to be pretty balanced.

  Q397  Chairman: Balanced in what sense?

  Ralph Tabberer: We have looked at the data, following the vice-chancellor's comment to the Committee that around 70% of those getting three A*s would come from independent schools. From our modelling, we anticipate that something like 1,180 independent school pupils would get three or more A*s from a total of 3,053, so 70% is far from the figure that we are talking about.

  Annette Brooke: We have to wait and see.

  Jim Knight: Yes.

  Annette Brooke: That sounded a very reasonable hypothesis.

  Q398  Chairman: The examination boards also said that A*s will make sure that fewer kids from less privileged backgrounds get into the research-rich universities. That is what they said. Although you are not responsible for higher education, Minister, every time you put up another barrier, bright kids from poorer backgrounds are put off from applying. You know that.

  Jim Knight: Yes. We had some concerns about that, which is why we examined actual achievement in A-level exams, and we were pleased to see that particular result. The difficulty when we took the decision, just to help the Committee, was that we could see the logic in providing more stretch at the top end for A-level. The issue was not about universities being able to differentiate between bright A-level pupils, because we can give individual marks on modules to admissions tutors; it was genuinely about stretch. Should we prevent pupils, in whatever setting, from having that stretch, just because of that one worry about independent and maintained-sector pupils? We took a judgment that we had a bigger responsibility than that, which is to stretch people in whatever setting they are in. That is why we took that decision, but we were reassured by the evidence that the situation is not as bleak as Steve Smith, whom we respect hugely as a member of the National Council for Educational Excellence, might have at first thought.

  Q399  Annette Brooke: I disagree with you on that point. Many years ago, for my generation, we had S-levels, so you had the opportunity for stretch. Why do we need this so tied in? Maybe sometimes we should put the clock back.

  Jim Knight: We had S-levels—I was very pleased with my Grade 1 in geography S-level. We replaced those subsequently with the Advanced Extension Award, in which I was delighted by my daughter's result. However, not many people took them, and they were not widely accepted—they did not seem to be a great success. S-levels were extremely elitist—it was fine for me, in my independent school, to get my Grade 1. I am sure that you did very well in whatever setting you were in.

  Annette Brooke: I did not go to an independent school.

  Jim Knight: They were introduced in the era of an elite education system. Integrating something into the A-level is a better way forward than introducing something marginal.

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2008
Prepared 13 May 2008