Select Committee on Children, Schools and Families Minutes of Evidence

Examination of Witnesses (Questions 100 - 119)



  Q100  Stephen Williams: That is based on the aggregate of the results for each child in the school, so would you reject alternatives completely?

  Dr Boston: It depends on the purpose. If your purpose is to find out whether children are writing and reading as well as they did 10 years ago nationally, the best test is to give a sample of them virtually the same test as was given to a sample of them 10 years ago. That will tell you whether we have gone up or down. If you want to report on the performance of a school this year in relation to the school next door, the sample will clearly not do that, but the full cohort test will. It is again about purpose. Both purposes are legitimate and some of the difficulties with the testing programme or the examination programme when looking at whether standards have changed significantly year on year or over a 20-year period, are that the curriculum, teaching methods and other things such as class size have changed. If you really want to know whether people are better at reading than they were 20 years ago, give them the same test.

  Q101  Stephen Williams: I see that you have had time to look at the note that has been passed to you.

  Dr Boston: This is Dr Horner's contribution in handwriting. The report of the task group on assessment and testing in 1989—while I was still elsewhere, Barry—decided on a 10-point scale and proposed a graph of progress that identified Level 4 at age 11. That report was then published, presumably. We are now on an eight-point scale—aren't we?—so that 10-point scale has been reduced. That does not fully answer your question, Stephen.

  Q102  Stephen Williams: No, four out of 10 is 40% and, on the same scale, four out of eight is 50%, so it seems to be a completely different target. Perhaps we are getting a bit too technical. I was trying to get at whether the Government were feeling the QCA's collar in respect of how we set the standards for children. Therefore, is it right that we have a separate regulator of standards for the future?

  Dr Boston: Barry, I should very much like to give you a written statement tomorrow in answer to this question.[5]

  Chairman: Okay.

  Q103  Stephen Williams: Before the note, I was going to mention the difference between how a child's performance is assessed and how a school's performance is assessed. You were going into how children's performance was assessed over time. Is there another way in which you can assess a school's effectiveness apart from the league table mentality that we have at the moment? Is there an alternative? After all, they do not have league tables in Wales or Scotland.

  Dr Boston: You can certainly assess the performance on the basis of teacher reporting, as against the school reporting its performance against a template of benchmarks, perhaps, as occurs in some other countries, and reporting its testing of its students against national averages in literacy, numeracy and so on. I know of cases where that occurs.

  Q104  Stephen Williams: You are a man of international experience. Do you think that anywhere else does it better than England—whether a state in Australia or anywhere else—without this sort of national frenzy every August, with people wondering whether things are going down the pan or the Government saying, "No, things have only ever got better"?

  Dr Boston: I think England is pretty rare in the way it does this in August, although I would not say unique.

  Q105  Chairman: Is that good or bad?

  Dr Boston: The annual debate about whether too many have passed and whether standards must have fallen is a very sterile debate and I would be glad to see the back of it. If it is right that this new regulator will lead to the end of that, it is a good thing. We are not so sure that it will. There are other, better, ways of celebrating success and achievement—not questioning it.

  Q106  Stephen Williams: Do you think that any particular country does it a lot better than England?

  Dr Boston: No. I think that in other countries where the results come out there is less public criticism of youngsters on the basis that, because they have three A grades, the result must be worthless. Such criticism is a very bad thing. From my previous experience, I have a great interest in Aboriginal education. There was 200 years of Aboriginal education in Australia with absolutely no impact on the performance of Aboriginal kids until we introduced full cohort testing and reporting at school level. Then, suddenly, people took Aboriginal education seriously and it began to improve.

  Q107  Stephen Williams: If nobody else wants to come in on this section, I should like to ask one last question, going back to the start, about introducing the new regulator. We have only had the report, Confidence in Standards, from the Secretary of State today, and we have not been able to digest it fully yet. When were you consulted about the split in the QCA's responsibilities? Was it before, during or after the August round of exam results that we had a few months ago?

  Dr Boston: It was after.

  Q108  Fiona Mactaggart: You have been talking clearly about the difficulty of tests fulfilling 14 different purposes. The fact is that they fulfil some of those inadequately. You suggested that the best way to see if children over time are able to achieve the same standard is through sampled testing. Do we do very much of that, and if not, why not?

  Dr Boston: Those tests will tell you whether performance on a particular task has improved over time. We do not do that as a country. We pay a lot of attention to PIRLS and PISA and the national maths and science study. In developing its Key Stage tests from year to year, the QCA does pre-test. Part of those pre-tests in schools, which youngsters think are just more practice tests, are pre-tests for what we will use in 18 months' time. In them we often use anchor questions, which are the same questions that have been asked a few years before or in consecutive years. They might be only slightly disguised or might not be changed at all. That is to help develop tests that maintain standards so that Level 4 is Level 4 year on year. The boundary between the levels is set by the examiners. It might be 59 in one year and 61 in another, but they know that in their judgment that is a Level 4. They draw on those tests.  We have not used the tests systematically enough to say, "We used these six questions for the past eight years and we know that students are getting better at reading or worse at writing," but that is the basis on which we develop and pre-test those emerging assessments.

  Q109  Fiona Mactaggart: I am struck by this. You are saying that we do it a bit to ensure the comparability of tests over time. We all accept that some of that kind of work is a necessary function of getting accurate summative tests, but there is a constant threat in debate in assessment about whether standards have changed over time. I do not think that I properly understand why we have not bothered to invest what does not strike me as a very large amount of resource in producing that kind of sampling over time to see whether standards are improving or weakening, and where. We would then have a national formative assessment about where the strengths and weaknesses of our education system are over time. Do we have a mechanism that is designed to do that? If not, why not?

  Dr Boston: No, we do not. We use PIRLS and PISA and in the recent results, they confirmed what we already know; for example, at PIRLS level, the line I talked about is not as steep as it was before. It has flattened off, but has not come to a plateau. The notion of a sampling programme is something that we have raised with Government. Some years ago, before I came into this job, there was the Assessment of Performance Unit, which did some of that work. That is no more. I do not know the background and the reasons why the work was not pursued, but it was work of this sort. It would seem to me that we need to be thinking not of either/or. That is the message that I really want to get across. We are not thinking of Key Stage tests or single level tests or sample tests. If we want to serve those 22 legitimate purposes of testing—I am sure there are more—we need a number of tests that will deliver between them all those things, but which are designed so that they are very close to what Paul Newton calls the design inference, where the user inference and the design inference are very close indeed.

  Q110  Fiona Mactaggart: What I do not understand about the proposed new system is that if we developed a wider range of tests to separate some of these functions more precisely so that we get more accurate information rather than trying to infer information from tests that are designed to do something else, which is what we do at present, who would take the lead in developing the sample tests and introducing them? Would it be the QCA or the new regulatory authority? I have not had time to read through the document, but I do not understand whose job is what.

  Dr Boston: It would be the QCA, and it would do its work partly through stimulating the private sector market and the awarding bodies to work with it. Presumably the QCA would take the initiative on remit from the Government. That would be critical: the Government would decide that they wanted a set of new tests. We did not go out and invent single level tests. We were remitted to do them. We produced them at Government request, and with our very strong support. So the initiative would rest fundamentally with the Government, but the body that would lead on it would be the QCA, or whatever the QCA might end up being called some time in the future. The regulatory authority is to ensure that, once the product—the assessment—is there, it delivers on standards and maintains standards. The regulator is not a development authority; it is an authority to regulate products and ensure their quality once they are there.

  Q111  Fiona Mactaggart: When you were remitted to develop the concept of single level tests, were you remitted to develop a test that was a one-way street, rather than a test that could be re-administered? I gather that the National Foundation for Educational Research is concerned about the fact that this is just a single pass test and that someone who chooses when they do it might pass then but might not necessarily pass it a month later.

  Dr Boston: We were remitted to produce a test which would be taken as a one-off. Further down the track if we get to a point, as I think we might, where single level tests are available virtually on line, on demand, we would need to go to a data bank of test items. What we have at the moment is a Level 3 test or a Level 4 test. A judgment is then made on the score you get about whether you are secure in Level 4. That test is then finished with. The time may come in the future, as with Key Stage 3 ICT tests, where there is a computer in the corner on which you can take at any stage your Level 4 or Level 5 reading test. That would depend on a data bank. In that sense it is constantly renewable, if I understand the question correctly.

  Q112  Fiona Mactaggart: It was not so much about whether it was renewable. If the teacher of the child can choose the moment at which the child takes a single level test and it is a propitious day for that particular child, the child may do well in the test and succeed, but it might still be rather a frail attainment. There is anxiety about whether that is a fully accurate picture of the child's capacity and the general learning level even though they can do it on a fair day with wind behind them.

  Dr Boston: I am remiss in that I have not fully explained the relationship between the Assessment of Pupil Performance and the tests. The APP programme is designed essentially to produce greater understanding among teachers about what is represented by a level—the profile of a Level 4 in reading, the profile of a Level 5 in reading and the difference between them. It represents the different indicators that show a child is either at Level 4 or Level 5, and the child is then entered for the test. The test is meant to be confirmation that the teacher has made the judgment correctly.

  Sitting suspended for fire evacuation.

  On resuming—

  Chairman: Dr Boston, we are back in business, although only briefly. I suspect that we will have to call you or your team back at some stage, because this has been unfortunate. I will give a question to each member of the team, and you will answer speedily. I will start with David, followed by Stephen, then Fiona, and I will finish.

  Q113  Mr Chaytor: On maintenance of standards, will the new A* grade at A-level have the same pass rate in all subjects across all examining boards?

  Dr Boston: No.

  Q114  Mr Chaytor: Does the existing A-level threshold have the same pass rate in all subjects?

  Dr Boston: No.

  Q115  Mr Chaytor: Does that cause a problem?

  Dr Boston: No.

  Q116  Mr Chaytor: Will there not be a huge discrepancy between different subjects in different boards?

  Dr Boston: The A/B boundary is set by professional judgment. The reality is that subjects are different; there is no attempt to say that, for example, 10% must pass or have an A grade in every subject. No country in the world achieves precise comparability between subjects in terms of standards. Australia tries to do so: it takes all the youngsters who get a certain grade in, for example, English, geography, and art, and, if they find that a lot of the youngsters who are taking those three are getting higher grades in geography than in the other two subjects, then they deflate the mean of geography. Some pretty hairy assumptions underlie that. Here, an A/B boundary is set by professional examiners broadly at the level that a hard-working, well-taught, student who has applied himself or herself fully would achieve on a syllabus or specification.

  Q117  Mr Chaytor: Are the thresholds for subjects on examining boards matters of public record? That is, is the percentage score that triggers a B, an A or an A* on the record and available to pupils and parents?

  Dr Boston: The answer is no, I believe.

  Q118  Mr Chaytor: My next question is, should it be?

  Dr Boston: I would think not.

  Q119  Mr Chaytor: Why not?

  Dr Boston: The essential point is that you might have a harder paper one year than another, in which case the boundaries might change significantly. The point is not the numerical score where the boundary is drawn. The fundamental point is the professional judgment of the examiners, who decide where the A/B boundary is and where the E/U boundary is. They do that on the basis of their experience and past statistical evidence using papers of similar demand.

5   Note by witness: In 1988 the Task Group on Assessment and Testing (TGAT) designed the assessment system for the national curriculum. This included the development of a then 10 level scale to cover the years of compulsory schooling. Level 4 was pitched as the reasonable expectation for the end of the primary phase, to ensure pupils could move on with confidence in their skills to tackle the secondary curriculum. Back

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2008
Prepared 13 May 2008