|
WEDNESDAY 27 NOVEMBER 2002 __________ Members present: Mr Barry Sheerman, in the Chair __________ Memorandum submitted by OFSTED Examination of Witnesses MR DAVID TAYLOR, Director of Inspection, MISS KATH CROSS, Divisional Manager School Improvement Division, MR TIM KEY, Head of the Standards and Research Unit, and MR MIKE RALEIGH, Divisional Manager Secondary Education Division, OFSTED, examined. Chairman
(Mr Taylor) We are happy for you to ask questions. (Mr Taylor) Our take is essentially that inspection data provides something different to put alongside this much contested performance data and are not subject to the same interpretative difficulties, though they may have other interpretative difficulties. So what we tried to do in the papers which very late last night we managed to get across to you after short notice was to explore how different ways of categorising schools can be sorted in terms of the key indices that inspectors use in reaching overall judgments about the quality of information. We think it is important to put these into the frame alongside the performance data, not least because of the interpretative difficulties to which you and I have referred. We think that often these focus on key aspects of the quality of schools and whether improvement is happening and how it is happening, which are not necessarily shown by the performance data, so that we focus, for example, on what inspectors as a professional judgment say about the quality of leadership and management which we have argued consistently is one of the principal leaders of change and if we find that, in some kind of schools, the data from all the thousands of inspections we do each year suggest that certain types of school overall are doing slightly better on some of these categories, we think those are important pieces of information to bring to your attention. (Mr Taylor) I would like to ask Mr Raleigh to update us quickly. As you say, the report is now over a year ago and, since that, we have analysed the data. This is a change of landscape. It is often said that inspectors have a snapshot view but, on this, we are very much taking a longitudinal view on ever changing landscapes. I think it is important that Mr Raleigh sketches in some of the ways in which the data have or have not been done since our working task group. (Mr Raleigh) Looking at inspection data on specialist schools that have had that status for at least two years - and that is quite important - in our previous report, we did the same thing: we looked at those that had been established for some time to see what effects the programme was having on their provision generally and on achievement in schools. So, looking again at the specialist schools that have had that status for at least a couple of years, the inspection evidence for the last two years indicates that a proportion of teaching judged to be good or very good has grown for schools in the specialist category, although it is not the same across the different types of specialism, and that is another important point. This is not a homogeneous group. The technology colleges and the languages colleges tended to have a higher proportion of good or very good teaching than the arts or sports colleges. Among the other features that come out from the inspection data are the high proportion of schools where the quality and range of learning opportunities were high. The proportion of schools where management was judged to be good or very good was also a distinguishing factor. Generally, those specialist schools looked at over the last couple of years have shown greater improvement since their previous inspection. That is the case for other schools, although it is also a matter of concern that a number of schools have not made any improvement nor in fact have deteriorated. In looking at particular features of what schools are doing now compared to those we looked at a couple of years back, we are seeing some positive signs including more widespread and more pertinent use of target setting, the specialist subjects - and this is a particular issue in the specialist schools - being used more widely to improve teaching and achievement across the range of budgets. We have seen the development of a more systematic approach to school self-evaluation and we have seen more purposeful work in those specialist schools with their primary partners. So, there are improvements in relation to the recommendations that were made in our report in 2001 although one would have to say that all of those or virtually all of those are features of school improvement which you would expect to find in any school which is improving. We also looked at the GCSE results in 2002, if you would like me to go on to that subject, again for those schools that have had the status for at least two years and, that shows that with the exception of sports colleges - (Mr Raleigh) These are 521 schools. There are some 900 operating now, so obviously the numbers have gone up considerably. With the exception of sports colleges, the other categories of specialist schools achieved higher GCSE average point scores than schools nationally, with language colleges having the highest proportion of five or more A to C GCSE grades. The trend of improvement - and I think this is an important fact - since 1997 for those schools is broadly similar to the national picture, although we should say that not all the schools in that 521 have been specialist schools since 1997. Quite a lot of them have but not all. So, while they have been getting better, we are not talking about dramatic transformation. Their trend of improvement is about the same as nationally. It is also perhaps worth pointing out that technology colleges, which form an easily biggest group of specialist schools, have shown a slight fall in their improvement trend since the analysis we undertook in 2001. I hope that is a picture which updates our report on 2001. Chairman: That gives us a good background. Jeff Ennis (Mr Raleigh) This programme is of course a school improvement programme, it is designed to improve schools. I think it is fair to say that unless improvement programmes are universal and unless they are universally well implemented, then they will have the effect of leading to greater improvement in those schools than others, all other things being equal. At the expense of others could be - and I imagine it is - interpreted as those schools becoming more popular because they are specialist schools. When we did our detailed visits of 56 schools that we included in our 2001 report, we were keen to try and get to the bottom of that. It is actually extremely difficult to dissociate the contribution that specialism may make to the expression of preferences by parents. When pressed, the heads of about one-third of the schools thought that there was a significant contribution at some level but largely people, including the parents that we talked to and pupils, were pointing to other factors. These were after all schools that had been enabled to become specialist schools because their track record was good. They had purposeful plans for improving work in the specialist subjects and so on. So, they were schools which were on the up. The reasons people were giving for increased popularity - and it was the case that these schools were all more popular than they had been some years back - were wide, varied but common to the preferences that, generally speaking, parents have expressed. So, whether that means 'at the expense of other schools' is actually a rather tricky question. You could apply exactly the same question to excellence in cities as an initiative where secondary schools in that initiative have been improving at a faster rate than others. Not all schools are in the programme. There are schools in neighbouring authorities which are not. You could argue that because that programme is specifically targeted on need, that it is leading to greater improvement and, in a fashion, one could say, though I think it is a long stretch, that that is at the expense of other schools. (Mr Raleigh) Yes. It is not at all easy as I am sure you have gathered from the research material and the academics to whom you have spoken. (Mr Raleigh) I believe that, in 1999, the Department did introduce somewhat greater flexibility and indeed reduced the level at which matched funding was needed. (Mr Raleigh) One of the issues here is whether we are talking about local business, which is generally what the case was with the early schools in the programme, or business more generally and indeed a wider range of organisations. So, greater flexibility has been introduced. It is still the case, as I know from business at school, that raising that sum is a significant issue. The Department, were it here, might say that that is a good test of the school's determination and persistence and a good test of its enthusiasm to connect with business. (Mr Raleigh) The programme has changed over time and it is changing now. For this current round, the criteria have been broadened to deliberately look to applications from a group of schools or from pairs of schools. It is also focused more generally on the community role of specialist schools, what they will do for and with other schools and for the wider community. Local education authorities are being encouraged to produce a strategic plan within which new bids can be set. So, I think that development is very much in train. Paul Holmes (Mr Raleigh) All three! They are good components of an answer! The fact that schools are encouraged/prompted to build on strength is an important issue; the connection with business and more generally looking outwards; to have a deliberate management plan which is focusing on improvement not only in that subject but how that subject is going to contribute to improvement across the board; having such a plan and having targets; an expectation that that plan is reviewed rigorously; and an expectation that practice is spread deliberately. All that is extremely helpful and is a prompt for school improvement and you can see elements of those characteristics of improvement programme in other activities that the Government have prompted. So, it is a combination of those elements. There is no question but that the funding is helpful, although it is a complicated picture when you take into account the basic funding in schools, so it is not always, so to speak, as much as it looks. If the school's basic funding is pretty low, then what it can do with the additional £123 per pupil is a little more restrictive than elsewhere and that is an important feature. The range of basic funding of the total funding that we looked at among the 56 schools went from £2,000 per pupil to £3,300, which is a big gap. So, the money is important but it plays in within a more complicated picture. (Mr Taylor) That would seem to be a good question to put to the Government. Paul Holmes: I have done! Chairman (Mr Taylor) It was a polite way of reminding you of the difficulty. (Mr Taylor) We have not been asked to give an opinion on whether there should be a universal allocation. Paul Holmes (Mr Taylor) But we have not been asked by Government to give that advice, so we have not given it. Chairman (Mr Taylor) We would advise you to take that matter up with the Government because it is the Government that make the decision. (Mr Taylor) Our function is to provide the evidence about the quality of education in these schools. It is for others to draw the conclusions as to whether that is right. (Mr Taylor) I am not; I am feeling extremely comfortable. (Mr Taylor) I think we are satisfied that the evidence we have shows some encouraging features such that Mr Raleigh has already outlined, and the data we have shown you include many encouraging features, but I would not say that that is a sufficient basis for radical change of Government policy and I would also say that it is not OFSTED's function to advise on whether a particular policy ought to be taken forward in that way. (Mr Taylor) We have come to a number of judgments about improvements that Mr Raleigh has outlined; we have not come to a judgment that the inference to be taken from those findings is such as to influence Government policy in a particular way. Paul Holmes (Mr Raleigh) It is a large number of schools. The biggest growth has been in arts and sports colleges rather than technology and languages, a higher proportion of the schools having been in the programme for a short period remembering that some of the original schools go back to 1992. (Mr Raleigh) We looked at the examination trend of 1997 to 2002 and, where the schools were inspected, we checked, as indeed the inspection does always, the progress against the school's previous inspection. (Mr Raleigh) I am not sure that I could give you an answer on five years ago. I could certainly come back to you on that. I imagine that what you are asking is, was the trend of improvement ... (Mr Raleigh) You would absolutely expect that the trend of improvement in GCSE performance had not been there for three years before the bid was made because that was part of the piece, that we needed to show a track record of that kind. The criteria did not say how much, it said "some improvement over three years." (Mr Raleigh) The schools are self-selected and their bids were approved on the basis of criteria including those that I have outlined, and they were supported both through the resources and by the advice and guidance they were receiving from the Department and elsewhere. So, it would be a surprise and a disappointment if they were not improving and continuing to. (Mr Taylor) There are no self-fulfilling prophesies in examination results. (Mr Raleigh) Yes. I think our basic conclusion in the report - (Mr Raleigh) Our basic conclusion in the report last year was that the programme has helped to accelerate improvement. These were schools which were improving. (Mr Raleigh) Yes, across the range of the schools, you would expect inspectors to dig into the detail of different types of colleges and so on and the picture is more varied. Could I just come back to the value for money question since it is something that we pointed to in our report last year. What we said was that for most - and that meant 80 per cent - of the schools demonstrating good value for money in their general spending but, in particular, on their use of the revenue funding and the capital allocation of specialist schools - a different group of schools, but it turned out still to be 80 per cent - our conclusion was that one in five schools was making disappointing use of opportunities and resources and that asks the question of the Department for Education & Skills as to how, if it proposes to go ahead with the implementation of policy, they are going to make sure that it is going to be 100 per cent who make very good use of their additional resources. Mr Simmonds (Mr Raleigh) Among the 20 per cent that we did not think were making good use of the opportunities of the programme, there were certainly some whose commitment to it did not seem at all strong, and that might be for a number of reasons - the advantage of the status without actually living it through, the opportunity to get £100,000 worth of capital programme and apply it within the rules of the scheme - but they were not pushing it through. (Mr Taylor) What I would say is that is one of the ways in which OFSTED, on the evidence, can be rather helpful in a sense of providing a checking balance on what might be just seen as a bit of extra buckshee funding and we can point to the fact that if schools have entered into the scheme in that spirit, then they are unlikely to get as good a positive outcome on the inspection report as those which are committed not just to getting extra money, which most people would like, but to the real underlying principles of the scheme. (Mr Taylor) That was indeed one of the things we looked at because plainly it is a concern that, if you give money for a specific purpose and it is being used openly for more general purposes ... We did analyse that. (Mr Raleigh) It is quite a tricky balance as to how much you spend in a number of ways on that subject as opposed to trying to generalise the applications of better teaching and learning across the whole subject and, under the regime of this programme from 1999, 30 per cent of the funding needs to be spent on the community dimension, so you work with the other schools and you have to try and balance that. Mr Pollard (Mr Taylor) I do not think we would say that pride and an ability to celebrate or willingness to celebrate the status necessarily is the same as elitism, but I think any system like this has the potential for creating a tendency towards sheep and goats and therefore I think this Committee is right to be pursuing whether, in the desire to promote diversity, you risk hierarchical schools. That does seem to be an important question. Those excluded from a particular group may well see that group as inherently desirable to be a member of and therefore want to be part of it and apply for it. It does not necessarily follow that those outside that group are inferior in any way, but it does mean that they are recognising that membership of this club is potentially one which has benefits. I think the fact of having to go through the process and being subject to fairly strict controls is one that means that any bounty hunters are going to be unsuccessful because the process is designed precisely to weed out people who just think that it did not work under category A, so they will try under B and C and see if they can get the money that way. I think it is important that the specialist status is related to known and recognisable achievement within the specialist area for which the application is being made. Chairman (Mr Taylor) I think it has always been one of the risks and I think it is very important to come back to one of the points that Mr Raleigh was making earlier, that we look at how well it is situated within a local community, how well it is collaborating both with its feeder schools and the neighbourhood schools and the more that can be done to overcome any kind of image that suggests it is an oasis in a barren desert, the better. Mr Chaytor (Mr Taylor) Our general line would be that, if it is only focused on those schools which are relatively high performance or are improving well, then the strategy is incomplete and therefore what you are inviting us to say is that, both for the Department and for us really, it is important to look at those schools which are most in need of improvement and the effectiveness of the various strategies that we have, and both Mr Raleigh and Miss Cross have been involved in aspects of monitoring this, Miss Cross, for example, through the monitoring of the schools facing challenging circumstances, which is the target group of exactly the type of schools that you are talking about and I wonder if we could ask her to say a word about that. (Miss Cross) We have been monitoring schools on that list of schools facing challenging circumstances, which is now about 600 schools, and our evidence so far is that the schools are using the money sensibly to tackle the things that lead to improvement. It is a little slow in some places, slower than we would hope to see, but, for about a quarter of the schools that we visited last year, there had been an increase in the GCSE and other key stage four results, but sadly the school which has difficulties and low attainment, the five A* to C grade, is usually the last one to actually show improvement, but there were many signs of improvement. However, these schools are working in difficult circumstances as are other schools that we visited in terms of special measures and certainly we know there that increases in support for the school and funding has helped those schools to deliver some of the things they have been wanting to do. (Miss Cross) Yes. A year ago, which was the first time we had done this, we produced a report for ourselves and shared it with the DFES and we have updated that. (Miss Cross) It is not a public document yet but we were waiting until we had sufficient evidence, but it also features in the - (Mr Taylor) It features in the annual report of this year. (Miss Cross) And last year as well. (Mr Raleigh) In 2000, we produced a report for improving city schools which focused on the more affected schools in disadvantaged circumstances and we are proposing to update that report, which will include some but not all of the schools that are in the schools facing challenging circumstances group. (Mr Taylor) We also report on the excellence in cities programme. (Mr Raleigh) There is one general thing that I wish to say - and it is a rather dull thing to say as well - and that is that the things that drive improvement when you look across these different programmes are pretty common. So, if one is looking for, how much value for money do you get out of this as opposed to that, it is important to recognise that the mainsprings of improvement come down to the few extremely important features of good management in schools and that is always the answer to, why is this not producing value for money: that it is not well managed; that it is not well-integrated into the rest of the work that the school needs to do; this management team is not well-informed about what is actually going on; they are not making very good use of assessment data and so on. So, that rather short list which appears in different guises in our annual reports perhaps relentlessly: good management, good teaching, good personal support, good relations, good links with parents is the list that we find looking at the effectiveness of improvement programmes. (Mr Raleigh) That will be coming out in the New Year. It is a report on an exercise looking at excellence in cities and other major schemes including education action zones over a two-and-a-half year period covering primary and secondary schools. We hope it will be informative. (Mr Taylor) I think the answer is "no", if I can put it as simply as that. I do not think that we would have had the evidence to say that going for diversity in itself necessarily drives up practice standards more than a single system, but it does depend a little on what you mean by "system" because we have analysed that data both in terms of the foundation of the school, in terms of selection and in terms of designated status and, in all of these ways, you get pockets of difference which are interesting to pursue, but you do not get a 42 type solution. You do not say that because we have looked at it like this, we can say to the Government confidently, "Go for diversity rather than uniformity." It does not look as clear-cut as that. Chairman (Mr Taylor) It is undoubtedly right that, whichever system you have, some of the most intractable problems are those relating to inherited disadvantage or really the kinds of very hard indicators to shift. Alas, the free school meals indicator continues to be very consistent in its effect on performance and achievement, whichever kind of system of schooling you have. Therefore, unhappy as I am with relying on free school meals data - and I think this is why people level data will take us further - it is undoubtedly the case that, when you look at the kinds of charts we produced on free school meals, it is only relatively rare outriders that buck the trend that links performance and free school meals incidence. I do not know if any of my colleagues can make specific comment on our data on free school meals. (Mr Key) In chart 15 - it is not particularly clear because of the preponderance of red squares - two points might be drawn from that chart: one would be, as Mr Turner said, the general relationship between free school meals and GCSE average point score; secondly, at any given level of free school meals, the range within at any level of disadvantage, which is quite considerable. Mr Chaytor (Mr Key) We recognise the shortcomings of using free school meal data and they are frequently drawn to our attention by head teachers. Nevertheless, infuriatingly, they do work as a proxy for disadvantage. You are mentioning other indicators and, yes, we are working pretty hard to see whether we can put together a package of indicators including, for example, pupil mobility which might feed into a more acceptable although not necessarily more useable measure of disadvantage. (Mr Taylor) Certainly the changes that Mr Chaytor referred to have pushed us further along the road of looking for alternative ways of measuring the effect of disadvantage. Mr Turner (Mr Taylor) The terms are slippery ones, are they not! I think I want to say that we probably intend to use the first three of your quartet as being rather loosely interconnected, but that social segregation is a horse of quite a different colour and that, to some extent, was clearly one of the factors in some of the early studies in grammar schools in how successful they were or were not in providing a different kind of segregation which was designed to overcome the disadvantage of economic and social segregation. I think what we want to say is that social segregation if seen to produce a kind of elite socially and in terms of status is perhaps not what Government policy can be directed at. It should be directed at overcoming disadvantage and poverty and making sure that the entitlement to access to high quality education is there for everyone and that the risks of some kind of hierarchical system, which any diverse provision might create, are minimised. I do not know if that is an answer or not! (Mr Taylor) If we are talking specifically on specialist schools and non-specialist schools ...? (Mr Taylor) What would you see as an example of that in order that we are sure what we are talking about? (Mr Taylor) I know that the CTC, for example, has made great store out of stressing that it is not in that way socially divisive and that it is taking intakes fully representative of the local community. (Mr Raleigh) It is fiercely difficult to get to the bottom of this as Professor Gorard would have indicated. I think he was using ward level data and the match between that and the school that happens to sit in that ward is nowhere near exact. I think that the point to come back to is the one that Mr Taylor has made that, from our evidence including looking at research, there is not a straightforward connection between the extent of diversity and the quality of education and the outcomes in an area, nor is there a straightforward relationship between uniformity, relatively speaking, and outcomes and quality. We are talking about a number of factors here. Age range of schools is quite often regarded as significant by parents, 11 to 16 as opposed to 11 to 18, single sex and so on. You can play in probably a dozen factors in looking at the mix of the schools and there are examples in not necessarily the whole LEA area but smaller areas of systems which are, in one respect or another, highly diverse which overall do not produce very good outcomes, certainly within the maintained sector and vice-versa. Relatively, uniform systems do not produce consistently high levels of outcome. My feeling is that talking about it across the nation is actually a rather abstract notion and that it only starts to have reality when you are looking at a particular area and the question of, is this the right mix of schools for this patch? Rightly or wrongly, this is a matter for local decision and the Members here will no doubt have experience of the difficulties with the decision-making process when local authorities or others involved in our organisation try to get that mix to be different. The final point is that it is not just a matter of what the name on the gate subtitle is of those individual schools but the relationships between them and, when you look at those areas which have more consistent outcomes across the piece, where there is a higher level of parental satisfaction, certainly when you look at the parent preferences and the degree to which they are met, the number of appeals and so on, the fact that there is not friction of a particular sort between schools, the fact that there is co-operation among them in the provision of courses, in staff development and in management development, and the fact that they are looking not always but a lot of the time at how this system is providing for the most vulnerable, those who under-achieve, and they are looking at that together are very strong characteristics of a good mix which tends to correlate with a high level of satisfaction and credibility with parents. It is a system that carries conviction for them. So, it is not just which schools in the mix but how those schools relate. (Mr Raleigh) Yes, except that the schools in that local mix change over time. The designation 'specialist schools' has been one fairly new example but there are others with new designation or involvement. Schools change; they get better; some get worse. Parents' views of them change; they do not necessarily relate to what OFSTED inspection might say about the quality of the school, parents have other lines of inquiry. So, the local systems are really stable or really absolutely stable and there are some good examples of re-organisation proposals which, over time, have done exactly what the local council was hoping, that it has improved the quality of the education and the outcomes. (Mr Taylor) One thing we definitely cannot say is that a stable system would work better or worse because we have not had one! (Mr Taylor) Miss Cross has looked at some of the new school setups, fresh start, which is one type of drastic change. (Miss Cross) They have not normally changed in their population, so we do not have that sort of evidence. We are looking at schools that will become city academies and going to the predecessor schools to look at those and then we can help to give information to the city academies union in order that they can advise schools to try to improve themselves. So, we will be monitoring those in the future and getting that sorted. If I may just make a comment in relation to schools that have been in special measures and using the term "deprived areas". The key to those schools changing as well as the list of things that Mr Raleigh gave you about leadership, management and quality of teaching is that the culture changes to one where the teachers and people in the school believe that those pupils can achieve instead of saying, "What do you expect pupils to achieve when they have these sort of difficulties in their lives?" The most important thing for school improvement, when you are looking at some of the weakest schools, is that there is a focus on the pupils and it is a focus on the pupils' achievement and that drives everything and then you need decent leadership and management to pull that together and you have commitment from everybody involved in the school including governors, parents and pupils to work together to say, "We can do better than we were doing before." (Mr Taylor) On the rapidly expanding schools, I have just checked with Mr Key and we do not have any specific evidence though I think that if there is any, we should look at it. My gut instinct, which is the thing I am supposed to leave behind when I come in here, tells me that actually rapidly expanding schools tend to do rather well because they have a kind of buoyancy of confidence and the funding that goes with it and that it is much better to be in the school that is doing that than one which is losing numbers drastically as a result of other factors. Chairman: Those of us who went to the George Dixon School in Birmingham had that feeling of a buzz, growth and so on. Ms Munn (Mr Taylor) In broad terms, you are right. It is wrong to expect quick fixes albeit that some schools do actually achieve remarkably quick improvements under the right ideal circumstances. We would normally see improvement as something which should be expected to take two years or so. Again, from Miss Cross's evidence, schools in special measures often do make quite remarkable achievement within that two year period but they are not improvements that show themselves in dramatic transformations in test data, which is again why what we are trying to do is pick up on schools which we visited that have had that status for at least two years and look at these broad range of indicators about the quality of behaviour, quality of management and the quality of teaching and ask, how is this group of schools doing under the kind of things which we as inspectors, which are the same things that parents and pupils will see when they are in schools, are judging alongside exam data and are often not proxies for performance but the growth points which will lead to that performance in a couple of years. (Mr Taylor) Certainly the progress within the specialism was one of the key factors, which Mike Raleigh said a little bit about in our report, but again it is difficult to disentangle features that were already there to a large extent and that were what earned schools status in the first place from things which are generally innovative as a result of the programme. What it has normally done is given the spur to existing good practice and taken it on that extra level. We certainly quoted in our report some examples of very good specialist practices. (Mr Raleigh) We did indeed, which did build on what schools were doing otherwise they would not have had a basis for the difference they made but it enabled them to do things faster for greater numbers and in some cases different things. A fantastic example I saw in a sports column was what they managed to do in engaging a very large number of young people who would not necessarily have seen themselves in leadership roles in school or in a community context, and junior sports leaders and who worked their way up ---- (Mr Raleigh) To be honest, some of the sports performances we saw there were of such high quality you just wondered why we were not on the map already. (Mr Raleigh) I had better not attribute that to OFSTED! (Mr Taylor) I think we are directly responsible for the Rugby Union. Chairman: What about the cricket? Mr Pollard (Mr Taylor) Obviously the COVEs, as they are called, Centres of Vocational Excellence, which are now being placed on the FE sector largely and joined with developments in the 14-19 curriculum, are designed to ensure that there is a broad basis but only within the specialist college designation. (Mr Raleigh) They are all firmly expected to develop work-related learning to a high level within their specialism. In the new specialisms we have science, business and enterprise, mathematics and computing, and engineering. (Mr Taylor) Not plumbing though? Chairman (Mr Taylor) The same is true of maths teachers in secondary schools and everything else. Paul Holmes (Mr Taylor) Yes, we have, but prefaced by the obvious observation that for every Jesson you will find an anti Jesson there as well! (Mr Key) I was going to look more widely and say, as David Taylor just hinted, that it depends who you speak to and it is interesting talking to people from the countries which you mentioned, either with the selective system or with the non selective system, both will defend their particular systems on the basis of PISA results or higher attainment. Certainly in Scandinavia, and maybe Canada and Australia as well, there will be a strongly held feeling that streaming arrangements may disadvantage lower attaining children and therefore those countries are pretty strong in their desires to stick with the fully comprehensive systems they have. (Mr Taylor) If you ever get the chance to get to the back of the pack of charts with which we inundated you, you will find that there are three or so on teaching and selection by general ability, and we like these scatter gun type ones with dots all over the place, and once you start working out what it means, which takes a bit of help from Tim - for me anyway - you do find some extraordinarily interesting things but they never lead to you the point of saying, "Well, that proves that a selective system is better than a comprehensive one". (Mr Key) I have chart 28 in front of me and if I thought we would have been able to cope with another layer of dots and different colours I would have added one which showed comprehensive schools, of course. That is a fascinating range of dots to superimpose on this which really stretches from just about the highest of the blue diamonds for grammar schools down to way amongst the red squares for the secondary moderns, and shows that there are secondary modern schools which are doing better than some of the comprehensive schools and there are comprehensive schools up with the best of the grammar schools. (Mr Taylor) But the top secondary moderns are doing better than the worst grammar schools, which we found interesting to look at. (Mr Key) I think that is absolutely right because we have talked about schools that are located in a particular place, and I think it is important to look at what is the choice within the area that we are looking at. Have we got fully comprehensive? Have we got comprehensive? Have we got selective secondary modern and grammar? What is the mix? And how close are they to each other? So it is a very difficult picture to generalise over. (Mr Taylor) All of that is right, precisely, and that is why it is interesting that the red blobs at the left hand end, which is the good end, are also in very low school meal areas, so once again it is another index of how important this class pupil level data is going to be to enable us to drill down to those very specific factors which are affecting achievement. I think once we get that tracking of individual pupil data fully worked through the system, some of these great big questions which have been teasing us about system-wide improvement will be easier to answer. (Mr Key) Sure. Chairman: I am worried that our next group of witnesses will not have time, so could we move on?
Mr Simmonds (Mr Raleigh) The numbers are so small ---- (Mr Raleigh) Of the ones we have looked at in detail, I think only four ---- (Mr Taylor) Can we get back to you on that? We can look at the 47. (Mr Raleigh) I think they are all language colleges. Chairman (Mr Taylor) We have infinitely more data. Chairman: Thank you very much. Memorandum submitted by National Foundation for Educational Research Examination of Witnesses DR IAN SCHAGEN, Head of Statistics, and DR SANDIE SCHAGEN, Principal Research Officer, National Foundation for Educational Research, examined. Chairman (Dr Sandie Schagen) I understand you have been given a number of our papers so I do not want to take up your time and obviously you want to ask questions but, very briefly, we have been working together on the area of diversity for the past year or two, myself as a researcher and Ian as a statistician. We became very interested in this subject and we have looked at not only specialist schools but faith schools, selection, single sex schools and so on, and we have looked at it by using national value-added datasets so our research has been focused exclusively on academic performance, and we need to acknowledge that obviously there are other claimed advantages of different types of schools but we have not looked at those. Neither have we looked at change over time, improvement over time. We have been focused specifically on looking at the difference in value-added performance and different types of schools. With that introduction we are quite happy to take your questions. (Dr Sandie Schagen) That is not something we have looked at. (Dr Sandie Schagen) No. We have not looked at that. (Dr Sandie Schagen) That is true. (Dr Ian Schagen) I think I had concern and it was shared by Professor Goldstein that the quality of the research and the type of analysis carried out was not of sufficiently high standard to warrant some of the conclusions being drawn. That is not to say that the conclusions are wrong but that they were not backed up by the best quality research and the best quality analysis of the available data. We now have very good comprehensive data, national value-added datasets, which it is possible to analyse in a sophisticated and more valid way which is the kind of work we have been doing, and therefore it seemed that, if you like, going back to doing something which was simple is not good enough. I think one of my favourite quotations I believe is by Einstein which is, "Keep everything as simple as possible but no simpler" - in other words, produce analysis which is based on a simple-minded look at the data and which, to be honest, seemed to be mainly polemical rather than trying to look at what is the data telling us about different kinds of schools, and that the possible impact they may be having on pupils' performance in a value-added sense was quite wrong. Having said that, the best statistical analysis in the world can tell you what relationships appear to be there and can tell you, for example, what the apparent impact of specialist schools is but it cannot tell you why there is that impact; whether it is a pre-existing effect; whether it is caused by the fact that specialist schools are getting more money; or whether it is caused by the fact that good schools are chosen to be specialist in the first place. So we had concerns about methodology, if you like, from an academic point of view and also from the point of view that it seemed to be getting a lot of publicity and a lot of government policy was being based on something which we felt, from a statisticians' point of view, was not the best way of looking at the data. (Dr Ian Schagen) The job of academics is not the same as the job of government departments or even members of Parliament. Our job is to try to understand as deeply as possible the underlying processes and the data which we have. Our aim is, at the end of that, to come up with evidence which is based as closely as possible on the data and which is as objective as possible and which people can then use to build policy. However, if you are looking for quick and simple answers to complicated and dirty problems, you may be disappointed. (Dr Ian Schagen) Yes. (Dr Sandie Schagen) Indeed. Mr Simmonds (Dr Ian Schagen) It has had a very great impact and one of the reasons we got into this field was the availability from DfES and QCA of linked value-added datasets initially from key stage 2 to key stage 3 and then from key stage 3 to GCSE outcomes, but more recently a full dataset going from key stage 2 right up to GCSE 2001 has enabled us to use the methodology, which has been around for a long time, to look at what are the impacts of different kinds of schools when you take account of not only prior attainment of pupil level, free school meals at the school level as a kind of somewhat crude issue, as we discussed earlier, a proxy for social deprivation, and trying to build models which enable to us look in a great deal of detail at what is left over when you control for everything else. The problem with looking fairly crudely at the school level - at some schools getting these sort of outcomes but some schools getting those - is that that is telling you a great deal about the prior attainment, the abilities of the pupils on intake to those schools, and not a great deal about what the schools are doing with those pupils that they have, and you have to look at what schools are doing given the pupils that they have, and so that is why we do value-added analysis while we control for prior attainment, but using multi level modelling because some of the things we have to take into account are at the school level and not at the pupil level. I do not want to get technical about the reasons for that but we now have the methodology and the data and you put the two together. We have not answered all the questions and the models we are developing, as we get more data, from PLAS, the People Level Annual School census, will enable to us control more things at pupil level to see what is happening with different ethnic minorities, what is happening at pupil level for those eligible for free school meals or whatever. We hope we will be continuing and also to build in the longitudinal dimension of how these things are changing over time. It is quite an exciting time to be an educational statistician, and to answer questions which people have bandied around for quite a while. (Dr Ian Schagen) We have not got that data yet. We hope to get it from PLAS. All we have from free school meals is the percentage eligible for free school meals. But our research over several years has shown that, even when you do value-added analysis, when you control for prior attainment, there is still an impact of free school meals. Schools with higher levels of deprivation tend to make less progress, and if you do not take that into account it can be confounded with the effects of different school types. (Dr Ian Schagen) I am sure it is by no means the perfect measure of deprivation. There will be better ones, I would have thought. Unfortunately it happens to be the one we have at the moment and it does show a relationship with outcomes and therefore we continue to use it, because not to would be worse than to use it. But if you or anybody can come up with a better measure which we can collect reliably for DfES and pass on then it would be helpful. (Dr Ian Schagen) At the moment we have not factored that in but it is an area we ought to be looking at. The models that we build are in a sense always provisional. There are always ways in which they can be improved and ways in which they can build in extra factors, and so pupil mobility is something of concern to many people. (Dr Sandie Schagen) It is worth mentioning the fact that, because we are using matched pupil level data, for pupils who move around a lot it is probably more difficult to collect and match their results at different times, and therefore it is possibly the case that more highly mobile pupils are not included in our analysis, but hopefully that would be overall and therefore should not have, one would hope, a confounding impact. Mr Pollard (Dr Ian Schagen) I am sure that is something we could do if we were able to identify the schools on the schools database. It is not something that we have included at this stage. In a sense because NFER is funded through the research it does for other bodies mainly, we tend to do research, first of all, that people like the LGA commission us to do. Secondly, we do some of our own internal research because we feel it is of general interest. If there was an organisation or we felt there was something that we as a public body should do anyway then I am sure that is something we should take into account for future modelling. (Dr Ian Schagen) I do not think it is a total surprise that, even when you are doing value-added analysis, there are still some subjects in which boys make a bit more progress than girls. Contrariwise, most of the general indicators which are not maths and science tend to show slightly more progress made by girls. We did do some work looking at the impact of single sex schools -- (Dr Sandie Schagen) -- and that tended to slightly reduce the sex stereotyping particularly in girls' schools in terms of the subjects they were entered for certainly. Chairman (Dr Sandie Schagen) Not on that issue. We looked at his work on social division - polarisation as he calls it. Chairman: I may have been doing him an injustice. It could have been Dr Taylor or even Professor Pring who said that, but we will send you the transcript. Jeff Ennis (Dr Sandie Schagen) Evaluation of EIC is going to be our biggest project at the moment. (Dr Ian Schagen) We have done the analysis of the first year's round of data and some of those papers that we produced last December are in the public domain, though obviously that was very much a baseline. We are currently waiting on the final version of the PLAS data from DfES so we can do the analysis of this year's data. We decided there was no point in doing things half-baked so we are waiting for the full dataset and then we will do the analysis. Obviously there will be similar sorts of analyses of national data to a lot of what we are reporting here, but in addition we have been collecting from EIC schools surveys which give us a lot more information about people attitudes, and part of the aim of the EIC evaluation is to look not only at outcomes but at attitudes because we feel that, if an initiative is to have an impact, the first sign of that will be changes in attitudes and in what pupils are thinking and doing, and that may follow on later to changes in performance. So that is where we are coming from on that. (Dr Ian Schagen) When we do the analysis for national data for the EIC project we include in that national dataset information about specialist schools, faith schools, grammar schools and whether they are in education action zones and so forth, so all that is in the model and it is just a question of when we run them all and get the outcomes ---- (Dr Ian Schagen) Not at this stage. (Dr Sandie Schagen) Are we talking about grammar schools or social --- . (Dr Ian Schagen) Where you have a large amount of between-school variation? (Dr Ian Schagen) I would say that, generally speaking, England does have a large amount of between-school variation compared with some other parts of the world. Partly this is due to the differing prior attainment of people going to school, so there probably is more segregation in the intakes of schools. Whether, if you could re-run the educational history of the last 200 years with a less segregated educational system, you would get better results, it is hard to say. Ms Munn (Dr Sandie Schagen) We did not look at that when we did the original research we are talking about here. We just included, as specialist, schools that had become specialist by 1999 so it was a yes/no factor. We did not include length of time in the programme. Obviously that is one refinement we could incorporate but we have not done so as yet. (Dr Ian Schagen) Yes, I think that is probably fair. There is some evidence from a lot of the school improvements in school effectiveness and literature that there is a link between outcomes and attitudes. In fact, some of the analysis we have done for our value-added work for schools has shown that if you measure attitudes and outcomes you can link the two together fairly clearly. Chairman (Dr Sandie Schagen) I think what surprised us most was right at the start. Interestingly, it was in contradiction to what was mentioned at the end of the previous session because it was concerning grammar schools. What we found was that the grammar schools seemed to work not by enhancing the performance of the most able which is sometimes suggested but by greatly enhancing the performance of what we call borderline children - those who just managed to scrape into grammar schools. There are two theories about borderline children: within a selective system there is a view that they do better in secondary modern schools where they can be at the top of the pile rather than struggling at the bottom of grammar schools, but there is also the view that they may get pulled up within a grammar school, and certainly our evidence showed very strongly the latter. We were quite amazed when we saw the difference in performance of children with the same starting point, the same key stage 2 results, and what they would get by key stage 3 in a grammar school compared with another school. The difference was really quite staggering, and it did quite amaze us and was really partly why we looked further into these issues. That was probably the thing that did surprise us most. (Dr Ian Schagen) And it was obviously why we went into the specialist and faith schools - firstly, because of current government generally and, secondly, to see whether there were any similar effects we could detect. (Dr Sandie Schagen) We were then asked by the LGA to do initially a literature review of the impact of specialist and faith schools but we realised there was probably not a great deal of literature available yet - indeed, faith schools have obviously existed for a long while and there is not much formal analysis of their impact so, having seen the results of the work we had done on selection, I suggested that we did a similar analysis for specialist and faith, and perhaps we can come back to your point at the beginning: as regards specialist schools, and consistently in our more recent work, we did find that there does appear to be a positive impact of specialist schools on all of the outcomes we looked at, so from that point of view we are not necessarily disagreeing with David Jesson's findings. It is his methodology we are concerned about. We also found a positive impact but I think to be fair we need to say this: that the differences we see are, firstly, that the impact we found is not as big as is sometimes claimed. It is relatively small in terms of total point score, for example. I think it was just under two points' difference between pupils in specialist schools and pupils in other schools, so it is not so big. The other point is making the inference which we feel is not justified that because pupils in specialist schools appear to do slightly better in value-added terms one cannot therefore assume that that difference is due to the fact that they are in specialist schools, because there are clearly other factors that could be at work which, for various reasons, we have not been able to take account of. (Dr Sandie Schagen) I have to say he uses as his main measure the 5 plus A-Cs and we do not at all; we regard it as unstable because it can be influenced by the performance of just a few children. We all know there are schools which deliberately focus on children who are on the C/D borderline to try to push up their results. There is no secret that that happens, and we feel that from a statistical point of view the outcome measures you use should be ones that reflect the performance of all the pupils in the school and cannot just be influenced by the performance of a few. (Dr Ian Schagen) Picking up that point, I do not feel happy as a statistician in using any single measure to look at schools and say, "This is the measure by which schools are judged". I think the schools are complex, multi-faceted organisations. You need a range of outcome measures to look at different schools which is why we use about seven or so and there are a number of others. You can look at schools subject by subject and find that certain schools are doing well in certain subjects and not so well in others. That kind of information is more of value in terms of driving school improvement than in publishing league tables. If you can tell schools where they are doing well and not so well compared with what you might expect given their prior attainment and other circumstances you can give them a lot of valuable information which can help them to improve, which may do more for them in some ways than just saying, "Here is some money; go and be a specialist school". But that is a personal opinion. Mr Chaytor (Dr Ian Schagen) Not to my knowledge. (Dr Sandie Schagen) The Technology Colleges Trust. (Dr Sandie Schagen) Yes, and he works for them on a regular basis. (Dr Ian Schagen) I would not like to say what was normal! (Dr Sandie Schagen) To be fair we have to say our reports which we have been reading have not been peer reviewed either. We have been submitting papers, again in this area, to academic journals which are being peer reviewed.. (Dr Ian Schagen) And which have been accepted for publication. (Dr Sandie Schagen) Yes, after peer review, but we do not always wait for peer review before publishing. In fact, certainly speaking for the NFER, that would be unusual generally because our sponsors want the results and could not wait for that lengthy process, so I do not think it would be fair to criticise purely on those grounds. (Dr Ian Schagen) No. It is useful and we try wherever possible to get results published in academic journals where they will be peer reviewed. However, as you know, because the needs of the sponsored research community are such that we need to produce results, we have internal review processes. Obviously everything that goes out from the NFER is internally reviewed and there is a quality assurance procedure, but I do not want to get in an argument about whether our methods for review are better than other people's. (Dr Ian Schagen) To my understanding he was using the same national value-added datasets that we were. (Dr Sandie Schagen) There is one difference in that in his analysis he excluded selective schools. He only looked at comprehensive or secondary modern specialist and non specialist schools. We did wonder at one point if that was one reason for the difference between us, so we looked at the proportion of both specialist and non specialist schools that were grammar schools and they were roughly the same, so we concluded that that was not sufficient to account for the difference between us. (Dr Sandie Schagen) Absolutely. (Dr Sandie Schagen) That is partly why we look at lots of different outcomes rather than just one. For the original research on specialist and faith schools we used five and on the more recent one we used seven or eight, but the interesting thing is to look at both total point score and average. People argue for one or the other but, if you look at both together, it can give you an idea of whether a good performance in total point score is due to a good all-round performance, or to be doing extra subjects. In fact, there was an interesting point - that when we did the original research, by comparing those two outcomes it seemed to us that for both specialist and faith schools their advantage was not enormous anyway but in terms of point score was not matched by an equivalent advantage in terms of average score, and that suggested that the advantage was at least partly due to taking extra GCSEs. On our more recent research when we updated we used as a specific outcome the average number of GCSEs entered, and that confirmed what we had inferred from previous research - that specialist schools in particular do enter their children for more GCSEs. On average it was about a quarter of GCSE extra compared with children in non specialist schools. (Dr Ian Schagen) No. There are a lot of things that are on the "To Do" list, so to speak, with this data, and especially when we get fuller data. What we have not looked at as part of the EIC evaluation is to look at performance in specialist subjects or specialist-related subjects and whether there are any differences there, but I think the suspicion for the faith schools was that it was probably likely to be RE. (Dr Sandie Schagen) And we also thought with specialist schools that it might be science. Again this is hypothesising but they did well in terms of total science score, so we thought maybe they were encouraging pupils to take an extra science, but there is no direct evidence of that. Mr Turner (Dr Ian Schagen) What would be interesting and nice, and what we have not yet achieved, would be to get OFSTED to provide us with some of their numerical scales from their inspections where they look at leadership and so forth, which was something they were just talking about. What would also be nice, although there are a number of technical problems not least to do with the cycle of inspections, would be to try and relate some of those OFSTED scales to the value-added datasets and include them in multi-level modelling. I did some work with OFSTED a few years ago looking at their numerical inspection database trying to pull out factors which seemed to be related to high performing schools in one sense or another, and certainly leadership was one of them. There were others as well which it would be interesting to apply to the latest national value-added dataset. The thing about doing this kind of work is that there is an awful lot of noise - in other words, that you have all this data for all these pupils but most is down to individual pupils. You can predict what a pupil will get at GCSE from what they get at key stage 2 but not brilliantly. You can explain about 50 per cent of the variants in technical terms. In other words, about half of what somebody does at GCSE is down to what they were like when they came out of secondary school, and the other half is down to what happens to them at school - and obviously some people will go up and down but it shows what they did. In all of that there is a small amount of school effect. Schools do make a difference but compared with all the noise it is quite small, and if you take away all the other factors that you can account for you can explain something like 50 per cent of the variants between pupils and about 80-90 per cent of the differences between the schools, so if you have a league table with all the schools spread out in terms of their outcomes, if you take account of all the other things you can collapse that. So there is less difference between schools than you would think when you take account of other factors, but there are still significant differences between schools, and there are significant differences between schools with differentially different outcomes - some schools are very good at English, some at science, some at mathematics. Obviously what you want to do is try and look at what things within schools make that difference like leadership management and ethos - maybe those things that explain those differences between schools. (Dr Ian Schagen) Changing the tack slightly, we have been doing some work with Professor John Grey of Homerton, Cambridge based on data collected over many years of QCA on the results of the optional year 3, 4, 5 - going into primary education - and because we have longitudinal data on what levels pupils achieve in key stage 1 - i.e year 2, year 3, year 4, year 5 and then year 6 - we can draw tracks of these pupils and some are quite fascinating in that, unlike what people used to draw which is nice steady progress, you get waves. It is like a bowl of spaghetti sometimes. Some pupils are going down; they regress in year 3 and suddenly shoot up and do really well in years 4 and 5 and then come down again in year 6, so the path taken by individual pupils through education is not clear cut and simple. Yes, they can regress. There can be periods in which they apparently, depending on your measures, regress because normally we only measure the start and the end and draw a straight line between them. (Dr Ian Schagen) If I was trying to predict an individual I would find it amazingly difficult but since what we are doing with is getting on for half a million individuals and what we are interested in is not the individuals themselves but what is happening in the system as a whole and what is happening in different types of schools, we can put up with a bit of noise that the key stage 2 levels may not be accurate. They may be accurate to a particular individual but broadly they give you a measure which ties down with prior attainment within a particular cohort. In particular in the modelling, if you do not mind me getting slightly technical, we use not just the average key stage 2 level but the key stage 2 levels in English, maths and science, so three separate indicators for individuals who have a profile. We also allowed the relationship between key stage 2 performance and GCSE outcomes to vary from school to school because we are interested to see whether some schools are accelerating the progress of the most able when they come in and not doing so well with the least able and vice versa, and also looking to see whether that is related to school types so we found a small amount of evidence to say that specialist schools seem to have a slightly steeper slope - ie, that they are doing best with the most able to a small degree. So we find the key stage 2 data is fit for our purpose; obviously if you want to look in more detail at what individuals are doing you may want to have other tests of prior attainment because there are issues about key stage 2 and how well it is transmitted from primary to secondary school. I think there have even been rumours of people cheating in those exams but hopefully that is going to be at sufficiently low level not to disturb our analysis. Chairman (Dr Sandie Schagen) It could be just for that purpose. We have not done so because that is not part of our remit, but clearly you can do this kind of analysis - you could use it to identify schools that are performing well above expectations given other background factors. (Dr Ian Schagen) What we have done for many years is produce feedback to certain schools and LEAs on their value-added performance. We have produced what I call school residuals which is how well schools are doing above or below what you would expect taking account of everything else, and we do it for a range of outcomes and we put confidence intervals about those residuals, because you come up with a number for a school but obviously because it is a statistical model there is a range of uncertainty about that number, so we produce for schools "In total score you are significantly above expectations"; "For average score you are not significantly different"; "In English you are significantly below expectations", for example that kind of information which they can then use for their self improvement. So yes, in principle, the national value-added datasets can be used in that way. In fact, I am supposed to be at a meeting of what is called the Value-added Methods Advisory Group for DfES on how to take forward the whole production of value-added measures for schools, and I hope to be able to make a contribution to that. (Dr Ian Schagen) Indeed. (Dr Ian Schagen) That would depend on what dataset you want. If you want it on the dataset we have already we could do it within a week or two. (Dr Sandie Schagen) 2001 is the latest we have at the moment. (Dr Ian Schagen) Yes. We are still waiting for the 2002 data. (Dr Ian Schagen) There have been some problems, I understand, in publishing the 2002 data and we are still waiting to receive it. As I say, part of the excellence in cities evaluation we are waiting for and we hope to be allowed to use it for wider research. We will make the usual request to the appropriate people in QCA and DfES to have access to that data for our research purposes. Paul Holmes (Dr Sandie Schagen) Yes, I think probably the words were mine! I am not sure we would say that is making a recommendation - that is putting it too strongly - but I deliberately phrased it negatively in saying that we do not feel there is an assumption, on which this diversity programme I understand is built, that creating more specialist schools would improve performance across the board. Our research does not appear to indicate that. It shows, as we have said, that if you compare those specialist schools designated up to 1999 against all other schools they are slightly ahead but, first of all, it does not indicate the reasons for that and, secondly, when we looked within LEAs at the specialist and non specialist schools and then at whole LEAs taking into account the proportion of pupils in each that were in specialist schools, there was no evidence to suggest that LEAs with more specialist schools did better. (Dr Ian Schagen) It may be interesting once we get results on some of these international comparisons. Part of the problem is you have different outcomes between England and Scotland so it is difficult to make a direct comparison. We have some data from the Pearls international study which is the reading literacy study which we intend to analyse for England, and it may well be that if we can get hold of international data we can do some sort of comparison between England and Scotland. One of our problems is we cannot just do whatever comes into our heads - well, up to a point we can but there is a limit to how much we can do unless someone is prepared to fund it. Chairman (Dr Ian Schagen) It is to improve education and training through research. (Dr Sandie Schagen) Yes. (Dr Sandie Schagen) Not really. We will deal with them afterwards! Believe it or not NFER, as opposed to the two of us, does go into schools an awful lot -- (Dr Sandie Schagen) It is an independent foundation. (Dr Sandie Schagen) No. It is not linked to any university and, contrary to what a lot of people assume, it is not linked to government either. We are an independent body and 80/85 per cent of our research is for outside sponsors, of whom the DfES is by far the biggest, but we work for other government departments. We work for a whole range of sponsors including private companies like BT and Nat West, so basically we work for whoever pays us, more or less. (Dr Ian Schagen) Including the TCT. (Dr Sandie Schagen) And the Local Government Association. There is a regular fund from them and they chose to sponsor the research we are talking about now. (Dr Sandie Schagen) That is fine, but the point is you have asked us to talk about a particular subject and particular area of research that Ian and I have worked on and which has been virtually wholly statistical, but research projects that NFER gets involved in can vary an awful lot. They are not all heavily statistical. We have a number of research projects that do not involve any statisticians at all because they are based entirely on qualitative research, going into schools, in-depth visits to schools interviewing a whole range of staff, maybe talk to pupils - whatever is deemed appropriate for the particular project. Other projects may involve questionnaire surveys to schools, telephone interviews, desk work research, looking at what other people have said, etc. (Dr Sandie Schagen) Ian mentioned that we worked for the TCT. Again, that was a project within my department although I was not personally involved and that was a qualitative project. In fact, originally they essentially gave us lists of schools that had been identified by them on the basis of David Jesson's work as being particularly good specialist schools and we were then asked to go into those schools and do in-depth qualitative research to identify the reasons why those schools were good. As I said, neither of us were personally involved in that research although we were aware of it and I know what the outcomes were, and it is quite similar in a way to some of the other research which James Tooley and others did which I have mentioned in the report and I know he was here last week, but they were looking specifically at schools already identified as high performing and the factors they came out with as reasons for those schools' success were very similar to factors identified as generally being good in terms of school improvement. In other words, they did not seem to me to relate specifically to the fact that the schools were specialist: they related to the fact that they were very good schools. What would be interesting would be to take an equivalent group of non specialist schools and see if the findings were the same but obviously that has not been commissioned by the TCT. Jeff Ennis (Dr Sandie Schagen) I do not think all policy is based on research, much as we think it is right that it should be! I do not think we can really comment on the basis for those decisions which are essentially political decisions, but I was thinking about when we were talking earlier about comparisons because you probably have a better basis for comparison there with Wales and England because they have the same outcomes and, as you say, they do not have specialist skills. (Dr Sandie Schagen) No. (Dr Ian Schagen) Not to my knowledge but that is not to say there is not. (Dr Sandie Schagen) Our official title is the National Foundation for Educational Research in England and Wales, and we have been known to venture into Scotland and Northern Ireland occasionally but not very often. A lot of our projects span England and Wales but those are commissioned by the DfES, or the LGA for purely England, and I am certainly not aware of anyone doing equivalent studies in Wales, but it would be quite possible for us to do given the same data. (Dr Ian Schagen) The Welsh Assembly is currently collecting similar national value-added data to the data that has been collected in England, but whether it would be available to general research I am not sure. Chairman: I am surprised the Scots are not looking at comparisons against devolved areas. Mr Pollard (Dr Sandie Schagen) I think there are three questions there in one and I will take them in reverse order, if I may. On the last one, I know this will sound like a cop-out but I said at the start that we looked specifically at achievement and not at those other issues so it may well be on the basis of our research alone, and I could not deny this, that there may be other differences between faith schools and non-faith schools, if I can call them that. It may be that they are better in terms of ethos and behaviour, and I know Andrew Morris has done some work using OFSTED data on the basis of which he claims to find that there are those kind of differences, and that seems to me a not unreasonable supposition and a not unreasonable finding. What I have to say, however, is that on the basis of our research, looking exclusively at achievement, there is not any evidence at all to suggest really that increasing the number of faith schools will improve the level of achievement. We suspect that the reason why, and this is why value-added is so important, there is certainly a perception that church schools are doing really well because they often do appear at the top of league tables is that they are based on raw results, and that is why it is so important to look at intake. Our finding is that basically, when you apply value-added analysis, that advantage all but disappears which suggests that the difference is based on intake. Interestingly, you can hypothesise that if they do have better ethos and better behaviour and so on that would lead to better achievement, but we did not find any evidence that that is so. Chairman (Dr Ian Schagen) I do not think we have any evidence about the Surestart and other poverty reduction measures. There is a lot of evidence that there is a relationship between poverty and education attainment - not a completely mechanistic one. Everything is statistical on average so we may have anecdotal evidence from people who come from very poor backgrounds and do well, or go to grammar school and are borderline and do not do so well, but on average there is a relationship so if you take that relationship as being causal you would expect reductions in poverty would lead to higher attainment. There are other things - there is a lot of work in school effectiveness and school improvement literature on how to counteract poverty, effects of class sizes, etc, so I think there are things that can be done even with leaving existing levels of poverty, but probably you might find that reducing poverty would have as big as or even bigger impact than that. Mr Chaytor (Dr Ian Schagen) Probably a bit lower down than that. Probably 30 or 40 depending on the selection in the LEA. (Dr Sandie Schagen) And the selection varies tremendously, as I am sure you know, from an LEA where there is just one grammar school and everyone else is in so-called comprehensives to some LEAs which have up to or even slightly above 40 per cent in grammar schools. The range is huge, which is why we try to take some account of that in research. Coming back to Mr Pollard's point, obviously we are looking at children overall and there will be exceptions and I am sure there are many others like you and I think I would have thought that myself, which is why these results surprised us so much. The difference between ourselves and David Jesson is that his work on grammar schools specifically looked at GCSE outcomes and, to be honest, our findings on GCSE outcomes were not enormously different to his. Again, we did not look at the 5 A-Cs but as far as looking at total point score and so on our findings were not that dissimilar. We also looked at key stage 3 as well because at the time we did the original research we got to look at the two key stages separately, and it was key stage 3 where we saw this enormous impact overall of borderline children, and that intrigued us so much that we then, in order to find the reason for this, wondered if it was because of higher expectations, which was a fairly reasonable hypothesis, and we tested that, because there is information on the national value-added datasets about entry to tiers at key stage 3, by logistic regression to estimate the probabilities of children being entered for high tiers in maths and science at key stage 3 controlling for their prior attainment to key stage 2, and the results of that probably shocked us even more because the chances of a child in a grammar school being entered for a higher tier at key stage 3 varied with the subject and the level of percentage of children in grammar schools but it was in the range of 9-20 times higher than a child of the same ability in a comprehensive school, and that was what really made us think that this research, as we see it, is of interest beyond the relatively small number of selective authorities because there must be an issue there, within comprehensive schools, of what can be done to ensure children of that particular ability range -- (Dr Sandie Schagen) That was one, obviously. (Dr Ian Schagen) We are hypothesising something to do with expectations - that if you are within a system where you are expected to achieve a certain level, then by and large that is what you will achieve. (Dr Sandie Schagen) That is certainly my view, although I have not been able to go into grammar schools which I would like to. What I would like to do is be able to track some year 7 pupils in grammar schools and others to identify where the difference lies, because it would require that kind of research. You cannot get that from numbers alone but that is certainly our suspicion, backed up by some work we did for an individual LEA before we did any of this. As part of that research it was not just the statistical analysis we did but I also interviewed all of the head teachers within the LEA and certainly a number of them voiced that opinion: that it was very much being in the context of a group of high achieving children or, on the other hand, a group of not so high achieving children, and I am sure that is one of the major reasons. Mr Chaytor: But if there are lessons that can be transferred to other schools and other systems, either in terms of the critical mass issue or the key stage 3 entry policy issue, then I think that is very important for our inquiry, and if you could produce something in a bit more detail for us that would be helpful. Chairman (Dr Sandie Schagen) Yes. Paul Holmes (Dr Sandie Schagen) That was again looking within LEAs. We defined LEAs as having a high percentage of pupils in faith schools, a low percentage, or no pupils in faith schools. There are not many in the latter category but there are a few, and we looked at the relative impact of having a lot of children or a smaller number of children in faith schools compared with that, and also within LEAs we looked at the performance of those children in faith schools and those other children within the same LEA, because if you adopt the kind of hypothesis of a social split, as it were, or at least an academic segregation, then your hypothesis would be that children in faith schools would do better than the norm, and children in not competing but, say, non faith schools in the same area, if it is the case that the brighter children are being sent to the faith schools, then you would expect those children to perform below the norm, and we did find a little bit of evidence to suggest that might be the case but not a great deal and the differences were quite small. Chairman: Doctors Schagen, thank you very much for your attendance. I hope you did not think it was a discourtesy that some members had to leave but this has overrun our normal time and some members had to get to other meetings. We found it fascinating; we have learnt a great deal; and we hope we can remain in communication. Thank you very much. |