House of Commons - School Accountability - Children, Schools and Families Committee

School Accountability - Children, Schools and Families Committee Contents

4 ACHIEVEMENT AND ATTAINMENT TABLES AND THE SCHOOL REPORT CARD

162. Achievement and Attainment Tables, formerly known as performance tables, are compiled and published by the DCSF. The tables are a major part of the school accountability system and provide statistical information on the school cohort, test results, a CVA score,[249] a series of comparative annual data on test scores, statistics on absence, and statistics on pupils with special educational needs (SEN). Tables of one sort or another have been the subject of controversy for many years, not least because many feel the information they provide gives only a partial view of a school's overall performance.[250] The Government's proposal for a new school report card is an attempt at presenting more information on a wider range of performance indicators. We consider the consequences of the Achievement and Attainment Tables, how the school report card might change the accountability landscape and how useful these tools are for parents and others who are interested in a school's performance.

ACHIEVEMENT AND ATTAINMENT TABLES

163. With the introduction of the National Curriculum and National Curriculum testing following the Education Reform Act 1988 came the compilation of performance tables which permitted the ranking of schools according to their results. In 2004 these performance tables became known as Achievement and Attainment Tables and a 'Value Added' measure was included, which gave some context to a school's raw test scores. The Department expressed the general view that it was important for performance data to be publicly available in order that schools could be held accountable for their performance. It argued that the data should be accessible to everyone, presented in a comprehensible format, and suitable both for informing the general public about the quality of education provision in their area and for assisting parents in making school choices.[251] The Department believed that the current Achievement and Attainment Tables had strengthened the accountability of schools through a focus on standards and attainment; and that there was a strong positive association, demonstrated by OECD research, between public attainment data and stronger results.[252]

164. The Achievement and Attainment Tables do not rank schools, but the data presented, particularly the raw test scores, are processed by the press and others to produce league tables of schools.[253] DCSF acknowledged that the press used this information to produce ranked league tables but argued that, if the Government did not publish the data it holds, it would be forced to do so in any event under Freedom of Information legislation.[254] Nevertheless, Professor John MacBeath told us that there is precedent for governments ceasing to publish accountability data on the internet. He cited the case of Hong Kong, where the government has done just that, having recognised that the published information was having a demoralising effect on teachers.

165. The value of these tables in guiding parents in their choice of schools has been questioned, for instance on the grounds that the most recently published information on a school's performance relates to a cohort of pupils who entered the school several years previously, rather than the current cohort.[255]

166. Some witnesses have also argued that public reporting of accountability data in performance tables should cease. The NUT and the NASUWT both noted that there are no performance tables or national targets linked to test results in Scotland, Wales and Northern Ireland; they were either not introduced at all or, after some negative experiences, subsequently abolished. The NUT believed that they should be similarly abolished in England.[256] The Royal Statistical Society preferred a 'private accountability' system under which feedback on strengths and weaknesses would be given directly to schools with a view to supporting them towards higher performance, rather than identifying failures in public. Any subsequent publication of results should occur only at the end of the discussion process and should recognise the provisional nature of any judgements, the statistical uncertainties inherent in performance data, and relevant contextual factors which may have influenced results.[257] The General Teaching Council for England stated that it would favour a system where schools would have more responsibility for accounting publicly for their performance via the school profile. This, it said, would be preferable to the de-contextualised and incomplete picture given in the Achievement and Attainment Tables.[258] The National Governors' Association took a pragmatic approach:

The reporting of school performance is a vexed issue. Governors are largely against the current system of league tables and to date are not supportive of any single measure that defines a school's performance. However, there is recognition that it is not possible to return to a place where there is no reporting, and so there is support for a balanced report card - as long as it measures more than attainment.

The Government now proposes that the School Report Card should "supersede the Achievement and Attainment Tables as the central source of externally verified, objective information on the outcomes achieved by schools in England".[259] However, the detailed performance data used to prepare the School Report Card would continue to be published.[260]

167. Performance data have been a part of the educational landscape in England for some years. Like it or not, they are a feature of the school accountability system and we recognise the manifest difficulties in retreating from that position, even if a watchful eye should be kept on the consequences of the abandonment of performance tables linked to test results in other parts of the United Kingdom. If such data is to be collected, much can be done to mitigate the more unfortunate aspects of the publication. We take a pragmatic view and believe that the focus of debate should move towards a more fruitful discussion of the types of data and information collected and the method of presentation.

THE EFFECTS OF ACHIEVEMENT AND ATTAINMENT TABLES ON SCHOOLS

168. The Association of School and College Leaders summed up concerns about the 30-year shift in accountability practices to the present day system:

In the 1970s it became accepted wisdom that schools were not accountable, and that there was too little information available about them outside their walls. This may have been true, but the subsequent tendencies for 'naming and shaming', for the publication of misleading 'league tables', for accountability systems to become more intrusive, and for them to distort educational practice, has been very damaging.[261]

Professor Stephen Gorard agreed, stating that schools and teachers were routinely rewarded or punished on the basis of flawed performance evidence publicly reported in tables. In his view, teachers would focus on, for example, departmental value-added figures which would then lead them to focus their attention on particular areas or types of pupil. As a result, he believed that teachers' professional practice was being distorted, pupils' education was being damaged, and parents and pupils were led to judge schools on the basis of evidence which, he argued, was largely spurious. He added that these measures were "associated with only a narrow understanding of what education is for" and that the complexity of measures such as CVA scores were, in any event, largely incomprehensible to the public and even some academics involved in school effectiveness research. Even the Chief Inspector, Christine Gilbert, has agreed that parents and "people who are really quite engaged in the educational debate" cannot understand the contents of the Achievement and Attainment Tables.[262]

169. The desire of some schools to maintain or improve their standing in performance tables can give rise to a variety of other practices which are not necessarily beneficial for individual pupils. In our report on Testing and Assessment, we commented at length on the perverse incentives, created by performance tables and targets, for schools to teach to the test, narrow the taught curriculum, and focus on candidates on the threshold of target grades, all in an effort to achieve the best possible test results. We noted that, where they occurred, these practices were harmful because pupils were forced to focus on a very narrow syllabus, taught in a manner best calculated to get pupils through the test successfully. The chances of engaging children's interest in learning are low with an approach which stifles creativity and focuses on assessment output. Moreover, those pupils who are achieving either well above or well below the target level set by the Government may be given relatively less attention in class compared with those children who are on the target borderline.[263] As might be expected, we received much the same evidence as part of this inquiry.[264] Even Ofsted noted that, although the best schools do not resort to such measures:

The publication of information about schools' performance through test and examination results can lead in some cases to teaching to the test and a narrowing of the curriculum in certain year groups.[265]

170. This evidence begs the question: why do some schools feel the need to maximise test and examination results in ways which involve teaching to the test and other harmful practices? Part of the answer must lie in schools' perceptions of the way in which performance data is used. The Department maintained that parents did not use performance tables in isolation and formed judgements based on a range of evidence, including inspection outcomes, the views of other parents and other local intelligence.[266] Yet other witnesses asserted that parents and teachers used the published data without any understanding of their inherent statistical uncertainty, driving schools to take extreme measures to improve their performance data.[267] The two views are not necessarily mutually exclusive: it is the fear of some schools that the data might be used without understanding which drives them to maximise their statistics. The way parents interpret performance data is significant because, if a school is unable to boost its standing in the league tables, it may face public censure and risk a reducing student population as parents seek to send their children to a different school.[268] Parents aside, the interpretation of performance data by teachers may also have profound consequences for a school. A deteriorating reputation can present serious problems for a school in terms of recruiting and retaining the talented staff necessary to turn the school around.[269]

171. Many of the concerns registered in relation to Achievement and Attainment Tables have related to the 14-16, or Key Stage 4, phase of education. There is considerable diversity in the range of qualifications on offer in this phase, yet only certain qualifications are recognised for the purposes of the Achievement and Attainment Tables. We were told that maintained schools are less likely to offer those qualifications not recognised, even though they might be more appropriate for certain students, particularly those less engaged with academic learning. Witnesses stated that vocational qualifications recognised in the Tables have been "forced into alignment with academic qualifications", thereby reducing their usefulness and uptake amongst those learners for which they were originally designed. At the more challenging end of the academic spectrum, IGCSE is an example of a qualification not recognised in the Achievement and Attainment Tables, and there has been widespread debate about their use and standing in relation to other qualifications, in particular the GCSE.[270]

172. Once qualifications have been recognised, it is not straightforward to compare standards in education across such a diverse set of provision in order that this information can be presented in a simple set of tables, so a concept of equivalence has been developed over time. In this context, Cambridge Assessment provided a technical analysis and referred to some "serious issues" in relation to the compilation and management of the Achievement and Attainment Tables.[271] The rating process for determining equivalence and locating new qualifications within the existing framework of recognised awards is detailed, complex and lacks transparency. Cambridge Assessment stated that this process "contains substantial elements of judgement [and is] not subject to coherent regulation or scrutiny". A "very small team" in what is now the Qualifications and Curriculum Development Agency (QCDA)[272] allocates ratings for qualifications based on a flow of data from schools and the advice of officers in QCDA and DCSF, "on the basis of 'fit' and avoidance of anomalies". Cambridge Assessment added that the rating team was "heavily driven by the 'internal logic' of previous decisions and allocations" and that they did not carry out extensive empirical studies of the effect of ratings system on institutional behaviour.[273] Cambridge Assessment would like to see the ratings and equivalence process allocated to Ofqual, the independent examinations regulator, rather than the Government's QCDA. Cambridge Assessment believes that Ofqual can bring to the process a greater degree of transparency, sophistication and sensitivity than it receives at present.

173. These equivalence ratings have practical consequences. The assumption inherent in the Achievement and Attainment Tables is that the school is the correct level at which to measure performance and direct mechanisms for improvement in order to improve individual pupil learning. We were told that this, when combined with the equivalence mechanism inherent in the Tables, can give rise to perverse incentives for schools to choose 'easier' qualifications in which it is more likely that their students will achieve higher grades. Qualifications which, in the view of Cambridge Assessment, have "different societal status and currency for progression" are nevertheless deemed equivalent for the purposes of the Tables. Schools looking to improve their standing in the Tables can migrate towards those qualifications in which students are more likely to get a higher grade and reject qualifications deemed equivalent but which are actually academically more challenging. We note that the gap between the proportion of students getting any 5 A*-C grades at GCSE and the proportion getting 5 A*-C including English and mathematics has broadened from 10% ten years ago to around 20% last year.[274] This is likely to be at least partly due to schools choosing easier qualifications for some students. Cambridge Assessment sees no clear evidence that Achievement and Attainment Tables impact beneficially on classroom practice and, indeed, there is evidence to suggest that "more superficial learning approaches have been adopted in a misguided attempt to maximise examination performance".

ACCOUNTING FOR THE FULL RANGE OF A SCHOOL'S WORK

174. Many witnesses have called for a more rounded picture of schools to be presented in place of the current Achievement and Attainment Tables.[275] A survey of parents by the National Confederation of Parent Teacher Associations found that 96% of parents wanted to see a wider range of measures used. Ofsted noted that the current tables reflected only a narrow picture of a school's performance and could appear daunting to users:

Performance tables reflect a narrow although important part of schools' work. Currently, the range of public information on schools' performance can be confusing and in practice, parents may rely more on Ofsted inspection reports than the Achievement and Attainment Tables, because the reports provide a more holistic evaluation of the school.[276]

VT Education and Skills[277] made a case for inclusion of a broader spectrum of judgements in performance tables in order to do justice to schools which were performing well but would not achieve a good rating on the basis of raw test scores or CVA measures. Despite a relatively poor showing according to the current criteria, the performance tables would not show that such schools may be improving their test results steadily as well as improving their performance in areas which were harder to evaluate, such as "inclusion, collaboration with other schools, contribution to community cohesion and quality of multi-agency working".[278]

175. In a bid to move the Government away from a system which presents such a narrow view of school performance, our Report on Testing and Assessment recommended that the use of national testing for multiple purposes—measuring pupil attainment, school and teacher accountability, and national monitoring—should be stopped. For the purposes of national monitoring, we favoured a national sampling approach up to age 14, which would be much less burdensome on schools than the saturation testing which was, at the time of our Report, carried out at Key Stages 1, 2 and 3. In line with our recommendations, in October 2008, the Secretary of State announced that Key Stage 3 testing would no longer be compulsory.[279] Then, in its response to our report on Policy and delivery: the National Curriculum tests delivery failure in 2008, the Government stated that it would bring in sample testing for the purposes of national monitoring at the end of Key Stage 3. We welcomed these changes as a step in the right direction. The Association of School and College Leaders has also expressed its approval, noting that what had seemed a politically difficult move was not so difficult in reality:

It is politically difficult to move away from some of these measures. The retention of the school league tables and the overblown testing regime in particular seemed to have become a test of political machismo. Yet when the KS3 tests were abolished in 2008 there was relatively little adverse comment and a good deal of praise for the decision.[280]

176. The Achievement and Attainment Tables present a very narrow view of school performance and there are inherent methodological and statistical problems with the way they are constructed. For instance, they are likely to favour independent and selective schools, which have a lower intake of deprived children or of children with Special Educational Needs. It is unsurprising, therefore, if such schools consistently top the academic league tables. Yet most of those who may wish to use the Tables, particularly parents, remain unaware of the very serious defects associated with them and will interpret the data presented without taking account of their inherent flaws. As a result, many schools feel so constrained by the fear of failure according to the narrow criteria of the Tables that they resort to measures such as teaching to the test, narrowing the curriculum, an inappropriate focusing of resources on borderline candidates, and encouraging pupils towards 'easier' qualifications, all in an effort to maximise their performance data. There is an urgent need for the Government to move away from these damaging Achievement and Attainment Tables and towards a system which gives a full and rounded account of a school's provision.

SCHOOL REPORT CARD

THE PROPOSALS

177. The school report card, announced by the Secretary of State in October 2008, will be introduced from 2011, with a two year pilot starting in autumn 2009.[281] It is intended partly to answer criticisms of the narrow evidence base of the current Achievement and Attainment Tables, and the Department stated that it "will provide our key statement on the outcomes we expect from schools, and the balance of priorities between them, ensuring more intelligent accountability across schools' full range of responsibilities." The Department told us that:

The data currently available is heavily weighted towards academic attainment and while data which places pupil and student attainment and progress into context - in particular, Contextualised Value Added—is published by the Government, it is typically not reported by the press, or given much lower prominence than "raw" attainment scores. In developing School Report Cards, the Government hopes to make sure that accountability arrangements are made sharper and more comprehensive.[282]

178. The school report card will be published annually, with the results of more recent Ofsted inspections being incorporated when they are available. DCSF will compile and publish the school report card nationally and will provide an electronic copy to schools for them to publish locally.[283] The school report card will set out the key outcomes expected of schools, to include pupil attainment, progress and wellbeing; reducing the impact of disadvantage; parents' and pupils' perceptions of the school and the support they receive; and, possibly, partnership working. The progression measure will be contextualised in order to account for schools with differing intakes. In line with a recommendation by the Expert Group on Assessment, the school report card will supersede the Achievement and Attainment Tables as the main source of accountability information and the detailed performance data used to prepare the report card will be published.[284]

179. It is intended that the report card and Ofsted inspection report will be "complementary and different evaluations of the school's work".[285] DCSF and Ofsted are working together to establish a consistent set of priorities for schools which will be reflected in the school report card, Ofsted inspection report and school self-evaluation form (SEF). The report card will present quantitative information on an annual basis; Ofsted's inspection report, normally less frequent, will present more qualitative information resulting from an inspection which is a snapshot of a school's performance.[286]

180. The Department states that the report card is aimed at:

Parents and carers: to provide a clearer, balanced, comprehensive account of performance, which complements Ofsted's inspection reports; to inform parent choice and improve school accountability to parents; and to provide information in a format which is more understandable and accessible than at present.
Schools: to provide a single, clear, prioritised set of outcomes against which schools will be judged with predictable consequences; to recognise the value of work across all outcomes and to hold schools to account for those they can influence; to provide a balanced account of outcomes achieved and the degree of challenge faced by a school.
Government: to provide a means of supporting the vision for school reforms; to hold schools predictably and consistently to account for what is most important; and to incentivise schools in the right way and remove perverse incentives.
Ofsted: to support the school inspection process.[287]

181. The precise links between the school report card and Ofsted's inspection framework are unclear. The joint DCSF-Ofsted publication A School Report Card: Prospectus states that a majority of respondents to the consultation on the report card agreed that a common set of indicators should be used for the school report card and for Ofsted's risk assessment (previously referred to by Ofsted as a "health check" and, more recently, as an "interim assessment").[288] The Prospectus envisages that a common view of the relative importance of different outcomes should be reflected in both the school report card and the inspection framework.[289] The document goes on to state that:

Our intention is that the indicators that underpin the School Report Card will form the core of the process of risk assessment that Ofsted will use to select schools for inspection. … In the short term, Ofsted will use the selection process developed for the launch of the new inspection arrangements in September 2009.[290]

In both the Prospectus and the White Paper, the Government recognises that the school report card will not be the sole source of information Ofsted uses in coming to a final decision on whether to inspect a school.[291] The Minister, Vernon Coaker MP, was at pains to reassure us that the Department had worked "very closely" with Ofsted in producing the latest proposals for the school report card. He went so far as to say that, if the final version of the school report card did not work for Ofsted as the basis of its risk assessment, then he was sure that the Chief Inspector would decline to use it as such.[292]

182. In giving evidence to us, Ofsted has been less clear about the role of the school report card in its inspection processes, despite the fact that it has joint authorship of the Prospectus. The Chief Inspector told us that:

I do not see [the school report card] as aimed at Ofsted. I see it as primarily aimed—a number of people would use it—at parents, the public and pupils themselves.[293]

When we questioned Ofsted on its involvement with the development of the school report card, we were told that Ofsted's expertise was being drawn on by the Department, but we were certainly not given the impression that Ofsted were in equal partnership with the Department, despite the claims made in the Prospectus.[294] As part of the 2009 inspection framework, Ofsted has developed an "interim assessment" for schools previously judged 'good' or 'outstanding' which will not be inspected within three years of the previous inspection (inspection for such schools will take place at up to five-year intervals). The Department clearly envisages that the school report card will be used by Ofsted for risk assessment purposes, yet the Chief Inspector told us in May 2009 that:

We had been expecting to do our own—we did initially call it a report card or a health check—but we would not be doing both. It doesn't make sense to do both, so we felt comfortable working with what was emerging, as long as we could have some say in what was emerging.[295]

The end of this statement seems to imply that if the eventual form of the school report card does not work for Ofsted, inspectors may not use it to replace the existing interim assessment. In the 2009 inspection framework published in July 2009, Ofsted said only that "This interim assessment may be superseded by the proposed School Report Card" [emphasis added].[296]

183. There seems to be some confusion over the links between the school report card and Ofsted's inspection framework. In principle, we see the sense in developing a common set of indicators and a common view of the relative importance of different outcomes, leading to the potential for Ofsted to use the school report card in place of its current interim assessment when deciding which schools to inspect. However, it seems that the Department is driving the development of the school report card and that Ofsted is taking an advisory role.

184. We urge the Government to work closely with Ofsted in order to produce a model of the school report card appropriate for use by the inspectorate. However, if in Ofsted's view the school report card ultimately takes a form which is unsuitable for the purpose of risk assessment, as an independent regulator, Ofsted should not feel compelled to adopt the school report card as a replacement for its interim assessment.

CONTENTS OF THE SCHOOL REPORT CARD

185. The Prospectus sets out the basis on which the pilot of the school report card will commence, with modifications to be made as necessary during the pilot phase. The categories to be included in the school report card are:

Pupil progress;
Pupil attainment;
Pupil wellbeing;
Pupils' perceptions;
Parents' perceptions; and
Narrowing gaps in pupil performance.

There will be both a score and a rating for each performance category (e.g. a score of A and a rating of 8/10). The intention is to provide both an overall indicator of performance and an indication of the priority attached to the different elements of performance. The scores attained in each of these categories will go towards calculating an overall score for the school's performance. The Prospectus recognises the tension between recognising the full range of a school's performance and keeping the number of indicators to a manageable minimum. The weighting attached to each indicator will be key in this respect.[297]

186. The annual data presented in the school report card will be contextualised in certain ways. There will be some indication of how a school's performance in each category has changed since the previous year.[298] There will also be some basic information about the school on the school report card, such as whether it has a special unit for SEN pupils, and information about the particular characteristics of the school, and its ethos statement.[299] The Prospectus states that every school, regardless of its local circumstances, should have an equal chance to achieve a good score on the report card. However, it emphasises the importance of "absolute outcomes" for children. Indicators of pupil attainment will, therefore, not be contextualised. The pupil progress indicator will be the means of contextualisation in measuring a school's academic performance and the pilot will test three types of measure: progression, value added and contextual value added.[300] As pupils start at different levels of attainment, there is a degree of contextualisation inherent in this measure. It is noted that the weighting given to absolute and contextualised measures in the school report card will be fundamental to the fairness of the system. Later in the pilot study, contextualisation of other indicators—pupil well-being, parents' perceptions and pupils' perceptions—will be considered.[301]

187. In searching for an appropriate well-being indicator, the intention is to capture a measure of a school's contribution to pupil wellbeing. The framework of the five Every Child Matters outcomes is commonly used and a range of quantitative and qualitative wellbeing measures will be piloted from September 2009. The pilot will draw on work being carried out by Ofsted on well-being indicators, including the possibility of deriving pupil well-being data from parent and pupil perception surveys.[302] The indicators for the parents' perceptions and pupils' perceptions categories on the school report card will be derived from these surveys.[303]

188. The school report card will be used as a lever for narrowing the gap in achievement between certain under-performing groups and their peers. Factors which correlate closely with under-achievement at school include those relating to membership of some ethnic groups, levels of household income, special educational needs and being looked after by a local authority. The Prospectus states that the accountability system should be used to provide "positive incentives" to schools to identify and improve the progress of these children. In order to measure the attainment gap, indicators based on established Key Stage 2 and 4 thresholds[304] will be designed to address under-performance correlated with poverty or ethnicity. The attainment gap to be measured will be the difference between the proportion of disadvantaged pupils reaching target thresholds compared with the proportion of their peers reaching the same target. An alternative measure of income deprivation will also be tested, which combines eligibility for free school meals with the Income Deprivation Affecting Children Index, a postcode-based deprivation indicator currently used in the calculation of CVA.

189. In a similar vein, the school report card will contain a measure reflecting a school's "success in securing positive outcomes for children with SEN" and reflecting the views of children with SEN and their parents. The pilot study will seek a means of identifying outcomes for SEN children without creating perverse incentives to over- or under-identify SEN as a means of influencing scores.[305]

190. Finally, the Prospectus gives a commitment to the inclusion of partnership working on the school report card and promises further consultation.[306] In line with the increasing emphasis on this aspect of a school's activity, Ofsted's revised inspection framework for September 2009 will also give a judgement for the effectiveness of partnership working.

191. The school report card will be introduced into mainstream primary and secondary maintained schools in the first instance, including Academies. The model will be refined and developed for special schools, Pupil Referral Units and alternative provision. It will also be developed to reflect early years and sixth form provision and, in relation to the latter, how it might reflect the Framework for Excellence report card.[307]

APPRAISAL OF THE SCHOOL REPORT CARD

192. If witnesses to this inquiry have been largely critical of the current Achievement and Attainment Tables and their effect on schools, with some exceptions they have generally welcomed, at least in principle, the concept of the school report card.[308] We asked Jon Coles, Director General of the Schools Directorate at DCSF, what evidence the Department had for the effectiveness of the school report card. He said that there was very good evidence from New York that the report card there had challenged and raised performance in the lowest performing schools.[309] However, even those witnesses who welcome the school report card in principle have not shared the Department's confidence in the effectiveness of the specific report card model proposed.

193. First, there has been some concern at the prospect of the Government borrowing a policy from another jurisdiction without consideration for social, structural and cultural differences which may affect implementation of the policy and its ultimate effects. An international comparative analysis of accountability and children's outcomes in a variety of jurisdictions carried out by the School Accountability Review Group[310] concluded that:

There is widespread and increasing interest in monitoring the performance of education and children's service systems and in monitoring performance across an increasing range of outcomes. There is extensive policy borrowing. Government needs to recognise that policy structures and cultural practices are different in different systems.[311]

194. The NUT states that the key purpose of the school report card in the New York system is to focus resources on those schools most in need. It observed that The New York report card is published with a caveat that it does not provide information about student performance on a number of measures which are, nevertheless, valued by the community; and it added that these issues do not appear to be recognised in proposals for the English school report card.[312] Fundamentally, the NUT believes that the English school report card proposals are set within the existing "high-stakes, punitive inspection regime which is data driven", rather than being used as part of a revised accountability strategy. Such a revised regime should, according to the NUT, effect real change by encouraging schools to focus on their wider responsibilities to improve the life chances of pupils with the freedom to innovate.[313] The NUT considers that the report card proposals fail to take account of schools' self-evaluation work and states that it will merely add to the existing accountability burden, rather than reduce or rationalise it, as the Government claims.[314]

195. Second, there is a concern about the lack of evidence-based policy making. Professor Peter Tymms argued that, although we might consider the New York system and the way the report card is said to work in that jurisdiction, it was not possible to know for sure what effect the report card had had in New York because there was no way of making an evidence-based assessment. There, as here, many reforms and changes had been introduced simultaneously and, because of that, it was not possible to draw causal links between a single initiative and an outcome or set of outcomes.[315] If Professor Tymms' views are accepted, the Department's claims that there is good evidence for the effectiveness of the report card in New York seem hard to sustain.

196. We welcome in principle the introduction of the school report card as a rationalisation of current accountability mechanisms and an attempt at providing a broader evidence base for assessing schools' performance. However, the Government must take care in developing its proposals that it tailors the school report card to the particular needs of the English schools system. Lessons can be learned from international practices and the case of the New York school report card will be particularly relevant; but the Government should not assume that what works elsewhere will necessarily work in the English system.

197. We set out below issues raised by witnesses under a variety of headings. The list is not necessarily exhaustive, but should be taken as an indication that the school report card requires a considerable amount of work before it is suitable for use as a fundamental part of the English school accountability system.

WHAT IS THE SCHOOL REPORT CARD FOR AND WHO IS IT AIMED AT?

198. There is a question mark over what the school report card is for and who it is aimed at. The National Association of Head Teachers and the Association of School and College Leaders welcomed the school report card as having the potential to provide parents and others with a holistic view of a school not currently available to the public.[316] The National Governors' Association considered that the School Report Card was potentially a sound basis for informing parents about a range of measurable outcomes as well as describing the school's priorities. However, the Association warned that overcomplicating the school report card would defeat its purpose and make it incomprehensible to parents.[317]

199. Both the General Teaching Council for England and the Local Government Association (LGA) considered unrealistic the aspiration expressed by DCSF that the school report card could be a single accountability tool for all parties, including parents, carers, schools, government and Ofsted.[318] The LGA also questioned how the current proposals for a school report card would fit in with models and processes connected with Ofsted, Framework for Excellence and the Comprehensive Area Assessment.[319] Edexcel did not necessarily approve of the form of school report card currently proposed but saw potential for it as a means of rationalising information currently available from a variety of sources, some of which may be contradictory and inaccessible to some sections of the community. It considered that a school report card should be used to draw together in one place and in a coherent form a range of indicators of well-being, progress and achievement. These would be represented by a blend of quantitative and qualitative data, including parent and pupil surveys and a professional assessment of the quality of services, adjusted to reflect local circumstances.[320] The Association of School and College Leaders thought that the school report card was an attempt to:

… address some of the weaknesses of the present system by drawing different indicators together to offset one perverse incentive against another and to limit accountability measures to a single list. As such it is welcome, but ASCL is not convinced that it will not simply be added to the existing system rather than replacing it, or that it will not also grow without limit as every interest group adds its particular favoured element.[321]

200. There were also concerns about the likely complexity of the school report card. During our visit to New York, we were told that the report card used there was considered too complex for many parents to understand. Witnesses representing the Awarding Bodies in this country also remarked on this problem with the New York report card. Both Simon Lebus, Group Chief Executive of Cambridge Assessment, and Jerry Jarvis, Managing Director of Edexcel, said that generating large quantities of data for the purposes of a report card can become highly complex and lead to difficulties in making judgements about a school's performance. They both considered that much work had to be done before the English school report card would be in a position to work around these complexity issues.[322]

WHAT SHOULD BE IN THE SCHOOL REPORT CARD?

201. Although the principle of a scheme which gives a broader account of school performance was welcomed, witnesses have expressed differing views about what they want to see in the school report card. Some want it to provide plentiful, in-depth information about a school; others want it to provide just a few headline objectives ranked on a simple scale.[323] The Centre[324] suggested that both objectives could be achieved if the school report card were presented on a website with a simple front page for each report card and the ability to click on a category to access much more detailed information.[325] The Centre welcomed the school report card, but it cautioned that its success would depend on the right categories being chosen and the full data on which the report is based being made available to parents should they wish to access it.[326]

202. The Local Government Association (LGA) considered that the proposed categories of performance for the school report card—attainment, pupil progress, wider outcomes, narrowing gaps, parents' views, pupils' views—seemed sensible. However, the LGA was awaiting further detail on how the indicators were to be constructed before passing final judgement. The Audit Commission, the General Teaching Council for England (GTCE) and others were also positive about the inclusion of wider outcomes and an indicator of narrowing gaps in achievement.[327] Indeed, the Audit Commission thought the school report card should go even further to include comment on out of school activities and other services, such as use of the school nurse and therapist.[328] However, the Association of Teachers and Lecturers (ATL) did not believe that the proposal to collect data on wider outcomes and narrowing gaps would be any more valid or reliable than the more traditional, and in the ATL's view flawed, indicators of pupil attainment and progress.[329]

203. The Prospectus makes clear the Government's commitment to the view that raw test scores, regardless of a school's context, are extremely important indicators of a school's performance. Contextualisation is introduced in only one measure on the school report card, 'pupil progress'. The GTCE wished to see all indicators on the school report card contextualised, on the basis that the focus on raw scores could negatively impact on schools working in challenging circumstances and fail to give them credit for good work.[330] The National Association of Head Teachers argued that the school report card should be viewed alongside the school's self-evaluation form and development plan in order to provide context.[331]

204. The Prospectus gives a commitment to the inclusion of partnership working on the school report card and promises further consultation.[332] The Youth Justice Board welcomed this as a means of promoting partnership working.[333] The LGA also approved. The GTCE, however, saw a contradiction between, on the one hand, the vision set out in the 21^st Century Schools consultation document of the school as a community hub, involved in a variety of networks and partnerships connected with the wider outcomes for children and young people; and, on the other hand, the notion of a school report card which may or may not include a description of the partnerships in which the school is involved.[334] According to the GTCE, further work was needed to explore how partnership working should be reported.[335] The ATL was concerned about the school report card making schools accountable for matters beyond their control, particularly in terms of well-being. The ATL saw responsibility for children's well-being as shared between schools and other local services and worried that the shared nature of the responsibility would not adequately be reflected in the measure of partnership working included in the school report card.

205. It is proposed that the school report card will use a 'credit system' in relation to work with disadvantaged children. Schools with sufficient numbers of disadvantaged pupils would gain credit to the extent that they achieve continuous improvement for all pupils whilst at the same time narrowing the gap between disadvantaged groups and others. The GTCE supported this focus on a school's contribution to narrowing achievement gaps.[336] The Government is considering penalties for increasing gaps, but it is acknowledged that this may result in unfair penalties for schools due to a change of intake rather than performance.[337] James Liebman, Chief Accountability Officer at the Department of Education in New York, told us that the New York authorities considered it important to give credit to schools which make progress with under-performing groups of pupils and to ensure that there was no disincentive in the system which would motivate schools to avoid taking on challenging pupils. Reflecting this concern, the Youth Justice Board of England and Wales urged the Government not to include in the school report card any measure which might have the effect of discouraging schools from taking their "fair share" of pupils with challenging behaviour and from disadvantaged backgrounds.[338]

206. Schools should be strongly incentivised by the accountability system to take on challenging pupils and work hard to raise their levels of attainment. To this end, we support the proposals to introduce credits on the school report card for narrowing the gaps in achievement between disadvantaged pupils and their peers. However, we strongly caution the Government against the introduction of any penalties for increasing gaps in achievement. If the Government were to attach such penalties, it is likely that schools would seek to deny school places to challenging pupils in order to avoid the risk of a lower school report card score. They might also create incentives for schools not to push gifted and talented students to reach really high levels of achievement.

THE OVERALL SCORE

207. The Government remains committed to the inclusion of an overall rating on the school report card. It recognised in the White Paper that how the score was constructed would be crucial to the way it was received.[339] The Prospectus states that a summary score on the school report card is an important mechanism for ensuring clarity and transparency in priorities for school performance and it must be underpinned by a balanced method of representing a school's performance; but it is recognised that developing a scoring model that is robust, credible and accepted will be challenging.[340]

208. The debate about the inclusion of an overall score in the school report card is essentially an argument about complexity. If it is thought that parents and others can understand complexity in school performance reporting, then there is no need for a summative school report card which provides an overall grade. On the other hand there is the view that complexity is not normally grasped by lay users of performance data and it is better to take control centrally by presenting one, privileged view of school performance—one which focuses on a number of measures judged to be the most important and weighted according to defined criteria. According to this view, there will always be other ways of judging a school, but one formula is chosen over all others for policy reasons and an overall grade is awarded on the basis of a single view of what should be prioritised. A possible middle way would be to grade performance in the chosen categories, say on a scale of 1-10, and leave it at that, so that no overall grade is awarded and parents and others can choose the measurements that are most important to them. Individuals do this routinely in other areas. For example, when buying a car consumers have a range of information on which to base their choice. They prioritise what is important to them—price, NCAP rating,[341] engine size, fuel type and consumption—and make their choice without recourse to an overall rating.

209. John Bangs, representing the NUT, expressed another view of the complexity issue. He thought that the overall score was more about providing simplicity to government than to parents:

We have to understand where the single grade comes from. It comes from the Government's approach to public sector reform. It is a flight from complexity. It is about giving Ministers simple solutions to complex problems, but … those are often wrong solutions. A single grade does not drive up motivation for institutional improvement. What it does is tell the best people in the institution to leave, especially if it is a really bad grade, because it can't differentiate between those who are effective in the institution and those who are not. It is a crude blunderbuss approach that can lead to the best people leaving the institution. Perhaps [a balanced school report card] is a holy grail, but it is achievable: the key issue is to have a simple summary of the effectiveness of the institution, looking at the key concerns and issues, without having a single grade bracketed into four separate tiers that actually has the effect of demoralising individual people who are really making a difference in the institution.[342]

210. There was little support in the evidence for an overall score on the school report card. A survey of parents by the National Confederation of Parent Teacher Associations found that 56% favoured the inclusion of both an overall score and individual scores for each measure used to assess performance.[343] However, many witnesses thought that an overall score would be misleading or risk an over-simplification of a complex set of data.[344] The Audit Commission stated that the school report card should include only measures which gave a balanced view of school performance. Parents could then prioritise for themselves which indicators they thought were most important for their child and focus on the relevant part of the report card. Both the General Teaching Council for England and the National Association of Head Teachers strongly opposed the overall rating and were concerned that it may undermine the legitimacy and usefulness of the school report card.[345] Edexcel expressed concern that, without an appropriate explanatory narrative underpinning the report card, there was a danger that over-simplification "could easily provoke public and media over-reaction to a single summative grade, as has been the case in the US".[346] The LGA was against the inclusion of an overall score on the basis that it risks gross over-simplification of a complex set of indicators. The Advisory Centre for Education was also opposed, arguing that the overall score could be misleading. The Association of Teachers and Lecturers focused on problems inherent in the data on which the overall score would be based, stating that the school report card would "replicate the problems of the current accountability system. Individual grades will be allocated based on an accumulation of flawed data, but will be reported as if they offer meaningful information and comparison".[347]

211. We have been struck by the weight of evidence we have received which argues against an overall score on the school report card. It is true that Ofsted comes to an overall judgement on a four point scale, but this judgement is meant to be the result of a very extensive analysis of a school's provision across the board, relying on quantitative and qualitative evidence and first-hand experience of the school at work. A school report card is not, and in our view never can be, a full account of a school's performance, yet the inclusion of an overall score suggests that it is.

212. The range of discrete measures proposed for inclusion in the school report card certainly present a broader picture of a school than the current Achievement and Attainment Tables; but they cannot be the basis for a definitive judgement of overall performance in the same way as we are entitled to expect an Ofsted judgement to be. On balance, we think that parents and others should be able to decide for themselves those measures of performance most important to them. We approve of the proposal both to grade and rate performance in each category on the school report card, but we are not persuaded of the appropriateness of and need for an overall score.

CONSISTENCY

213. There are at least two types of consistency which are relevant to the school report card. First, in order to allow for a meaningful analysis of a school's year-on-year progress, it should be an instrument which is consistent over time, so far as possible. The White Paper certainly presents the school report card as a means of providing a consistent picture of schools' performance. However, we heard from the Office of Accountability in the New York Department of Education that the reporting mechanisms for the New York report card were constantly being modified and refined as a result of challenges from schools who considered them unfair. James Liebman, Chief Accountability Officer, admitted that the system was not perfect, but he maintained that it could be constantly improved and he considered the report card was still fundamentally useful in that it concentrated the minds of school leaders on improving performance. Nevertheless, serial changes to reporting criteria present a barrier to tracking year-on-year performance of schools on the basis of a consistent set of indicators.

214. Second, the school report card should be consistent with wider elements of the accountability system so that conflicting judgements and priorities are not perpetuated from the existing system into the reformed system. The overall score on the school report card is intended to be complementary to the Ofsted judgement.[348] Although the Prospectus envisages that a common view of the relative importance of different outcomes should be reflected in both the school report card and the inspection framework,[349] the NUT had some objections to this. The alignment of the school report card with Ofsted inspection criteria suggested, according to the NUT, that Ofsted would be following Government policy on 'standards' and school intervention which, in turn, suggested an erosion of Ofsted's independence from Government.[350]

215. The Government accepts that there will be times when the school report card score and Ofsted inspection judgement do not give the same message about a school's performance.[351] Ofsted also accepts this will happen, explained by the fact that the school report card will present backwards-looking data, whereas an Ofsted inspection judgement represents a real-time snapshot of the performance of a school which takes into account a different set of performance indicators.[352]

216. We recommend that the Government guards against serial changes to reporting criteria for the school report card once it is introduced nationally. The ability to track school performance on a range of issues over time is potentially a valuable feature of the reformed system, but this will not be possible if the reporting criteria are in a constant state of flux.

217. There is potential for substantial confusion to be introduced if the reasons for differences between scores on the school report card and Ofsted judgements are not clear, leading to a perception of incoherence in the accountability system. This would be unfortunate, as the success of any accountability system depends on the extent to which users have confidence in it. We recommend that DCSF and Ofsted work together to find a way to eradicate, or at least minimise the impact of, this problem. If the Government accepts our recommendation not to include an overall score in the school report card, the potential for conflicting accounts of school performance would be greatly reduced.

SUITABILITY OF PROPOSED INDICATORS AS A PROXY FOR UNDERLYING PERFORMANCE

218. According to the White Paper and Prospectus, three of the measures proposed for the school report card—pupil wellbeing, parents' perceptions and pupils' perceptions—may be based wholly or largely on responses to 'perception surveys' and, in the case of well-being indicators, nationally collected data on attendance, exclusions, post-16 progression, the amount of sport provided and the uptake of school lunches.[353]

219. There is no indication of what thought has been given to the possibility that response rates for some schools, particularly from parents, might be quite low and we have noted in Chapter 3 some serious methodological concerns with survey evidence.[354] Witnesses such as Aspect have certainly expressed concern about the validity of conclusions drawn from unrepresentative samples.[355]

220. We are also concerned about the well-being indicators and the extent to which they can really be accurate, based as they are on a limited set of loosely-related quantitative data and problematic survey evidence. Beyond the recent Ofsted consultation on well-being indicators, there is very little evidence to indicate just how accurately such indicators can represent a school's true performance in relation to pupil well-being. The Association of Teachers and Lecturers did not believe that the proposal to collect well-being indicators was sound.[356] The Independent Schools Inspectorate found the proposals for including "qualitative judgements in numeric form" alarming. It thought that the evidence may be thin, it was not clear how the views of pupils and parents would be reported appropriately, and yet such evidence could have a significant negative impact on a school.[357] The NASUWT pointed to the possibility that views of pupils and parents on the quality of school provision may differ significantly depending on their individual circumstances and urged caution over the use of survey evidence.

221. Professor Stephen Gorard of the University of Birmingham has considered the current model of school effectiveness, dominant in research, policy and practice in England, which is based on assessment results. He has noted that there is a need for more research on school effectiveness in terms of wider outcomes for children:

Schools are mini-societies in which pupils may learn how to interact, what to expect from wider society, and how to judge fairness (Gorard and Smith 2009). Schools seem to be a key influence on pupils' desire to take part in future learning opportunities, and on their occupational aspirations (Gorard and Rees 2002). All of these outcomes have been largely ignored in three decades of school effectiveness research. It is time to move on.[358]

222. The Government must address the methodological problems inherent in basing important indicators on survey evidence. It is unacceptable that schools with the most challenging intakes might suffer skewed performance scores because of a low response rate to surveys for the purposes of the school report card.

223. Academic research in the field of school effectiveness is lacking in the field of pupil well-being and wider outcomes beyond assessment results. In the absence of robust, independent research evidence, the Government should exercise great caution in pursuing its otherwise laudable aim of widening the accountability system beyond simple test scores.

224. We do not believe that the indicators based on parent and pupil surveys, together with data on attendance, exclusions, the amount of sport provided and the uptake of school lunches, provide a balanced picture of a school's performance. In the absence of a set of performance indicators which are able to provide a fully rounded and accurate picture of how well a school is supporting and enhancing the well-being and outcomes of its pupils, the school report card should not purport to give a balanced view of a school's overall performance in this or any other area. The Government should make clear on the face of the school report card that its contents should only be considered as a partial picture of the work of a school. This is not to say that we do not consider the inclusion of well-being indicators to be a welcome development: we are merely concerned that parents and others should understand the limits of the information which is presented to them on the school report card.

SUMMARY CONCLUSIONS ON THE SCHOOL REPORT CARD

225. We are pleased that the Government is now moving away from the Achievement and Attainment Tables based on a narrow set of measures of academic achievement derived from test results. We believe that the move towards the broader evidence base proposed for the school report card is a step in the right direction. However, we reiterate our warning to the Government that it should not make claims for the school report card which do not stand up to scrutiny. It will never constitute a definitive view of a school's performance but it might, if properly constructed, be a useful tool in assessing a broader range of aspects of a school's performance than is possible at present.

226. At the start of the pilot study of the school report card, it is too early for us to make detailed recommendations about its precise contents. At this stage, we simply urge the Government to take account of the concerns raised by witnesses to this inquiry. There is still much work to be done in developing the school report card into a workable format.

249 Contextual Value Added score. See footnote 7 for an explanation. Back

250 See further Testing and Assessment, Third Report of the Children, Schools and Families Committee, Session 2007-08, HC 169-I and HC 169-II Back

251 Ev 146 Back

252 Ev 152 Back

253 Ev 197 Back

254 Ev 153 Back

255 Leckie, G. and Goldstein, H. (2009)The limitations of using school league tables to inform school choice Back

256 Ev 17 Back

257 Ev 178 Back

258 Ev 4 Back

259 Your child, your schools, our future: 21^st Century Schools White Paper, DCSF, Cm 7588, para 4.24. See also Explanatory Notes to the Children, Schools and Families Bill, Bill 8 of Session 2009-10, para 92. Back

260 Your child, your schools, our future, DCSF, Cm 7588, para 4.25 Back