Education and Adoption Bill

Written submission submitted by Dr Rebecca Al len, Director, Education Datalab (EAB 01)

 

Background on author and organisation

Education Datalab, part of the non-profit company FFT Education Ltd, produces independent research on education policy and practice. Dr Rebecca Allen is Director of Education Datalab, on leave from her academic position as Reader in Economics of Education at UCL Institute of Education. She is an expert in the analysis of large scale administrative and survey datasets, including the National Pupil Database and School Workforce Census.

Summary

In this note I address the question of how coasting schools might best be defined. I argue that the social gradient in the distribution of inadequate Ofsted judgements has produced a perceived need to address underperformance of schools serving more affluent communities. Progress 8 cannot straightforwardly be used to identify coasting schools because it also displays a social gradient in its distribution. I offer two alternative metrics – contextual value-added and families of schools comparisons. I discuss how best to avoid the problem of volatility in small primary school performance. I offer some additional data metrics that should be used to determine which schools fall into the coasting schools group.

The legislation on coasting schools is needed because Ofsted’s remit and guidance to inspectors is not well-aligned with public policy goals .

1. It is generally agreed that there are schools who provide a perfectly adequate education, but who could do a great deal better for the pupils they educate. They are far more likely to be situated in more affluent communities where there is a steady of supply of teachers willing to work at the school, where pupil mobility is low, and where the school does not have to routinely compensate for significant social dysfunction in families.

2. Ofsted does not judge these schools to be inadequate. This is because inspectors observe lessons and practices that are satisfactory or good, compared to the typical school they visit. The result is a very strong social skew in Ofsted judgements:

· 2 out of 380 secondary schools with a free school meals (FSM6) proportion below 10% were most recently judged as inadequate; 45/353 schools with FSM6 over 50% were judged as inadequate.

· 221/380 secondary schools with FSM6 below 10% were judged as outstanding; 53/353 schools with FSM6 over 50% were judged as outstanding.

· 14/2493 primary schools with a free school meals (FSM6) proportion below 10% were judged as inadequate; 78/2915 schools with a free school meals (FSM6) proportion over 50% were judged as inadequate.

· 664/2493 primary schools with FSM6 below 10% were judged as outstanding; 293/2915 schools with FSM6 over 50% were judged as outstanding.

3. This social skew in Ofsted judgements is not a priori wrong if we want the judgements to reflect the quality of teaching and learning they observe. But an alternative approach would be to ask Ofsted to judge whether schools are providing a high quality education, compared to other schools that operate in similar social circumstances.

4. This would lead to so-called ‘coasting’ schools being judged as poor, since their exam performance, and the quality of observed practices would be poor, relative to schools inspectors have visited who have similar social circumstances. In my opinion, this modification of Ofsted’s remit and guidance would remove the need for new legislation to tackle coasting schools in the manner described in the draft Bill.

A definition of coasting schools based on Progress 8 replicates the social gradient problem of Ofsted judgements

5. The social gradient in Ofsted judgements highlights how important it is that the data used to define a school as coasting is capable of identifying relative underperformance of more affluent schools. This implies that it must be based on some measure that takes into account the social setting of the school, rather than simple raw outcome measures. Assuming we follow the Secretary of State’s indication that it will be based on three years of data (e.g. a measure falling below a particular value in each of the last 3 years), there are three broad approaches worth discussing for secondary schools:

Progress 8

6. Progress 8 will be used by Government as the main accountability metric from 2016 onwards. It judges how each pupil performs at GCSEs in maths, English, any 3 of the English Baccalaureate subjects (sciences, history or geography, languages, computer science), and any 3 other subjects by comparison with pupils who achieved the same Key Stage Two fine grade at age 11. Any school scoring below -0.5 (i.e. an average of half a grade below expectations for each pupil) falls below the floor target. About 300 schools would have fallen below the floor in 2014.

7. Progress 8 would seem to be a good candidate to define coasting schools because it is being introduced into the accountability framework anyway and is well-aligned with a curriculum that Government would like schools to deliver. However, it is actually highly unsuitable, for two reasons.

8. First, it cannot be used for a number of years. To define coasting schools we need a lagged measure of performance. Schools cannot be held accountable for attainment on Progress 8 for the period of time where their curriculum may not have been well-aligned. This means coasting schools cannot be identified on this metric until Autumn 2018 (when data for 2016, 2017 and 2018 is available). And during its introduction it is important to bear in mind that the distribution of Progress 8 scores achieved by schools is likely to substantially alter as subject-entry patterns change and as GCSEs are re-scaled onto the 0-9 scores.

9. Second, Progress 8 displays exactly the same social gradient across schools that it is important to avoid. [1] Schools in more affluent areas will, on average, achieve higher Progress 8 scores and so it would be impossible to use it to label them as coasting. To give an idea of the magnitude of the problem, in 2014: [2]

· Just 42/380 with FSM6 below 10% had a negative Progress 8 score

· 191/347 with FSM6 over 50% had a negative Progress 8 score

10. It is not always obvious why the social gradient in Progress 8 should emerge, given that it accounts the pupil’s Key Stage Two prior attainment. There are many reasons why, of which the most important is that pupils in schools have clustering of social circumstances so that a low attaining pupil who attends a relatively affluent school is likely to have a supportive home environment and thus do better at GCSE than a low attaining pupil in a less affluent school.

Contextual value-added

11. Contextual value-added (CVA) was a performance measure used in the latter part of the last Labour government that uses regression analysis to ‘remove’ the social gradient in school performance, thus giving schools with affluent and deprived intakes an equal ‘chance’ of doing well. [3] The formula for doing so is complex for schools to understand because it includes measures of the child’s socio-demographic background and prior attainment and the school’s average socio-demographic make-up.

12. By forcing all schools to have an equal chance of doing well across the spectrum of school intake characteristics it makes an enormous assumption that there are equal numbers of effective schools serving deprived communities as there are serving affluent communities. We have good reasons to suspect this is not the case. However, for the purposes of identifying coasting schools this would seem to be a nice property.

13. Although a pupil’s best 8 GCSE results were originally used in the measure (with a double count for maths and English), any outcome measure could be used to calculate CVA, including Attainment 8. It would even be possible to use a gradual transition towards Attainment 8, so that the outcome measure is:

· 2014: English, maths (both with a double count) and any other 6 GCSEs or equivalents

· 2015: English, maths (both with a double count), any 2 EBacc subjects and any other 4 GCSEs or equivalents

· 2016 onwards: English, maths (both with a double count), any 3 EBacc subjects and any other 3 GCSEs or equivalents (i.e. Attainment 8)

Families of Schools comparisons

14. The Department for Education, the Educational Endowment Foundation and others have created Families of Schools that allow judgement of whether a school is doing well, relative to other schools with a similar pupil intake. Creating these Families of Schools is not without controversy because choices have to be made about the size of a group and factors to determine inclusion. However, they could provide an intuitively appealing way to judge whether or not a school is coasting.

Identifying coasting primary schools is more difficult

15. It is not possible to consider what Key Stage Two outcome measures should be used to identify coasting primary schools until the Standards and Testing Authority release further details of the tests. But regardless of the measure used, primary schools experience higher volatility in exam results for reasons entirely out of the control of the headteacher. This is especially true for single form entry primary schools where numbers sitting tests are small.

16. The impact of this volatility obviously generally means that any performance metric is less well correlated with the true quality of school teaching and practices in any particular year. If we set a coasting schools definition that requires performance below a particular level in each of 3 years, it is likely that truly coasting schools escape identification because the volatility in the metric raises their performance above the threshold for an individual year.

17. To provide a magnitude of the problem, just 5.7% of primary schools fall into the bottom 25% of the CVA distribution for each of 2012, 2013 and 2014. By comparison, for secondary schools this figure is 7.6%.

18. Any solutions to dealing with volatility are rather unsatisfactory. It would generally be much better to measure primary school performance using a 3-year rolling average. But to take 3 separate 3-year rolling averages would stretch inclusion of information on a primary school’s last 5 years of performance.

Identifying the performance of pupil sub-groups increases volatility of measure

19. If the coasting schools definition includes monitoring of the performance of particular sub-groups, such as Pupil Premium children, this necessarily reduces the number of children that the metric is calculated on and so increases the volatility of the measure. So, this should not be done for primary schools and I would recommend it is not done for secondary schools because it would substantially reduce the reliability of the measure.

Distinguishing coasting from struggling to stay afloat

20. If coasting is solely defined by performance data, it will misidentify schools that are ‘paddling hard to keep their head above water’. Some schools face dynamic pressures such that they expend an enormous amount of energy just to achieve mediocre exam results year after year. Some of these dynamic pressures are identifiable in administrative data. One is levels of pupil mobility, particularly receiving children who are migrants or who have been excluded from other schools in the past. The other is schools situated in towns or counties with structurally high teacher mobility.

The arrival of a new headteacher must re-start the coasting schools clock

21. We should think of the arrival of a new headteacher as an intervention in itself. That individual should have three years to implement changes before the school can be deemed to be coasting. The consequences of not including this break-clause in the definition are very serious indeed since otherwise no prospective headteacher would be prepared to join a school with a year or more of relative underperformance.

How should we choose how many schools we label as coasting?

22. The debate about which schools are subject to intervention, or to receive extra support, has always been in terms of how many are failing, or coasting – defined in terms of not hitting a certain standard. I would turn the question around and ask: at any given point in time, how many schools can we reasonably support to improve? And how can we make sure those schools get help? [4]

23. There are very real resource constraints – excellent headteachers looking for a new school, time of the Regional Schools Commissioners, inspectors with hours to devote to school improvement or proven academy sponsors pursuing growth.

24. In any case, the judgement of ‘coasting’ is fundamentally a relative one, as all judgements we make on schools are. This is true in data – our perspective on what it is possible for a school to achieve is necessarily revealed through the performance of others. And equally inspectors’ perspectives on what they see when they walk through a school’s door is shaped by what they’ve seen elsewhere.

June 2015