Memorandum submitted by the National Foundation
for Educational Research
EXECUTIVE SUMMARY
1. NFER is Britain's leading independent
educational research institution and is submitting this paper
to the Children, Schools and Families Select Committee to inform
its inquiry into school accountability.
2. The evidence presented by the Foundation is
based on work commissioned by a number of clients wishing to examine
elements of the school accountability system. The findings portray
a relatively positive attitude (from schools) to the new accountability
mechanisms and suggest a number of areas where further attention
should be focused. The key findings from our work are summarised
below.
— Section 5 inspections are seen as
"fit for purpose".
— Schools value inspection as a tool for
school improvement.
— That the perceived burden of inspection
has reduced on moving from Section 10 (S10) to Section 5 (S5)
inspections.
— An independent inspectorate has a positive
influence on school standards.
— Schools would welcome a greater emphasis
on lesson observation and detailed feedback rather than an undue
emphasis on data to aid school improvement.
— School self-evaluation, whilst time-consuming,
is seen by schools as a good way of identifying strengths and
weaknesses.
— Data-led interpretation of school performance
must take into account the circumstances of the particular school:
Every Child Matters has brought this more sharply into the foreground.
— In a minority of cases, schools feel
that inspection findings are not specific or can lead to direct
action to address concerns raised by inspectors.
INTRODUCTION
3. The Children, Schools and Families Select
Committee is conducting an inquiry into school accountability.
This submission by the National Foundation for Educational Research
(NFER) aims to provide evidence and recommendations that we hope
will support the committee in its inquiry.
4. NFER is Britain's leading independent educational
research institution. The Research, Evaluation and Information
Department is one of two research departments at the NFER, and
specialises in providing high-quality, independent research and
evaluation in all areas of education and training. The Department
for Research in Assessment and Measurement is the second research
department of the Foundation. It specialises in test development
and research into assessment-related questions. This submission
draws solely on the Foundation's experience with respect to research
and evaluation concerning school accountability issues. Further
information about the NFER is provided in the accompanying NFER
Credentials document.
RESEARCH CONDUCTED
BY THE
NFER
5. The following sections take the lead
from the structure set out in the call for evidence by this Select
Committee. The evidence submitted by NFER covers only the elements
pertaining to Accountability and Inspection.
ACCOUNTABILITY
Is it right in principle that schools should be
held publicly accountable for their performance?
6. Research carried out by NFER suggests
that it is indeed right that schools should be held publicly accountable
for their performance. Indeed, evidence collected from school
respondents across various projects indicates that school staff
themselves would very largely accept this principle.
7. Issues arise, however, when questions are
raised about the extent of this accountability and to whom
schools should be accountable (see below). Most school personnel
would stress strongly that their accountability should be based
on valid and reliable data and upon comprehensive information
about the full circumstances of the school, including its
geographical and socio-economic context. In other words, if the
information used to judge a school's performance is comprehensive
and fair, then a school would accept this judgement. Evidence
supporting these statements is provided below in the findings
from our research on school inspections.
What should be the fundamental purposes of an
accountability system for schools and, in particular?
— what should be the consequences?
8. The accountability system should have
three fundamental purposes: (1) to ensure that there is a degree
of regular external accountability; (2) to ensure that areas for
improvement are identified; and (3) to ensure that schools are
provided with appropriate support and guidance on how to
improve.
Is the current accountability system of inspection
and performance reporting for schools broadly fit for purpose?
9. Given that a majority of schools are
satisfied with the Section 5 (S5) inspection process (see
below) it would seem that this process is broadly fit for purpose.
INSPECTION
10. Since 2006 a research team at NFER
has had a unique opportunity to carry out a detailed and independent
evaluation of the new inspection process. The central aim of the
research has been to assess the extent to which schools feel that
the new inspections have contributed to school improvement. The
evaluation, commissioned by the Office for Standards in Education,
Children's Services and Skills (Ofsted), commenced in February
2006 and will be completed in March 2009. The research methods
included:
— A survey of all schools inspected between
October 2005 and March 2006 (subject to minor exclusions):
1,597 schools responded to this survey—a 67% response
rate.
— Case-study visits to 36 schools where
interviews were conducted with headteachers, senior managers,
governors and parents, usually individually, along with 243 pupils,
usually in small discussion groups.
— Follow-up interviews with headteachers
or senior managers, to provide a longitudinal perspective.
— Statistical modelling of survey responses
and various school background factors, using satisfaction with,
and perceived impact of, inspection as outcomes.
— A desk-top review of key case-study school
documents and test and examination results.
11. The key finding from this evaluation
has been that, overall the vast majority of schools are satisfied
with the inspection process and this process is generally perceived
as contributing to school improvement. Over half of the schools
surveyed (52%) were "very satisfied" with the inspection
and more than a third (36%) were "quite satisfied".
A minority, 10% of survey schools, were "not at all satisfied"
with the S5 inspection.
12. A substantial majority of survey respondents
(83%) thought that the actual monetary costs incurred due to inspections
were minimal and were certainly less than those incurred during
the previous S10 inspection process. Furthermore, three-fifths
of school respondents thought that the new S5 inspection
process was less stressful than the previous system. Further evidence
and findings from this evaluation are presented at the relevant
points below.
Is an independent inspectorate an appropriate
mechanism for holding schools to account?
13. It is difficult to see how schools could
be accountable without some form of independent inspection. The
NFER believes that the withdrawal of an independent element of
inspection or accountability checks would be likely to lead to
a decline in academic standards in some schools.
14. Furthermore, despite the largely negative
impression of inspection experiences, based mainly upon anecdotal
and personal opinions, evidence collected by the NFER clearly
indicates that schools themselves are largely satisfied with,
and understand the need for, independent external inspections.
What is the impact of the inspection process on
school performance, including confidence, creativity and innovation?
15. The impact of the inspection process
on school performance is largely indirect, but nonetheless important.
This is because there are many drivers of school improvement,
including the school's management approach, school improvement
or development planning, the use of data, and the impact of specific
educational initiatives. The main benefit of inspection was perceived
to be that the inspection had been valuable in providing external
confirmation of schools' own self evaluation (86% of survey respondents
took this view). Additionally, other benefits were perceived to
be that inspection boosted staff morale (42%) and, as well as
providing confirmation, it also stimulated improvements (33%).
In these respects inspections do contribute to confidence, creativity
and innovation.
16. Although schools were generally satisfied
with the inspection, just under half made suggestions for changes.
These most often related to perceptions that the (self-evaluation
form) SEF should be simplified, more time should be allowed for
inspectors to observe lessons, inspections should be less data-driven
and that there should be more consistency across inspection teams.
Are inspectors appropriately qualified and trained
to carry out inspections, particularly in the light of the need
to report against Every Child Matters outcomes?
17. With respect to schools inspections
up to the present time, the evidence collected by the NFER indicates
that those who have been inspected have a very positive view of
the professionalism and qualifications of inspectors.
18. Oral feedback from the inspection team was
found to be a vital part of the inspection process. Over half
(60%) of the survey respondents found the oral feedback very useful,
and further 32% fairly useful. The research also revealed a positive,
and statistically significant, relationship between constructive
oral feedback and overall satisfaction with the inspection process.
Is it appropriate for inspection reports to be
placed in the public domain?
19. The majority of survey respondents and
interviewees agreed with the inspection report recommendations
and valued the contribution to school improvement in terms of
the confirmation, prioritisation and clarification of areas for
improvement. Over three-quarters of case-study school interviewees
believed the inspection team's diagnosis, and the written report,
to be fair and accurate. Over half of survey respondents found
the written report helpful for identifying areas for improvement.
The majority of interviewees found the written report to be useful,
helpful and easy to read. A few interviewees believed the report
to be too brief and generalised. However, it was perceived to
be accessible and parents appreciated an independent assessment
of schools.
20. On the whole pupils liked the letter from
the inspector and valued involvement in the process. Recommendations
for improvement were, on the whole, considered to be helpful and
sufficiently specific and follow-up interviews showed that almost
all case-study schools were implementing all, or most, of their
recommendations.
21. The area of greatest perceived impact,
from the S5 recommendations, was in assessment, monitoring
and tracking. Nearly two-thirds of survey respondents and just
over half the case-study interviewees considered that the inspection
had contributed to school improvement. The main way it had contributed
was by confirming, prioritising and clarifying areas for improvement,
rather than by highlighting new areas.
22. Above all, schools recognised that however
reliable their own self evaluation was, it was useful for parents
and the local communities, as well as for their own staff, to
have their judgements confirmed by an external and objective body.
How often should inspections be carried out and
how long and detailed should these inspections be?
23. There needs to be a careful balancing
act here. There appears to be more support from schools for the
five-year cycle with shorter inspections and concise reports,
as opposed to the three-year cycle with longer inspection visits
and more detailed reports. However, school respondents have told
us that they would like more lesson observations and that they
appreciated detailed feedback and discussion.
24. On this basis, the current Section 5 approach,
with a three-year cycle is appreciated by schools—if the
visits required by this approach were to be made more detailed,
then any expansions should be in the observation (which could
mean more teachers or more subjects) and feedback elements.
How much notice, if any, should a school receive
of an upcoming inspection?
25. The shorter notice system is more popular
among schools than the previous Section 10 approach, with
a longer period of notice.
In the context of an inspection, what is the value
of:
26. All of these are clearly important parts
of inspection considerations. School self-assessment and school
self-evaluation have grown dramatically in importance in the last
10 to 15 years. So much so that they now have an official
and substantial place in the inspection process, in the form of
the SEF and all stakeholders now appear to appreciate the usefulness
of this.
27. Although the majority of interviewees
reported that it was time-consuming to complete the SEF, there
was also a strong view that the SEF had been effective as a means
of identifying school strengths and weaknesses. Inspection teams
made good use of the SEF and it provided a focus for the inspection.
Self evaluation generally was regarded as having improved and
the SEF framework had contributed to this improvement.
28. In the first year's use of the SEF,
school staff complained about the time that was required to fill
the SEF in, but they also recognised how important this was and
how useful the process is as a means of "getting to know
your school". Schools welcome the fact that they can now
present their own data and that the inspection team should take
due account of this information. The use of self-evaluation is
a crucially important part of the dialogue that takes place between
the school managers and the inspectors.
29. The NFER has evidence to show that school
self-assessments mostly tend to be in line with external assessments,
suggesting that, in the main, schools are realistic and sensible
in their self-judgements. Filling in the SEF required schools
to make their own self-assessment of their "overall effectiveness"
and a number of other categories such as "achievement and
standards".
30. School survey respondents were asked
to compare their own SEF grades with those awarded by the inspectors:
two-thirds of survey respondents reported no differences between
the S5 and school's SEF grades, indicating a large degree
of consistency between the two sets of judgements.
31. The results of national tests are also
clearly of importance to inspection as they provide an objective
assessment of pupils' progress at certain stages in their school
careers. One of the disadvantages of "raw" test results,
however, is that they do not take account of the effects of prior
attainment, gender and the socio-economic background of pupils.
Value added and contextual value added scores can take account
of these factors, and schools generally welcome their use, though
there have been some concerns about the way school inspectors
have used both "raw" and "value added" data.
…and how much weight should be attached to
these elements in the inspection report?
32. A common complaint from schools (in
a context of mostly positive views about the Section 5 inspections)
in our research was that inspection was "too data driven".
Clearly attainment data needs to feature significantly in inspection
considerations because they indicate a child's academic progress
and the extent to which a school is performing the function of
providing qualifications and an appropriate academic education.
33. However, many school staff would argue firstly
that there has been too much emphasis on the attainment inspection
grade, at the expense of other inspection grades and, secondly,
that in the context of ECM and the Children's Plan, a school's
provision is now about much more than academic qualifications,
indeed it is about the five outcomes and creating responsible
citizens and much more besides.
In an inspection, how should emphasis be balanced
between educational attainment and other aspects of a school's
provision, such as the Every Child Matters outcomes?
34. It would seem that attainment indicators
are universal and should be maintained, but not to the exclusion
of other indicators. The time is now right for other indicators
to receive greater emphasis and for more sophisticated indicators
of pupil progress and school improvement to be developed. Assessment
has to become more sophisticated in the 21st century.
Should inspections be tailored to the current
performance levels of the specific school being inspected and,
if so, to what extent?
35. The tradition in local authority intervention
and support for schools is, of course, intervention in "inverse
proportion to success". With regard to inspections there
seems to be general agreement that low performing schools benefit
from more regular inspections, more detailed inspections and customised
support from the local authority and others. With respect to high
performing schools the picture is not so clear.
OTHER ELEMENTS
OF INSPECTION
36. In the work undertaken by NFER, a number
of other factors have been identified as being important to schools
and should be shared with the Committee. These are described below.
USE OF
DATA
37. Concern was expressed over data interpretation
by some schools inspected throughout the period from October 2005 to
March 2006. This suggests that there may be a need for more evenness
and consistency in terms of the way data is used, particularly
in relation to fully understanding the school context. With the
introduction of RAISE online, there should be opportunities to
ensure that consistencies in data use and interpretation are further
promoted and strengthened.
IMPORTANCE OF
ORAL FEEDBACK
AND DIALOGUE
38. In view of the importance that schools
placed on the oral feedback, Ofsted should maintain and perhaps
even enhance the central position which oral feedback has in the
inspection process. Schools appeared to welcome the opportunity
for, and were responsive to, dialogue, especially as these conversations
provided opportunities to explain the broader school context.
USING POSITIVE
TERMINOLOGY
39. Though it was widely accepted that inspectors
had to work within the agreed standard framework of gradings,
there was some dissatisfaction in schools with the terminology
used to describe the "overall effectiveness" grades,
especially in relation to the "satisfactory" grading.
Whilst it was accepted that parents and other stakeholders should
be provided with a clear, comparable, external, objective assessment
of a school's performance, some school interviewees expressed
a view that the terminology was too negative and too rigid. Several
respondents suggested that inspectors should look for further
ways of providing praise and encouragement for staff: and it might
be possible to do this through the oral feedback and the lesson
observation elements of the inspection process.
REFINING INSPECTION
RECOMMENDATIONS
40. The vast majority of survey respondents
agreed with the inspection recommendations, found them helpful,
and felt that they were sufficiently specific. Only one in 10 schools
found the recommendations "not at all helpful". Where
this latter view was present, the reasons were usually along the
lines of: (1) the recommendations were not specific enough; or
(2) the recommendations lacked practical guidance. It might be
worth bearing these two points in mind when any further advice
on drafting recommendations is given to inspectors.
What is the impact on schools of league tables
published by the press?
41. The importance of terminology and function
is worth re-stating here. The Department for Children, Schools
and Families (DCSF) produces school and college achievement and
attainment tables, free of any attempts to rank schools or colleges.
The transformation of these data into league tables published
by the press tend to cause more harm than good. This is particularly
true of tables based on raw results. Such information can be very
damaging to a school which in fact is doing well and adding value
to pupils' experience by supporting them to achieve better than
expected test or examination results.
CONCLUSIONS
42. A range of research and evaluation exercises
have been carried out by NFER focusing upon the school accountability
system in England. The predominant findings are about the positive
impact which having an independently guided inspection system
which has a clear focus on school improvement is welcomed by schools.
There are undoubtedly challenges in squaring a simple and low-burden
system with the increasingly complex range of activities which
schools are expected to be accountable for (emphasised by the
scope of Every Child Matters). The research which NFER has undertaken
is broadly supportive of the changes from Section 10 to Section
5 inspections.
February 2009
SOURCESArnold, R
(1998). Target Setting: School and LEA in Partnership.
Slough: NFER, EMIE.
Arnold, R (2000). Education Strategic Plans in
Wales: "Building Excellent Schools Together" (EMIE
Report No.58). Slough: NFER, EMIE.
Arnold, R (2002). The School Improvers: Survey
of LEA Advisory, Inspection, and School Improvement Services
(EMIE Report 65). Slough: NFER, EMIE.
Arnold, R (2002). Good Practice in the Midlands:
a Report Prepared for the Society of Midlands Education Officers.
Slough: NFER, EMIE.
Arnold, R (2004). Transforming Secondary Education:
the Beacon Council Scheme. Round 4 (EMIE Report No.78). Slough:
EMIE at NFER.
Atkinson, M, Gulliver, C, Lamont, E and White, R
(2006). The New School Funding Arrangements 2006-07: the Local
Authority Perspective (LGA Research Report 2/06). Slough:
NFER.
Atkinson, M, Lamont, E, Gulliver, C, White, R and
Kinder, K (2005). School Funding: a Review of Existing Models
in European and OECD Countries (LGA Research Report 3/05).
Slough: NFER.
Atkinson, M, Springate, I, Johnson, F and Halsey,
K (2007). Inter-School Collaboration: a Literature Review.
Slough: NFER [online]. Available: http://www.nfer.ac.uk/publications/pdfs/downloadable/ATPreport.pdf
[27 February, 2009].
Benton, T (2003). "Measuring schools' value-added
performance: the importance of picking the right outcome."
Paper presented at the British Educational Research Association
Annual Conference, Heriot-Watt University, Edinburgh, 12 September
[online]. Available: http://www.nfer.ac.uk/publications/other-publications/conference-papers/measuring-schools-value-added-performance-the-importance-of-picking-the-right-outcome.cfm
[27 February, 2009].
Benton, T, Hutchison, D, Schagen, I and Scott, E
(2003). Study of the Performance of Maintained Secondary Schools
in England: Report for the National Audit Office. Slough:
NFER. [online] Available: http://www.nfer.ac.uk/publications/other-publications/downloadable-reports/pdf_docs/NAO.pdf
[27 February, 2009].
Chamberlain, T, Lewis, K, Teeman, D and Kendall,
L (2006). Schools' Concerns and their Implications for Local
Authorities: Annual Survey of Trends in Education 2006 (LGA
Research Report 5/06). Slough: NFER.
Davies, D and Rudd, P (2001). Evaluating School
Self-evaluation (LGA Research Report 21). Slough: NFER.
Derrington, C (2000). The LEA Contribution to
School Improvement: a Role Worth Fighting For (LGA Research
Report 9). Slough: NFER.
Easton, C, Knight, S and Kendall, L (2005). Annual
Survey of Trends in Primary Education: Survey of 2004 (LGA
Research Report 8/05). Slough: NFER.
Easton, C, Knight, S and Kendall, L (2005). Annual
Survey of Trends in Primary Education. Digest No. 17: Survey of
2004. Slough: NFER.
Fisher, D (1999). Partnership for Progress: Support
for Underachieving Schools. Slough: NFER, EMIE.
Fowler, J and Waterman, C (2005). Digest of the
Education Act 2005. Slough: NFER.
Hackworth, C and Barson, A (2007). School Improvement:
Beacon Scheme Round 8 (EMIE Report 89). Slough: EMIE at NFER.
Halsey, K, Judkins, M, Atkinson, M and Rudd, P (2005).
New Relationship with Schools: Evaluation of Trial Local Authorities
and Schools (Research Summary). [online]. Available: http://www.nfer.ac.uk/research-areas/pims-data/summaries/new-relationship-with-schools-evaluation-of-trial-leas-and-schools.cfm
[27 February, 2009].
Harland, J, Kinder, K, Lord, P, Stott, A, Schagen,
I, Haynes, J, Cusworth, L, White, R and Paola, R (2000). Arts
Education in Secondary Schools: Effects and Effectiveness.
Slough: NFER.
Hendy, J (1998). Fourth Survey of LEA Advisory
and Inspection Services. Slough: NFER, EMIE.
Knight, S, Taggart, G and Kendall, L (2006). Annual
Survey of Trends in Secondary Education: Report for 2005 (LGA
Research Report 1/06). Slough: NFER.
Mann, P (1995). Third Survey of LEA Advisory and
Inspection Services. Slough: NFER, EMIE.
McCrone, T, Rudd, P, Blenkinsop, S and Wade, P (2006).
Impact of Section 5 Inspections: Maintained Schools in
England [online]. Available: http://www.ofsted.gov.uk/Ofsted-home/Publications-and-research/Browse-all-by/Documents-by-type/Thematic-reports/Impact-of-Section-5-inspections-maintained-schools-in-England
[27 February, 2009].
McCrone, T, Rudd, P, Blenkinsop, S, Wade, P, Rutt,
S and Yeshanew, T (2007). Evaluation of the Impact of Section
5 Inspections. Slough: NFER.
National Foundation for Educational Research (2002).
Evaluation of Excellence in Cities Policy: Updated June 2002.
Slough: NFER.
Pritchard, K (2005). National Standards for School
Improvement Professionals: the Legacy and the Future (EMIE
Report No.83). Slough: EMIE at NFER.
Rudd, P, Davies, D, Jamison, J, Rickinson, M and
Johnson, R (2001). Beacon Schools: Further External Evaluation
of the Initiative. Final Report. Slough: NFER.
Rudd, P and Davies, D (2002). A Revolution in
the Use of Data? The LEA Role in Data Collection, Analysis and
Use and Its Impact on Pupil Performance (LGA Research Report
29). Slough: NFER.
Rudd, P and Dartnall, L (2003). 50 Websites
for School Improvement. Slough: NFER.
Rudd, P (2007). School Inspections: International
Literature Review [online]. Available: http://www.nfer.ac.uk/research-areas/pims-data/outlines/school-inspections---international-literature-review.cfm
[27 February, 2009].
Saunders, L and Stradling, B with Rudd, P (2000).
Raising Attainment in Secondary Schools: a Handbook for School
Self-Evaluation (New Updated Edition). Slough: NFER.
Saunders, L (1999). "Value Added" Measurement
of School Effectiveness: a Critical Review. Slough: NFER.
Saunders, L (1998). "Value Added" Measurement
of School Effectiveness: an Overview. Slough: NFER.
Savory, C, Walker, M and Rudd, P (2005). National
and Local Government Raising Standards Across Schools: a Literature
Review (LGA Research Summary 5/05). Slough: NFER [online].
Available: http://www.nfer.ac.uk/research-areas/pims-data/summaries/shared-priorities-raising-standards-across-schools.cfm
[27 February, 2009].
Scanlon, M (1999). The Impact of OFSTED Inspections.
Slough: NFER.
Schagen, I and Benton, T (2003). "The relationships
between school and pupil background factors and progress from
Key Stage 1 to Key Stage 2." Paper presented at the
British Educational Research Association Annual Conference, Heriot-Watt
University, Edinburgh, 13 September [online]. http://www.nfer.ac.uk/publications/other-publications/conference-papers/the-relationships-between-school-and-pupil-background-factors-and-progress-from-key-stage-1-to-key-stage-2.cfm
[27 February, 2009].
Schagen, I and Weston, P with Hewitt, D and Sims,
D (1997). Hitting the Targets: Evaluation of Progress Towards
Foundation Targets 1 and 3 by English Region. Slough:
NFER.
Schagen, S, Davies, D, Rudd, P and Schagen, I (2002).
The Impact of Specialist and Faith Schools on Performance
(LGA Research Report 28). Slough: NFER [online]. Available: http://www.nfer.ac.uk/publications/pdfs/downloadable/specialistandfaithschools.pdf
[27 February, 2009].
Sharp, C, Handscomb, G, Eames, A, Sanders, D and
Tomlinson, K (2006). Advising Research-Engaged Schools: a Role
for Local Authorities. Slough: NFER.
Sturman, L, Lewis, K, Morrison, J, Scott, E, Smith,
P, Styles, B, Taggart, G and Woodthorpe, A (2005). General
Teaching Council Survey of Teachers 2005: Final Report. Slough:
NFER [online]. Available: http://www.gtce.org.uk/research/tsurvey/2005survey
[27 February, 2009].
Wilkin, A, Atkinson, M, Halsey, K, Johnson, A, White,
R and Kinder, K (2005). New Roles for Local Authorities in
Education: Opportunities and Challenges (LGA Research Report
9/05). Slough: NFER.
Wilson, R, Sharp, C, Hemsley-Brown, J and Easton,
C (2002). Using Research for School Improvement: the LEA's
Role (Interim Summary). Slough: NFER.
Wray, M (2002). Local Authority Reactions to the
OFSTED Inspection Process (LGA Research Report 32) [online].
Available: http://www.nfer.ac.uk/publications/other-publications/downloadable-archive-reports/local-authority-reactions-to-the-ofsted-inspection-process.cfm
[27 February, 2009]
| 
