House of Commons - Education and Skills

Select Committee on Education and Skills Minutes of Evidence

Memorandum submitted by Professors Pam Sammons, School of Education, University of Nottingham[7] and Peter Matthews, Institute of Education, University of London[8] (OFS 7)

This memorandum from the joint authors of the report, Improvement through Inspection,[9] to which the Committee referred in its last report on the work of Ofsted,[10] offers comments on:

— the degree of independence of the Ofsted/Institute of Education report;

— frames of reference used in evaluating the impact of Ofsted's work;

— issues in gauging Ofsted's value for money; and

— two matters raised in the Committee's Sixth report on the Work of Ofsted

PROBITY OF THE OFSTED EVALUATION REPORT

1. Although the Committee has not questioned it directly, we feel that members should be apprised of the degree of independence of the evaluation report, which has been misconstrued in some sections of the media. Ofsted's intention to evaluate its own effectiveness had been welcomed by Committee previously. The findings of the evaluation stem from the analysis and interpretation of a wide range of evidence that included: published research; evidence furnished by Ofsted; the views of providers who have experienced inspections, and additional research and surveys. From the outset, the study planned to involve an external assessor so as to provide a critical independent view. This work was commissioned through a competitive tender, won by the Institute of Education which nominated a professorial researcher to engage with the evaluation from January 2004. The study thus gained from an important external perspective, particularly in the analysis and weight given to interpreting the various sources of evidence. The report should not therefore be seen simply as an internal self-assessment.

2. Following the practice of previous early studies of the quality of the school inspection system which also used external assessors,[11] the assessor was invited to decide—on the strength of the evidence and the use made of it—whether to write a separate commentary in the report or to act as co-author. In the event, case, the assessor was sufficiently satisfied with the approach taken to the study to participate as co-author.

3. It should also be recorded that the publication was not, at any stage subject to any editorial direction or control by Her Majesty's Chief Inspector (HMCI), the Board of Ofsted or any of its staff, nor have they sought to do so. Having commissioned the evaluation and agreed the main areas to be covered, it has been treated by Ofsted as an independent study that was given a free hand in the way it was designed and conducted and open access to any records, data and correspondence. All the main Division contributed evidence to the evaluators. The draft report was subjected to external critical scrutiny as detailed in the publication. The responsibility for the contents of the report, however, rests solely and entirely with the authors, who are named on the report for this reason, as an exception to Ofsted's customary practice.

FRAMES OF REFERENCE

4. The evaluation set out principally to examine Ofsted's impact, both on individual institutions and on the education system, and to explore evidence for "Improvement through Inspection". As is the case in most areas of social and educational enquiry, the evidence of cause and effect is largely associative, for Ofsted is seldom the direct agent of improvement. The report identifies inspection as acting in conjunction with other measures to have the net effect of promoting improvement: the so-called "cocktail effect". The findings are reinforced by trends in educational quality and standards and evidence from those who have been inspected, who report on benefits that significantly outweigh disadvantages.

5. The statutory remit for Ofsted, our second frame of reference, gives little emphasis to the aspiration of improvement through inspection except in the case of providers whose performance causes concern. Legislation sets out duties that include: implementing particular inspection arrangements, public reporting, the Annual report to Parliament, and provision of advice to the Secretary of State. Our evaluation covers the discharge of these duties as well as HMCI's specific terms of reference, modified from time to time by remit letters. Ofsted fulfils expectations when measured against what Parliament expects of it. Its performance and practices, which include many improvements to inspection processes and innovations such as the issuing of detailed performance analyses annually to providers and the development and promotion of self-evaluation, have resulted in benefits that are well supported by evidence.

6. Thirdly, the evaluation took considerable account of the principles for inspectorates, published by the Prime Minister's Office for Public Service Reform (OPSR) in summer 2003.[12] Indeed, it decided to award performance grades for Ofsted's work against each of those principles, based on the evidence available. Grading is a feature of all Ofsted's inspections and it was considered that the evaluation should adopt the same principle.

7. Through concentrating on the quality and effectiveness of Ofsted's inspection regimes and advisory functions, the evaluation—which was time limited—did not attempt to provide coverage of Ofsted's internal working, staff surveys, leadership, organisation and management, or the commercial aspects of school inspections through which statutory requirements are applied.

VALUE FOR MONEY

8. In its recent report, the Committee expressed its interest in a more searching examination of value for money. Value for money was not intended as a main focus of the Ofsted evaluation, but we felt it important to explore this aspect as far as we could. The evaluation of value for money presents a number of difficult methodological problems. Value for money is not necessarily susceptible to precise quantification since value is essentially more of a judgement than a measure. There are few benchmarks and the nature, context, conduct and outcomes of different educational inspection systems introduce a complex set of variables. We share the Committee's view that the inspection costs can be quantified in relation to the large public (and in some sectors private) investment made in the different sectors inspected, recognising that costs are but part of the equation. We presented data to put inspection costs in context in relation to the inspection of different sectors, for example, by giving the expenditure per head per pupil. We also identified some pointers for further explorations of this area which we feel are worth following up.

9. In the four point scale used in the evaluation report, the judgement that value for money was good (second point down the scale) took account of the good or very good efficiency and effectiveness of Ofsted in most of its spheres of operation, balanced against economy (especially costs) which, at worst, could be viewed as fair. The evidence is clear that Ofsted operates effectively—and in many areas very efficiently—within its means; nonetheless, areas for improvement are indicated, for example, where greater synergy between HMI thematic and subject surveys and institutional inspections offers an opportunity for better use of inspection resources. The different sections of the report analyses a wealth of evidence relevant to the judgement of Ofsted's effectiveness, and also deals with the efficient delivery of inspection programmes and fulfilment of qualitative and quantitative targets.

10. The Committee's most recent report quoted comparative data on expenditure. The aggregation of all Ofsted's inspection work into a simple measure[13] is of limited usefulness, since Ofsted to date has functioned more as a range of separate inspectorates than a single entity. Inspection costs as a proportion of total public expenditure vary considerably across the different sectors inspected by Ofsted. Earlier Treasury data furnished ratios of 0.4% for school inspections and 0.7% for early years. In the case of early years, the ratio would be much lower if account was taken of the considerable amount of private money that parents pay for nursery education and childcare over and beyond publicly funded Sure Start, nursery places and the childcare element of the tax credit system.

11. This type of quantitative analysis is of limited value without consideration of the nature and purpose of inspections. Some regimes focus more on compliance, others on quality. We made little progress on international benchmarking of inspection costs in the time available because of the differences in the systems explored and the need to arrive, in agreement with the other inspectorates, at indicators that overcome the disadvantages described above.

12. The report also makes it clear that inspection is differentially effective. In schools for example, it is inferred that value for money is greatest in stimulating rapid change in weaker schools but may be somewhat lower in relation to the inspection of schools that make little use of the inspection findings to improve their performance. Greater investment in following up the inspection of such schools, in which leadership and management commonly lack impact, would add to costs but probably increase the value of the inspection. Ofsted's proposals for more frequent inspection and the issue of notices to improve may achieve the same end.

13. Guidance to departments from the OPSR and HM Treasury interprets the need for inspectorates to "have regard for value for money" as "delivering benefits commensurate with cost, and has issued guidance on reviewing the cost-effectiveness of inspection activity through a bottom-up approach to cost benefit analysis.)[14] This approach first considers "the costs and benefits of inspection on a sample of service units, before extrapolating the results for a particular sector and reviewing the results at an aggregate level". Our evaluation tested this approach in a sample of schools and found that that the amount of preparatory work required which is not intrinsically useful to them has been greatly reduced, particularly since the self evaluation form and procedures were introduced for all inspections in September 2003.

14. The new inspection arrangements that are currently being piloted, together with the substantial restructuring of Ofsted, are likely to lead to economies but will merit a fresh examination of impact. The report points to the need for caution over reliance on self-assessment given the tendency of some providers to rate themselves more positively than external evaluators. There is a case for researching international and cross-inspectorate comparisons and refining the methodology for examining organisational value for money (including the use of contracted inspectors) over the next 12 months in preparation for more rigorous analysis when the new inspection systems and regional structures are in place. Improvement through Inspection provides many indicators against which to assess the future improvement of Ofsted, particularly in terms of the primary beneficiaries of inspection.

OTHER MATTERS RAISED BY THE COMMITTEE

15. We wish to clarify two matters that have been highlighted by the Committee. Firstly, we did not find any pattern of evidence which can add weight to the claim[15] that inspection leads to a systematic spiral of decline for some schools. Rather, the reverse is a much more typical pattern, in arresting or reversing weaknesses that may have been indigenous. In most schools exhibiting serious weaknesses, as well as those requiring special measures, there is much evidence and data to show that inspection provides a significant catalyst for improvement and generates additional support and resources as well as pressure to improve. Changes in inspection grades provide important indicators of improvement but these are supported by data on national assessment results and examination trends. The extent of improvement in weaker schools has been of particular benefit to more disadvantaged groups of pupils who are over-represented in schools judged to offer poor quality education. There is evidence that high proportions of special measures schools improve and subsequently remain out of any of the categories of schools causing concern. We estimate that more than one million pupils have passed through these schools and benefited from their improvement.

16. Secondly, in the light of the Newcastle University findings—quoted by the Committee—we commissioned an analysis, reported in Improvement through Inspection, to re-examine the effect of inspections on the results of secondary schools at the time of their inspections and up to two years afterwards. The analysis of results from 1995-2003, indicate that there is little difference between the results of schools inspected and those of all schools except in two years where the cohort of schools inspected contained a higher proportion of low attaining schools than the national sample. The evidence suggests that inspections in recent years have not had a significant effect, deleterious or otherwise, on the results of secondary schools around the time of their inspections. [16]Overall, results have risen consistently during this period.

October 2004