Memorandum submitted by Richard Cooper


1. Introduction

1.1 I write as a professional with over 25 years practice in managing for quality. I am qualified in Quality Management and statistics and have extensive experience of consulting in industry. My concern is that the government's policy-making and target-setting in education (as in other areas) is not founded on either sound evidence or robust theories of how people in organisations work in the real world. Tony Blair said of Iraqi WMD "I only know what I believe" - it seems that facts have no value for this government whose policy is driven by dogma.

2. Executive Summary

2.1 Government policy seems to be based on theories that are unsupported by evidence. Arbitrary targets are set for attainment in schools, seemingly based on what the government would like to happen (85% of children attaining a certain level), rather than on the capability of the present system of education. Targets for improvement also seem to be based on wishes rather than planned improvements that have been tested and found effective.

2.2 There is no evidence that arbitrary targets bring about improvements, and extensive evidence to suggest they do not. Alfie Kohn (The Schools Our Children Deserve, Houghton Mifflin, 1999) is one author who has conducted research in this area, and the famous statistician W. Edwards Deming (Out Of The Crisis, Cambridge, 1982) wrote and taught extensively about his experience of the futility of setting targets without providing the means to meet them and that where there is fear there are also phony figures.

2.3 There is no system that a manager can design that an intelligent worker cannot subvert. If a manager says "the result must be this number" then the worker will find a way to make it so, especially if a bonus or promotion depends on it. Any measurement can be distorted, even when (as in manufacturing) the measurement instruments are precise, tightly calibrated and highly accurate. In education, testing instruments are coarse and inaccurate, and each measurement is subject to a teacher's personal interpretation. When teachers' and schools' reputations are on the line, how can any test result be expected to mean anything?

3. Recommendations for Action

3.1 The way to improve the performance of an organisation is to change the system. This means understanding the capability of the system, planning and testing changes to improve it and only then instituting them. Of course outcomes must be checked ('summative' assessment) but they must be subordinated to process measurements ('formative' assessment) that lead to improvements. Setting targets without the means to achieve them is fruitless and harmful. As Simon Caulkin pointed out, "targets actively corrupt" (The Observer, 26 May 2007). Instead, the system for education must be studied and improved. I implore the committee to advise the government to do this.











General Issues



Why do we have a centrally run system of testing and assessment?

Unknown. Possibly because the DfES wants to retain control over testing. In a more democratic society, local authorities would be able to decide for themselves what tests and assessments should be used. In a competitive environment (which is a government mantra), there would be competing testing and assessment bodies among whom local authorities could choose.


What other systems of assessment are in place both internationally and across the UK?

Unknown, but there is talk of baccalaureates in Europe and there are the miserable and meaningless SATs in the US.


Does a focus on national testing and assessment reduce the scope for creativity in the curriculum?

It depends on what is meant by creativity. If creativity includes cheating then a focus on testing encourages it by encouraging schools to find ways to cheat the system by preparing children for the test.


Who is the QCA accountable to and is this accountability effective?

Unknown. How would effectiveness be measured? If the members of the QCA could be fined or fired for their incompetence then accountability could be effective.


What role should exam boards have in testing and assessment?

Exam boards should set exams, and local authorities should be able to choose which exam board their schools used.


National Key Stage Tests - The current situation



How effective are the current Key Stage tests?

It depends on what effect is intended. The current intention seems to be that all children are subjected to standardised tests based on standard models of what must be learned. In that case, these tests are effective in setting the criteria, but there is no evidence that either the criteria are useful or that the tests are consistent, reliable, accurate or precise enough to be of any use.


Do they adequately reflect levels of performance of children and schools, and changes in performance over time?

Unknown. If the 'true' level of performance could be known, then it could be judged whether the tests accurately reflect this. But there can be no 'true' level of any measurement - all measurements are subject to measurement error due to variation in the instrument and its operator.


Do they provide assessment for learning (enabling teachers to concentrate on areas of a pupil's performance that needs improvement)?

No. Obviously, because they are 'summative'. Assessment for learning is called 'formative'.


Does testing help to improve levels of attainment?

No. Testing cannot improve attainment. Only teaching can do that. You cannot improve a child's attainment by testing it, any more than you can fatten a pig by chasing it round its pen trying to weigh it.


Are they effective in holding schools accountable for their performance?

In part, to the extent that a school's aim is to have pupils pass the test. However, a school's aim should be education, not exam results.


How effective are performance measures such as value-added scores for schools?

No more than any other measure. Just like any other measurement they are subject to instrument and operator error. Test administrators aiming to demonstrate high 'value-add' will underestimate when administering the test to incoming pupils and overestimate outgoing pupils.


Are league tables based on test results an accurate reflection of how well schools are performing?

As above, only to the extent that a school's objective is to have pupils pass a test. Tables inevitably lead to cheating when funding is at stake - witness the state of professional football; and education is not a game.


To what extent is there 'teaching to the test'?

Unknown. However, in a climate of fear induced by public flogging of 'failing' schools, it is likely that pupils and test administrators will all do everything they can, including lying and cheating, to avoid being flogged.


How much of a factor is 'hot-housing' in the fall-off in pupil performance from Year 6 to Year 7?

Unknown. This question could only be answered by conducting a test with and without 'hot-housing'. All else is conjecture.


Does the importance given to test results mean that teaching generally is narrowly focused?

It seems likely that if pupils and teachers are given to understand that their success depends on passing the test, they will do all they can to pass, including avoiding spending time on anything unconnected to the test.


What role does assessment by teachers have in teaching and learning?

Formative assessment is crucial to teaching. However, learning is accomplished by pupils, not teachers, so it unlikely that learning can be affected by an assessment by a teacher.


National Key Stage Tests - The future



Should the system of national tests be changed?

Yes, since the current testing system is part of an education system that results in children continuing to leave school with skills that employers find inadequate and an education that leads them to vote for someone who says "I only know what I believe".


If so, should the tests be modified or abolished?

Abolished. Local testing will better meet the needs of local communities because schools and the communities they serve will have a larger voice in its design and operation.


The Secretary of State has suggested that there should be a move to more personalised assessment to measure how a pupil's level of attainment has improved over time. Pilot areas to test proposals have just been announced. Would the introduction of this kind of assessment make it possible to make an overall judgment on a school's performance?

The Secretary of State must be an ignorant fool with no training or understanding of education or statistics. The Secretary of State should ask professionals trained in education and statistical measurement what kind of testing is required. Formative assessment by the teacher should of course be personalised; summative assessment by examination obviously cannot be.


Would it be possible to make meaningful comparisons between different schools?

Unknown. It is unclear why this should be needed. Parents and children do not have any meaningful choice of school (any more than they do over which rail company they use or which NHS they use). Knowing that a school has less good exam results than another merely affects demand - it cannot affect supply, since schools have limited teachers and teaching facilities. The only possible purpose is choosing candidates for public flogging. Resources would be better spent on improving teaching, not improving testing.


What effect would testing at different times have on pupils and schools? Would it create pressure on schools to push pupils to take tests earlier?

Impossible to predict. It seems likely that administrative chaos would result if different schools choose to run the same test at different times, with leakage of test papers. If each test has a different paper, then resources will be wasted designing different tests for different dates.


If Key Stage tests remain, what should they be seeking to measure?

They should seek to measure the extent of pupils' understanding of and skills in 'Key' subjects.


If, for example, performance at Level 4 is the average level of attainment for an eleven year old, what proportion of children is it reasonable to expect to achieve at or above that level?

Unknown. It depends on the variation and distribution of values in the sample. If the data formed a Gaussian distribution with a standard deviation of one 'Level', then 84% would be at Level 4 or above. Of course, no real data can ever be Gaussian and the measurement tools are too coarse to know what the standard deviation might be.


How are the different levels of performance expected at each age decided on? Is there broad agreement that the levels are appropriate and meaningful?

Unknown. Perhaps they are decided by some educationalists writing criteria based on experience. However, the data base is unclear and so agreement is irrelevant.


Testing and assessment at 16 and after



Is the testing and assessment in "summative" tests (for example, GCSE, AS, A2) fit for purpose?

It depends on the purpose. If the aim is to frighten and annoy pupils and teachers, then possibly they are. If the aim is to provide qualifications that pupils can be proud of, then obviously not, since GCSEs are handed out like napkins at MacDonalds. If the aim is to give an indication to universities and employers of what a pupil is capable, then clearly not, because universities complain that there is insufficient granularity in the results (everyone gets an A or A-star) and employers complain that people who have passed English and Maths are functionally illiterate and innumerate.


Are the changes to GCSE coursework due to come into effect in 2009 reasonable? What alternative forms of assessment might be used?

They are irrelevant. With widespread use of the internet, the use of coursework for a qualification has become a test in concealing plagiarism, not a test of understanding.


What are the benefits of exams and coursework? How should they work together? What should the balance between them be?

Exams provide an answer to the question "can the pupil answer the questions put?" and coursework answers the question "can the pupil create something that contains the required information?". It would be better if exams asked the question "does the pupil understand the subject?" and coursework was seen merely as a means of practising using the tools and exploring the knowledge that underlies the subject.


Will the ways in which the new 14-19 diplomas are to be assessed impact on other qualifications, such as GCSE?

Unknown. Not clear how the diplomas are to be assessed. If by exam, then they will tend to subvert or replace GCSEs. If by interview, then they will add to it.


Is holding formal summative tests at ages 16, 17 and 18 imposing too great a burden on students? If so, what changes should be made?

It is unclear why summative tests should be needed each year, when it is formative tests that inform teaching (see next answer).


To what extent is frequent, modular assessment altering both the scope of teaching and the style of teaching?

The need for constant assessment is inherent in proper teaching. Formative assessment should be a continual process. Summative assessment should be conducted rarely, and only when a pupil reaches a milestone such as a move from primary to secondary or secondary to tertiary education or employment or unemployment.


How does the national assessment system interact with university entrance? What does it mean for a national system of testing and assessment that universities are setting entrance tests as individual institutions?

They are disconnected, as demonstrated by universities setting entrance tests. This alone should be sufficient to demonstrate even to someone as ignorant as the Secretary of State that the system is failing and a system developed and operated by experts, preferably at a local level, is needed.


June 2007