1.1 I
write as a professional with over 25 years practice in managing for quality. I
am qualified in Quality Management and statistics and have extensive experience
of consulting in industry. My concern is that the government's policy-making
and target-setting in education (as in other areas) is not founded on either
sound evidence or robust theories of how people in organisations work in the
real world. Tony Blair said of Iraqi WMD "I only know what I believe" - it
seems that facts have no value for this government whose policy is driven by
dogma.
2.1 Government
policy seems to be based on theories that are unsupported by evidence.
Arbitrary targets are set for attainment in schools, seemingly based on what
the government would like to happen (85% of children attaining a certain
level), rather than on the capability of the present system of education.
Targets for improvement also seem to be based on wishes rather than planned
improvements that have been tested and found effective.
2.2 There is no
evidence that arbitrary targets bring about improvements, and extensive
evidence to suggest they do not. Alfie Kohn (The Schools Our Children Deserve,
Houghton Mifflin, 1999) is one author who has conducted research in this area,
and the famous statistician W. Edwards Deming (Out Of The Crisis, Cambridge,
1982) wrote and taught extensively about his experience of the futility of
setting targets without providing the means to meet them and that where there
is fear there are also phony figures.
2.3 There is no
system that a manager can design that an intelligent worker cannot subvert. If
a manager says "the result must be this number" then the worker will find a way
to make it so, especially if a bonus or promotion depends on it. Any
measurement can be distorted, even when (as in manufacturing) the measurement
instruments are precise, tightly calibrated and highly accurate. In education,
testing instruments are coarse and inaccurate, and each measurement is subject
to a teacher's personal interpretation. When teachers' and schools' reputations
are on the line, how can any test result be expected to mean anything?
3.1 The
way to improve the performance of an organisation is to change the system. This
means understanding the capability of the system, planning and testing changes
to improve it and only then instituting them. Of course outcomes must be
checked ('summative' assessment) but they must be subordinated to process
measurements ('formative' assessment) that lead to improvements. Setting
targets without the means to achieve them is fruitless and harmful. As Simon
Caulkin pointed out, "targets actively corrupt" (The Observer, 26 May 2007).
Instead, the system for education must be studied and improved. I implore the
committee to advise the government to do this.
|
No
|
Questions
|
Answers
|
|
1
|
General Issues
|
|
|
1.1
|
Why do we have a centrally run system of testing and assessment?
|
Unknown. Possibly because the DfES wants to retain
control over testing. In a more democratic society, local authorities would
be able to decide for themselves what tests and assessments should be used.
In a competitive environment (which is a government mantra), there would be
competing testing and assessment bodies among whom local authorities could
choose.
|
|
1.2
|
What other systems of assessment are in place both internationally and
across the UK?
|
Unknown, but there is talk of baccalaureates in Europe
and there are the miserable and meaningless SATs in the US.
|
|
1.3
|
Does a focus on national testing and assessment reduce the scope for
creativity in the curriculum?
|
It depends on what is meant by creativity. If
creativity includes cheating then a focus on testing encourages it by
encouraging schools to find ways to cheat the system by preparing children
for the test.
|
|
1.4
|
Who is the QCA accountable to and is this accountability effective?
|
Unknown. How would effectiveness be measured? If the
members of the QCA could be fined or fired for their incompetence then
accountability could be effective.
|
|
1.5
|
What role should exam boards have in testing and assessment?
|
Exam boards should set exams, and local authorities
should be able to choose which exam board their schools used.
|
|
2
|
National Key Stage Tests - The current situation
|
|
|
2.1
|
How effective are the current Key Stage tests?
|
It depends on what effect is intended. The current
intention seems to be that all children are subjected to standardised tests
based on standard models of what must be learned. In that case, these tests
are effective in setting the criteria, but there is no evidence that either
the criteria are useful or that the tests are consistent, reliable, accurate
or precise enough to be of any use.
|
|
2.2
|
Do they adequately reflect levels of performance of children and
schools, and changes in performance over time?
|
Unknown. If the 'true' level of performance could be
known, then it could be judged whether the tests accurately reflect this. But
there can be no 'true' level of any measurement - all measurements are
subject to measurement error due to variation in the instrument and its
operator.
|
|
2.3
|
Do they provide assessment for learning (enabling teachers to
concentrate on areas of a pupil's performance that needs improvement)?
|
No. Obviously, because they are 'summative'.
Assessment for learning is called 'formative'.
|
|
2.4
|
Does testing help to improve levels of attainment?
|
No. Testing cannot improve attainment. Only teaching
can do that. You cannot improve a child's attainment by testing it, any more
than you can fatten a pig by chasing it round its pen trying to weigh it.
|
|
2.5
|
Are they effective in holding schools accountable for their performance?
|
In part, to the extent that a school's aim is to have
pupils pass the test. However, a school's aim should be education, not exam
results.
|
|
2.6
|
How effective are performance measures such as value-added scores for
schools?
|
No more than any other measure. Just like any other
measurement they are subject to instrument and operator error. Test
administrators aiming to demonstrate high 'value-add' will underestimate when
administering the test to incoming pupils and overestimate outgoing pupils.
|
|
2.7
|
Are league tables based on test results an accurate reflection of how
well schools are performing?
|
As above, only to the extent that a school's objective
is to have pupils pass a test. Tables inevitably lead to cheating when
funding is at stake - witness the state of professional football; and
education is not a game.
|
|
2.8
|
To what extent is there 'teaching to the test'?
|
Unknown. However, in a climate of fear induced by
public flogging of 'failing' schools, it is likely that pupils and test
administrators will all do everything they can, including lying and cheating,
to avoid being flogged.
|
|
2.9
|
How much of a factor is 'hot-housing' in the fall-off in pupil
performance from Year 6 to Year 7?
|
Unknown. This question could only be answered by
conducting a test with and without 'hot-housing'. All else is conjecture.
|
|
2.10
|
Does the importance given to test results mean that teaching generally
is narrowly focused?
|
It seems likely that if pupils and teachers are given
to understand that their success depends on passing the test, they will do
all they can to pass, including avoiding spending time on anything
unconnected to the test.
|
|
2.11
|
What role does assessment by teachers have in teaching and learning?
|
Formative assessment is crucial to teaching. However,
learning is accomplished by pupils, not teachers, so it unlikely that
learning can be affected by an assessment by a teacher.
|
|
3
|
National Key Stage Tests - The future
|
|
|
3.1
|
Should the system of national tests be changed?
|
Yes, since the current testing system is part of an
education system that results in children continuing to leave school with
skills that employers find inadequate and an education that leads them to
vote for someone who says "I only know what I believe".
|
|
3.2
|
If so, should the tests be modified or abolished?
|
Abolished. Local testing will better meet the needs of
local communities because schools and the communities they serve will have a
larger voice in its design and operation.
|
|
3.3
|
The Secretary of State has suggested that there should be a move to more
personalised assessment to measure how a pupil's level of attainment has
improved over time. Pilot areas to test proposals have just been announced.
Would the introduction of this kind of assessment make it possible to make an
overall judgment on a school's performance?
|
The Secretary of State must be an ignorant fool with
no training or understanding of education or statistics. The Secretary of
State should ask professionals trained in education and statistical
measurement what kind of testing is required. Formative assessment by the
teacher should of course be personalised; summative assessment by examination
obviously cannot be.
|
|
3.4
|
Would it be possible to make meaningful comparisons between different
schools?
|
Unknown. It is unclear why this should be needed.
Parents and children do not have any meaningful choice of school (any more
than they do over which rail company they use or which NHS they use). Knowing
that a school has less good exam results than another merely affects demand -
it cannot affect supply, since schools have limited teachers and teaching
facilities. The only possible purpose is choosing candidates for public
flogging. Resources would be better spent on improving teaching, not
improving testing.
|
|
3.5
|
What effect would testing at different times have on pupils and schools?
Would it create pressure on schools to push pupils to take tests earlier?
|
Impossible to predict. It seems likely that
administrative chaos would result if different schools choose to run the same
test at different times, with leakage of test papers. If each test has a
different paper, then resources will be wasted designing different tests for
different dates.
|
|
3.6
|
If Key Stage tests remain, what should they be seeking to measure?
|
They should seek to measure the extent of pupils'
understanding of and skills in 'Key' subjects.
|
|
3.7
|
If, for example, performance at Level 4 is the average level of
attainment for an eleven year old, what proportion of children is it
reasonable to expect to achieve at or above that level?
|
Unknown. It depends on the variation and distribution
of values in the sample. If the data formed a Gaussian distribution with a
standard deviation of one 'Level', then 84% would be at Level 4 or above. Of
course, no real data can ever be Gaussian and the measurement tools are too
coarse to know what the standard deviation might be.
|
|
3.8
|
How are the different levels of performance expected at each age decided
on? Is there broad agreement that the levels are appropriate and meaningful?
|
Unknown. Perhaps they are decided by some
educationalists writing criteria based on experience. However, the data base
is unclear and so agreement is irrelevant.
|
|
4
|
Testing and assessment at 16 and after
|
|
|
4.1
|
Is the testing and assessment in "summative" tests (for example, GCSE,
AS, A2) fit for purpose?
|
It depends on the purpose. If the aim is to frighten
and annoy pupils and teachers, then possibly they are. If the aim is to
provide qualifications that pupils can be proud of, then obviously not, since
GCSEs are handed out like napkins at MacDonalds. If the aim is to give an
indication to universities and employers of what a pupil is capable, then
clearly not, because universities complain that there is insufficient
granularity in the results (everyone gets an A or A-star) and employers
complain that people who have passed English and Maths are functionally
illiterate and innumerate.
|
|
4.2
|
Are the changes to GCSE coursework due to come into effect in 2009
reasonable? What alternative forms of assessment might be used?
|
They are irrelevant. With widespread use of the
internet, the use of coursework for a qualification has become a test in
concealing plagiarism, not a test of understanding.
|
|
4.3
|
What are the benefits of exams and coursework? How should they work
together? What should the balance between them be?
|
Exams provide an answer to the question "can the pupil
answer the questions put?" and coursework answers the question "can the pupil
create something that contains the required information?". It would be better
if exams asked the question "does the pupil understand the subject?" and
coursework was seen merely as a means of practising using the tools and
exploring the knowledge that underlies the subject.
|
|
4.4
|
Will the ways in which the new 14-19 diplomas are to be assessed impact
on other qualifications, such as GCSE?
|
Unknown. Not clear how the diplomas are to be
assessed. If by exam, then they will tend to subvert or replace GCSEs. If by
interview, then they will add to it.
|
|
4.5
|
Is holding formal summative tests at ages 16, 17 and 18 imposing too
great a burden on students? If so, what changes should be made?
|
It is unclear why summative tests should be needed
each year, when it is formative tests that inform teaching (see next answer).
|
|
4.6
|
To what extent is frequent, modular assessment altering both the scope
of teaching and the style of teaching?
|
The need for constant assessment is inherent in proper
teaching. Formative assessment should be a continual process. Summative
assessment should be conducted rarely, and only when a pupil reaches a
milestone such as a move from primary to secondary or secondary to tertiary
education or employment or unemployment.
|
|
4.7
|
How does the national assessment system interact with university
entrance? What does it mean for a national system of testing and assessment
that universities are setting entrance tests as individual institutions?
|
They are disconnected, as demonstrated by universities
setting entrance tests. This alone should be sufficient to demonstrate even
to someone as ignorant as the Secretary of State that the system is failing
and a system developed and operated by experts, preferably at a local level,
is needed.
|
