Inquiry on Testing and Assessment
Executive Summary to City & Guilds submission
1. City & Guilds approaches
the discussion from a vocational qualification standpoint.
2. Assessment should be seen as
a rare event that put demands and responsibilities on both the designer and the
learner.
3. The use of awarding bodies
within the English/UK system is untypical in comparison with European
countires. However, they provide a valued assurance of independence and
professional expertise to the consumer.
4. Through a pair of professional
associations awarding bodies are successfully reducing examination bureaucracy.
5. The twin requirements of
validity and reliability should always govern the choice of assessment methods.
6. While formative and summative
assessments have different purposes there is potentially valuable feedback
available from both which can aid learning for all parties.
7. Coursework should not be
abandoned in favour of examination. The design of coursework should be
improved.
8. To reduce the examination
burden between 16 and 18 teachers should be given a greater role in summative
assessment of performance.
9. Using multiple approaches to
assessment increases the reliability and accuracy of the assessment of the
learner's knowledge and skills.
10. There is too little time for
there to be the development of innovative approaches to assessment for the new
aspects of the Diploma.
11. While employers should be
encouraged to train to awards or units on the national qualifications
framework, involving awarding bodies can ensure that bespoke qualifications are
of a high standard and are portable.
12.
Introduction
12.1 I am the Director
of Assessment and Quality at City & Guilds. I have worked in education for
over 30 years principally in vocational settings. I have been an assessment
specialist within City & Guilds for over 20 years.
12.2 It must be made
clear at the outset that City & Guilds is a vocational awarding body that
has as its primary focus the assessment and certification of vocational
knowledge and skills. Our market is generally 16+ and the average age of our
candidature is around 30.
12.3 We have over 500
qualifications on offer and deliver to around 6500 centres in the UK. A centre
can be anything from a FTSE 100 employer, to a College of Further Education, a
Sixth Form college, a private training provider, to small employers. We issue
around 1.5 million certificates a year. We believe that about 1 in 5 adults
within the UK hold a City & Guilds certificate.
12.4 While our history
would associate us strongly with traditional craft skills like agriculture and
horticulture or construction and building services our broad portfolio of
products reaches to retail, care, IT and ESOL (English for Speakers of other
Languages) and much beyond. The City & Guilds Group also includes the
Institute of Leadership and Management offering awards in over 2,200 centres.
Across the Group the range of awards extends from Entry Level to the equivalent
of Level 8 of the QCA National Qualification Framework. Our portfolio is also
delivered in about 100 countries internationally through some 3500 centres
worldwide.
12.5 City & Guilds
has been an examining body since 1878 and was awarded a Royal Charter in 1900.
It has necessarily acquired considerable skills in curriculum and assessment
design and delivery.
12.6 With regard to the
interests of the Select Committee we seek to offer some general observations on
the principles and purposes of testing and assessment, and on the role of
awarding bodies.
13. General Issues
13.1 The UK stands out
in comparison with both its local and more distant neighbours in that
independent awarding bodies carry out the process of examining and
certification for national awards. These bodies have become centres of
considerable expertise in these processes and carry a major financial and moral
burden of expectation in terms of accuracy and prompt delivery. This situation
is a consequence of historical decisions and a presumed wish by past
governments not to take on the administration and associated significant costs
of running this national system.
13.2 The management of
this system through regulation provides a reassuring degree of independence to
the system that we believe is valued by the consumer. There is more
accountability in that the awarding body's reputation depends upon their
ability to deliver and market forces exert continued pressure on the system to
ensure high levels of quality assurance. As opposed to being run by what could
so easily be perceived as a bureaucratic government department.
13.3 Recent initiatives
by awarding body consortia, the Joint Council for Qualifications (JCQ) and the
Federation of Awarding Bodies (FAB), are making progress in the fight against
bureaucracy. In particular the JCQ (comprising Edexcel, AQA, OCR, City and
Guilds, WJEC, CCEA and SQA) supports an initiative it has called The Eight
Pledges. These are principally eight areas where the awarding bodies will
collaborate to reduce the complexity in the qualifications system by reducing
the administrative burden caused by assessment and quality assurance with the
purpose of simplifying the relationships within and between awarding bodies and
centres. FAB is a consortium of over 30 awarding bodies concerned with
vocational qualifications.
13.4 In the awarding
body system we have collaboration and competition both have roles to play. In
that much of the marketplace is buying national awards then collaboration is
critically important to retain customer confidence and economy within the
system. The opportunity for competition enables awarding bodies to fine-tune
their approaches to satisfy a wide marketplace and provide specific and
differentiating customer benefits.
13.5 The English
regulator, QCA has contributed to this situation. On the one hand when
discussing major initiatives it ensures wide representation by awarding bodies
but its supervision of the awarding process remains specific to each awarding
body. Where there has been confusion in recent times it has been when QCA has
adopted roles already performed by awarding bodies. For example, in the
development of curriculum for say GNVQ or more recently the management of
question banks for Key Skills.
13.6 Awarding bodies are
centres of expertise. The larger awarding bodies, in particular those in
membership to the JCQ, carry specific technical expertise in curriculum design
and assessment practice. The JCQ itself through its committee structure also
promotes the further development of these skills and addresses the technical
issues of assessment and standards setting for general qualifications.
14. Testing and assessment at 16 and after
14.1 Assessment is about
the collection and validation of specific evidence from or about a learner.
Assessment in any of its forms is intrusive and for many unwelcome, un-looked
for and unpleasant. Consequently, there is considerable obligation on the
designer of tests or assessments to make them as efficient and meaningful as
possible. Assessment opportunities should be seen as rare events during which
the assessment tool must be finely tuned, accurate and incisive. To conduct a
test that is inaccurate, excessive, unreliable or inappropriate is
unpardonable. Moreover, it is an insult to the hard work and anxiety of the
learner to waste their time or be needlessly over demanding. Economy it time,
effort and cost is imperative.
14.2 Assessment can be
put to two principal purposes; namely formative and summative. These have
different roles in the learning process and in the ultimate recognition of
achievement. Put briefly, formative assessment provides feedback to the
learning process by identifying both progress and gaps in learning. Used in
this sense it is often regarded as diagnostic. Summative assessment is usually
conducted at the end of a learning process and is focussed on assessing
learning against a known standard for the purposes of certification.
14.3 There is a strong
argument to suggest that all assessment should support or promote learning.
This will depend upon the opportunity for feedback post assessment. Currently,
this is the strongest division between the two types of assessment. Formative
assessment generally operates at a local level and is built into a learning
programme. The outcomes of assessment are not high stakes but provide staging
posts to further or remedial learning. It is assumed that summative assessment
only has value in terms of the final result (pass, merit, distinction or grades
A-E etc). Given suitable analysis or interpretation much value can be extracted
from a candidate's examination paper or final practical assessment. Considering
the significant effort that goes into the final examining process by all
parties the current under-use of this data is a travesty. Some awarding bodies
are now developing analytical software associated with on-line access to enable
some benefit to be drawn from this available data. This information can be of
use to the awarding body, the examiner, the centre as well as the candidate.
14.4 For summative
assessments like GCSE, AS or A2 enabling the outcome of assessment to support
learning would make demands on both the examination structure and the marking
process. However, the potential benefits for the candidate are significant. It
would also exert additional rigour on the assessment process to achieve greater
detail and accuracy. Some attempt has been made to compensate for the scant
regard given final assessment through the provision of the opportunity for
centres and learners to review examination scripts for general qualifications.
14.5 The discussion of
examination and coursework may benefit from a brief consideration of two of the
technical issues within assessment, namely validity and reliability. Put
simply, the 'test' of validity seeks to confirm that the form of assessment
used adequately reflects or accesses the skills to be measured. You will have a
more appropriate assessment of the skills of baking by setting the task of
baking rather than setting an essay question on how to bake. The 'test' of
reliability requires the assessment designer to show that assessment will
repeatedly produce the same outcome, that there is no inherent bias or
variability in the assessment instrument. Examination and Coursework should be
regarded as two separate assessment instruments that reside in the assessment
designer's toolbox which carry different degrees of validity and reliability.
14.6 In considering the
balance between examination and coursework it may be interesting to reflect
upon the current situation in vocational qualifications. In particular the
national vocational qualification (NVQ) as regulated by QCA. The NVQ is a
performance-based qualification underpinned by specific occupational standards.
The assessment is almost totally locally supervised against assessment
schedules prepared by awarding bodies in association with sector skills
councils (SSCs) and accredited by QCA. Some NVQ do involve additional knowledge
tests issued by the awarding body. The coursework is evidenced through a
portfolio, a physical or electronic document that maps the learner's progress
of performance/skills demonstration through the various units of the award. A
locally based, occupationally competent assessor who has the opportunity to ask
questions, challenge and reconfirm the performance carefully monitors and
confirms the learner's achievements.
14.7 It is interesting
to note that were workplace qualifications are concerned, where the country's
economic performance is essential and of keen government interest the primary
assessment decisions are made at local (supervisor) level. Whereas for school
based qualifications which may lead to initial employment, further education
and training or higher education an elaborate system of external examinations
and near total independence from the local centre of learning is required. Many
other countries invest considerable importance in the professional judgement of
their trained teachers with regard to summative assessments of achievement.
14.8 It has been noted
in some studies of the vocational education and training settings that a
proportion of trainees experience difficulties with the development of their
NVQ portfolio because of the tight examination based schooling they have
received which has failed to inculcate the independence of thought and action
needed within vocational education and training and employment.
14.9 The recent
anxieties expressed over coursework and the opportunities for plagiarism are
not unknown in vocational awards (VQs) though the types of incidence are
different. The quality assurance system for VQs depends upon occasional visits
to centres to essentially undertake an audit of local assessment practice. This
process supported through regulation is called external verification. The
process seeks to ensure that centres follow the required guidance and maintain
the performance standards and criteria.
14.10 One particular issue
to be considered with coursework is the nature of the task being demanded. If
the task is one which can be easily downloaded from the internet and passed off
as the new learner's work then one can rightly suggest that the original task
was poorly conceived or set out. Assessment techniques must of necessity move
with the times. If new technologies make accepted assessment practices
redundant then new ways of accessing the required skills must be devised. This
takes time but the solution is not to rely upon one form of assessment, as this
should be regarded as poor practice.
14.11 It is important to
recognise that learners differ from each other and it is likely that acquiring
curriculum content or specific skills demands a range of skills on behalf of
the learner. Consequently using only one or at best two assessment techniques
limits the type and value of the evidence one is collecting. There is a risk
that the process will not do the learner justice. Over reliance on one or other
forms of assessment cannot be regarded as good practice. However, it must be
conceded that getting the appropriate weighting between assessment methods is
also a difficult process. Dispensing with coursework is not the answer to
plagiarism, as this would over-focus the teaching programme on the final
examination to the diminution of those auxiliary skills the curriculum sought
to develop. This also returns us to the question of the validity of the chosen
assessment tool.
14.12 The weighting issue
has been of particular concern in the development of the new Diploma, in that
the final award has to be derived from performance in a range of specified
elements. What has been more problematic has been the difficulty in securing
clarity over the purpose of the qualification and its underpinning curriculum.
The tension between the general and vocational themes will also be played out
in terms of assessment regimes. The vocational trend would be for more emphasis
on performance evidence (ie 'can do') whereas the general trend goes for
knowledge-based evidence (ie 'knows and understands that'). The vocational aim
for the Diploma was to get learners out of the classroom or at least to make
learning relevant beyond the classroom. It is still too early to say how the
Diploma will work out in practice. A full and energetic evaluation of the pilot
programmes is essential.
14.13 It is difficult to say
whether or not Diploma assessment methods will have a consequential effect on
GCSE. This will in part depend upon the innovation awarding bodies are able to
bring to the assessment of the new elements in the Diploma. Given current
timescales there has been insufficient time for Component Awarding Bodies to
research and develop new assessment approaches. It can easily be argued that
for a new qualification which is to be the standard bearer for a subtle blend
of general and vocational skills that new and innovative approaches would be
needed in both curriculum and assessment delivery. Accepting some of the
earlier points in this submission some of this innovation should pay attention
to improving feedback after assessment following the principles of assessment
for learning.
14.14 There can be little
doubt that the years 16,17 and 18 are a great challenge and pose a significant
demand on all learners as they face an intense 3 year examination period. It is
a shame that the regulated examining systems are unable to make greater use of
the expert judgement of teachers and tutors and that increasing reliance is put
upon single shot examinations. One of the benefits of the NVQ assessment system
is the requirement to observe performance over time. Consequently random poor
performances can be weighed against evidence of more consistent performance.
Repeated demonstrations of good performance are also required rather than a
single inspired one.
14.15 It must be
recognised that the general qualifications process remains a highly competitive
one and a learner's success depends heavily upon the school or college they
attend and the resources that school or college is able to secure. In this
sense it is not a 'fair' system. It will not be 'fair' until all learners are
able to access equivalent resources delivered to a common standard. While the
same criticism can be made of vocational qualifications the system has embedded
procedures to reduce the variability. All occupationally competent assessors
must also hold a nationally approved assessors qualification. Every centre
delivering an NVQ must meet regulated approval criteria. Each centre is
regularly visited by a representative of the awarding body to ensure that the
centre is maintaining the occupational competence standards in its assessment
practices.
14.16 The same is true
even where employers choose to deliver NVQ within their staff training
programme. The benefit being that staff trainees will ultimately receive a
nationally recognised qualification which will be truly portable rather than a
training package locally conceived by the employer which may have no relevance
to any other (future) employer. A training programme endorsed by the employer
alone, except where that employer has achieved considerable brand credibility,
will have only limited value.
14.17 Where awarding
bodies collaborate with major employers to produce bespoke qualifications the
staff trainees have the benefit of knowing that they will receive a properly
validated qualification based on the experience, expertise and professionalism
of the awarding body. The portability is based upon the recognised
professionalism of the awarding body. It is also most likely that the awarding
body will retain, in a lasting archive, full records of past achievements. City
& Guilds, for example, goes back 100 years. Few commercial businesses last
that long or would wish to retain records of long departed employees.
Murray Butcher
Director of Assessment and
Quality
City & Guilds
May 2007
