Inquiry on Testing and Assessment

 

Executive Summary to City & Guilds submission

 

1. City & Guilds approaches the discussion from a vocational qualification standpoint.

2. Assessment should be seen as a rare event that put demands and responsibilities on both the designer and the learner.

3. The use of awarding bodies within the English/UK system is untypical in comparison with European countires. However, they provide a valued assurance of independence and professional expertise to the consumer.

4. Through a pair of professional associations awarding bodies are successfully reducing examination bureaucracy.

5. The twin requirements of validity and reliability should always govern the choice of assessment methods.

6. While formative and summative assessments have different purposes there is potentially valuable feedback available from both which can aid learning for all parties.

7. Coursework should not be abandoned in favour of examination. The design of coursework should be improved.

8. To reduce the examination burden between 16 and 18 teachers should be given a greater role in summative assessment of performance.

9. Using multiple approaches to assessment increases the reliability and accuracy of the assessment of the learner's knowledge and skills.

10. There is too little time for there to be the development of innovative approaches to assessment for the new aspects of the Diploma.

11. While employers should be encouraged to train to awards or units on the national qualifications framework, involving awarding bodies can ensure that bespoke qualifications are of a high standard and are portable.

12.
Introduction

12.1 I am the Director of Assessment and Quality at City & Guilds. I have worked in education for over 30 years principally in vocational settings. I have been an assessment specialist within City & Guilds for over 20 years.

 

12.2 It must be made clear at the outset that City & Guilds is a vocational awarding body that has as its primary focus the assessment and certification of vocational knowledge and skills. Our market is generally 16+ and the average age of our candidature is around 30.

 

12.3 We have over 500 qualifications on offer and deliver to around 6500 centres in the UK. A centre can be anything from a FTSE 100 employer, to a College of Further Education, a Sixth Form college, a private training provider, to small employers. We issue around 1.5 million certificates a year. We believe that about 1 in 5 adults within the UK hold a City & Guilds certificate.

 

12.4 While our history would associate us strongly with traditional craft skills like agriculture and horticulture or construction and building services our broad portfolio of products reaches to retail, care, IT and ESOL (English for Speakers of other Languages) and much beyond. The City & Guilds Group also includes the Institute of Leadership and Management offering awards in over 2,200 centres. Across the Group the range of awards extends from Entry Level to the equivalent of Level 8 of the QCA National Qualification Framework. Our portfolio is also delivered in about 100 countries internationally through some 3500 centres worldwide.

 

12.5 City & Guilds has been an examining body since 1878 and was awarded a Royal Charter in 1900. It has necessarily acquired considerable skills in curriculum and assessment design and delivery.

 

12.6 With regard to the interests of the Select Committee we seek to offer some general observations on the principles and purposes of testing and assessment, and on the role of awarding bodies.

 

13. General Issues

13.1 The UK stands out in comparison with both its local and more distant neighbours in that independent awarding bodies carry out the process of examining and certification for national awards. These bodies have become centres of considerable expertise in these processes and carry a major financial and moral burden of expectation in terms of accuracy and prompt delivery. This situation is a consequence of historical decisions and a presumed wish by past governments not to take on the administration and associated significant costs of running this national system.

 

13.2 The management of this system through regulation provides a reassuring degree of independence to the system that we believe is valued by the consumer. There is more accountability in that the awarding body's reputation depends upon their ability to deliver and market forces exert continued pressure on the system to ensure high levels of quality assurance. As opposed to being run by what could so easily be perceived as a bureaucratic government department.

 

13.3 Recent initiatives by awarding body consortia, the Joint Council for Qualifications (JCQ) and the Federation of Awarding Bodies (FAB), are making progress in the fight against bureaucracy. In particular the JCQ (comprising Edexcel, AQA, OCR, City and Guilds, WJEC, CCEA and SQA) supports an initiative it has called The Eight Pledges. These are principally eight areas where the awarding bodies will collaborate to reduce the complexity in the qualifications system by reducing the administrative burden caused by assessment and quality assurance with the purpose of simplifying the relationships within and between awarding bodies and centres. FAB is a consortium of over 30 awarding bodies concerned with vocational qualifications.

 

13.4 In the awarding body system we have collaboration and competition both have roles to play. In that much of the marketplace is buying national awards then collaboration is critically important to retain customer confidence and economy within the system. The opportunity for competition enables awarding bodies to fine-tune their approaches to satisfy a wide marketplace and provide specific and differentiating customer benefits.

 

13.5 The English regulator, QCA has contributed to this situation. On the one hand when discussing major initiatives it ensures wide representation by awarding bodies but its supervision of the awarding process remains specific to each awarding body. Where there has been confusion in recent times it has been when QCA has adopted roles already performed by awarding bodies. For example, in the development of curriculum for say GNVQ or more recently the management of question banks for Key Skills.

 

13.6 Awarding bodies are centres of expertise. The larger awarding bodies, in particular those in membership to the JCQ, carry specific technical expertise in curriculum design and assessment practice. The JCQ itself through its committee structure also promotes the further development of these skills and addresses the technical issues of assessment and standards setting for general qualifications.

 

14. Testing and assessment at 16 and after

14.1 Assessment is about the collection and validation of specific evidence from or about a learner. Assessment in any of its forms is intrusive and for many unwelcome, un-looked for and unpleasant. Consequently, there is considerable obligation on the designer of tests or assessments to make them as efficient and meaningful as possible. Assessment opportunities should be seen as rare events during which the assessment tool must be finely tuned, accurate and incisive. To conduct a test that is inaccurate, excessive, unreliable or inappropriate is unpardonable. Moreover, it is an insult to the hard work and anxiety of the learner to waste their time or be needlessly over demanding. Economy it time, effort and cost is imperative.

 

14.2 Assessment can be put to two principal purposes; namely formative and summative. These have different roles in the learning process and in the ultimate recognition of achievement. Put briefly, formative assessment provides feedback to the learning process by identifying both progress and gaps in learning. Used in this sense it is often regarded as diagnostic. Summative assessment is usually conducted at the end of a learning process and is focussed on assessing learning against a known standard for the purposes of certification.

 

14.3 There is a strong argument to suggest that all assessment should support or promote learning. This will depend upon the opportunity for feedback post assessment. Currently, this is the strongest division between the two types of assessment. Formative assessment generally operates at a local level and is built into a learning programme. The outcomes of assessment are not high stakes but provide staging posts to further or remedial learning. It is assumed that summative assessment only has value in terms of the final result (pass, merit, distinction or grades A-E etc). Given suitable analysis or interpretation much value can be extracted from a candidate's examination paper or final practical assessment. Considering the significant effort that goes into the final examining process by all parties the current under-use of this data is a travesty. Some awarding bodies are now developing analytical software associated with on-line access to enable some benefit to be drawn from this available data. This information can be of use to the awarding body, the examiner, the centre as well as the candidate.

 

14.4 For summative assessments like GCSE, AS or A2 enabling the outcome of assessment to support learning would make demands on both the examination structure and the marking process. However, the potential benefits for the candidate are significant. It would also exert additional rigour on the assessment process to achieve greater detail and accuracy. Some attempt has been made to compensate for the scant regard given final assessment through the provision of the opportunity for centres and learners to review examination scripts for general qualifications.

 

14.5 The discussion of examination and coursework may benefit from a brief consideration of two of the technical issues within assessment, namely validity and reliability. Put simply, the 'test' of validity seeks to confirm that the form of assessment used adequately reflects or accesses the skills to be measured. You will have a more appropriate assessment of the skills of baking by setting the task of baking rather than setting an essay question on how to bake. The 'test' of reliability requires the assessment designer to show that assessment will repeatedly produce the same outcome, that there is no inherent bias or variability in the assessment instrument. Examination and Coursework should be regarded as two separate assessment instruments that reside in the assessment designer's toolbox which carry different degrees of validity and reliability.

 

14.6 In considering the balance between examination and coursework it may be interesting to reflect upon the current situation in vocational qualifications. In particular the national vocational qualification (NVQ) as regulated by QCA. The NVQ is a performance-based qualification underpinned by specific occupational standards. The assessment is almost totally locally supervised against assessment schedules prepared by awarding bodies in association with sector skills councils (SSCs) and accredited by QCA. Some NVQ do involve additional knowledge tests issued by the awarding body. The coursework is evidenced through a portfolio, a physical or electronic document that maps the learner's progress of performance/skills demonstration through the various units of the award. A locally based, occupationally competent assessor who has the opportunity to ask questions, challenge and reconfirm the performance carefully monitors and confirms the learner's achievements.

 

14.7 It is interesting to note that were workplace qualifications are concerned, where the country's economic performance is essential and of keen government interest the primary assessment decisions are made at local (supervisor) level. Whereas for school based qualifications which may lead to initial employment, further education and training or higher education an elaborate system of external examinations and near total independence from the local centre of learning is required. Many other countries invest considerable importance in the professional judgement of their trained teachers with regard to summative assessments of achievement.

 

14.8 It has been noted in some studies of the vocational education and training settings that a proportion of trainees experience difficulties with the development of their NVQ portfolio because of the tight examination based schooling they have received which has failed to inculcate the independence of thought and action needed within vocational education and training and employment.

 

14.9 The recent anxieties expressed over coursework and the opportunities for plagiarism are not unknown in vocational awards (VQs) though the types of incidence are different. The quality assurance system for VQs depends upon occasional visits to centres to essentially undertake an audit of local assessment practice. This process supported through regulation is called external verification. The process seeks to ensure that centres follow the required guidance and maintain the performance standards and criteria.

 

14.10 One particular issue to be considered with coursework is the nature of the task being demanded. If the task is one which can be easily downloaded from the internet and passed off as the new learner's work then one can rightly suggest that the original task was poorly conceived or set out. Assessment techniques must of necessity move with the times. If new technologies make accepted assessment practices redundant then new ways of accessing the required skills must be devised. This takes time but the solution is not to rely upon one form of assessment, as this should be regarded as poor practice.

 

14.11 It is important to recognise that learners differ from each other and it is likely that acquiring curriculum content or specific skills demands a range of skills on behalf of the learner. Consequently using only one or at best two assessment techniques limits the type and value of the evidence one is collecting. There is a risk that the process will not do the learner justice. Over reliance on one or other forms of assessment cannot be regarded as good practice. However, it must be conceded that getting the appropriate weighting between assessment methods is also a difficult process. Dispensing with coursework is not the answer to plagiarism, as this would over-focus the teaching programme on the final examination to the diminution of those auxiliary skills the curriculum sought to develop. This also returns us to the question of the validity of the chosen assessment tool.

 

14.12 The weighting issue has been of particular concern in the development of the new Diploma, in that the final award has to be derived from performance in a range of specified elements. What has been more problematic has been the difficulty in securing clarity over the purpose of the qualification and its underpinning curriculum. The tension between the general and vocational themes will also be played out in terms of assessment regimes. The vocational trend would be for more emphasis on performance evidence (ie 'can do') whereas the general trend goes for knowledge-based evidence (ie 'knows and understands that'). The vocational aim for the Diploma was to get learners out of the classroom or at least to make learning relevant beyond the classroom. It is still too early to say how the Diploma will work out in practice. A full and energetic evaluation of the pilot programmes is essential.

 

14.13 It is difficult to say whether or not Diploma assessment methods will have a consequential effect on GCSE. This will in part depend upon the innovation awarding bodies are able to bring to the assessment of the new elements in the Diploma. Given current timescales there has been insufficient time for Component Awarding Bodies to research and develop new assessment approaches. It can easily be argued that for a new qualification which is to be the standard bearer for a subtle blend of general and vocational skills that new and innovative approaches would be needed in both curriculum and assessment delivery. Accepting some of the earlier points in this submission some of this innovation should pay attention to improving feedback after assessment following the principles of assessment for learning.

 

14.14 There can be little doubt that the years 16,17 and 18 are a great challenge and pose a significant demand on all learners as they face an intense 3 year examination period. It is a shame that the regulated examining systems are unable to make greater use of the expert judgement of teachers and tutors and that increasing reliance is put upon single shot examinations. One of the benefits of the NVQ assessment system is the requirement to observe performance over time. Consequently random poor performances can be weighed against evidence of more consistent performance. Repeated demonstrations of good performance are also required rather than a single inspired one.

 

14.15 It must be recognised that the general qualifications process remains a highly competitive one and a learner's success depends heavily upon the school or college they attend and the resources that school or college is able to secure. In this sense it is not a 'fair' system. It will not be 'fair' until all learners are able to access equivalent resources delivered to a common standard. While the same criticism can be made of vocational qualifications the system has embedded procedures to reduce the variability. All occupationally competent assessors must also hold a nationally approved assessors qualification. Every centre delivering an NVQ must meet regulated approval criteria. Each centre is regularly visited by a representative of the awarding body to ensure that the centre is maintaining the occupational competence standards in its assessment practices.

 

14.16 The same is true even where employers choose to deliver NVQ within their staff training programme. The benefit being that staff trainees will ultimately receive a nationally recognised qualification which will be truly portable rather than a training package locally conceived by the employer which may have no relevance to any other (future) employer. A training programme endorsed by the employer alone, except where that employer has achieved considerable brand credibility, will have only limited value.

 

14.17 Where awarding bodies collaborate with major employers to produce bespoke qualifications the staff trainees have the benefit of knowing that they will receive a properly validated qualification based on the experience, expertise and professionalism of the awarding body. The portability is based upon the recognised professionalism of the awarding body. It is also most likely that the awarding body will retain, in a lasting archive, full records of past achievements. City & Guilds, for example, goes back 100 years. Few commercial businesses last that long or would wish to retain records of long departed employees.

 

 

Murray Butcher

Director of Assessment and Quality

City & Guilds

May 2007