Education CommitteeWritten evidence submitted by the Advisory Committee on Mathematics Education (ACME)

About ACME

The Advisory Committee on Mathematics Education (ACME www.acme-uk.org) is an independent committee, based at the Royal Society and operating under its auspices, that aims to influence Government strategy and policies with a view to improving the outcomes of mathematics teaching and learning in England and so secure a mathematically enabled population. The ACME response has been informed by input from the mathematics community through the ACME Outer Circle, a group assembled to encompass a breadth of knowledge, support and influence which we consult on key issues.

Summary

1. Key features of any curriculum and examination system include effective regulation, strong subject leadership, capacity for collaboration and innovation, and transparency of operation. The dominant consideration for qualification and assessment development should be the nature of a subject and its educational purpose and goals.

2. ACME has concerns about the powers and responsibilities of a regulatory body and suggests that the question of regulation or scrutiny should therefore form part of this Select Committee inquiry.

3. Consistent accuracy in setting and marking examination questions is only one facet of the overall quality of an examination paper. The current system does not always adequately lead to the production of high quality examination papers.

4. ACME is concerned about the commercial relationships that exist between awarding organisations and publishers. Awarding organisations should be prevented from using their names or logos on textbooks, and publishers should not divulge the fact that an author is also an examiner for a particular awarding organisation.

Introduction

5. There has never been “a golden age of assessment”. Each change to the examination system has aimed to address issues that have arisen in a previous system. The current examination system and its regulation require review and further changes in order to ensure that all qualifications are respected and valued.

6. There is a wide range of mathematics qualifications taken by pupils in the 15–19 age bracket, including GCSEs, AS and A-levels, Functional Skills, Free Standing Mathematics Qualifications (FSMQs), iGCSEs and many more. A large number of mathematics qualifications are taken each year. For example, in England, each year approximately 700,000 students sit GCSE mathematics and 75,000 sit A-level mathematics.1 Whilst ACME welcomes the existence of a range of qualifications that suit individuals’ needs, this landscape needs to be clear to students, parents/carers, teachers and employers.

7. GCSE mathematics is currently a gate-keeper qualification at the end of compulsory schooling. Employers and universities use a grade C in GCSE mathematics to help them to decide whether individuals will be able to succeed in their chosen career or university course. Thus it would seem essential that the examination system maintains GCSE mathematics at an appropriate standard and has the confidence of employers and higher education. However, many from across schools, colleges, universities and employers feel that the present system has shortcomings.2 ^, 3

8. In order to resource the examination system, each year schools across the country pay the awarding organisations’ examination entrance fees,4 at total of £281 million in 2008–09. This figure represents a significant increase on the previous six years, partly because of the pressures on students and schools and colleges to increase their grades (for example at the C/D boundary at GCSE) through re-sitting examinations and/or modules. It is important that this budget is spent effectively.

9. The figure above for the costs of examination entrance fees does not include any fees for support materials provided by the awarding organisations. Awarding organisations also provide or endorse a range of resources for schools. These resources include lesson plans, student worksheets, textbooks and training opportunities. Some are provided without charge.

10. It is not possible to provide evidence on the range of awarding organisations and the quality of the process and their commercial activities without also considering the broader system of relationships in which national examinations at 15–19 operate. The awarding organisations do not exist in isolation; they are regulated, are required to respond to changes in the National Curriculum (at least in the case of GCSEs in National Curriculum subjects), and government accountability measures impact significantly on qualifications and their take-up in schools.

11. The effectiveness of these relationships is crucial, no matter how many awarding organisations there are. At the moment, this system is out of balance. At times, the assessment system, and the regulations which govern it, undermine the broader aims of the National Curriculum and the A-level subject criteria. The development of a national assessment or examination system should take place alongside curriculum review, and should reflect the full aims of the curriculum.

The Role of Regulation

12. Whatever examination system is in place, effective regulation or scrutiny must be designed to support high quality provision of qualifications and not, as at present, hinder it. Such scrutiny could be undertaken by a quango, a national board, or ultimately parliament itself. Currently, regulation is undertaken by Ofqual, with awarding organisations being responsible for internal quality assurance.

13. ACME has concerns about the current approach to regulation and suggests that the question of regulation or scrutiny should therefore form part of this Select Committee inquiry. In particular we are concerned that:

(a)There is no subject expertise within Ofqual. For nationally important key subjects, such as mathematics, we are concerned that Ofqual is not able to draw on dedicated subject experts within its own organisation. This lack of expertise results in assessments which are not always consistent with a subject’s learning demands, have widely differing interpretation of the criteria and do not mirror the full range of the stated learning aims, resulting in a progressively “lowest common denominator” interpretation. Any external regulator should maintain its own expertise in mathematics and its assessment and there should be mechanisms to validate subject content in cases of appeal. The regulation process should consider whether assessments are well designed, cover the curriculum and its learning objectives appropriately, and enable pupils to demonstrate what they know and are able to do.

(b)Because of this lack of internal expertise, the regulator appoints external subject experts. However, there is no transparency in Ofqual’s operation in terms of the choice of external subject experts consulted. This results in a lack of confidence in Ofqual’s ability to regulate key national examinations effectively. Any regulator of qualifications should appoint its advisors in a transparent way and bodies such as ACME and the subject associations should be able to challenge any advice in mathematics that they feel is unsatisfactory.

(c)There has been a tendency for a “one size fits all” approach to the regulation of subjects. By treating subjects generically, little room is left for the vital aspects that distinguish one subject from another.5 Ofqual’s overall approach to regulation and its enforcement stifles innovation in the system. The regulator should ensure that assessment structures for mathematics harmonise with subject content and the curriculum aims.

(d)Even when Ofqual does find failures have occurred within awarding organisations, its powers appear limited. The regulator can place an awarding organisation under greater scrutiny in the future; but it is unlikely to prohibit an awarding organisation from offering key national qualifications. Regulation should place a significant emphasis on preventing problems with examinations and/or awarding organisations emerging in the first place. The regulator needs to have the power to impose appropriate sanctions on awarding organisations when security is breached, or when questions are shown to be inaccurate or even incorrect.

Models for a National Examination System

The status of awarding organisations

14. Before discussing various models for the examination system, we note that the five awarding organisations in the UK that award GCSEs and A-levels are set up in different ways and therefore have different characteristics. The awarding organisations themselves have complex histories, but today they describe themselves as follows:

AQA—a registered educational charity.6

CCEA—a non-departmental public body reporting to the Department of Education in Northern Ireland.7

Edexcel—a subsidiary company of Pearson,8 which is a commercial organisation.

OCR—Oxford Cambridge and RSA Examinations is a company limited by guarantee registered in England, and9 part of the Cambridge Assessment Group, a not for profit organisation, which itself is a department of the University of Cambridge.

WJEC—a registered charity, and a company limited by guarantee, owned by the 22 local authorities in Wales.10

15. The differing nature of the awarding organisations is perceived to affect how they respond to the various incentives that the system promotes, and it is worthwhile bringing this element into the Committee’s inquiry. In particular, it should be noted that “commercial interests” of awarding organisations can prevent them from working together—particularly in pioneering the development of high quality new qualifications.

The current model

16. The current model for GCSEs and A-level examinations is a regulated market. This model has some strengths, including:

(a)Competitiveness in the administration of examinations should, along with effective regulation, keep costs low for schools (although it is not obvious whether this effect is being realised).

(b)The quality of the administrative service to schools should be enhanced by competition between examination providers.

(c)Multiple awarding organisations should maintain flexibility in the scale of the system and should enable it to cope with large numbers of entrants to individual qualifications.

(d)Having an external regulator and multiple awarding organisations maintains a separation, to some degree, between short-term political priorities and the need for longer term qualification development.

17. However, the current model has raised some serious concerns:

(a)Competition between awarding organisations may have led to a downward auction in standards. This is particularly so in the case of large entry specifications, such as mathematics, which inevitably subsidise smaller specifications.

(b)Confidence in the system may be undermined by the variety of specifications attracting the same qualification title.

(c)Expertise in setting effective assessments is limited in some areas: this is particularly obvious where sophisticated contexualised questions are required several times a year, as with the Functional Skills qualifications.

(d)The awarding organisations are perceived to be under pressure from schools and colleges to minimise change to syllabuses and assessment methods. Meaningful development in assessment fully supports intended changes in learning experiences for students.

18. The curriculum and assessment models adopted by each awarding organisation are in some cases currently very similar so, where this is the case, nothing is being gained by the existence of several awarding organisations. Moreover, even if there were substantial differences, it is in most cases the institutions rather than the students which choose the awarding organisation providing each qualification. In these cases, students do not benefit from the variety on offer:

(a)Examination entrants on courses or modules with low participation may be spread between awarding bodies, leading to inefficiencies in the system.

(b)There is duplication of effort involved in the development of new specifications.

19. The current model could be made more effective, without reducing the number of awarding organisations, particularly through improved collaboration. The awarding organisations operate as an oligarchy at the top level concerned with the protection of intellectual property. However, people who work at subject level are used to working together as teachers or examiners. There is a clear mismatch here.

20. It is possible to work more collaboratively. Four awarding organisations have been involved in the development of the linked pair of mathematics GCSEs (AQA, Edexcel, OCR and WJEC). These organisations, facilitated in the first instance by ACME, have been collaborating to improve the quality of the pilot examinations. However, this collaboration has been post hoc. It would be invaluable if the awarding organisations tried to develop questions collaboratively so as to improve the quality of the setting of examination questions, the consistency of the interpretation of assessment objectives and the actual number of marks awarded for similar types of question. ACME believes that its experience with the linked pair of mathematics GCSEs demonstrates how a collaborative approach can produce innovation and quality, and this should be extended to formal collaboration in the development of new specifications and qualifications.

21. The awarding organisations each independently seek advice from a range of subject associations, mathematics education researchers and mathematics organisations, which is not the most efficient way to function. For instance, some awarding organisations are now holding regular meetings or seminar series relating to mathematics assessments. If the current number of awarding organisations remains the same, we would welcome significantly more collaboration between them in order to improve the efficiency for all concerned. A standing mathematics council could be introduced, which could have a remit wider than qualification development (see paragraph 30b).

22. All awarding organisations should be required to commit to innovation as part of normal practice, and they should work with groups of schools, subject associations, charitable bodies and others to do this. They should be able to show how such projects advantage learners, and that such projects are informed by advice from a range of stakeholders. In other words, the assessment system should be part of ongoing curriculum review.

23. It has been argued that such innovation is stifled by the awarding organisations having to work to restrictive guidelines. If the system continues to include multiple awarding organisations, the guidelines would need to be reviewed in order to ensure that they support, rather than undermine, innovation.

Alternative Models for a National Examination System

Rationalising the number of awarding organisations

24. As long as Academic and Vocational qualifications remain separate, it may be inevitable that there will be distinctly different types of awarding organisation. A separate question is whether there should be a smaller number of organisations awarding GCSEs and A-levels. The number of organisations awarding GCSEs/A-levels has already reduced over the years; the question now is what the implications would be of taking this further.

25. This could result in a single, national awarding body such as in Scotland. Alternatively there could be a franchise model, where a single qualification is operated by an awarding organisation for a given length of time.

26. A single awarding body would:

(a)Improve consistency in terms of examination setting; confidence in qualifications would therefore increase.

(b)Result in less time spent making the same qualifications comparable across different organisations.

(c)Be able to innovate, as new specifications could be developed (in suitable time scales) with groups external to the single body. Choices of specification could also still be offered, much as awarding organisations do currently.

(d)Operate more efficiently through economies of scale.

27. However, counter arguments might be:

(a)There would be no efficiency incentive for the awarding organisation – processes could be bloated and expensive. There would be a monopoly on examination fees for national examinations.

(b)If something goes wrong with an examination, or a specification is in some way flawed, it would affect the entire cohort of students, potentially over 700,000 individuals in the case of GCSE mathematics.

(c)If, for any reason, the qualification is not deemed fit for purpose by schools, parents, employers of HE, there remains a risk that alternative qualifications may be created by other providers, hence undermining the national qualifications, although this is possibly unlikely in the case of high currency qualifications like GCSE Mathematics.

(d)Examination specifications risk becoming fossilised, unless requirements to develop and innovate are contractual.

National Curriculum assessment model

28. An alternative model of developing national examinations could be modelled on the End of Key Stage tests. When the End of Key Stage tests were first introduced, QCA (as then was) had responsibility for their development, including pre-testing and quality control, and the administration and marking of the tests. The Development body was also responsible for the appointment and training of markers and the standardisation of results from year to year.

29. The development of the assessment questions was undertaken by academic teams based in a university or the NFER.11 The test items were then trialled on thousands of schoolchildren and then discussed by a large test review group of teachers, mathematics officers from within QCA, HMI subject association appointees and national mathematics advisers.

30. The present practice of setting and marking GCSE and A-level qualifications, as well as other national qualifications is very different, particularly in terms of the amount of development and trialling of assessment items. However, it should be noted that GCSE and A-level qualifications are high-stakes examinations, and this may impact on the effectiveness of this model, and the style of questions may not suit trialling. Also, there are issues regarding security (or the lack of it) of pre-trialled questions.

University-led model

31. A university-led model, in which control of GCSEs and A-levels is through one or more higher education institutions, may be attractive to some. A large amount of expertise resides in higher education, employers, learned societies, subject associations and the teaching community. Universities should continue to be included in any reviewing groups set up to approve papers in A-level Mathematics and Further Mathematics.

32. However, the majority of mathematics students will go into employment or study subjects other than mathematics, but will still need an understanding of mathematics.12 In general, universities may know very little about the mathematical needs and capabilities of these, or younger, students. If examination boards become dominated by university perspectives, this might return us to a situation when there was little that was relevant in the mathematics curriculum for large numbers of students.

Desirable features of a curriculum and examination system

33. In order for the assessment system to function properly, the following factors should be considered:

(a)Effective regulation—The regulation of the examination system should be transparent and accountable to subject communities, and not hinder high quality provision of qualifications (see paragraph 9 above).

(b)Leadership—There should be subject specific leadership across curriculum and qualifications. Consideration should be given as to whether a standing mathematics council, owned jointly by the subject community and the government, could be established.

(c)Coherence—Curriculum development should operate within legislated parameters of a fixed timetable of curriculum reviews. Assessment should be considered alongside curriculum development.

(d)Collaboration—If multiple awarding organisations exist, they should be required to collaborate on the development of qualifications and assessment instruments.

(e)Innovation—Awarding organisation(s) should be encouraged to innovate, working with groups of schools, subject associations, charitable bodies and others.

(f)Subject expertise—awarding organisation(s) should draw on subject expertise in higher education, schools, subject associations, in a clear, transparent, and meaningful way.

(g)Transparency—The roles and relationships between awarding organisations, regulators, government and subject associations should be clear and transparent.

(h)Clarity—The system of qualifications must be clear to students, parents and carers, teachers, employers and HE.

(i)Accountability mechanisms—Accountability mechanisms should not distort the development or use or qualifications. They should support good teaching and learning in schools and colleges.

Accuracy in Setting Papers and Marking Scripts

34. Consistent accuracy in setting and marking examination questions is important, but must be considered in the wider context of their overall quality. ACME is concerned about this: occasional instances of conceptual flaws or the poor posing of questions are accompanied by questions which are inconsequential, predictable or lack imagination. The current system does not always lead to the production of high quality examination papers.

35. A national mathematics body responsible for all curriculum and qualifications development, and for developing high quality examination questions, could improve the quality of examination questions (see paragraph 30b). Such a body should have a permanent team of recognised subject experts and subject assessment experts and should draw on talent throughout the land to stimulate all developments and provide quality-control over these developments. At the very least, specifications and assessment instruments should be extensively piloted, and then evaluated, before being introduced into schools.

36. Issues of quality and accuracy have been exacerbated by the number of assessments each awarding organisation has had to develop each year. The proposed changes to the number of examination series may help to address some of our concerns.

Commercial Activities of Awarding Bodies

37. ACME is concerned about the commercial relationships that exist between awarding organisations and publishers, which allow textbooks to be sold as having been written or endorsed by the chief examiner. Setting exams should be a position of trust not a commercial opportunity. The use of examination boards’ logos to advertise books also places them at an unfair competitive advantage against other, possibly better books.

38. Awarding organisation logo endorsements or chief examiner authorship encourage schools to make purchasing decisions on the basis of author affiliation rather than the quality of the resource itself in terms of the teaching and learning it supports. Endorsed textbooks are often seen as the safe option by schools. This is often not in the best interests of the learners. These texts tend to be too closely aligned with the assessment of the syllabus with a focus on passing the examination, rather than encouraging exploration of related topics and applications, and the development of mathematical thinking.

39. ACME recommends that awarding organisations are prevented from using their names or logos on textbooks and that publishers do not divulge the fact that an author is also an examiner for a particular awarding organisation.

40. Endorsed textbooks also have to be reprinted (and sometimes changed) when specifications change. This culture of reprinting can lead to further costs for schools and colleges, and, potentially, could compromise the quality of the textbooks and related resources. Timelines during specification development are tight, leading to very short turnarounds for changes to endorsed specifications. In contrast, high quality textbooks have often been produced by special projects, such as the School Mathematics Project (SMP), Nuffield Mathematics, and Mathematics in Education and Industry (MEI) over longer periods of time. This high quality is due to the books growing out of a vision of a course which is itself the result of a deeply considered curriculum and qualifications development programme for mathematics. The timescale for revision and implementation of specifications should be lengthened and be related to the review of the National Curriculum on a fixed 10 year cycle.

41. Chief examiners also run paid-for training sessions which risk being focused on coaching participants on how to pass the examination, further encouraging “teaching to the test” in schools and creating an incentive for the examiner to set questions in such a way as to reward those who attend the course. Such training offered by awarding organisations should be closely monitored against standards for professional development. This could be achieved, possibly through the NCETM CPD accreditation system.

In Conclusion

42. The present system has many critics. There is the potential for improving the ways in which the mathematics curriculum is developed and assessed and how qualifications are developed and regulated. The ways in which this is done will depend on the form of the overarching structures and mechanisms that are put in place; the success of the system relies on all of these working to the benefit of the students. The factors that need to cohere include regulation, curriculum review, innovation, safeguarding of standards, the setting and marking of examinations, and the development and marketing of suitable resources and textbooks. At present the way the system operates tends to favour institutions and agencies rather than the students themselves.

November 2011