House of Commons - The administration of examinations for 15-19 year olds in England

The administration of examinations for 15-19 year olds in England - Education Committee Contents

4 The way forward

Introduction

61. A recurring criticism of the current system of multiple exam boards is that they compete by lowering their standards, in order to increase market share, in the so-called "race to the bottom". While this may not be overt, many have expressed concern about the two-fold downward pressure on the system generated by competition between exam boards and the accountability system (of which more in chapter ten).

62. The concept of exam standards is complex, involving the content of the syllabus and the demand of question papers and tasks, as well as marking and grading.

Syllabus content

63. Exam boards develop syllabuses based on qualification and subject criteria that are set by Ofqual. These criteria provide parameters within which exam boards may develop distinctive courses. Ofqual approves or accredits new syllabuses against the qualification and subject criteria. GCSE subject criteria are based on the Programmes of Study laid down in the National Curriculum, where applicable.

64. In the current system exam boards compete intensively on syllabus features, offering a variety of syllabuses so that "teachers can select approaches tailored to the needs and interests of their students".[97] Examiners, many of whom were also practising teachers, reported that exam boards conduct extensive market research to help them identify which aspects of subjects (for example, texts in English, periods of history) are preferred by teachers. The choice available to teachers within the current system is such that teachers can choose the most appropriate syllabus for their classes or school cohorts, rather than for individual pupils. The distinctiveness of syllabuses varies between subjects, with duplication of very similar syllabuses in some subjects.

65. Learned bodies were critical of a lack of innovation in syllabus development across the exam boards. SCORE told us that the "current model is not supportive of innovation [...] sharing best practice and collaborative working are not embraced".[98] Professor John Holman of the Wellcome Trust put it more bluntly, telling us that "innovation is not engendered in the current system because people aren't encouraged to talk to each other".[99] We believe that it is unrealistic to expect significant collaboration between exam boards on syllabus development in the current system, as exam boards compete vigorously at this stage for market share on the basis of syllabus features.

66. The DfE has also expressed its concerns about content standards:

Central to our concern is that the nature of competition seems to present significant risk of awarding bodies producing more 'accessible' specifications, with content that is less intrinsically challenging, in order to capture market share.[100]

This concern was echoed by Jon Coles, former director general for education standards at the DfE, who told a recent Cambridge Assessment conference that exam boards should stop marketing qualifications as "accessible". Mr Coles suggested that the tactic has promoted a "culture in which it is seen to be acceptable to say to schools, 'do this [exam] because it is easier' [...] even if you do not use those words, that is what schools are hearing".[101] Mr Coles also accused the exam boards of developing exams that "barely meet" Ofqual's minimum requirements and called for the boards to have the "moral courage" to ensure their qualifications have the depth and quality to exceed Ofqual's minimum requirements.[102]

67. Beyond Jon Coles' "moral courage", no-one was able to help us identify incentives that exist in the current system for exam boards to exceed the minimum requirements. The evidence pointed instead to downward pressures in the system, with the Minister for Schools telling us that we have "a system that rewards awarding organisations with an incentive to provide the most accessible [...] examination in order to increase market share".[103] SCORE argued that the commercial concerns of exam boards and their need to maintain or increase market share have taken precedence over educational ones, affecting syllabus content (chosen to be easily assessable) and the demand of question papers (with fewer questions requiring higher level skills of analysis, synthesis and evaluation).[104]

68. Even if exam boards do not compete overtly on standards, we agree with a TES forum post that "there is every incentive for the exam boards to ensure their exams are no harder than anyone else's".[105] Strikingly, this issue was also raised as an area of concern by a senior exam board official: AQA's Andrew Hall called for a "focus on how we develop content standards for examinations", arguing that the "safest thing for delivering the most secure improvements for our students is to really tackle content standards".[106]

69. There were recurring calls in evidence for Ofqual to be more robust in its regulation of content standards, specifically to have more rigorous and transparent accreditation procedures, with appropriate use of subject experts.[107] Glenys Stacey, Chief Executive of Ofqual, described the accreditation process as "our people at Ofqual trying almost to second-guess the experts in awarding bodies".[108] We suspect that this task may be made more difficult by a lack of assessment expertise within Ofqual. As Professor Jo-Anne Baird has pointed out "few individuals in [...] Ofqual have training or even experience of designing and delivering educational assessments".[109]

70. We welcome the recent action taken by Ofqual on content standards, requiring the exam boards to "tighten" GCSEs in geography, English literature, history and mathematics. This is to ensure that all students have to study the subject in appropriate breadth and depth.[110] However, the fact remains that the syllabuses that needed "tightening" had been approved by Ofqual's accreditation procedures. We were concerned at Glenys Stacey's analysis that "accreditation at the moment is a process that has its worth, but it is part of the way things were done".[111] This seems to imply that Ofqual is moving away from its current approach to accreditation. We return to Ofqual's regulation of content standards, in particular its accreditation procedures, in chapter five.

71. It is our view (and we have heard little to persuade us otherwise) that the incentives in the current system lead to downward competition on content standards. Strong incentives exist for exam boards to reduce content to a minimum or to offer "content-lite" routes through their syllabuses. The problem as we see it is that, as Glenys Stacey herself stated, "there are no incentives to go in a different direction".[112] We agree with Rod Bristow, President of Pearson UK, that it is fair to ask "what incentives can be put into the system to raise standards".[113] We have therefore considered two options for change which would aim to provide such incentives: the first is national syllabuses and the second is franchising by subject.

NATIONAL SYLLABUSES

72. National syllabuses would be accredited by Ofqual and then regarded as a national resource that could be examined by any of the English exam boards. There could be one accredited national syllabus in each subject or more than one to provide some choice for schools. This would remove the incentive for exam boards to compete on syllabus content, while retaining current incentives for operational efficiency, innovation and quality of service to schools. Too often, competition on content adds little value and, in some subjects, creates duplication of effort for little genuine variation.

73. Such national GCSE and A level syllabuses could be designed by exam boards in conjunction with representatives from higher education, learned bodies and/or employers. Increasing the role of learned bodies (and universities and employers) in syllabus development would be one way to encourage collaboration and a focus on best practice. Under a system of national syllabuses, exam boards could be required to consult with a relevant learned body or, at A level, higher education representatives, as part of their syllabus development process. The syllabus could be endorsed by the learned body or universities, as the Government envisages for A level. Exam boards would be required to have at least one syllabus accredited (in any subject) in order to offer question papers linked to the national syllabuses developed by another exam board. Ofqual would need to monitor question papers, but its grading standards work (see chapter five) would take account of, and adjust for, any differences in demand between question papers.

74. An alternative approach to the creation of national syllabuses, which would then be examined by the existing boards in competition, would be to commission the syllabuses directly. Successful bidders in this process should be required to be consortia of examining boards, learned bodies, higher education and/or employers.

FRANCHISING OF SUBJECTS TO EXAM BOARDS

75. The second option is franchising, which would involve rather more fundamental changes to the way the system is organised. In common with national syllabuses, it would offer a way of eliminating competition on content standards and allegations of a "race to the bottom", while retaining multiple exam boards. Unlike national syllabuses, it would involve exam boards developing particular areas of subject expertise, as they would be contracted to offer only certain subjects at GCSE and A level. A franchised system would make it easier for learned bodies, higher education and employers to engage (they would only need to engage with one exam board per subject) and would also avoid the duplication of syllabuses and dilution of examiner expertise that exist in the current system.

76. There are some very significant issues relating to franchising that would need to be taken into account. Franchising would require substantial Government and regulatory input to draw up and award contracts to providers. Glenys Stacey of Ofqual told us that experience (of regulating National Curriculum assessments and in other sectors) would suggest that

you have to make a very real and significant investment in getting the detail of the specification, the contract, and the bidding and tendering processes right [...] my advice would be to concentrate very hard on getting the legislation right, and getting the mechanics and technicalities of it right. It would be a significant and complex matter.[114]

The burden would be on those who procure to specify correctly, as the success of the system would depend upon it. Exam boards would have to devote significant resource to the bidding process, which requires a different skill-set to that involved in running exams. The impact on unsuccessful bidders should also be considered.

77. Glenys Stacey also advised that the Government should "truly evaluate the risks and [...] recognise the trade-offs." Pricing, in a franchised model, would need close monitoring: "one might expect that pricing would be a dark art, or would be lacking in transparency" and "there is likely to be an increase in pricing", in comparison to the current system.[115] Exam boards would include a premium for risk and to cover the implications of changes in government policy (which, recent experience suggests, would be highly likely during the lifetime of a contract).

78. A particular concern would be how to encourage innovation, and the associated financial investment, such as improvements to question paper setting processes. Innovation is commonly cited as one of the benefits of multiple competing exam boards. As Cambridge Assessment observed, "whenever one board achieves an advance, the others compete vigorously to catch up and overtake".[116] Within a franchised system, innovation linked to service delivery would need to be built into the terms of a contract, as would the provision of small entry subjects.

79. As well as removing competition on syllabus development, one of the main benefits of franchising, acknowledged by many, would be the concentration and development of subject expertise within an exam board.[117] This would encourage a focus on enhancing the quality of syllabuses and their associated materials, rather than "just meeting" the minimum regulatory requirements. The flip side is that, as Tim Oates argued, "over a period of time, the expertise becomes entirely concentrated in individual institutions [...] your choices then become profoundly limited".[118] Glenys Stacey described this as "a one-way street. When you get towards the end of a franchise period, it is much harder to attract true competition and real bidders".[119]

80. Several witnesses pointed out to us that there are salutary experiences of how a franchised system can go wrong to be found in the history of National Curriculum tests.[120] Should the option of franchising be pursued, the Government needs to consider the legal implications of a franchised system very carefully, including the need to constrain schools' choices of syllabuses to those provided by the franchisee and to prevent other exam boards from marketing alternative "equivalent" qualifications. Any major structural change to the exam system needs to be managed with extreme care in order to minimise the disruption caused, bearing in mind that the greatest threat to standards is at moments of change and that the bigger the change, the bigger the risk.

CONCLUSION ON OPTIONS FOR CHANGE

81. We believe that the current system incentivises downward competition on content standards and we recommend that the Government act immediately to change these incentives. We consider that national syllabuses would offer a way of addressing downward competition on content and provide reassurance on standards, without the risks, lost benefits and disruption involved in moving to a single board. The Government should begin by piloting a national syllabus in one large entry subject as part of the forthcoming A level reforms. Ofqual should review the effectiveness of the pilot, with a view to extending the approach across GCSE and A levels if appropriate. We believe that national syllabuses, coupled with a stronger Ofqual and greater involvement of subject communities in GCSEs and A levels, should help to maximise the benefits of having multiple competing exam boards while minimising the shortcomings.

82. While we can see that the second option we outline—franchising of subjects to exam boards—offers a way to address downward competition on content, we have concerns about the long-term impact and suggest that there may be serious downsides to such a change that need to be better understood before it can be recommended.

Grading standards and grade inflation

83. The Wellcome Trust told us that "it is important that the public, employers and universities understand and have confidence in the process by which grades are arrived at. Yet little is known publicly about how awarding bodies proceed from marked scripts to final grades".[121] The issue of grading standards is complex and the awarding of grades a technical process, involving a blend of the professional judgement of examiners with sophisticated statistical data. It is a difficult area, which quickly becomes impenetrable to non-specialists. Therein lies much of the challenge for the exam boards and Ofqual when seeking to provide reassurance about grading standards. Furthermore, as Glenys Stacey observed,[122] the language used to discuss grading standards, including commonly used terms such as 'grade inflation', is emotive and loaded. Recently, Ms Stacey has been more outspoken, stating that there has been "persistent grade inflation" at GCSE and A level.[123] It is an area of long-standing debate and one that has strong influence over public confidence in the exam system.

84. When considering grading standards, we were concerned with several questions. First, whether increasing numbers of students achieving higher GCSE and A level grades reflect genuine improvements in standards of attainment; second, how the system of competing multiple exam boards has contributed to the year-on-year increases in results. Third, how the system should balance the requirement to maintain standards over time with the need to adapt to ensure that GCSEs and A levels remain fit for purpose.

INCREASING NUMBERS ACHIEVING HIGHER GRADES

85. Rising GCSE and A level pass rates and increases in the numbers of candidates achieving good grades are well-documented and have prompted ongoing debates about grade standards. Figures 1 and 2 below show the trends in A level and GCSE pass rates since their introduction in 1951 and 1988 respectively.[124] Professor Alan Smithers observes that in the past 30 years, the A level pass rate has increased from 68.2 per cent in 1982 to 97.8 per cent in 2011. He notes that since norm-referencing was superseded by criterion-referencing in 1987, the proportion of candidates achieving a grade A has increased from under 10 per cent to 27 per cent in 2010.[125] Professor Smithers comments that the GCSE has seen "almost continuous grade improvement in the twenty-four years of its existence".[126]

Figure 1: Trends in A-Level Entries and Passes

Source: A levels 2011, Alan Smithers, Centre for Education and Employment Research, University of Buckingham, 2011

Figure 2: Trends in GCSE Passes

Source: GCSEs 2011, Alan Smithers, Centre for Education and Employment Research, University of Buckingham, 2011

86. The National Association of Schoolmasters/Union of Women Teachers (NASUWT) told us that "there is no evidence that standards in the qualifications used most regularly in the 14-19 sectors have fallen over time".[127] However, few other stakeholders, including assessment specialists, appeared so confident. Doubt is cast by the critical comments from universities and employers (see chapter two) and by the contrasting static performance of English pupils on international surveys such as PISA. Tim Oates of Cambridge Assessment stated in a debate on exam standards in 2010 that there is "sufficient evidence from diverse sources to generate concern" about grade drift.[128]

87. The DfE pointed to the findings of research by Robert Coe and Peter Tymms at Durham University, which, it told us, "raise important questions about grading".[129] The research found that between 1996 and 2007 the average grade achieved by GCSE candidates of the same "general ability" rose by almost two-thirds of a grade; for A level candidates the increase between 1988 and 2007 was over two grades.[130] The Secretary of State warned last October and again recently that "there will be years, because we are going to make exams tougher, when the number of people passing will fall".[131] This has led to discussion in the media of the prospect of "grade deflation".[132]

88. AQA's Centre for Education Research and Policy (CERP) argued that grade awarding procedures and checks, along with associated research exercises, are "robust, efficient and effective" and that, given the rigour of the current grading system, "confidence in standards in general, and in inter-year and inter-AB [awarding body] comparability in particular, should be high".[133] Exam boards explained to us how they share data with each other and the regulator at key stages prior to the publication of results. While exam board chief executives defended their awarding procedures as robust, they acknowledged that "whilst actual grades have been going up, our performance on international league tables has not" and that "there has been a creep in grading standards".[134]

89. AQA suggested to us that

the progressively high-stakes nature of general qualifications partly explains the year-on-year increases in results, which amount to a handful of extra students in each school exceeding the grade boundary each year. Teachers focus intensively on supporting borderline students, aided by the increased availability of mark schemes, past papers, information and support, and transparency as to the skills and knowledge required and how to demonstrate them [...] as education becomes increasingly centred on passing examinations, outcomes go up.[135]

90. Glenys Stacey was recently reported as saying that "the grade inflation we have seen [at GCSE and A level] is virtually impossible to justify".[136] We explored with assessment experts the elements that have most contributed to increasing numbers achieving higher grades. Several factors were cited. These seemed to us to fall into four broad categories: genuine improvements (for example, more efficient teaching, better quality teachers, harder-working pupils and pupils with better support from home), changes in the structure and design of qualifications (for example, modularisation, re-sits), factors linked to underlying drivers such as the accountability system (a culture of increasing results and expectation of increased performance, schools focusing heavily on GCSEs, teaching to the test) and finally factors linked to exam board procedures (greater transparency, improved reliability, examiners giving students the "benefit of the doubt", tolerance of small increases year-on-year). We note the point made by Tim Oates, that there is a lack of evidence identifying which of these factors lead to actual improvements in underlying attainment (as well as improved grades).[137]

91. Increasing numbers of students achieving higher grades in GCSEs and A levels is an undisputed trend. What is less clear is whether this is matched by improvements in knowledge, skills and understanding among students. While we appreciate that this is a technically complex area, we agree with SCORE that "the impression that standards are either slipping or becoming incomparable between year groups cannot be ignored".[138] More young people may be doing well because they are better prepared to pass their GCSE and A level exams. However, we feel that there is sufficient evidence from a variety of sources, such as universities, employers, England's flat PISA profile and research by Durham University cited above, to cast doubt over whether GCSEs and A levels indicate improved preparation of young people for further education and employment and whether higher grades reflect genuine improvements in their underlying knowledge, skills and understanding. The findings of Ofqual's most recent set of GCSE and A level standards reviews cast further doubt in this respect. Ofqual reported that many of the reviews "raise concerns about the maintenance of standards" and that in several cases changes to the way content was assessed had reduced the demands made by exams.[139] This doubt devalues qualifications and young people's achievements and undermines public confidence in the system. Ofqual should continue to investigate grading issues as part of its programme of standards reviews and to engage publicly with debate on exam standards. Ofqual needs to be able to account for what AQA's Andrew Hall described as the "creep in grading standards", particularly in the commercially significant large entry subjects at GCSE, which are key to schools' performance in league tables and also in large entry A level subjects, commonly used for university entrance.

92. We can see that the requirement on Ofqual and the exam boards to maintain standards over time may present challenges when trying to ensure that GCSEs and A levels adapt and remain fit for purpose. A levels, for example, cater for a broader ability range, with larger numbers going on to university, than they did thirty years ago. As Professor Nick Lieven of Bristol University told us, "the median area that you are trying to focus the qualification on has shifted: it has to, in order to be fair".[140] We suggest that occasional explicit recalibration of grading standards may be required and is preferable to slow creep downwards or upwards. Slow drift or creep in grading standards, such as that seen over recent years, is difficult to interpret and leads to the system courting controversy, with obvious consequential risks to public confidence. We return to this in chapter five.

COMPETITION ON GRADING STANDARDS

93. There is clearly a public perception that exam boards compete on grading standards. The Wellcome Trust told us that "it seems likely that grade [boundaries] have reduced in part because the awarding bodies are competing for custom and teachers are likely to choose those qualifications that will yield the best performance for their schools and for their students." The Trust cites evidence of one exam board marketing their syllabus as "proven to help improve grades".[141]

94. According to the Wellcome Trust, "Mark Walport, chair of the Science and Learning Expert Group, observed that when giving evidence [to his inquiry], awarding bodies openly admitted that they struggle to avoid competing with each other on grade standards".[142] This was not our experience. Exam board chief executives were emphatic that they do not compete on grading standards, telling us that "there is no scope for competing on standards" and that "it is the standards agenda that brings us into a collective, but we compete on everything else".[143] A key factor in this appears to be recent action taken by Ofqual to control grade inflation. Since summer 2010 Ofqual has monitored the interim outcomes from summer awards to secure comparability between exam boards and minimise any grade inflation. As Tim Oates explained, "the regulator has increasingly said, 'if you're erring, err in this direction, not that direction'".[144] The approach seems to have been broadly welcomed by assessment experts and the exam boards. AQA's Andrew Hall told us "we share data—we share it with the regulator—and we have made a great leap forward now".[145] Ofqual has said that it is "committed to containing grade inflation, whilst making sure that awards reflect accurately students' achievements"[146] and that it plans to adopt the same approach to the award of GCSEs in summer 2012.

95. We welcome Ofqual's recent action to regulate grading standards and recommend that it continue with this approach for A level and, from summer 2012, for GCSE. The effect of this action is twofold: first it helps to control grade inflation and second it provides reassurance that the exam boards are not competing on grading standards. We recognise that the effect will take time to filter through the system and to help increase public confidence.

97 Ev 114 paragraph 2.6 (AQA) Back

98 Ev 153 Back

99 Q445 Back

100 Ev 170 Back

101 Jon Coles, speaking at Cambridge Assessment event "Learning comes first: shifting the focus from examining to the curriculum", 31 January 2012 Back

102 Ibid. Back

103 Q646 Nick Gibb Back

104 Ev 155, paragraph 7 bullet 2 Back

105 TES forum post, 27 October 2011 Back

106 Q537, Andrew Hall, Back

107 See for example, Ev 128, Ev w78, Ev 153 Back

108 Q603 Back