Education CommitteeWritten evidence submitted by the Oxford Cambridge and RSA Examinations Board (OCR)

Introduction

1. OCR is a major, not for profit, UK Awarding Body and part of the Cambridge Assessment Group.

Ensuring Accuracy of Setting, Marking and Awarding

2. The select committee inquiry uses the concept of “accuracy” in setting, marking and awarding and we have focussed our evidence accordingly. Nevertheless, there are important wider principles of assessment design, especially, validity and reliability. These are enshrined in “The Cambridge Approach” which sets out the fundamental principles of assessment followed by OCR. The approach links validity and reliability closely:

While validity relates to whether the assessment is assessing what it is intended to assess, and whether the inferences drawn from the results are well-founded, reliability relates to the stability of the assessment, for example, whether on different occasions or using different markers the same outcomes are related in the same way. If validity is poor, reliability in the assessment will be to little effect. If reliability is poor—and results are unstable—validity is compromised.

3. Getting the design of assessments right and ensuring the fairest possible outcomes for learners are fundamental, but so is deciding what should be assessed in the first place. The syllabus or programme of learning is key to qualification development. No amount of precision in print, or accuracy in marking will deliver a good education.

Accuracy of Setting

4. During the summer, three OCR papers included errors, which were not identified until the examinations were being taken and had the potential to unsettle candidates during the examination. Although the errors should be seen in the context of an industry which sets over 60,000 questions a year, we accept that everything must be done to prevent such errors in the future.

5. OCR conducted a thorough internal investigation into its processes for developing and quality assuring exam papers. A report of that investigation and its conclusions can be viewed at http://www.ocr.org.uk/download/news/ocr_62383_news_qp_errors.pdf. We also set out publicly on our website the steps taken during the awarding process to ensure that any candidates affected received the grades they deserved. This process involved analysis of individual scripts and the way that an error may have impacted on overall performance. We also took into account the achievements of candidates on other papers. These practices are long established and have been developed over years to account for occasions when, for example, severe weather, have prevented candidates from sitting an exam.

6. The errors and the publicity surrounding them generated many enquiries from anxious parents and students; our arrangements for double checking and then appealing against results were widely shared. However, the number of formal inquiries did not rise dramatically and the number of formal appeals seems to be fewer than last year.

	14 October 2011	15 October 2010
Number of appeals accepted	92	120
Number of appeals finished at stage 1	30	53
Number of appeals progressing to Stage 2	4	0
Number of appeals closed at stage 2	2	0
Total closed	28	53
% of completed/logged	30.5%	44.2%
% of Result Enquiry appeals competed within QCA timescale (50 days)	100%	100%

7. Results enquiries:

Enquiry Type	Number of Enquiries June 2011	Number of Enquiries June 2010
Priority Service 2—for candidates entering HE	4,553	3,901
Service 1—clerical re-check on the script	456	623
Service 2—clerical re-check and a review of marking	34,817	33,154
Service 3—review of moderation	691	638
Total	40,517	38,316
	Increase of 5.74%

8. Ofqual has carried out its own investigation, and it concurs that there was no evidence of non-compliance with established processes or of any systemic breakdown in processes. The evidence shows that each error occurred at different points in the process with no single root cause. We will work with the regulators to ensure we adopt more finely tuned risk management and clearer lines of accountability in the checking process. We may conclude that current regulations, set out in the Code of Practice, tie us into particular processes that do not allow us to eliminate risk through alternative processes.

Accuracy of Marking

9. Where a candidate or school is surprised by an exam result, they are likely to suspect that something has gone wrong with the marking of the paper. Indeed, the vast majority of inquiries and appeals are about marking reliability. Much effort has been spent in securing consistent and reliable marking across examiners, and approaches have been developed and refined over many years with substantial investment in new technologies considerably increasing the ways in which performance of individual examiners can be monitored and quality and consistency of marking improved.

10. Marker-related reliability has been an important strand of Ofqual’s Reliability Programme. As part of this programme, Cambridge Assessment carried out significant research into marker agreement (Bramley & Dhawan, 20101). The study confirmed the view that the level of marker reliability depends on the nature of the subject. Examination units/components consisting of structured, short answer questions with more constrained mark schemes were marked more reliably than those consisting of longer-answer essay questions with more open-ended mark schemes. Using both data collected in the “old” paper-based system and the “new” on-screen marking system, comparisons with previous research studies dating back to “O” Levels supported, or at least did not run counter to, the claim that marking of public examinations has become more reliable over the years.

11. The increasing use of on-screen marking technology for both paper and computer-based assessments has made it possible to introduce significant improvements. These include the ability to monitor the quality of marking of each examiner against a set of common, “definitively” marked scripts on an on-going basis; the ability to anonymise and randomise the allocation of scripts; the availability of more detailed evidence upon which to decide whether an individual examiner’s marking is “aberrant”; and the ability to allocate re-marking immediately without the delays caused by the need to despatch paper scripts.

12. But there is a philosophical point about how far we seek to design papers which elicit absolute reliability from examiners. This can lead to detailed, prescriptive mark schemes that reward compliance over originality, and encourage teaching to the test. Evidence from OCR’s Higher Education forums indicates a concern that schools teach highly formulaic approaches to writing essays. Mechanistic assessment may be accurate, but it doesn’t always encourage deep learning.

Quality and Supply of Examiners

13. We monitor the quality of examiner performance closely using processes already outlined. Any examiner not marking to standard is stopped from marking, and their marking is remarked. The performance of the remaining examiners is measured and graded on a scale of 1-5. Currently:

80% of examiners are graded 1 and 2; and

15% of examiners are graded 3.

Examiners graded 4 are offered given additional support and are closely monitored. The small number graded 5 are not invited to mark again.

14. The sourcing of sufficient, high quality examiners is critical to the success of the examination system and often identified as an area of potential risk. Annually, OCR engages 13,000 examiners of which the vast majority are practising teachers (appendix A).

15. Our planning and recruitment consistently secure sufficient, qualified examiners but we are dependent on the support and encouragement of schools and colleges in promoting the benefits of examining to their staff.

Accuracy of Awarding

16. The awarding process involves expert judgement informed by comprehensive statistical and qualitative evidence. During awarding meetings experts review evidence including: exam papers from past series, archives of past scripts, information about mark distributions, estimated grades submitted by schools, and details of previous candidate performance. The full range of evidence that must be used in awarding is set out in the Ofqual Code of Practice, provided as appendix B. During awarding meetings the grade boundaries are set, subject to review and sign off by a named accountable officer

17. Consistency of awarding between awarding bodies is, in part, secured by common design criteria, at qualification-type and subject level. Also, JCQ reviews percentages of grades achieved across Awarding Bodies by subject and informs them of any variance. As the regulator, Ofqual monitors any variation between boards.

18. The increasing use of new technologies provides much richer data and the capacity for complex algorithms to assist judgements.

Recommendations

19. The awarding body industry needs to strive to be as open and transparent as possible about the workings of the exam system, its virtues and its limitations.

20. It is critical for teachers to be involved in the assessment system. We recommend that Heads and Principals release teachers more easily and more frequently to allow their individual development and better system integrity.

21. Awarding Bodies must continue to invest heavily in new technology. We believe such major investment is best managed by the awarding bodies; there is no evidence that this major development would be better managed as a government IT project.

22. Ofqual should concentrate on the core function of monitoring standards as this is critical to securing an effective market in qualifications.

Commercial Activities

23. This section sets the context for the commercial environment of the exams system and provides information about our fees, our relationship with publishers and our support for teachers.

Qualification Fees

Context

24. The scale of the examination system is vast. OCR alone despatched 8.5 million question papers this year, scanned more than 60 million pages of candidate scripts, processed 95,500 requests to access scripts, allocated and monitored, trained and remunerated 16,000 assessment personnel (appendix C).

25. OCR has developed new technologies to support these activities and technology will continue to be a major source of investment for OCR—we have embarked on an ambitious five year programme to develop capabilities which anticipate a transformation in the way learning and assessment will look in the future.

26. We also face costs from government intervention and an increasing regulatory burden. Over recent years we have been subject to numerous initiatives including:

the introduction of the A*;

the breaking of A Levels into six units and then reducing them to four;

fluctuating apprenticeship frameworks;

literacy and numeracy qualifications yoked to GCSE maths and English and then unyoked at the last minute;

the introduction of a complex and flawed Qualifications and Credit Framework;

the redevelopment of about 50 GCSEs to a modular structure, awarded the first time this year and to be replaced, as an interim measure, with more linear versions, alongside new Spelling, Punctuation and Grammar requirements (although Wales and Northern Ireland may opt to continue with the modular GCSE, doubling the number of GCSE examinations awarding bodies would have to run);

new 14-19 Diplomas complete with a new IT system to support it;

the introduction of “controlled assessment” to replace coursework;

fluctuating rules on re-sits;

a stalled Foundation Learning Tier;

a fudged withdrawal of NVQs; and

the advent and passing of GNVQs.

27. Meanwhile we are preparing for completely new GCSEs to accompany a new National Curriculum in 2014. Some of these developments are entirely reasonable or, like a new National Curriculum, hold out the prospect of greatly improving our education system, others have been highly misguided. But the main point here is that this level of flux brings with it high operational and development costs and potential risks to the system. At the same time, government is rushing through legislation to give Ofqual powers to fine awarding bodies should anything go wrong.

28. We have been subject to some form of regulation for nearly 20 years. The past is littered with organisations such as the Schools Assessment and Curriculum Authority (SCAA), the National Council for Vocational Qualifications (NCVQ), QCA, QCDA and now, Ofqual. Each iteration has added new burdens and layers of criteria and codes of practice. Ofqual plans to be a strategic, high level regulator, but, with its 52 “general conditions of recognition” incorporating 161 criteria, it is mired in detail.

29. Fortunately, the market for qualifications has been strong in recent years: public funding policies and school performance indicators have rewarded qualification achievement; the modular design of qualifications has increased the amount of assessment that leads to a qualification. This has enabled us offset costs and to keep its prices in line with inflation (appendix D). However, it is clear that limits on public funding, changes to funding models and a shift away from modular assessment mean OCR is budgeting for a loss over the next few years. At appendix E we also provide tables to indicate the proportion of income generated by re-sits and a comparison of charges between OCR and its main competitors at appendix F.

Arrangements with Publishers

30. OCR’s focus is syllabus development, and assessment. However, we recognise the importance of support materials and text books in supporting teachers. Therefore we ensure, through partnership, and for no financial gain, that a full range of the best publishers develop and make available this support.

31. OCR works with publishers in two ways:

Formal partnership.

Endorsement.

Publisher Partnerships

32. The arrangement is simple—for subjects covered by partnership, formal partners have access to OCR staff and the qualifications they are developing from the start of qualification development through to completion.

33. There are benefits to both partners: the publisher can get to market early, having had early sight of the qualification; OCR is assured that it can launch a new qualification complete with quality support materials.

34. There is a further benefit to learners. In agreeing partnerships, OCR is able to negotiate with publishers the range of subjects they will cover. This guarantees that there are support materials for all the range of qualifications on offer, not just those high up-take subjects likely to provide publishers with a high return.

35. OCR currently has formal partnerships with three publishers. The partnerships were originally awarded in 2006 after a formal process involving expressions of interest from 33 publishers. The number of large educational publishers has declined significantly in recent years due to mergers; without being definitive, about six remain independent of each other. One belongs to a competitor, another had, until recently, an exclusive deal with another competitor.

Endorsement

36. There are benefits to publishers in having their materials endorsed by OCR. At the same time, OCR wishes there to be a strong and varied range of support materials for all its qualifications. That is why we actively promote the endorsement process and charge minimum costs.

37. OCR will “endorse” text books and other support materials where they provide appropriate support to an OCR qualification. Endorsed materials are allowed to carry the OCR logo.

38. For a publication to be endorsed, it must go through the endorsement process. This process is used to confirm that the content has sufficient coverage of the OCR qualification at the appropriate level.

39. Any publisher or individual author can submit materials for endorsement. OCR charges a small administrative cost and for the time taken to review the materials. The average charge is £230.

40. Publisher partners must also use the endorsement process before any of their materials can carry the OCR endorsement.

e-books

41. OCR recognises that the cost of text books is a big issue for our customers so, in a recent initiative we are providing e-text books free of charge to schools and colleges. There is no charge to schools taking up the offer, nor do we require them to be a customer of OCR’s to access the free books. The arrangement involves us paying a fee to the relevant publisher for e-books provided. Since we introduced the offer in September we have had orders for over 150,000 e-books from about 1,500 schools and colleges.

Training for Teachers

42. OCR provides around 1,200 training events for teachers annually, serving approximately 20,000 delegates. We have four kinds of events, Get Ready—compares a previous OCR specification to a new OCR specification, Get Started—an introduction to an OCR specification, Get Ahead—for teachers already experienced in teaching the qualification and Lead the Way—training designed to improve and inspire teachers with new approaches to teaching subjects (not necessarily qualification linked).

43. We recognise the importance of supporting teachers in starting out with new qualifications so our policy is to provide free training for teachers during the first 18 months of a new specification. This means that about half of all our events are provided free of charge and overall our programmes of training run at a loss.

Conflicts of Interest

44. We take potential conflicts of interest very seriously. When we contract with examiners we require them to sign up to a set of rules which prevent them from exploiting their position as an OCR examiner in any way. For example, if they have authored a book they are not allowed to use the fact that they are an OCR Examiner in publicising the book. Nor are they allowed to publicise any training they might offer privately to teachers as coming from an OCR assessor. We have similar rules to control the risk of disclosure of any confidential information about exam papers under development. Any breach leads to dismissal. The relevant clauses in examiner contracts are provided in appendix G.

November 2011