The administration of examinations for 15 to 19 year olds in England

Written evidence submitted by The Mathematical Association

1. The Mathematical Association is the oldest of the subject associations and the largest such association in Britain supporting mathematics teachers in their endeavour to promote good mathematics teaching and learning. We are pleased to be offered the opportunity to present our views on such an important matter as the administration of examinations. The views and evidence presented below have been gathered and constructed by the association’s Teaching Committee, chaired by Dr Chris Pritchard. The association is a broad church and not every suggestion made below is universally held across its members.

2. The Mathematical Association believes that in considering the best approach to the administration of examinations the over-riding factor should be the raising of standards so that the system attracts the confidence of higher education, of business and industry, and indeed of the general public, whilst the quality of teaching and learning is driven upwards. Further, we feel that the three specified questions cannot be answered without first clarifying the prerequisites for any model for the administration of examinations.

3. As will be seen from the arguments below, we see potential advantages in working towards a single awarding body. However, there may prove to be considerable barriers to such a move in the short to medium term. So whilst expressing a general preference for a single body, we also suggest an alternative, competitive model which we believe would be a significant improvement on the status quo.

Principles for constructing a model for the administration of examinations

4. When considering the formalisation of examinations, thought should be given to philosophical questions relating to each subject area, such as, for mathematics, ‘Are we teaching the subject because it is an intellectually worthwhile and culturally valuable subject to learn, with a unique perspective on the generation of knowledge? This will set the context within which the design work is undertaken.

5. The examination system should operate for the benefit of the learners (and their carers) and end-users of the public examination system; this will not only influence the model but should ideally reduce the range of choices available to centres.

6. Policy formation and analysis should emerge naturally out of a permanent core of subject expertise within the Department, with teachers and universities making a significant contribution and full interaction with the wider community, including drawing on the expertise developed within the awarding bodies.

7. Candidates’ results should reflect their performance. That performance must be judged against standards and be comparable to similar performances both across awarding bodies, if more than one exists, and over time.

8. There should be an understanding that examining is a craft rather than a science and that examination results are never wholly reliable. An effective system requires an effective bureaucracy; but this is never sufficient. There is a need for competent professionals to exercise judgement, both at the setting/checking and at the marking stages.

9. To avoid the mechanisation of examination administration and decrease the chances of mistakes being overlooked, the processes of setting and marking should focus on content rather than procedures.

10. The model must be responsive to, if not anticipate, changes in society and in education and must have built in, the opportunity to innovate. Such opportunity should not be confused with the opportunity to make capital by constantly changing examination specifications so that new textbooks are needed.

11. The whole system must have transparency. There must be a climate in which questions are answered openly. Key meetings should not be held in camera (as was the case with the QCA Annual Meeting, for example, during the A-Level grades crisis in 2002).

The Current Competitive Model

12. The Mathematical Association is concerned that the current competitive model may be creating downward pressure on genuine standards.

13. The market for the administration of examinations is a lucrative one with each awarding body seeking to maximise its share of the cake, particularly in large-entry subject areas. In order to attract as high a proportion of each examination cohort as possible to take its papers, the ‘products’ need to be as appealing as possible. There is evidence, partly demonstrable, partly anecdotal, that the awarding bodies tout for business on the basis of lower cost, easier papers and more sympathetic marking régimes. The first of these incentives has a clear financial attraction to schools and local authorities and need not influence standards, whilst the last two offer the potential for better examination statistics, though at the expense of standards.

14. It is possible that this downward pressure on standards exists without market forces actually affecting choice. Although we have multiple specifications, awarding bodies cleave to the middle of the market to maximise market share; that is, there is little variety amongst the choices available. Furthermore, it is a major upheaval for a school or college to change awarding body and that makes not only for a viscous market but one where there is not much competition on price. (Ofqual’s market reports show very little movement in market share over time but they do not indicate the degree of churn, the net effects.) If it is true that in seeking a larger share of the market, question difficulty and marking standards are depressed, whilst the differential in price across the awarding bodies is too small to convince schools to switch, then there is a hit on both educational standards and the public purse.

15. With regard to the maintenance of standards and confidence in standards, perception is at least as important as substance. We recognise that the examinations of yesteryear, designed for an élite intellectual group within each cohort, cannot be compared with their ‘equivalents’ today, written for a broader achievement spectrum. Nevertheless, there is evidence of decline, see for example the work of Robert Coe in Durham and Tim Oates at Cambridge Assessment (presentations by both of them at the Ofqual conference on 13/10/11 are available on the Ofqual website).

16. One further specific point perhaps worthy of note is that the current arrangements appear to suffer from reduced interactions between awarding bodies and both university academics and experienced school teachers. Links with universities have largely been severed and teacher-involvement has changed markedly, so that the best of those who contribute no longer feel that their professional competence is being valued. The resulting process is increasingly driven by bureaucratic concerns relating to managing somewhat mechanistic marking processes and towards removing the need for candidates to ‘join the dots’ for themselves. Furthermore, the unnecessary multiplication of specifications dilutes the pool of examiner expertise.

17. In summary, it would appear that the current situation, in opening the way for continuously decreasing standards, challenges the view that public sector activity benefits from competition and private sector input.

Case for a Single Awarding Body

18. The Mathematical Association can see potential advantages in working towards a single awarding body. (In making the case, we acknowledge that such a position is not held universally in the association.)

19. Examination boards were established by the universities in the second half of the nineteenth century and the early years of the twentieth. By the 1960s there was a huge number of boards for CSE, O Level and A Level and the trend since has been to consolidate, right down to the current three bodies. The move to a single body could be seen as the inevitable and logical conclusion of such a process. And indeed, it would be hard to believe that should we be establishing a system for the first time now, we would consider anything but a single awarding body model.

20. Across the world, it is far from unusual for countries to have a single awarding body, including countries larger than England. Of smaller nations, our nearest neighbours, Scotland, Wales and Northern Ireland operate single body models successfully. Interestingly, the Scottish Qualifications Authority commands a high level of public support and respect despite the occasional highly-publicised problem arising.

21. Calls for a single body are not new. The inspectors’ reservations about GCSE standards in 1992 led to calls for a single awarding body, and these calls were repeated following Mike Tomlinson’s review of the 2002 crisis. And there is precedent for a lack of choice: the KS3 tests were common to all students and did not seem to be controversial in terms of there being no choice of examination.

22. The advantages of a single body include,

· simplification: a single awarding body would likely lead to fewer specifications, removing at least those that exist for purely competitive reasons,

· reduced costs: the production and administering of one set of papers is inevitably less than those of three sets,

· supporting innovation: the reasons for a minimal change model would be removed,

· concentrating expertise, which also brings further efficiencies,

· consistency: year-on-year consistency will never be fully under control but the need to make comparisons across multiple bodies would be eliminated.

23. However, we are mindful that should only one organisation produce papers, it is essential that the available level of expertise is such as to ensure that they are of the highest quality, encourage good teaching and support innovation. This may imply a long lead time to such a change.

An Alternative Competitive Model

24. We attempt below to describe a competitive model different in nature to the current one. It is not expected that such a model would be adopted in its entirety, but hope that elements of it will resonate with those designing the new model.

25. According to this model, examinations would continue to be packaged as commodities, though those elements relating to standards would be reserved to public control (but not necessarily execution). The writing of specifications would be in the hands of user communities under public co-ordination and supervision. This would include provision for experimental specifications.

26. Those elements of examining a specification which are standards-specific (paper setting, mark scheme writing, standardisation of marking, grading) would be separated out. A public body (perhaps DfE or Ofqual) would award contracts for these for fixed periods to contractors for specific subjects or groups of subjects. The price structure of the contracts should be such as to give no incentive to the contractor to increase or decrease the uptake of their subject(s). The present awarding bodies would be able to bid for the contracts but would need to separate the operation from their other activities (e.g. no shared staff). Others would also be able to bid, perhaps NFER or institutes of higher education, or suitable consortia. The elements being included in the contract should be such that there would be no need for a large infrastructure so as to open up the market. The contracts would be awarded on a quality basis rather than cheapest tender. The awarding bodies would continue to badge the papers and market specifications (more than one could offer the same specification), distribute question papers, mark them, and distribute results and certificates, as well as providing customer service.

27. Measures would be put in place to assure standards of marking across awarding bodies, and this would be supervised rigorously by Ofqual, possibly including an element of cross-marking between awarding bodies (on an anonymous basis). The awarding bodies would licence the question papers etc from the public ring-holder and therefore would have no direct commercial dealings with the contractors. Under such an arrangement awarding bodies would compete on price and quality of service. Changing awarding body would be less traumatic than at present and this would lead to a more fluid market in which centres or consortia of them could seek to drive down cost. New players might also enter the market.

28. The interface between the contractors and awarding bodies would be well-defined and include suitable penalties in contracts to ensure it is in the financial interests of all parties that the system works well. Such an approach would suggest it was important to ensure the market is sufficiently diverse that players do not emerge who are so big that their failure would capsize the system.

Franchising individual subjects

29. The option of franchising individual subjects appears to encourage inter-subject competition and have the potential to exert a downward pressure on standards. If one body has the franchise for Subject A and another has the franchise for Subject B, then to attract the floating students who are wavering between A and B, each awarding body will want its examinations to be perceived as the easier. To protect their income, both bodies would be inclined to drop their standards in response, thus triggering a downward spiral.

Ensuring accuracy in setting papers, marking scripts, and awarding grades

30. We offer views below on four issues: criterion referencing of questions, the practice of marking, the drawing of grade boundaries and the handling of causes for concern.

31. Accuracy in matching assessment items to curricular expectations is difficult to sustain and regulate when the awarding body is driven by the economic imperative of passing as many candidates as it can. A single body responsible for the development, trialling and implementing of examinations for all students at a given assessment stage is therefore preferred. The body should be directly responsible to the Department, thereby reducing bureaucracy, and all who work within in it should be qualified in assessment.

32. The criteria and often the specification are often laudable, but the questions and mark schemes frequently fail to reflect them fully. There should be less predictability in what candidates face and greater freedom for markers to exercise professional judgement when awarding marks. The arrangements should actively dissuade teachers from advising candidates that when in doubt, present two or more conflicting solutions in the hope that one of them might be marked correct.

33. Grade boundaries would be determined on an annual basis by the experts working in the examination team. They would draw on moderated samples of students’ work and, importantly, a process of matching with curriculum expectations, custom and practice.

34. For decades, concerned teachers have written to awarding bodies each year complaining (often with considerable justification) of ‘unfair’ or inaccurate examination questions; and each year the exam board would consider carefully how to reply, and seek to learn whatever lessons needed to be learned. More recently such ‘errors’ have been communicated directly to the national press. It may well be that the number of serious errors is not very different from what it has always been. Or it may be that the massive increase in the extent of assessment (as a result of modularisation and repeat sittings) has increased the stresses on the checking system, leading to more frequent errors. We encourage the Committee to talk to awarding bodies to get an accurate picture of actual trends.

The commercial activities of awarding bodies, including examination fees and textbooks, and their impact on schools and pupils.

35. There is some concern about the detrimental effect on secondary education of the increasing involvement of awarding bodies in the preparation, endorsement and publication of textbooks.

36. We are also uneasy that textbooks are prepared with contents limited to what is needed for a specific examination and then endorsed by the awarding body. Lacking depth and covering a limited range of subject material, such textbooks give students little incentive or opportunity to engage with broader and richer material; they do not foster an appreciation of the subject's subtleties. This practice results in reducing the intellectual endeavour of learning to uninspiring repetition aimed squarely at passing the relevant examination; it narrows students’ horizons, sets for them false criteria of personal achievement, and de-professionalises teachers.

37. To this has recently been added another concern – namely the increasing involvement of awarding bodies in Continuing Professional Development (CPD) of teachers. This trend combines with the links between examination boards and production of textbook to reinforce the tendency to ‘teach to the test’. CPD sessions run by awarding bodies are likely to focus on explaining to teachers how to train students to maximise marks in examinations set by one particular board, while a textbook endorsed by the same awarding bodies leaves the teacher no option but to coach students for the relevant examination.

38. In summary, we would welcome amendments to the regulatory framework that forbid an awarding body from producing materials for which it charges, and requires that a textbook author’s link to an awarding body be publicised. We would also welcome clear constraints on the kinds of CPD courses and events examination boards are free to advertise. In particular we would welcome an introduction of a rule (similar to the one existing in the Civil Service) that senior employees and examiners of awarding bodies should be forbidden to publish educational material or deliver certain forms of CPD (based on their inside knowledge) both during their employment in an awarding body and for a specified period after termination of their contract.

November 2011