Education CommitteeWritten evidence submitted by DRS, Data Services Limited
Executive Summary
This submission presents the case for multiple Awarding bodies in England on the basis that it encourages innovation which results improved quality, standards and efficiency. This document demonstrates that through the use of innovation, (ie online marking of examination of scripts) marking quality has been improved, marking efficiency has been increased and costs driven down. This innovation should be encouraged to track the complete assessment process from examination creation to awarding in support of the human endeavour and reduce the opportunity for errors.
Introduction to DRS
Established in 1969 and publically listed in 1993, DRS has over 40 years of experience in providing technology based, assessment services to the Education sector both in the UK and overseas. DRS has won both the Queen Award for Export and Industry Awards for its scanners.
As a pioneer of online marking solutions at the start of the last decade, DRS e-Marker® offers a variety of modular and scalable electronic marking services for awarding and professional bodies.
DRS has provided online marking solutions to all the UK unitary awarding bodies during its history and this summer provided services to the Northern Ireland awarding body, CCEA (Council for the Curriculum, Examinations and Assessment), the Welsh awarding body (WJEC/CBAC) and the English awarding body AQA (Assessment and Qualifications Alliance) making it the most used system by script volume during 2011. DRS is now engaged in taking this solution overseas where it has run a number of successful pilots in Africa, Malaysia and the Caribbean.
Based in Milton Keynes DRS employs 200 core staff.
Response
The advantages of multiple awarding bodies
(1) The question of whether there should be a single, national examination board in England has arisen before. The arguments against such a step are as follows:
Competition drives innovation, quality and continuous improvement.
Competition provides customer choice.
National Bodies tend to be risk averse and their rate of change towards greater efficiencies and improvements is slower.
National projects do not have a good track record in delivery.
Administrative errors occur in both models but a progressive system is more likely to resolve issues more quickly.
The move to a single body would be a complex and major exercise with no certainty of positive change.
How to ensure accuracy in setting papers, marking scripts, and awarding grades
(2) In order to demonstrate how to ensure accuracy in setting papers, marking scripts and awarding grades, it is necessary to summarise the examination process. This process for unitary awarding bodies marking scripts physically is shown in Appendix 1.
Marking Scripts Physically
(3) Steps 1-5 are associated with the creation of the examination paper, its secure printing and distribution to the Examination Centre by the Awarding body.
(4) Step 6 Once the examination is sat the Centre send the scripts directly to particular markers who will mark that subject from that centre.
(5) Step 7 does not exist.
(6) Step 8 Before they can mark the scripts the markers must attend a Standardisation meeting where the markers are trained and tested on the mark scheme for each of the questions in the subject script. These meeting is booked for a particular day and hosted in a facility which has been hired for the purpose. Payments are made to the markers for subsistence, travel and loss of salary.
(7) Step 9 & 10 Markers then start to mark the scripts “live” and they select a fixed sample and send this back to the awarding body for quality checks to be carried out on the markers work. Costs are incurred in packaging up these scripts and posting them back to the Awarding Body.
(8) Step 11 A repeat of Step 10. A further sample, selected by the marker, of representative work for the Awarding Body to assess. Costs are the same as in Step 10.
(9) Step 12 Meanwhile the awarding body must manage the progress of marking with a limited view of what is happening with the thousands of markers working for it. Scripts will inevitably need to re-shipping from one marker to another for a variety of reasons – this cost time and resource.
(10) Step 13 The marker finishes their allocation and ships all the remaining scripts back to the Awarding Body. Cost of shipping.
(11) Step 14 The Awarding Body then carry out a series of checks on all the scripts, such as checking the addition of the marks that the markers have made, the transposition of marks from the body of the script to the front page and then enter of the marks into the Awarding Body Examination Processing System. Temporary staff are taken on to carry out this task on the millions of scripts.
(12) Step 15 Senior Markers then sample scripts from each marker separately to discern whether the marking has been conducted at the appropriate quality level and analysis is carried out on the marks to assess whether the marker has been generally too lenient or harsh. Scripts can be remarked completely or adjustments made to the marks if the marking is found to be below standard. This can be a costly exercise and resources are needed as the awarding date approaches.
(13) Step 16 For the awarding meeting, scripts totalling a certain mark must be found for each subject to assess where the grade boundary is. Finding a significant sample of scripts with a particular mark in a range of marks can be time consuming.
(14) Step 17 Throughout this process the awarding body must have sufficient storage for millions of scripts that allows quick and easy access.
(15) Step 18 Once the Award is made a percentage of candidates will question the grade they have been given and will ask for the script to be re-marked. This is called Enquiries about Results and generally requires the repeating of steps 14 and 15.
Marking Script Images
(16) Quality improvements and cost savings can be made by awarding bodies if they move from physically marking the scripts to marking images of the candidate’s responses online through an electronic marking service such as e-Marker® from DRS.
(17) Briefly the Examination Centres send all the scripts directly to the service provider who checks them in reconciling the candidate enrolments against the received scripts. The scripts are then scanned and images held in secure hosted facility. Markers are then able to attend their Standardisation meeting online and once they have passed the quality measure they are allowed to mark candidate responses live. Because the candidate responses are images, they can be segmented into itemised responses. This means that a marker can be allocated to only mark Q2, whilst another marker marks Q3. This brings a number of benefits but the key one is marker quality. This comes as result of marking opinion being spread across a particular candidate script (not reliant on a single marker) and a consistency of marking that comes as a result of working on one question type for a period of time. The biggest quality gain comes from the ability to apply a quality methodology that checks the marking quality of the marker as they are marking. This means that adherent marking can be spotted early and action taken. This then leads to the marks at the end of the process being reliable and fair, negating all the post mark checks normally carried out.
(18) The process when using an electronic Marking solution is shown in Appendix 2 which highlights areas where quality improvements, service improvements and cost savings are made.
(19) Steps 1-5 do not change.
(20) Step 6 the examination centre now sends all scripts to one address where the scripts are scanned. This more efficient, less prone to error, reduces shipping costs and is much simpler to administer.
(21) Step 7 The scripts are scanned.
(22) Step 8 Markers are standardised online. This is a major cost saving and means that the markers can conduct the standardisation at a time when it is convenient to them.
(23) Step 9 Markers are then able to mark online the candidate question responses live (such as Question 2), that have been allocated to them. There is no need to limit the number of items that the markers can mark and when they mark an item, Q2, a new one is randomly selected from the pool of Q2s. The items are totally anonymous.
(24) Step 10 During marking the markers are automatically checked for quality of marking without the marker being aware.
(25) Step 11 This step of self selection of scripts for quality checking purposes is not longer required.
(26) Step 12 The Administrators have a full picture of the marking progress.
(27) Step 13 The requirement to ship scripts back to the awarding body is no longer required.
(28) Step 14 The requirement for the awarding body to carry out a series of checks all the scripts is not longer required.
(29) Step 15 Marking review is no longer required.
(30) Step 16 Instead of having to physically retrieve scripts with the correct total mark for the Awarding Meeting this can all be done by selecting the images of candidates with the required marking through the electronic Marking system.
(31) Step 17 Storage is no longer required by the awarding body. Scripts can be securely destroyed after the Enquires about Results stage as the images are available if required.
(32) Step 18 Enquiries About Results is a much simpler process as the candidate’s script can be retrieved as an image for a marker to remark.
(33) Step 19 During the Electronic Marking process data is gathered that was never available during physical marking of scripts. This allows much more detailed analysis of exam and candidate performance:
The benefits of marking from images of scripts are as follows:
improves marking quality;
makes more efficient use of the marker workforce;
removes any bias from marking;
enables the remote training of markers (Standardisation);
allows the anonymity of candidates;
improve administrative visibility;
provides enhanced management information; and
allows for more progressive approaches to assessment.
(34) Steps 1-3 cover the process of creating an examination paper from scratch. These steps can be integrated into the electronic marking system with the following benefits:
Managed workflow with full audit trail reducing the possibility of errors.
A single, continuous system from examination creation through to grade awarding.
Data gathered early can be used later creating a more efficient system.
(35) Rather than reduce the English awarding bodies down to one organisation and destroy what makes them successful and distinctive potentially one might remove areas where there is duplication of activity between the awarding bodies. Some of these areas of duplication are:
Examination Processing System. This is the backbone of the Awarding Body. It manages the receipt of candidate entries, Centre details, Examiner details and payments. It holds the mark data and manages the awarding and enquires about results process. This system is unique to the Awarding Body and as such there are no commercial systems that meet the scale of requirements of an English Awarding body. Developments costs are considerable and the efficiency gains in sharing such a system are trivial.
Assessment Processing System. This system undertakes the marking of scripts and the management of the markers undertaking the exercise. All three awarding bodies use a different system but each share broadly the same principles, in that they all mark from image. There would be some efficiency gains by utilising one system across all three awarding bodies as well as a commonality of standards.
Scanning process. Each Awarding body scans their scripts to provide images for their marking system. There are efficiency gains in using one scanning facility, but to maximise the benefits each awarding body should also utilise the same examination paper formats.
Recommendations
(36) In order to encourage innovation that will increase quality, improve standards and reduce cost the key recommendation is to create a regulatory framework that will allow it. Multiple Awarding bodies support this case, not undermine it.
November 2011