|
Memorandum submitted by Professor Michael Arthur
1. This submission provides the committee with information on the National Student Survey (NSS), which I would like the Committee to be aware of prior to my oral evidence.
Summary of main points
2. The NSS forms part of HEFCE's quality assurance framework. Its purpose is to support potential students and their advisers make decisions about what and where to study. In addition, it provides detailed results back to institutions and students unions (SUs), which can identify areas of good practice and for improvement.
3. The survey is now in its third year with the 2007 NSS currently underway. For both 2005 and 2006 over 80 per cent of students were satisfied with the quality of their courses. The overall response rates are high at 60 per cent in 2005 and 56 per cent in 2006, demonstrating the robustness of the survey results.
4. Higher Education Institutions (HEIs) and Students Unions (SU) are already using the survey to improve the student experience. For example, the University of York is providing longer library opening hours and the University of the Arts London have asked all departments to create action plans in response to their results. HEFCE will be commissioning work to further analyse how institutions and students unions are making changes following the outcomes of the survey.
5. The recent review of the quality assurance framework (QAF) confirmed that the survey can make a valuable contribution to student choice. However, some improvements need to be made to meet this objective more fully. This is currently being implemented, with the appointment of UCAS to re-develop the Teaching Quality Information (TQI) website, where the NSS results are published, to make it more user-focused and friendly. A new site, with a new brand and identity will be released in early August 2007.
6. Sector wide analysis of the 2005 NSS results has been completed, which showed that students viewed their higher education courses very positively with over 80% of students agreeing that 'overall, I am satisfied with my course'. However, this report also highlighted that assessment and feedback is an area where students are overall satisfied, but relatively less satisfied in comparison to other elements of their experience. This is also the case for students from an ethnic minority, who overall are satisfied but are relatively less satisfied than their white counterparts. HEFCE is working with the Higher Education Academy and the Equality Challenge Unit to identify why this is the case and how we can support institutions to make improvements.
7. Analysis of the 2006 NSS results is currently underway and will be published during the spring. It will identify changes between the 2005 and 2006 NSS results.
Professor Michael Arthur
8. Professor Michael Arthur is Vice Chancellor of the University of Leeds (1st September 2004). He was formally Professor of Medicine (1992), Head of the School of Medicine (1998 - 2001) and Dean of the Faculty of Medicine, Health and Life Sciences in Southampton (2003-2004). He is a hepatologist with research interests in liver cell biology which were developed at the University of California, San Francisco (1986-1988) and more recently as a Fulbright Distinguished Scholar at Mount Sinai School of Medicine in New York (2002). Professor Arthur became a Fellow of the Academy of Medical Sciences in 1998. His national contributions have included roles at the Wellcome Trust (Cell and Molecular panel), at HEFCE (Strategic Research Committee) and at UUK (Research Policy Committee). He has recently become Chair of the Teaching Quality Information and National Student's Survey Steering Group.
The National Student Survey
Background
9. As part of the revised quality assurance framework that followed the completion of subject review, a Task Group chaired by Professor Sir Ron Cooke was set up in 2001 to make recommendations on the information about the quality and standards of teaching that the new quality assurance framework should generate. The NSS was first mooted as part of this new package of information.
10. The Task Group concluded that as part of the public information package, a national survey would need to be introduced to support students making choices about what and where to study. The Student Feedback Project Steering Group, also chaired by Sir Ron Cooke, was established in the summer 2002 to scope the new survey.
11. The NSS Pilot Steering Group (NSSPSG) made recommendations for the conduct of the first full scale survey in 2005, in the light of evidence from two pilot exercises and consultation outcomes.
12. In parallel to piloting the NSS, the Teaching Quality Information (TQI) website was developed by HERO as the mechanism for publishing the public information aspects of the new quality assurance framework. This included publishing the NSS results, HESA derived statistics (such as continuation data) and the Destination of Leavers of Higher Education (DLHE) survey.
13. A review of the Quality Assurance Framework (QAF) has now been completed, which evaluated TQI (which included the NSS). The review group concluded that these initiatives can make a valuable contribution to student choice, however the group recommended re-focusing the TQI site on the needs of applicants and their advisers, as the purpose of the site should be to help potential students choose what and where to study. The outcomes of this are currently being implemented, with UCAS being appointed to develop a new site with a new 'brand' and identity.
14. The National Union of Students (NUS) are a key partner in implementing the NSS and HEFCE has funded a post at the NUS to support Student Unions (SUs) to promote the survey and use the results to identify good practice and areas for improvement.
Scope and coverage of the survey
15. The coverage of the NSS is as follows:
a. all publicly-funded HEIs in England, Wales, Northern Ireland (and some institutions in Scotland and the University of Buckingham from 2006) are included. b. All full-time and part-time undergraduate students registered at these HEIs are surveyed, during their final year of study (or, for flexible programmes where the final year cannot be predicted, during their fourth year of study). c. Teaching and Development Agency (TDA) for Schools funded students were included in the NSS for the first time in 2006. d. NHS funded students are included for the first time in the 2007 NSS.
16. Currently the Scottish Funding Council does not fund the NSS, however some institutions (currently eight) are now voluntarily taking part in the NSS.
Questionnaire
17. The questionnaire (which was devised and thoroughly tested through the pilot process) consists of twenty one questions grouped into six 'scales' covering the following: The Teaching on my course Assessment and Feedback Academic Support Organisation and Management Learning Resources Personal Development Question twenty two measures overall satisfaction.
18. NHS funded students are asked an addition 'scale' of questions covering their experience of practice placements.
19. In addition, there is the opportunity to give positive or negative comments on the student experience.
20. In the 2007 NSS, a new bank of questions is being piloted. These were devised from institutions' own internal questionnaires by the Higher Education Academy. Institutions (if they choose to opt-in) were able to choose a maximum of 5 question scales (from a possible 12) and then students would be asked if they were happy to complete further questions, once the full NSS was complete. This was via the online method only. The results will be published back to institutions only and not publicly available.
21. HEFCE will be commissioning a full review of the questionnaire (including the additional questions for NHS students and the pilot bank of questions) so the steering group can consider how it will develop in the future following the 2007 NSS.
Methodology
22. The target list is generated automatically by the Higher Education Statistics Agency (HESA) at the point when institutions make their HESA return. Institutions then populate the target list with student contact details. HEIs have the opportunity to choose which week, over a five week period between January and Easter, when they would like to start the survey.
23. The survey uses three methods of data collection: online, postal and telephone. This is also the sequence in which data is collected. Ipsos MORI complete the survey on our behalf and contact students directly requesting a response.
24. Institutions and SUs are provided with marketing materials to promote the survey and encourage students to respond.
25. For the 2006 NSS significant changes were made to the telephone methodology to address concerns that some HEIs and SUs had regarding the number of attempts that were made to seek a response. The number of attempts to call a student has been reduced by half (to a maximum of 8), no number is called where more than ten students share the same number and no messages are left. An attempt is counted even where there is no answer to a call. This did result in the response rate dropping slightly for the 2006 NSS.
26. We are aiming to increase the online response rate over time, therefore reducing the need for telephoning. However, the analysis of the results (completed by Paula Surridge at the University of Bristol) concluded that students from an ethnic minority or those that have a disability are more likely to respond by telephone. It is important that this student group has the opportunity to respond to the survey, so telephoning will remain.
27. HEFCE will be commissioning a full review of the methodology so the steering group can consider how it can develop in the future following the 2007 NSS and the further analysis from the University of Bristol on the methodology of the 2006 and 2007 survey.
Publication of results
TQI website
28. The results of the survey are published via the TQI website. Users are able to compare results at subject and institution level on the site. Institutions are also able to make a commentary on the results of the survey and other data included on the site. Following the outcomes of the Quality Assurance Framework Review HEFCE[1] appointed UCAS to re-develop the website, so that it is focused on potential students and their advisers. The NSS will continue to form a key part of the data available on the site.
29. The NSS results are subject to a 'publication threshold' of 50% response rate and 30 students on the existing site. 93% of institutions had data published in 2005 and 2006 and we are working to achieve 100% coverage to ensure that potential students have as much information as possible when making choices: HEFCE has funded a post at the NUS, to support SUs to promote the survey and the use of the results. They are targeting those SUs where the response rate was below the threshold for publication in the previous surveys. Ipsos MORI (the survey agency) is also working with institutions to promote the survey and monitor response rates. HEFCE is working with HEIs to promote the survey and continually improve the quality of the HESA data and therefore increase response rates.
30. Following the outcomes of the QAF review, HEFCE has consulted with the sector on the development of the TQI website, to make it more user-friendly. A key issue was in relation to the threshold on the site: the proposal was to reduce it to 40% response rate and 20 students. This was supported by the sector, however further expert statistical advice is currently being sought to consider this issue further.
Results provided back to HEIs and SUs
31. Institutions and Students Unions have access to more detailed results of the survey, via the Ipsos MORI dissemination website. This has a data protection threshold of ten students, so more data is available including for those institutions which do not have data on the TQI website. This also includes the comments students included on the questionnaire (these are 'cleaned' so no individual can be identified).
32. This has enabled HEIs and SUs to enhance the quality of the student experience, such as the University of Essex's department of Biological Sciences which has established a student mentoring scheme. Evidence from the QAF review has also suggested that SUs found the survey extremely positive in securing beneficial changes for students such as anonymous marking. Work is currently underway with Ipsos MORI and the Higher Education Academy to make improvements to the dissemination website, to make it more useful to HEIs and SUs.
Analysis of results
33. HEFCE commissioned Paula Surridge from the University of Bristol to analyse the results of the NSS. The analysis of the 2005 results has now been published and focused on the institutional, student and subject characteristics of the results. This analysis found that overall the majority of students are satisfied with their experience of higher education with over 80% of students agreeing that 'overall, I am satisfied with my course'. However, this report also highlighted that assessment and feedback is an area where students are overall satisfied, but relatively less satisfied in comparison to other elements of their experience. In addition, although overall students from an ethnic minority are satisfied, they are relatively less satisfied than their white counterparts.
34. HEFCE is working with the Higher Education Academy and the Equality Challenge Unit to identify why this is the case and how we can support institutions to make improvements.
35. Analysis of the 2006 NSS results is currently underway, This will be published during the spring.
February 2007 [1] On behalf of HEFCW, DEL and SFC |