Government reform of Higher EducationWritten evidence submitted by the Economics Network, Bristol University

Mechanisms to Promote Teaching Quality in HE

Summary

This paper examines incentives to improve teaching quality in higher education.

It first examines the current situation and identifies some levers which are or have been used with mixed success.

It then goes on to identify three categories of lever: regulation; competition and market mechanisms; developing communities of practice with the culture of sharing innovation and good practice.

The paper argues that there is considerable scope within each of these three categories for enhancing teaching quality.

1. The current situation

1.1 Teaching is given a lower priority in most universities than research. Although there has been some improvement in teaching quality there is massive scope for improvement. Some improvements have already come about as a result of:

1.1aQAA subject review from prior to 2002. This was a peer review process with teams of academics from the same discipline doing the inspections of each subject, which gave credence to it with academics. It forced discussion at departmental level under highly relevant heads: curriculum design; teaching, learning and assessment; student progression and achievement; student support and guidance; learning resources; quality management and enhancement. The process was flawed, however, by having a crude scoring system, which led to a strategic compliance approach rather than a developmental approach to teaching quality. There is considerable scope for improving a reintroduced peer review scheme, but it could be costly.

1.1bUsing teaching ability and performance as a promotion criterion – normally alongside the quality of research output and administrative responsibilities and performance. The problem here has been that teaching has typically been judged at a relatively low threshold level, unsupported by robust evidence, with research output normally being the determinant of promotion. There is also a danger that in the search for evidence on a lecturer’s teaching ability weight will be given to pedagogical research. This would be a mistake as a pedagogical researcher is not necessarily a good teacher.

1.1cProfessionalisation of teaching, largely through the requirement that new lecturers pass an internal university-wide generic initial professional development course. Unfortunately, both at university level, and at national level (eg through the two-sided UK Professional Standards Framework document), professional standards have been far too generic with little perceived relevance to lecturers within their disciplines. Many new lecturers resent having to take such courses when they are trying to establish their research reputation and are under tremendous pressure to publish. Heads of Department also often resent freeing up staff time and budgets for such courses, which they too see as of little relevance.

 The exception to this has been the training and workshops provided both to new lecturers and established lecturers by the 24 subject centres (see separate submitted paper “The Abolition of Subject Centres: the case for keeping them”), originally the Learning and Teaching Support Network and currently part of the Higher Education Academy. Feedback from new lecturers attending such training is normally wholly positive and is contrasted unfavourably with generic courses in their home universities. Once of the large benefits of such national (or regional) subject-specific workshops is the opportunity for new lecturers to build networks with other new lecturers from the same discipline and to share ideas, experiences and solutions. The subject centres provide a hub for building such networks – networks that would be extremely difficult to retain if the subject centres are abolished in line with the plans of the Higher Education Academy.

1.1dThe National Student Survey, although deeply flawed as a survey, manipulated by universities and with crude results, has raised the perceived importance of teaching quality in the minds of students and university lecturers and managers. There is, however, considerable scope for improvement. For example, the survey could include more questions requiring textual response (at the moment there is only one); it could be given to all years and linked to student personal development planning; there could be student focus groups to reflect on the results and where improvements could be made. Such focus groups could be run by subject centres (and have been), with confidential reports provided to departments. Subject centres could also customise the surveys for their own particular discipline(s).

1.1eStudent fees. Students are seeing themselves increasingly as customers and are demanding a better service for the money they pay. Such demands are likely to increase from 2012 and are likely to be more co-ordinated. Such demands are likely to drive change, but for the change to be effective, lecturers require support and incentives for better teaching need to be appropriately aligned.

1.1fThe work of subject centres. These have now been operating for 11 years and have made a significant difference to the practices of lecturers. They have developed networks of practitioners to share ideas; support new lecturers through training and resources; run continuing professional development (CPD) activities for departments; provide large banks of resources on individual subject centre websites; fund mini projects; run subject-specific teaching and learning conferences; disseminate practice; run lecturer awards schemes; develop relationships with departments; publish case studies and articles in their own peer-reviewed journals and magazines; collaborate with subject and professional associations; develop relationships with key employers.

1.2 Despite these developments, teaching is still given a relatively low priority in many universities and the current incentive mechanisms drive this situation. Research output is relatively easy to measure with well-established metrics, such as journal rankings and citations, and with peer review at the heart of the research assessment/ evaluation process. It is difficult, but not impossible, to provide a parallel process to drive improvements in teaching quality, particularly if, like research, teaching quality is peer-assessed in a disciplinary context.

2. Options for improving teaching quality: regulation

2.1 Regulation: inspection. Teaching quality is less easy to measure than research quality and thus regulation which hinges on indicators is likely to be flawed and open to manipulation. This was the central problem with the QAA’s subject review process. University departments played a game of targeting the indicators in order to receive a maximum of 4 out of 4 for each of the six indicators. In many/most cases, it was the score they were after rather than genuine quality improvements. The asymmetry of information between internal staff and the visiting team allowed the team to be hoodwinked on many occasions:

2.1.aA revised version of subject review could be reintroduced that would encourage academics to focus on genuine improvements in teaching and learning. Peer review by fellow academics from the same discipline from other universities and with a professional chair would be retained, but scores would not be used. Instead there would be a simple pass/fail metric, with a relatively low threshold, and the visit would be seen as developmental. Internal staff could discuss issues in a protected environment and the visiting panel would advise. A report would be published, but it would be a formative report, not a summative one, and the report would go through a process of iteration between the team and the internal staff before the agreed report and action plan were published.

2.2Regulation: minimum standards. Universities could be required to meet minimum standards in terms of various indicators, such as contact time, staff teaching hours, class sizes, volume of assessment, proportion of funding devoted to teaching and amount of staff time spent on CPD activities. The problem with any of these indicators is that they are crude and subject to easy manipulation. What, for example is contact? Does virtual contact count? What is the role of graduate teaching assistants and is contact with them equivalent to contact with established staff? Also they are quantitative indicators, which are often a very poor proxy for quality.

2.3 Regulation: training. University staff could be required to undertake both initial training and CPD activities that lead to some form of qualification. The former has been taking place and most universities have established courses for new lecturers. The biggest weakness here, as argued above, is that they are generic and are dismissed by many lecturers as largely irrelevant to their needs as teachers of a specific discipline or specialism within that discipline. There is considerable scope here for building on the work of the subject centres to provide that discipline specificity and relevance:

2.3.aIn the case of CPD, if reluctant compliance by staff, which would lead to no real improvement in teaching quality, is to be avoided, CPD has to be targeted at the perceived needs of staff and thus should be negotiated with and within departments. In other words, it should be demand led. It could be linked with the process of programme review or with the outcomes of student/staff committees. It could be linked to dissemination activities of subject centres, where discipline-specific solutions adopted in one university could be adapted to the equivalent department in another university. It could involve attendance at national/regional workshops run by subject centres or other agencies.

2.3.bThere would need to a light overview regulation of this process, but this could easily be achieved by the QAA as part of institutional audit.

3. Options for improving teaching quality: competition and market mechanisms

3.1 There is considerable competition between universities in terms of research quality and there is a thriving market in research academics. There are two key reasons for this: first, there is transparency in terms of publications and well-established indicators, allowing league tables to be easily constructed (although, it must be noted, the new impact measure in the REF will be much more problematic); second, research output is prioritised over teaching quality, despite teaching income exceeding research income in the vast majority of cases.

3.2 How can competition be created in teaching? An important “market” failure here is the asymmetry of information. People outside a university have little knowledge of what goes on behind closed classroom doors, other than what they hear from student friends or sons/daughters at university. Even within a university, provided that there are no student complaints, colleagues, including heads of department, often have little knowledge of the quality of teaching of their fellow lecturers. What knowledge they do have is often anecdotal or based on crude student satisfaction surveys. Current league tables reflect data that poorly reflect the quality of learning and teaching.

3.3 Another market failure is perversity of incentives. The few incentives related to teaching quality are often related to a poor proxy for quality, such as student satisfaction surveys or pass rates. Students often like being spoon-fed with teaching directed to a highly predictable exam. Lecturers behaving in this way may be giving their students a poor quality learning experience and yet might get high satisfaction ratings. Teaching which challenges students with difficult learning tasks may be unpopular with students at the time but may well produce better (and more grateful) graduates with skill sets that make them more employable. As far as pass rates are concerned, easy and predictable assessment is likely to give higher pass rates, but again a poor quality learning experience based on rote learning. The incentive to produce easy and popular courses is stronger when the modules are optional ones. Lecturers clearly want students to choose their options – after all, they can use the same materials as last year and will not have to prepare for alternative modules. If they make these options easy to pass, this information will quickly spread to the next cohort of students.

3.4 Mechanism 1: increased information in the public domain. One way of increasing information would be to require universities to publish various teaching-related statistics by department as well as an aggregate for the university. These could include: funding received for teaching and funding received for research; staff time allocated to teaching and research (this information does normally exist as “hours” is the normal metric for allocating staff duties); student contact time by class type; class sizes; student/staff ratios; the proportion of classes taken by graduate teaching assistants; the proportion of classes taken by visiting lecturers (i.e. part-time hourly-paid lecturers); the nature of the learning opportunities and experiences available to students; assessment regimes; progression rates; entry qualifications; retention rates; proportions of students receiving each degree classification; CPD activities undertaken by staff. Although these indicators are often too crude to be used for regulation, they provide useful information for prospective students, parents and employers and could put universities in significant competition with each other.

3.5 Mechanism 2: teaching awards. There has been some development here in terms of (a) national teaching fellows (NTFs), (b) subject-centre awards to lecturers for excellent teaching and (c) institutional awards and teaching fellowships:

3.5.aThe former has already created some degree of competition between universities, and universities are generally keen for their lecturers to receive such an award. However, the scheme has the flaw that the award is generic and thus is difficult to relate to teaching quality within a department. There is usually little spill-over into the practices of colleagues. Nevertheless universities prize their staff receiving such awards and have helped to increase the importance attached to teaching quality by senior managers in universities. The number of NTFs should be increased. But it is important that these be seen as a prize and not carrying further expectations. The Nobel prizes would be a parallel. Also there is the issue that too many of the awards go to general educationalists. There must be much more recognition of excellence by individual teachers within the disciplines.

3.5.bSubject-centre awards have had some impact at departmental/subject level and often lead to projects at departmental level, but at the moment these are small scale. Competition could be significantly increased by extending awards at subject level, as it is at this level that universities compete (as they do in research). For example, a proportion of NTFs could be pre-allocated to each discipline (say a certain number for each subject centre to administer), would put departments into much more direct competition with each other.

3.6 Mechanism 3: Open Educational Resources. Most lecturers now use the virtual learning environment adopted by their university. The main ones are Blackboard, WebCT and the freeware product Moodle. This means that students can now access course materials – including syllabuses, previous assessments, lecture slides, notes, etc. – through a password-protected website. This, however, has reduced competition between universities, as materials that might previously have been on an open website can no longer be seen other than by the students on a particular module and their lecturer. They are even hidden from other students and lecturers within the university:

3.6.aA solution here is to make course materials publicly accessible under a creative commons licence. This process has already begun with the Open Educational Resources movement. Part of this is voluntary with individual academics simply wishing to share their materials; part is at university level, with universities such as Harvard and Nottingham putting their course materials on an open website; part is at subject level. The subject centres have been leading at this third level with HEFCE project funding distributed through JISC. Individual academics have been supplying materials and sharing them with academics in the same discipline or sub-discipline across the UK and the world. For example, in economics 15 wikis have been developed for 15 specialisms (such as development economics, health economics and monetary economics), each one led by a senior academic in the field.

3.6.bThese sub-disciplines often align with research communities where academics already know each other. Lecturers are encouraged to submit their materials and these can then be seen publicly. There has been a lot of interest in this and there are three key benefits. First, it gives ideas for lecturers developing a module and allows them to draw on best practice from across the country. Second, it puts competitive pressure on lecturers to develop the quality of their materials if they are to post them on the wiki and have them seen by others. Third, it puts competitive pressure on lecturers who do not submit to the wiki because their students can still compare what they get with what students elsewhere are getting.

3.7 Mechanism 4: clearer and better structured career routes through teaching excellence. Just as universities compete for top researchers, so it is desirable to create an equivalent market for top teachers. One part of the solution here is for clear progression routes for lecturers within universities linked to teaching excellence. This has a number of requirements. First, there must be senior appointments for teaching-only or teaching-predominantly staff. Some universities have already instituted such routes, with teaching fellows, senior teaching fellows and associate professors as a progression ladder. Second, there must be transparent criteria used for promotion, such as student evaluations, student performance, peer review, staff appraisal, scholarly output related to teaching, teaching innovations, work with subject communities outside the university, winning of funding from outside bodies such as subject centres or professional associations. Third, there must be educational leadership in universities that values teaching.

3.7.aOnce such career routes are widely established, this will create a market in excellent teachers with the market incentives of improving quality of teaching. Of course, this should not preclude promotion routes for academics who are both excellent teachers and excellent researchers.

3.8 Mechanism 5: stronger feedback loops between student evaluation of staff and changed staff practice. The market element here is an internal one within a university or department. Too often, feedback from surveys of students at module level is not acted on and lecturers are not held accountable for their performance. Whilst student surveys in themselves may give limited information on the quality of teaching and learning, as alluded to above, they can be backed up by consideration at staff/student committee level, where lecturers will be held accountable, not only to students, but to colleagues too. Evaluation can then form the basis of quality enhancement activities and the redesign of teaching for subsequent years. Student focus groups can be used to establish good practices as well as practices that do not work so well. Involvement of students in course planning is empowering and is likely to increase student pressure for genuine enhancement of teaching quality.

3.8.aExternal surveys could be used here. For example, the Economics subject centre has been developing and using student experience surveys tailored to economics teaching, with follow-up student focus groups. These capture a much richer set of information about teaching quality than the NSS or many internal surveys. Confidential reports are supplied to departments. Similar work is undertaken by other Subject Centres and could easily be extended to all disciplines.

3.9 Mechanism 6: alumni surveys. Students who at the time like easy and predictable courses with spoon-feeding, may look back after they have graduated and be very critical of them. They may have much more insight into the quality of their learning after graduation than when they were a student. Thus alumni surveys of modules and programmes, as well as focus groups with alumni, would be an important means of addressing information problems.

3.10 Mechanism 7: monetary carrots and sticks. Short-term pay enhancement could be given for developing teaching innovations, for reorganising problem courses or for taking on a consultative role with students to assess the teaching quality within the department and recommend improvements. Staff could be given a budget to attend teaching/learning workshops and conferences, just as many now have a budget for attending research conferences.

3.10.aAn alternative to pay enhancement would be timetable relief to develop new practices and disseminate them to colleagues.

3.10.bIt could be argued that performance-related pay is the optimum carrot/stick to improve performance. Of course, to be effective this requires a measure of performance, either quantitatively or through assessment by a manager. With the former, the problem is a lack of robust quantitative indicators. For the reasons given above, pass rates and average marks, retention rates and student satisfaction are all flawed and their use could create perverse effects. In terms of managerial solutions, the obvious monetary carrots are ones of promotion or tenure. The sticks could be non-renewal of contracts. Thus fixed-term but renewable contracts might be a useful mechanism for wielding sticks.

3.11 Mechanism 8: external degrees. One of the problems with the existing system is that lecturers set their own assessment, thereby removing an important potential area of competition between universities. If they all have different assessments, how are meaningful comparisons to be made between universities. The external examiner system is supposed to regulate for this. However, the system is not fit for purpose because too little resource is allocated to the system and there is a high degree of regulatory capture. Externals in some cases are useful advisors, but rarely ensure genuine comparability of standards, given that they do not set the syllabus or assessments. What is more, most university examination regulations preclude externals altering marks, except for a whole cohort, and this would involve a much bigger marking load for the external.

3.11.aA solution would be to institute a voluntary system (with incentives) of external degrees. The post-1992 universities, when they first began as polytechnics in the late 1960s, did not have degree awarding powers and their students took London University external degrees. The staff clearly did not know in advance what would be on the exam. This allowed genuine comparison between the polytechnics and put them into competition with each other. Clearly it would not be politically possible to remove degree-awarding powers from universities, other than in extremis, and so the solution would be to permit universities to use external degrees with positive incentives for doing so and possibly to introduce the negative incentive of tighter regulation for those preferring to use their own degrees.

3.11.bA practical way of doing this would be to allow universities to set up consortia with peer universities and set common exams that each member of the consortium would use. There may need to be some minimum number in a consortium. Not only would this provide competitive pressure between consortium members to improve teaching quality, as degree results could easily be compared, but it would also create competitive pressure between consortia to maintain standards, or to be distinguished from each other in terms of the type of degree being offered (e.g. more theoretical or more applied). A parallel here would be with A-level boards, which compete with each other. Clearly there would be security issues, as there are with A-levels, to ensure that those setting the exams did not reveal them to university staff.

3.11.cAlthough the system would be voluntary, there would be considerable incentives for universities to join such consortia in order to attract students, who would see such degrees as having considerable market value. Universities might be happy to take part in pilots with specific subjects.

4. Options for improving teaching quality: developing communities of practice

4.1 Students choose to study particular subjects at university. Academics see their primary allegiance as being to their subject. Employers may prefer to recruit graduates from certain subjects. As estates are developed, there is much interest in providing facilities which encourage subject-based learning communities, of students and academics, within universities. Academic subject “tribes” are an intrinsic feature of higher education and effective learning and teaching are best cultivated, recognised and celebrated within these.

4.2 Student communities. Students learn better in an environment where there is a sense of community, where they have joint responsibilities for improving their own learning and that of their fellow students. Some universities actively encourage the development of student subject societies, the production of student journals/magazines, the development of discussion groups outside formal classes and an active alumni association with a sense of responsibility towards the current generation of students. The careful use of group work and various classroom and assessment practices can also help to develop student communities and a sense of communal responsibility. There are plenty of examples of successful communities and information needs to be gathered and disseminated to help their development more broadly.

4.3 Communities of academics as teachers in their disciplines. The subject centres have developed networks of practitioners over the past 11 years (see separate paper “The planned abolition of subject centres”). These share ideas and innovations through various dissemination channels facilitated by the subject centres, such as case studies, workshops, discussion groups, special interest groups, the sharing of material (see open education resources above), conferences, departmental contact meetings (for dissemination by the contacts within departments), networks of associates with particular expertise. Lecturers engage willingly in such communities because they are from within the same academic “tribe”. Academics are often unwilling to engage with the teaching/learning enhancement agenda at a generic level as the language and generality is alien. However, they are often very willing to share practical solutions to issues, such as improving student performance, dealing with large numbers, tackling specific problem subjects or concepts, or relating research to teaching, with colleagues from other institutions but from the same discipline who speak the same language and experience similar problems. Developing and deepening such networks at the subject/discipline level is a way of engaging staff with the teaching/learning agenda and bringing significant quality enhancement and innovation.

5. Overcoming institutional barriers to change

5.1 The above represents a range of possibilities and they can be used in different combinations. A key factor affecting their feasibility is the attitudes of bodies, such as UUK and groups of universities, such as the Russell Group and the Alliance Group. Perhaps the key to unlocking resistance to change is the changed environment in which universities find themselves. Higher student fees, a strengthened NSS and greater transparency in terms of university performance and use of funding are all likely to drive universities towards giving better teaching a higher priority. In these circumstances, universities might be more responsive to mechanisms and regimes that allow them achieve better teaching and learning.

5.2 Appropriate incentive mechanisms are the key. These need to apply not only to individual lecturers, but also to universities.

9 March 2011