House of Commons - Health

Select Committee on Health Third Report

3 The 2007 crisis

Box 2: Implementation of MMC and MTAS: Timeline of key events in 2007

22 January - MTAS computer system opens for Specialty Training applications.

9 February - High Court rules that DH guidance on HSMP applicants is legal; British Association of Physicians of Indian Origin (BAPIO) launches appeal; DH decides not to enforce guidance in 2007 because of ongoing appeal.

5 March - The interview panel for ST3 level General Surgery in the West Midlands decides not to proceed with interviews due to "serious procedural flaw" with short-listing; DH announces "Douglas Review" review of ongoing ST selection process.

17 March - Junior doctors demonstrate in London and Glasgow about the MTAS system

31 March - Alan Crockard, head of MMC, resigns, blaming lack of leadership of and confidence in MMC and MTAS; National Clinical Advisor Shelley Heard also resigns.

4 April - The Douglas Review announces changes to the selection process: at least one interview is guaranteed for each candidate in an extended first round.

24 April - Sir John Tooke asked to lead independent inquiry into implementation of MMC and MTAS.

26 April - MTAS website is suspended following two separate security breaches which make some candidates details publicly available; the system is subsequently abandoned.

20 May - James Johnson, chairman of the BMA, resigns, following criticism of the organisation's failure to adequately represent junior doctors' interests.

23 May - RemedyUK's legal challenge is rejected at the High Court; however, the judgement describes MTAS as a flawed system and acknowledges the "disastrous consequences" of MTAS.

8 June - Revised plans for Round 2 of recruitment are announced; applicants are guaranteed employment in existing posts until the end of October 2007.

22 June - Round 1 selection is completed, more than 2 months behind schedule.

12 July - The final report of the Douglas Review is published, describing ST selection as "the biggest crisis within the medical profession in a generation".

1 August - Those accepted into run-through and FTSTA posts in Round 1, and those in GP training posts, begin their new programmes.

Introduction

40. The unheeded warning signs which appeared in 2006 were rapidly translated into crisis in 2007. The introduction of the new Specialty training arrangements in 2007 was strongly and widely criticised, both at the time and in subsequent reviews and reports. The Douglas Review described the events as "the biggest crisis within the medical profession in a generation".[28] The independent Tooke Review called the "failure and abandonment" of the national recruitment process "a deeply damaging episode for British Medicine".[29] The May 2007 judicial review described MTAS as a "flawed system" whose "premature introduction" had "disastrous consequences".[30] The Chief Medical Officer acknowledged that it was "a traumatic and highly unsatisfactory experience" and that "the technical aspects of implementation clearly went badly wrong."[31] This chapter examines the causes of the crisis and the key events of 2007.

Causes and triggers

41. The crisis of confidence in the MTAS system was caused by a number of interacting weaknesses in the planning and design of the national recruitment process. The most significant failing which, we examine below, were:

Serious flaws in the short-listing process;

An excess of eligible applicants because of the last-minute failure to restrict applications from non-EEA doctors; and

Problems with the design of both MMC and MTAS, leading to the perception of MTAS as a "one strike and you're out" process.

It was the combination of these shortcomings which plunged the recruitment process into crisis in March 2007.

THE SHORT-LISTING PROCESS

42. The national on-line specialty training recruitment system opened for applications on 22 January 2007. Candidates were given until 4 February to make up to four separate applications, choosing among the 59 specialties, 16 regional areas, and four levels of seniority available. This deadline was subsequently extended until 5 February after operational problems with the on-line system.[32] The short-listing process was carried out by consultants across the country over the next few weeks, with the interview process scheduled to take place in March and April. Round 1 of recruitment, intended to fill half of the available posts, was scheduled to finish by the end of April. A second round, beginning in May, would give candidates the opportunity to compete for the remaining posts prior to the start of the new programmes in August 2007.[33]

43. During February, there were widespread complaints both from candidates and assessors about the MTAS application form and the short-listing process. The key problems included:

The applications forms relied too heavily on "white space" questions, whereby candidates were asked to give examples of how they satisfied the particular competencies required for the post, rather than verifiable achievements or traditional CV-based information; Furthermore CVs were not available to assessors alongside application forms;[34]
The short-listing scoring system gave undue weight to "white space" answers relative, for example, to educational qualifications;
"White space" answers could easily be plagiarised: there were reports of websites offering "model" answers to applicants;[35]
There was a lack of information, training and time for assessors to carry out short-listing, as well as inconsistencies in the approach to assessment;[36] and
The same application form design and scoring methodology were used for candidates at all levels (i.e. from ST1 up to ST4), even though they were particularly unsuited to distinguishing between more experienced candidates.[37]

44. Witnesses were especially critical of the use of the same type of application form, relying heavily on "white space" questions, for short-listing process at all levels of the MTAS system. Work Psychology Partnership, the company responsible for the design of the application forms, pointed out that their product was only intended for the short-listing of relatively inexperienced doctors applying for ST1 posts:

The Yorkshire Deanery described the decision, made by the Department of Health, to extend the approach used in ST1 selection to higher levels of the training system as a "fatal mistake".[39] Officials did not clearly explain why, or by whom, this decision was made.[40]

45. The medical profession was more generally critical of the short-listing process. The Royal College of Surgeons dismissed short-listing as "fundamentally flawed", while the Royal Colleges of Radiologists called the application form "useless".[41] The BMA stated that the criteria for short-listing "were not acceptable to the profession".[42]

46. The Chief Medical Officer acknowledged the inadequacy of the application form, commenting that "serious aspects…were not fit for purpose, particularly for judging more senior trainees". The Department of Health stated in its evidence that:

Some consultants shortlisting for interview said they found it difficult to differentiate between applicants on the basis of the form. It was also felt that insufficient weight was given to academic achievement in the national shortlisting scoring. The high volume of applications exacerbated these problems…This led to serious concerns that some of the best applicants were not being shortlisted for interview.

Department of Health officials did not explain why they pressed ahead with the introduction of the new recruitment system in spite of these inadequacies.

BAPIO'S LEGAL CHALLENGE

47. Problems with short-listing were compounded by a surge in the number of eligible applicants in February 2007. The failure in previous years to establish which candidates were eligible to apply for training posts, which we describe in Chapter 2, meant that the status of non-EEA applicants was not resolved when Round 1 of applications opened at the end of January. This planning error had profound consequences: on 9 February, the High Court ruled that the Department's employment guidance giving preference to UK and EEA applicants was lawful but granted BAPIO the right to appeal against its verdict.[43] Because the appeal was not heard until October 2007, the Department was unable to implement its guidance during Round 1 of the 2007 recruitment process:

The Court's judgement was received after the recruitment process had started (9 February). It would have been logistically difficult, but possible and legally justified, to implement the guidance at this late stage. However, as shortlisting and invitations to interview were already underway, it was judged that the disruption would be too much, particularly given that Round 1 was expected to fill only 50 per cent of the posts and the option still remained to apply the guidance in Round 2.

48. The Department's decision made it possible for doctors from anywhere in the world who had gained HSMP status by the time applications opened to compete for UK training posts on an equal footing with UK and EEA doctors in 2007. This caused an immediate mismatch between the number of applicants and the number of available training posts, raising the prospect of large numbers of UK-educated doctors being unable to continue into specialist training. Figures from the Tooke Review show that the inclusion of overseas doctors meant there were almost 10,000 more applicants than training posts in 2007, an excess of more than 40%:

Figure 4: Comparison of applicants and training posts in Round 1 of 2007 MTAS process

Source: Aspiring to Excellence, p.67

49. This unanticipated level of competition for training posts radically changed the function of the MTAS selection system, and particularly the short-listing process. As Work Psychology Partnership stated:

50. In effect, the MTAS system had been designed to match candidates to posts on the assumption that there would be enough training posts for everyone. As Figure 4 demonstrates, there would have been more posts than candidates had non-EEA doctors been prevented from applying in 2007. With 10,000 non-EEA applicants, however, the system was required to operate a highly competitive selection process, especially in the most popular specialties and Deanery areas. This process was very different to the matching process for which MTAS had been designed, and for which it was used more successfully during Foundation programme recruitment. Thus the inclusion of non-EEA doctors exacerbated and drew attention to the weaknesses in the short-listing process for specialty training described above.

"ONE STRIKE AND YOU'RE OUT"

51. The inadequacy of the short-listing process and the unexpected inclusion of non-EEA applicants were the principle causes of the 2007 crisis. Other elements of the design of the MTAS system and the structure of MMC further compounded these problems, however, by fuelling the perception that Round 1 of the 2007 recruitment process would be the only opportunity for many candidates to enter the training system:

First, the "preferencing system" allowed each applicant four applications during Round 1. Given that the number of interviews available was fixed, this meant that many applicants received four interviews, reducing the number of opportunities for other candidates, and leaving many with no interview.[45]
Secondly, there was a widespread lack of understanding outside the Department of Health that only 50% of posts were intended to be filled in Round 1 of the process, with the remainder available in Round 2.[46] This increased the anxiety of candidates who were not offered interviews in Round 1.
Thirdly, the centralised application process meant candidates could only make one main application during the whole year, making this a particularly "high stakes" process. This contrasted starkly with the previous system where applicants could apply to many different jobs at many points during the year.
Fourthly, run-through training contracts, lasting up to seven years, were offered in all specialties. This added to the impression that unsuccessful candidates would not get a second chance to enter the training system. The prospect of guaranteed long-term employment also increased the number of applications from overseas doctors.[47]
Finally, a significant proportion of the available posts in 2007, around 23%, were in FTSTA posts, usually lasting for only one year. It was not clear to candidates offered FTSTA posts whether and how they would remain in the training system after these posts expired. Witnesses described the combination of run-through and FTSTA posts as a "two tier system", and characterised doctors in FTSTA posts as "the new lost tribe".[48]

52. Each of these elements of the design and structure of MTAS and MMC contributed to the impression of a "one strike and you're out" system, leaving many candidates who did not receive interviews in Round 1 feeling that they had been permanently excluded from the training system. Because of the perceived inadequacy of the short-listing process upon which decisions to offer interviews were based, many candidates felt that they had been unfairly prevented from continuing their careers. This increased the sense of outrage amongst applicants and assessors alike.

Key events in 2007

THE CRISIS ERUPTS

53. During February 2007, there were widespread reports of high-quality candidates not receiving any invitations to interview. On 27 February, the day before interviews were due to begin, the BMA issued a statement condemning the "shambolic recruitment system" which had "descended into pandemonium". It called for the interview process to be suspended until it could be shown that short-listing had been "consistent and fair".[49] Several days later, a letter from a large group of senior doctors to the British Medical Journal argued that profession had been "torn apart by an Exocet we should have seen coming". The group, which later formed the organisation Fidelio, commented on "top juniors flung on the scrap heaps after a decade of training" and was highly critical of the leadership of the medical profession, describing them as "hand-maidens to their own apocalypse".[50]

54. Strong words from the profession were soon accompanied by strong actions. On 5 March, an interview panel for ST3 general surgery posts in the West Midlands refused to conduct scheduled interviews. The panel stated that "the MTAS procedure for recruitment to ST3 in General Surgery, has not been implemented according to agreed guidelines. We have therefore declined to continue with the interviews today."[51] The group cited a range of specific problems, including unrealistic deadlines, unexpected numbers of applications, inconsistent marking and problems with the reliability of the computer system. It concluded that:

55. In response to the increasing levels of criticism, and in particular to the suspension of the interview process in the West Midlands, the Secretary of State held emergency meetings with leaders of the profession on 5 March. The following day, the Department of Health issued a statement commenting that:

In light of the concerns, the Department announced an immediate review of Round 1 of the selection process, to be completed by the end of March 2007.

THE DOUGLAS REVIEW

56. The Review Group was chaired by Professor Neil Douglas, Vice Chair of the Academy of Medical Royal Colleges. Its 16 members comprised representatives from the Department of Health, including the Chief Medical Officer and the head of the MMC Team, the Royal Colleges and the BMA. The group's task was to assess what had gone wrong with the selection process and to identify changes to be made both during Round 1 and in subsequent rounds.[54]

Whether to continue with MTAS

57. The immediate question facing the Review Group was whether to abandon the MTAS process altogether because of the reported inadequacy of the short-listing system. Professor Douglas told us that this option was seriously considered, describing it as "a very close call on several occasions".[55] However, the Review Group ultimately decided to proceed with Round 1 of recruitment. Department of Health officials argued that "on balance, the benefits of continuing far outweighed the benefits of stopping the process" and stated that the leaders of the medical profession had led the decision to continue with MTAS.[56] Professor Douglas confirmed this:

The Review Group's first statement therefore announced that the recruitment process would continue.

58. Some witnesses expressed considerable surprise that the Review Group did not agree to halt the recruitment process.[58] Fidelio, for example, stated that:

59. Several days later, on 17 March, thousands of junior doctors took part in demonstrations in London and Glasgow.[60] The protests, organised by the increasingly active junior doctors' pressure group RemedyUK, called for the recruitment system to be scrapped. The London demonstration was addressed by the Leader of the Opposition, who described the application system as an "utter shambles".[61]

Reforming the recruitment process

60. The Review Group did, however, announce changes on 9 March "to strengthen the interview process" in Round 1, including making CVs available to interviewers. It also offered candidates not selected for interview in Round 1 the opportunity to have their application form reviewed, with the possibility of an interview subsequently being offered. The group also announced changes to the short-listing process to take effect from the beginning of Round 2.[62]

61. The Douglas Review announced a number of further changes to the recruitment system during March and April. On 16 March, all eligible candidates for ST3 and ST4 posts were offered a single guaranteed interview, regardless of their short-listing score, while further reviews of applications forms were offered to ST1 and ST2 candidates.[63] On 22 March, the offer of one guaranteed interview was extended to all eligible candidates in England.[64] Applicants in Scotland, Wales and Northern Ireland, meanwhile, were guaranteed interviews for all four of their selected posts.[65] Further details were announced on 4 April: additional interviews would take place in May and Round 1 would be extended until the end of June, delaying the beginning of Round 2.[66] These week-by-week adjustments to the active recruitment process had profound implications for all applicants, making this a period of particular uncertainty and anxiety.

62. The guaranteed interview scheme was the most significant change introduced by the Review Group. The offer of at least one interview to all candidates mitigated the impact of the flawed short-listing process, but also created a need for thousands of additional interviews to take place. This extra burden was concentrated in popular areas such as London and the West Midlands and caused Round 1 to be significantly extended. This in turn meant that Round 2 was significantly delayed and the number of posts available significantly reduced. 85% of posts were ultimately filled in Round 1 in England, rather than the 50% originally intended.[67] Thus the Douglas Review ensured that the MTAS process would continue, but in a significantly adapted form.

63. Later in April, the Secretary of State announced a separate independent inquiry to look in more detail at the 2007 problem and to recommend changes to both MMC and MTAS for 2008 and beyond. The independent inquiry was to be led by Sir John Tooke, then Chair of the Medical Schools Council.

RESIGNATION FEVER

64. The decisions of the Douglas Review in March and April were not only unpopular with many doctors, but also with some members of the Review Group itself. Professor Alan Crockard, National Director of MMC and a member of the Review Group, resigned on 31 March 2007. His resignation letter stated that the programme "has lacked clear leadership from the top for a very long time".[68] Several days later, MMC's National Clinical Adviser, Professor Shelley Heard, also resigned after criticising the decisions and the outlook of the Review Group.[69] Professor Heard later told us that RemedyUK's proposal that all appointments in 2007 be for a maximum of one year alone should have been adopted by the Review Group.[70]

65. Further resignations from senior medical leaders followed in May. On 17 May, James Johnson, Chair of Council at the BMA, and Dame Carol Black, Chair of the Academy of Medical Royal Colleges, wrote jointly to The Times. Their letter endorsed the principles of MMC and expressed support for the Chief Medical Officer and the decisions of the Review Group. Following widespread criticism of the timing and content of the letter, Dr Johnson resigned as head of the BMA on 20 May, acknowledging that he had failed to represent the anger and uncertainty felt by most doctors at the time.[71] Dame Carol Black did not resign; she later told us that she was asked to write the letter by Professor Douglas.[72] Around the same time, Mr Bernard Ribeiro, President of the Royal College of Surgeons, resigned from the Review Group, criticising the "scandalous failure" to address problems with the training system.[73]

THE JUDICIAL REVIEW

66. Still more uncertainty was created when RemedyUK sought a judicial review of the MTAS process, asking for the decisions of the Douglas Review to be quashed. At the review, heard in the High Court on 17 May, RemedyUK argued that the changes to MTAS made by the Review Group were "conspicuously unfair"[74] and called either for the recruitment process to be abandoned, or for all posts offered in 2007 to be limited to a maximum of one year.[75]

67. The judicial verdict was passed just days before the first Round 1 posts were finally due to be offered to successful candidates. Thus applicants were unclear until the very last minute whether any valid offers would be made in 2007 and the length of the training contracts which might be offered.[76] The High Court's decision was made public on 22 May: RemedyUK's application was refused and the legality of the Review group's decisions was upheld. Mr Justice Goldring concluded that the Douglas Review had been tasked with satisfying "irreconcilable" competing interests and that the guaranteed interview system was "a possible rational solution and not conspicuously unfair".[77] His judgement was, however, strongly critical of the wider MTAS system:

THE SUSPENSION OF ASPECTS OF THE MTAS SYSTEM

68. At the same time as RemedyUK's legal challenge was taking place, another serious problem with the recruitment system emerged: the on-line application system suffered two security lapses. On the morning of 25 April, candidates' personal details were made available to employers through the MTAS website without adequate password protection. The breach, which made private details publicly available, was discovered in the early afternoon by Channel 4 News, which later informed the Department of Health. The details were subsequently removed and the Secretary of State later informed the House that:

There is no evidence that members of the public or other commercial interests, apart from staff at ITN and Channel 4 News, accessed the site. None the less, it was an extremely serious breach of security, as a well as a breach of contract between the IT provider and the Department of Health.[

69. The provider in question was Methods Consulting, the company responsible for building and operating the online applications system. Methods' Managing Director, Mark Johnston, later blamed the security failure on human error:

70. The following day, 26 April, a second, unrelated security lapse was discovered. Applicants with secure access to the MTAS system were able to randomly access confidential messages, for example regarding interview appointments, sent to other candidates. Following this second security problem, the MTAS website was suspended to allow a full security review to take place.[81]

71. A subsequent Ministerial Statement on 15 May announced that the security review had been completed and that "appropriate and sufficiently comprehensive action has been taken" to address the problems. However, the Secretary of State announced that while the MTAS website would be re-opened for internal use by the Postgraduate Deaneries, it would no longer be used as part of the Specialty training recruitment system. Instead:

72. The decision to abandon aspects of the MTAS system meant that Round 1 had to be further extended, this time until late June, meaning in turn that Round 2 could not be completed before the 1 August deadline. The change also created large amounts of unexpected work for Postgraduate Deaneries who had only a few weeks to design and co-ordinate a fair and efficient process for making offers to candidates, something which was intended to be done by the national online system.[83] The London Deanery told the Committee that its staff worked a total of 7,500 hours of overtime to implement the revised 2007 recruitment system.[84]

THE FINAL SCRAMBLE

73. Following the changes made by the Douglas Review, the decision to scrap aspects of the national application system, the resignations of the leaders of MMC and the completion of the judicial review, the first offers were made to Round 1 candidates on 25 May 2007, more than four months after the application system opened. The offers process, conducted locally by the Postgraduate Deaneries, continued until 22 June 2007 when Round 1 was completed, only a few weeks before successful candidates were to begin in their new posts.[85] The delayed and drawn-out nature of the process meant many candidates were forced to decide whether to accept offers in their second-choice post before knowing the outcome of their first choice application.[86]

74. On 8 June, the MMC team set out revised arrangements for Round 2 of recruitment. The process would be run entirely by Postgraduate Deaneries, using revised short-listing and application systems. 215 extra run-through training posts were to be created, although the total number available would still be far less than originally planned, due to the expansion of the Round 1 process by the Douglas Review. The over-run to the Round 1 process also meant that Round 2 would be conducted in large part after the 1 August deadline. The Department of Health therefore made a comprehensive commitment that all NHS doctors applying to Round 2 would have their contracts extended to cover the time taken by the recruitment process.[87]

75. Following the completion of Round 1 in late June, a Ministerial Statement on 12 July announced that 85% of available posts in England had been filled. Predictably, the fill rate for run-through training posts was higher (91%) than that for FTSTA posts (64%). In academic medicine, an area particularly badly affected by the weaknesses of the selection system, only 57% of posts were filled.[88] Nonetheless, on 1 August 2007, successful candidates in GP and Specialty training posts began work across the country.

Conclusions

76. The introduction of the new Specialty Training arrangements in 2007 was disastrous. The failure to restrict overseas applicants and the manifest weakness of the national recruitment system made the collapse of confidence in the selection process inevitable. The design of the initial application forms was particularly inappropriate, failing to recognise doctors' key achievements and giving undue weight to "white space" questions. The short-listing process, critical to the futures of so many, therefore descended into little more than a creative writing exercise. Candidates and assessors alike were justifiably outraged by the sheer inadequacy of MTAS.

77. The period between February and August 2007 was characterised by unrelenting chaos and severe anxiety for thousands of junior doctors. The repeated changes to the recruitment system, a High Court challenge and the failure to protect the privacy of candidates' personal information ensured that the process was miserable for all the applicants involved. The Review Group, faced with an impossible situation, was unable to restore confidence in the recruitment system. The wave of resignations by senior medical leaders and series of emergency Ministerial statements which followed were both acutely embarrassing for the Government. The reputations of both the Department of Health and the leaders of the profession were severely diminished by the events of 2007.

28 Final Report of the Douglas Review group, 12 July 2007, p.14 Back

29 Aspiring to Excellence, pp. 10 and 13 Back

30 Judicial Review Verdict, 22 May 2007, paragraph 139 Back

31 Qq 2-4 Back

32 Ev 14 Back

33 See MMC-Statement on recruitment and selection to Specialty Training programmes in 2007 Back

34 "White space" questions asked candidates to describe, for example, a situation where they had worked under pressure or made a complex clinical decision in 150 words of prose Back

35 Ev 123 Back

36 For example, forms were marked "vertically" in some areas, with one assessor marking the whole of each form, and "horizontally" in others, with each assessor marking a particular question on each form. Assessors also complained of difficulties printing out application forms, inconsistent pagination, and lack of time to complete assessments, in particular because the short-listing period coincided with the half-term holiday. Back