Memorandum by Work Psychology Partnership
(MMC 52)
INVOLVEMENT IN SPECIALTY SELECTION 2007
1. Specifically, our work in 2006-07 comprised
of advising on selection methodology for; (1) Foundation Programmes
in the UK; (2) the GP selection process (shortlisting tests and
selection centres) and; (3) selection into all other specialities
at ST1. The time scale for delivery was extremely tight and we
expressed our concerns at the outset (June 2006). We were also
concerned about the lack of piloting opportunity. The selection
methodologies for both Foundation and GP have been shown to work
well.
2. In relation to Specialty selection, The
Work Psychology Partnership (WPP) responded to a DH Invitation
to Tender in May 2006. The scope of work states; "The number
of applicants expected to apply for entry into Specialty Training
is approximately 6,000 and that applications will be via a single
electronic national portal entry system (separate project) the
working assumption for the closing date will be 5 January 2007".
We were informed that there would be 16 specialties and there
were existing Person Specifications submitted by the AoMRC. Although
there were 16 specialties in the original scope, in late autumn
2006, WPP were asked to deliver an additional 110 Person Specifications
(signed-off by College representatives) to include those in transition
at levels ST2, ST3, ST4 plus FTSTAs). At the outset we were asked
to deliver a shortlisting process for ST1. We were not asked to
deliver the selection methodology for doctors in "transition"
via ST2, ST3, ST4 and FTSTAs. We were led to believe transition
arrangements would be delivered via local processes.
3. In relation to shortlisting for ST1,
WPP recommended to COPMeD the use of a national standardised test
in 2007. However, we were informed that this was not the preferred
option for 2007. By contrast, in collaboration with the GP National
Recruitment Office, WPP designed the shortlisting test for GP,
which has shown to work well.
4. Given the time scale (approximately 12-16
weeks) we had no option but to use existing application form materials.
Our input into design of the application form for 2007 was directly
based on existing application forms previously used for entry
into specialty training (including those used by the London and
Yorkshire Deaneries). WPP were sent the application form template
by the Project Manager (then Ms Carole Mistry, latterly, Ms Janet
Brown).
5. On reviewing existing application forms,
"white space" questions were widely used. This is not
an invention from WPP. "White space" questions were
used in the past, and are used frequently for shortlisting purposes
in medical selection. However, our advice was that if used, they
would not be a long term solution given concerns regarding plagiarism
(see GP experience where our advice and work has involved the
development of other short listing solutions). We were clear that
the use of these questions was an interim solution and that other
short-listing procedures should be developed. Specifically, we
highlighted all the risks to the COPMeD steering group (including
need for training, standardised rating scales etc). In addition,
we were led to believe that MTAS would comprise software to detect
plagiarism.
6. Relevant experience, posts completed
and other CV related information are clearly important indicators
of competence and would normally be used to make selection decisions.
However, we were told that there should be no weighting of previous
relevant experience. This is unusual in a selection context. The
decision not to score experience and previous posts was not ours,
it was the brief given. This was guidance from COPMeD and MMC
team regarding PMETB standards. This is not an interpretation
or new development proposed by Work Psychology Partnership. Similarly,
we were advised that gaining a College exam could not be "scored/used
to rank individuals", even though these are clearly robust
indicators of competence. The shortlisting scoring framework received
legal and HR approval from the Department of Health. WPP were
not responsible for HR input at any stage. HR expertise was provided
by the DH.
7. We were led to believe that there would
be sufficient posts to applicants so that the vast majority of
applicants would gain a training post. Interview capacity was
determined locally. WPP were told that the competition ratios
would not be too high and so short listing would be relatively
"light touch". The actual numbers of applicants was
vastly underestimated by various stakeholders. In fact, the shortlisting
system had to sift over 128,000 applications for approximately
20,000 posts. This inevitably meant that several thousand competent
doctors would not be invited to interview, let alone offered a
post. Our scope of work was for ST1 applicants only and we believed
that the vast majority of applicants would be shortlisted.
8. All our work was signed off by the COPMeD
steering group and the Department of Health (we have a full contact
log to this effect). WPP can provide a full log of all communications
with all stakeholders on request. Both parties stated that they
were happy with our work and that we had fulfilled our contract.
9. WPP were not involved in any way with
the development of selection methodology for Academic posts. We
were asked to provide advice in November 2006 but we declined
because of our concerns regarding the lead time to develop appropriate
scoring frames for the academic community. We were also concerned
about the time scale for consultation. As a result, Professor
Thomas (Chair, COPMeD steering Group) informed us that this was
to be delivered by "Mark Walport's group". WPP had no
involvement in academic recruitment.
10. At the outset, given that the original
project manager left her post (Carole Mistry), there was no formal
discussion about contracting arrangements and we worked at risk
for several months. We tried to address this on several occasions
as the team was working at risk for several months. There were
gaps in project management and we were unclear about who was leading
our contract management. I asked for an urgent meeting with DH
staff in January 2007 as I had major concerns about how our contract
was being managed and the work demands therein. As suppliers,
throughout this project there was a lack of appropriate project
management and as a result, reporting arrangements were very unclear.
We were not presented with a contract to sign from the DH until
17 February 2007, which was nine months after we had commenced
the work. We expressed our concerns throughout this time as the
work plan shifted significantly out of scope.
11. In October 2004, Professor Patterson
presented a list of all the risks of inadequate preparation and
development of the selection methodology at a meeting with the
MMC team (then Dominic Hurndall and Dr Andrew Havers and others).
She recommended a selection methodology comprising selection centres
that could be validated over two years with large scale stakeholder
consultation. Following this meeting, Professor Patterson received
no further correspondence from the MMC team regarding selection
methodology, and no pilots were delivered.
12. On all counts, as suppliers, WPP delivered
our original scope of work plus major and significant out of scope
activities in an extremely tight timescale. The design brief was
changed significantly and our early advice on selection methodology
was not taken up. Within this time we made significant personal
efforts to consult with all key stakeholders, including presentations
to the AoMRC STC committee, and delivering an additional 19 workshops
with stakeholders in an 8 week period. WPP cannot be held responsible
for a reported lack of consultation. Our expertise and long-standing
reputation has been questioned as a result of this process. We
are particularly unhappy that detailed information regarding our
involvement and our responsibilities has not yet been made clear
by the various stakeholders involved, who were responsible for
setting, managing and "signing-off" our scope of work.
November 2007
| 
