The future of rail: Improving the rail passenger experience Contents

4Passenger satisfaction and TOC performance measures

90.Many passengers are sceptical about current official methods of measuring rail passenger satisfaction and TOC performance, in particular that the measures do not adequately reflect the real experience of rail travel.100 These concerns have been acknowledged by the DfT, the independent consumer body Transport Focus and the Rail Delivery Group (RDG, industry-wide body that includes TOCs and Network Rail), and work is under way to bring about improvements.101 Below we examine the concerns and assess the steps being taken to address them.

Measuring satisfaction: The National Rail Passenger Survey

91.The National Rail Passenger Survey (NRPS), conducted by Transport Focus, is the predominant measure of rail passengers’ satisfaction with rail travel. It surveys over 50,000 passengers per year across the GB rail network. Data from the NPRS can be disaggregated by a wide range of categories, including by TOC; type of operator (regional; London and South East; and long-distance); and journey type (leisure; business; and commuter). It surveys passengers’ satisfaction with their most recent rail journey—passengers are handed a paper questionnaire as they board the train. The survey is carried out in two main tranches, with the results published in Spring and Autumn each year. The NRPS was established in 1999 and provides data in a long time-series that can be used to identify trends.102

92.As noted in the introduction to this Report, the average level of overall rail passenger satisfaction appears high (80% in Spring 2016), but it is down from a peak of 85% in Autumn 2012. Disaggregated by journey type, the data show that long-distance (88%) and regional rail passengers (85%) are significantly more satisfied than those in London and the South East (78%):

Figure 2: Rail passenger satisfaction Spring 2011–Spring 2016, by sector

Source: Transport Focus, National Rail Passenger Survey (Spring 2016), June 2016

On average leisure passengers are the most satisfied (90%), followed by business passengers (82%). Satisfaction amongst commuters is significantly lower and falling—commuter satisfaction has fallen from 80% in Autumn 2012, to a five-year joint low of 72% in Spring 2016. When the data are disaggregated by individual TOC, they show that the least satisfied passengers were those using Southeastern Railway and Southern Railway commuter services in and out of London (both scored 69%):

Figure 3: Rail passenger satisfaction Spring 2011–Spring 2016, by journey type103

Source: Transport Focus, National Rail Passenger Survey (Spring 2016), June 2016

93.While the low ranking of Southestern and Southern Railways reflects the weight of evidence we received from these operators’ dissatisfied passengers, a number of Southern Railway’s London commuter passengers expressed incredulity about even the comparatively low figure of 69%; they simply did not feel it reflected the overwhelmingly negative recent experience of using Southern’s commuter services.104 It should be noted, however, that the negative sentiment of Southern’s many London commuter passengers will to some extent be averaged out by the more positive experiences of other types of passengers on other parts of Southern’s diverse network.

94.A commonly held view was that NRPS results were deliberately manipulated to produce artificially high satisfaction scores. Some witnesses were convinced that that NRPS was deliberately carried out at off-peak times or on quieter routes, to avoid the mass of dissatisfied passengers on the busiest trains.105 Some argued that the NRPS’s “snapshot” method—surveying satisfaction with a single journey on a particular day—produced skewed results, which “rarely reflect the true picture of daily problems faced by regular commuters”.106 They felt that the NRPS should “look at the overall experience, and not just the last journey made.”107

95.Some expert witnesses had a degree of sympathy for this argument. London TravelWatch, the statutory consumer body for the Capital, noted that Which’s annual rail consumer survey considered satisfaction over the previous 12 months, and was therefore able to track passenger sentiment over a longer period.108 Railfuture made a similar point; it believed that the Which? survey’s methodology was more likely to reveal commuter dissatisfaction.109

96.Transport Focus defended the NRPS’s methodology. It noted that its sample size was by far the largest: 50,000 passengers per year in two tranches compared to 7,000 participants in Which’s annual survey. It insisted that its sampling plan ensures an appropriate spread of rail journeys by: journey purpose; peak/off peak; weekday/weekend; and staffed/unstaffed stations. It emphasised that, while the Which? survey typically produced lower satisfaction scores, Which?’s ranking of TOCs in order of passenger satisfaction tended to be consistent with the NRPS. Transport Focus emphasised two key factors that contributed to the lower scores produced by the Which? survey:

97.Broadly, industry and expert witnesses agreed that the NRPS was well-established, methodologically robust and provided useful time-series data to track trends and compare performance.111

98.A number of possible enhancements to the NRPS were suggested, including adopting more modern techniques such as online surveys.112 The Urban Transport Group thought that the views of non-users of trains, including those who were unable to board a train due to overcrowding, were important but currently omitted from the sample.113 A common view was that the main tranches of the NRPS ought to be carried out more frequently.114

99.The DfT’s position was that the NRPS in its current form was useful but had limitations. It told us it was working with Transport Focus and the TOCs to move towards improved, broader and more frequent measures of passenger satisfaction.115

100.Transport Focus told us that it would soon be testing a new approach in relation to the recently let Greater Anglia franchise agreement, which is due to commence shortly. The new approach would focus on measuring three Key Performance Indicators (KPIs): customer experience, presentation of facilities and staff performance. Customer experience would be evaluated via passenger satisfaction surveys using the NRPS methodology; presentation of facilities and staff performance would be measured using “mystery shopper” exercises.

101.Transport Focus told us it intends to evaluate performance against each KPI in every Reporting Period of the franchise agreement i.e. at four-week intervals. Over 7,500 Greater Anglia passengers will be surveyed each year, providing the new operator with “more data in order to better improve the customer experience.” The mystery shopper exercises will cover broader issues, including “cleanliness, upkeep and repair, ticket buying facilities and the attitudes and helpfulness of staff.” The franchise agreement, when published later this year, is likely to include targets against each KPI; failures will result in Remedial Plans and/or financial penalties imposed on the TOC.116

102.The National Rail Passenger Survey (NRPS) is long-established and methodologically sound, but it could and should be improved to more accurately reflect the everyday experience of passengers. We therefore welcome the proposed approach to be tested in relation to the new Greater Anglia franchise, which we intend to further scrutinise in our broader inquiry into franchising.

103.The GB-wide NRPS’s principal weakness is that it is a relatively narrow “snapshot”, carried out in two tranches per year, of passengers’ satisfaction with a particular journey. Steps should be taken to address this, while maintaining the NRPS’s key strength—provision of reliable time-series data by which to identify trends and compare TOC performance, disaggregated by a number of logical categories. We recommend that the GB-wide NRPS be carried out at least quarterly, and that the DfT fund Transport Focus to do this. We further recommend that Transport Focus take careful steps to broaden the NRPS’s sample, including by piloting the use of online surveys and surveying the views of non-users of rail travel, including those who were unable to board a train due to overcrowding.

Measuring performance: Public Performance Measure

104.The headline measure of TOC performance tends to be the Public Performance Measure (PPM)—the percentage of trains that arrive at their destinations “on time” (defined as within five minutes of scheduled arrival time for regional and commuter trains and within 10 minutes for longer distance trains). Official data show that the PPM is over 87.4% across the whole GB network. Performance across franchised TOCs ranges from 70.4% (GTR) to over 97% (c2c).117

105.Broadly witnesses felt that PPM was too blunt a measure, which took little account of the passenger’s experience of train travel. Several pointed out that PPM records only what happens to trains, rather than outcomes for passengers, and in doing so creates perverse incentives for TOCs to take actions to maintain PPM figures, such as missing out stops en route to make up time to the destination station, to the detriment of some passengers. This appears to be a common experience.118

106.Transport Focus argued that better performance measures, which take greater account of the experience of all passengers, were required to improve trust in the industry. Its view was that performance measures should “make sense to passengers and drive behaviours that they want to see”. These included “hard targets” for “right time” punctuality; Transport Focus’s research has shown that passenger dissatisfaction starts to increase well within the five or 10 minute leeway of the PPM measures. For high frequency services, London TravelWatch favoured an “excess journey time” performance measure, as operates on London Underground, as a way of more accurately capturing the real experience of passengers:

This is the total additional time that passenger journeys would have taken compared with the scheduled time, i.e. taking account of the numbers of passengers travelling. For high frequency ‘metro’ services, this is probably a much better measure of reliability than a PPM based on whether trains arrive at their destination [ … ] within 5 or 10 minutes of their public schedule.119

107.The current Public Performance Measure (PPM) does not reflect the real passenger experience, and produces perverse incentives—most obviously, missing out intermediate stops to make up time in reaching the train’s destination—that often work against the interests of many passengers. This is unacceptable and must stop.

108.We recommend PPM be abandoned as the headline measure of TOC performance, and the DfT work with Network Rail and Transport Focus to establish updated “right-time” measures, which take account of punctuality at all stops on a train’s journey. We recommend that a replacement for PPM be implemented by summer 2017.

Insufficient focus on overcrowding

109.Given the huge capacity challenge facing the railway, it is perhaps unsurprising that many witnesses drew attention to uncomfortable, and potentially dangerous, levels of overcrowding. This was a major concern for passengers across the network, not just on the busiest London commuter routes. 120 The Campaign for Better Transport (CfBT) noted that reducing overcrowding was one of the highest priorities for passengers; overcrowding was uncomfortable, highly stressful and, in many cases, forcing commuters to travel to work very early or leave work late, to the detriment of spending time with their families.

110.CfBT emphasised recent evidence that a “troubling number” of individual services were operating at nearly double their recommended capacity.121 The DfT’s official statistics on overcrowding measure the percentage of passengers in excess of capacity (PiXC) and percentage of passengers standing, by TOC and 11 major city regions. These show that in 2015 London services were most overcrowded: on a typical autumn weekday morning during peak hours, trains were on average 6% over capacity, with 24% of passengers standing. The next most overcrowded trains were in Manchester, at 4% over capacity, with 12% of passengers standing.122 In addition, the DfT identifies the “top ten” most overcrowded individual train services in England and Wales. These statistics show that in Autumn 2015, the ten most overcrowded services were between 61% and 129% over capacity. Of the “top ten”, eight were in London and two in Manchester.123

111.A number of witnesses argued that levels of overcrowding should be more effectively encompassed in TOC performance measures. Professor Chris Baker and Dr Jochen Brandt, for example, both believed that trains that are overcrowded to the extent that some passengers are unable to board should be counted against TOC performance in a similar way to late and cancelled trains.124

112.Persistently high levels of overcrowding on particular services on some parts of the rail network are a concern for many passengers. The current official measure of overcrowding, the DfT’s percentage of passengers in excess of capacity (PiXC) statistics, illustrates the problem only in broad average terms, in 11 city regions. In addition, the DfT identifies the “top ten” most overcrowded individual train services. There is a very strong case for more clearly identifying, and taking action to alleviate, substantial overcrowding on specific services across the network. We recommend the DfT review and redesign its PiXC statistics, with a view to more clearly identifying particular train services operating at substantially over capacity. The Department should draw on these statistics to develop a more coherent strategy for tackling overcrowding, including by incentivising TOCs, through franchise agreements, to alleviate the worst examples of persistent overcrowding on particular services across the rail network.

100 See, for example, Iain Kernaghan (RPE0080); Adam Martin-Lawrence (RPE0125); Grahame Mitchell (RPE0041)

101 DfT (RPE0166); Transport Focus (RPE0198); RDG (RPE0202)

102 Transport Focus (RPE0198)

103 Transport Focus, National Rail Passenger Survey, Spring 2016, infographics

104 Caroline Clark (RPE0142); Adam Martin-Lawrence (RPE0125); Iain Kernaghan (RPE0080)

105 See, for example, Iain Kernaghan (RPE0080); Grahame Mitchell (RPE0041)

106 Caroline Clark (RPE0142)

107 Adam Martin-Lawrence (RPE0125)

108 London TravelWatch (RPE0134)

109 Railfuture (RPE0138)

110 Transport Focus (RPE0198)

111 See, for example, FirstGroup plc (RPE0208); ORR (RPE0220); London TravelWatch (RPE0134)

112 Virgin Trains (RPE0204)

113 Urban Transport Group (RPE2029)

114 DfT (RPE0166); Rail Delivery Group (RPE0202); Virgin Trains (RPE0204)

115 DfT (RPE0166)

116 Transport Focus (RPE0249)

117 Network Rail, ‘Performance’, accessed 24 August 2016

118 See, for example, Dallas Roulston (RPE0047); Ed Aldridge (RPE0129); Dr Tom Haines (RPE0131); Paul Sparks (RPE0213)

119 London TravelWatch (RPE0134)

120 See, for example, Tyne and Wear Public Transport Users Group (RPE0163); Sherborne Transport Action Group (RPE0174); Dr Jochen Brandt (RPE0177); Emma Sullivan (RPE0203); Shouvik Datta (RPE0222); Martyn Clark (RPE0227)

121 Campaign for Better Transport (RPE0206)

124 Prof. Chris Baker (RPE0022); Dr Jochen Brandt (RPE0177)

© Parliamentary copyright 2015

13 October 2016