Performance and efficiency

PERFORMANCE INDICATORS

58. The Department for Transport encourages the CAA to publish and maintain a statement of the service and standards which it provides to those it regulates.[88] Accordingly, the CAA publishes indicators on a six-monthly basis detailing its service and productivity, its financial performance and the safety record of UK aviation. The latest update was published in August 2006.[89] Box 4 lists the indicators published.Box 4:  CAA performance indicators

Financial indicators: Rate of return; Unit cost of CAA (per flight and per passenger); Operating costs; and Cost per ATOL-protected passenger. Safety indicators: Fatal accident rates: UK registered/AOC fixed wing passenger aircraft above 5,700kg UK registered/AOC fixed wing freighter aircraft above 5,700kg UK registered/AOC fixed wing public transport aircraft below 5,700kg UK registered/AOC public transport helicopters above 2,730kg UK general aviation below 5,700kg Service-level/productivity indicators: Pilot licence issues—average time to issue; Pilot licence issues—licences issued per staff member; Safety Regulation Group's performance against 19 separate service standards; ATOL applications—average time to decision for new applicants; ATOL applications—average time to decision for renewals; and ATOL applications—average caseload per staff member.

Source:  Civil Aviation Authority, CAA Indicators, August 2006

59. These indicators have shown almost universally strong performance in recent years and the CAA identified specific examples in its submission to us. Improvements detailed included reducing the time taken to process new ATOL applications from an average of 5.4 weeks in 1999/2000 to 3.8 weeks in 2004/05, reducing the average number of days taken to grant private flight crew licences and associated licence ratings from 6.5 days in 2001/02 to 4.8 days in 2004/05, and reducing the overall cost per passenger of the CAA's activities from 73 pence in 2001/02 to 54 pence in 2004/05.[90] The CAA also argued that the strong UK safety record demonstrated its effectiveness in discharging its responsibilities in respect of safety.[91]

60. In addition, the CAA made us aware of two benchmarking exercises designed to measure how efficiently it was performing. First, as part of the review of the Safety Regulation Group's costs and charges in 2004, a firm of consultants was employed to undertake an independent benchmarking exercise of the CAA's support/back-office functions.[92] The CAA interpreted the results as showing that the support functions provided an "efficient and high quality service", with the finance and procurement functions shown to represent "good value for money".[93] It told us also that, exceptional performance was reported for all of the functions in at least 30 per cent of the activities included in the review.[94] A second benchmarking exercise compared the CAA Safety Regulatory Group with the national aviation authorities of Germany, Sweden, Italy and Spain. The study found that the CAA was handling 50 per cent more approved organisations per inspector/surveyor than any other national aviation authority and was using the most efficient combination of technical and support staff.[95]

61. While the performance indicators reported by the CAA are encouraging, more information regarding how the outcomes had been achieved and the extent to which CAA policies had directly led to improvements would have been beneficial. For example, in detailing its unit cost performance, the CAA told us that some of the observed reduction flowed from efficiencies and some occurred as a result of staff reductions associated with the development of European regulation, but failed to provide any indication of the relative magnitudes of these two causes.[96] Similarly, with respect to the safety indicators, the CAA acknowledged that there were "many factors" influencing the safety record, but did not provide any estimate of the extent to which its own activities had directly contributed to the observed performance.[97] There was also a lack of detail in the presentation of the results of the benchmarking exercises: in the UK benchmarking study, no mention was made of the findings for the 70 per cent of activities in which exceptional performance was not reported, while the CAA failed to include any discussion of the lessons it had learnt in either instance.[98]

62. The CAA offers, on the whole, a good service, and has proper regard for its own efficiency. The CAA is benchmarking itself against other national aviation authorities and other organisations in the UK. We are encouraged to see that the CAA compares favourably and that, more generally, the performance indicators reported by the CAA are showing continued improvement. However, the CAA could provide further information when publishing assessments of its performance. We recommend that in future, when publishing assessments of its performance, the CAA include details of the level of influence its activities have had on the recorded outcome, the lessons it has learnt in conducting the exercise and the action it plans to take to improve performance further.

63. While most of the CAA's indicators point to strong performance and improving efficiency, we heard a significant amount of anecdotal evidence to the contrary. The British Microlight Aircraft Association argued that it was being asked to maintain standards that the CAA itself could not meet. It told us that, while it had fixed periods to respond to issues raised at audit by the CAA and faced suspension of its approvals if it failed to respond in time, it had waited just under a year for a response from the CAA to a proposed amendment to its Technical Procedures Manual, and that it had been waiting since June 2004 either for the re-issue of an exemption affecting hundreds of its members or for alternative arrangements to be made.[99] Manchester Airports Group also complained about lengthy processes, telling us that the time taken to secure regulatory approval of air traffic control-related equipment could be "unacceptably long", causing delays to projects and ultimately adding to its costs for keeping contracts or project teams supported for longer than might otherwise be necessary.[100]

64. Some of the CAA's customers clearly have dissatisfactions with specific aspects of the service provided. It is important that the CAA ensure it has in place effective systems for receiving feedback from those it regulates about which parts of its service could be improved, and that it responds to any complaints or suggestions in a timely manner and reports on these issues in an acceptable way.

PERFORMANCE TARGETS

65. In 2002-03, the House of Commons Public Administration Committee reported on the use of performance targets in Government departments and agencies. The report identified five Government aspirations for targets, and these are set out in Box 5.Box 5:  The Government's five aspirations for performance targets

Targets provide a clear statement of what the Government is trying to achieve. They set out the Government's aims and priorities for improving public services and the specific results Government is aiming to deliver. Targets can also be used to set standards to achieve greater equity. Targets provide a clear sense of direction and ambition. The aim, objectives and targets in each Public Service Agreement provide a clear statement around which departments can mobilize their resources. This helps in business planning and communicating a clear message to staff and to the various public bodies which contribute to delivering each department's programme. Targets provide a focus on delivering results. By starting from the outcome Government is trying to achieve, the targets encourage departments to think creatively about how their activities and policies contribute to delivering those results. They also encourage departments to look across boundaries to build partnerships with those they need to work with to be successful. Targets provide a basis for monitoring what is and isn't working. Being clear what you are aiming to achieve, and tracking progress, allows you to see if what you are doing is working. If it is, you can reward that success; if it isn't, you can do something about it. Targets provide better public accountability. Government is committed to regular public reporting of progress against targets. Targets are meant to be stretching. So not all targets can be hit. But everyone can see what progress is being made.

Source:   Public Administration Select Committee, Fifth Report of Session 2002-03, On Target? Government by Measurement, HC 62-I, pp 7-12

66. The CAA appears to make limited use of performance targets, with many of the indicators detailed in Box 4 having no associated targets. With regard to the financial indicators, the CAA does not set itself any targets, but it is required by the Government to achieve a certain rate of return each year, as discussed in Chapter 5 of this report. There are targets in relation to the safety indicators, with the CAA aiming to ensure that the frequency of fatal accidents in each category does not increase in line with forecast growth in traffic. Latest figures, covering the period 2003/05, show that fatal accidents are below the forecast levels for all categories of aircraft, other than general aviation helicopters and "other" general aviation aircraft, including gliders, microlights, gyroships, airships and balloons.[101]

67. In terms of the service-level indicators, the Safety Regulation Group has set itself the target of meeting 90% of its service standards at any one time. This target was missed in ten of the last 16 quarters, but the CAA's updated publication provides no indication of which of the 19 service standards have been missed, nor by what margin.[102] The Code of Practice for the ATOL scheme requires the CAA to make decisions within an average of eight weeks of receiving necessary information for both new applicants and renewals, and both of these targets have been consistently met: in 2005/06, new applications took an average of 3.5 weeks and renewals took an average of 1.3 weeks.[103]

68. Given the support within Government for the use of performance targets, it is surprising that the CAA does not make more use of them. In addition to not specifying targets for a number of the performance indicators measured, the CAA also appears to be failing to use the targets it does have in place in the proper manner. In the case of the length of time within which ATOL licence decisions are made, the margin by which the two targets have been passed—between 4.5 and 6.7 weeks—indicates that they are not stretching enough and unlikely to have any relevance to performance. The targets are therefore unlikely to help drive further improvements in performance. In the case of the Safety Regulation Group's performance against its service standards, the failure to separate out findings for each of the 19 standards undermines the accountability function of targets, as the observer is unable to determine in which areas the Safety Regulation Group is underperforming.

69. The CAA is not making proper use of targets as a means of focusing activity and driving improvements in performance. We recommend that the CAA review its performance indicators regularly and consider the value of establishing targets for those indicators currently without them. In addition, we recommend that the CAA review those targets already in place to make sure they are realistic and challenging enough to ensure the organisation and its staff are motivated to seek improvement. When publishing details of performance against its specified targets, it is important that the CAA produce sufficient detail to allow readers to observe the areas in which improvement is required, and the areas in which performance is strong.

TECHNICAL EFFICIENCY

70. Roger Wilshire of the British Air Transport Association told us that his organisation thought that the CAA could make use of new technology to improve its administrative efficiency, while TUI UK, a tour operator group, argued that the CAA should extend its acceptance of electronic documentation such as permits to fly. It also said that CAA website needed "a full review, updating and modernisation as a matter of urgency."[104]

71. The CAA has said that it expects its Electronic Document and Records Management System (EDRM) to "introduce modern technologies to the document lifecycle and CAA records".[105] As a first stage, the CAA launched an Electronic Records Management (ERM) programme. The CAA told us, however, that completion of phase 1 of the ERM programme was delayed by four months following a failure of the ERM software products to meet the acceptance criteria of the CAA. It said that phase 2 of the ERM programme was therefore expected to be similarly delayed, with completion forecast for July 2006. Despite these delays, the CAA told us that it still expected EDRM to be completed in December 2007 as initially intended.[106]

72. There is an opportunity to improve the administrative efficiency of the CAA through electronic means. We recommend that the CAA monitor closely the progress of the Electronic Document and Records Management System and take action as necessary to ensure that it does not fall behind its completion schedule of December 2007.

Transparency and accountability

GOOD PRACTICE

73. The Better Regulation Commission's (BRC) survey of independent regulators in 2003 found that most considered themselves accountable to Ministers and Parliament. In addition to this accountability, the BRC concluded that it was equally important that regulators were more clearly answerable both to those who they regulated and those on whose behalf they regulated. The BRC argued that this required regulators to operate in a more open and transparent way and it identified a number of examples of good practice in this area among the independent regulators it studied.[107] The CAA's use of some of these is discussed below.

ACCESSIBILITY OF INFORMATION

74. The CAA prepares and submits an annual corporate plan covering a five-year period, providing information on objectives, performance indicators, targets, timing and costs. The organisation is required to maintain management information and accounting systems which enable it to review its financial and non-financial performance against its budgets and targets, in line with its corporate plan. It reports its performance against these budgets and key targets in its annual report and accounts.[108]

75. Despite the production of these documents, we heard dissatisfactions from a number of witnesses about the amount of information made available by the CAA.[109] Sir Roy McNulty, accepted that scrutiny review, consultation and the transparency of the CAA could be improved, and he told us that the CAA had become aware that aspects of its role which it thought everybody recognised were in fact "not well understood by a lot of people."[110] One means by which the CAA has already improved its accountability is the publication of its performance indicators on its website in early 2006. The CAA told us, however, that not all of the performance data it gathered was made publicly available.[111]

76. It is apparent that those affected by the operation of the CAA would like to see more information made available. This perception of insufficient accountability is potentially damaging to the relationship the CAA has with its stakeholders. The CAA itself recognises that there is room for improvement. We welcome the publication of the CAA's performance indicators on its website as a step in the right direction, and we wish to see an extension of this to include all of the performance data collected by the CAA. In order to ensure that the CAA is accountable for its performance, not only to those it regulates but also to those on whose behalf it is regulating, we recommend that it enter into discussion with all its stakeholders about what information they would find useful and relevant, and that it publish this information in an appropriate and accessible way, providing a justification for any information which it considers it inappropriate to publish.

APPEALS

77. The BRC made clear the importance of having a simple appeals process in its 2003 report on independent regulators. It said that given the enormous power wielded by independent regulators, it should be open to those being regulated and those on whose behalf the regulating is being carried out to be able to appeal against decisions that significantly affect them. The BRC explained that appeals mechanisms among independent regulators fell largely into two groups: those which have an internal appeals mechanism or tribunal system; and those where appeals have to be made to the Secretary of State or thorough the court system. The report concluded that "the latter process may be too burdensome for some to use."[112]

78. We have heard that the absence of an independent body charged with reviewing the actions of the CAA means that those who are regulated must enter into a complicated complaints process. De Havilland Support Limited explained that, while there was the opportunity to make a complaint under the terms of Regulation 6 of the CAA Regulations 1991, the process was "onerous and hard for the layman to follow without qualified legal assistance."[113] It said that, as such complaints were subsequently judged by the CAA itself, the process inspired little confidence, and went on to argue that some form of independent authority was needed, (such as the Office of the Telecommunications Ombudsman (OTELO) which investigates the telecommunications industry), to whom complaints about the CAA's actions could be directed. It said that the appeals process needed to be simple and straightforward, and accessible to people without the costs associated with formal legal action.[114]

79. The lack of an accessible system for appealing the CAA's decisions is a source of concern for a number of organisations. We recommend that the Government conduct a consultation on the merits of establishing an independent authority, along the lines of the Office of the Telecommunications Ombudsman, to which appeals about the CAA's decisions can be directed.

Better regulation agenda

80. The amount of state intervention involved in regulation varies from model to model. The CAA is an example of an "independent regulator", defined by the BRC as a body established by Act of Parliament but operating at arm's length from Government.[115] Other examples include the Office of Rail Regulation, Ofgem, Ofcom and the Food Standards Agency.[116] The BRC encourages policy-makers to have regard for the full spectrum of regulatory options and ensure that any regulations imposed are "necessary, fair, effective, affordable and enjoy a broad degree of public confidence."[117] To this end, it promotes five principles of good regulation, as set out in Box 6.

Box 6:  Better Regulation Commission's five principles of good regulation

Proportionality	Regulators should only intervene when necessary. Remedies should be appropriate to the risk posed, and costs identified and minimised.
Accountability	Regulators must be able to justify decisions and be subject to public scrutiny.
Consistency	Government rules and standards must be joined up and implemented fairly.
Transparency	Regulators should be open, and keep regulations simple and user-friendly.
Targeting	Regulation should be focused on the problem and minimise side effects.

Source:  Better Regulation Task Force, Principles of Good Regulation, pp 4-6

81. The Chancellor of the Exchequer commissioned Philip Hampton in the 2004 Budget to consider "the scope for reducing administrative burdens by promoting more efficient approaches to regulatory inspection and enforcement".[118] The review covered the inspection and enforcement work of 63 national regulators and the final report, published in March 2005, concluded that the regulatory system was uncoordinated and that good practice was not uniform. The report recommended standardisation of regulatory practice and said that regulators should be held to account publicly for their performance against these standards.[119] It also included a number of recommendations for improving the efficiency of regulatory inspection and enforcement, as detailed in Box 7.Box 7:  Hampton Report recommendations for improving the efficiency of regulatory inspection and enforcement

Comprehensive risk assessment should be the foundation of all regulators' enforcement programmes. There should be no inspections without a reason, and data requirements should vary depending on the level of risks associated with particular business sectors. Resources released from unnecessary inspections should be redirected towards advice to improve compliance. There should be fewer, simpler forms. Data requirements, including the design of forms, should be coordinated across regulators. When new regulations are being devised, Departments should plan to ensure enforcement can be as efficient as possible, and follows the principles of this report. Thirty-one national regulators should be reduced to seven, more thematic, bodies.

Source:  Philip Hampton, Reducing administrative burdens: effective inspection and enforcement, March 2005, Executive Summary, pp 1-2

82. The CAA told us that it aimed to follow the five principles of better regulation . It argued, for instance, that its progressive withdrawal from the regulation of air fares and its adoption of more "flexible and simplified" approaches towards aviation data collection was evidence of its commitment to proportionate regulation.[120] It explained that it had contributed to, and supported the recommendations of, the review of the economic regulators conducted by the BRC in 2001, as well as the Hampton Report. Its assessment of the recommendations of the Hampton Report found that "many of the positive aspects of the report were already incorporated into the CAA's policies and working practices."[121] Sir Roy McNulty explained that the CAA had already changed some aspects of its operations in response to the Government's better regulation agenda, and that it expected this process to continue. He said that the CAA was working with the Department for Transport on a number of the aspects coming out of the Hampton Report and the work of the BRC, and he made clear that the CAA would apply better regulation wherever it could do.[122]

83. The CAA's approach to better regulation was welcomed by British Airways. It told us that it was heartened that the CAA had signed up to the better regulation agenda, but argued that there remained "a great deal of scope to reduce old regulatory burdens without undermining effectiveness."[123] The Chairman of the BRC explained that, while his organisation had not looked at the CAA and the aviation industry in particular, he was encouraged by what he had seen of the CAA's governance documents. He told us that the Sponsorship Statement included many of the aspects the BRC would want to see in it:

"[…] not least the commitment in any primary and secondary legislation to [regulatory] impact assessments, and to driving down the administrative burden, which is very encouraging. In the corporate plan of the CAA I think the words we read about adhering to the principles of better regulation and about the adoption of the principles of the enforcement concordat, all of that is very positive."[124]

84. We note that the CAA seeks to follow the five principles of good regulation and that it recognises the need to continue improving its regulatory performance. We are encouraged that it is willing to engage with the Better Regulation Commission and the Department for Transport in order to achieve better ways of working. We expect to see evidence that the CAA continues to lend its experience to the development of the Government's better regulation agenda.

Regulatory impact assessments

QUALITY AND EFFECTIVENESS OF RIAS

85. Regulatory impact assessments (RIAs) set out the costs and benefits of a proposal and the options for delivering the desired outcome. The BRC has described them as "a central and crucial part of the Government's drive to improve policy-making", and the Prime Minister said in January 2003 that "no proposal for regulation which has an impact on business, charities or voluntary bodies, should be considered by Ministers without a regulatory impact assessment being carried out."[125] The CAA's Sponsorship Statement states that:

"All primary and secondary legislation relating to the CAA will be subject to regulatory impact assessment. The CAA should seek to reduce unnecessary and over-detailed regulation and ensure that necessary regulation is clear and fair."[126]

The CAA told us that, as at January 2006, it had completed 59 RIAs since 2002: 51 within the Safety Regulation Group; six within the Directorate of Airspace Policy; and two within the Consumer Protection Group.[127]

86. The BRC was unable to provide us with any assessment of the CAA's RIAs. More generally, its Chairman told us that RIAs were being increasingly used by independent regulators, but that there were still "very considerable questions" about their quality.[128] In addition to these quality issues, the BRC has argued that RIAs need to be used more strategically and earlier in the decision-making process, in order to have the required influence on policy and in order to have credibility with stakeholders.[129] The overall use and quality of RIAs in Government is monitored by the National Audit Office, which produces an annual evaluation of a sample of RIAs across departments.[130]

87. Regulatory impact assessments (RIAs) are a key element of the Government's better regulation agenda. The assessments produced by the CAA must be of sufficient quality and developed in a sufficiently strategic and timely manner so as to have real influence on its policy-making. As yet, no organisation appears to have evaluated any of the CAA's Regulatory Impact Assessment processes. We therefore recommend that the Government, or the CAA itself, arrange for a selection of the CAA's existing RIAs to be reviewed as a matter of urgency by the National Audit Office or a similarly independent body, and that any lessons learnt are fed into future assessments. As part of this review process it is important that feedback is sought from those regulated by the CAA and those on whose behalf the regulation takes place.

APPROPRIATE USE OF RIAS

88. We heard claims from various witnesses, particularly from the general aviation sector, that the CAA should have produced an RIA in relation to its recent review of the Safety Regulation Group's costs and charges. For example, the Aircraft Owners and Pilots Association explained that, as the changes considered went beyond a known formula, such as the Retail Prices Index, it felt that the Government's guidance on the use of RIAs became relevant.[131]

89. The CAA told us that it firmly believed that it had complied with the principle supporting the RIA process, namely that of evidence-based policy-making. It explained that there was no proposed change in regulatory policy, but rather one in cost recovery methodology, and that the process of change had been initiated over a number of years via the Safety Regulation Finance Advisory Committee. It argued that the proposals arose principally as a consequence of the Joint Review Team (JRT), which comprised representatives from all sectors of the aviation industry, and had operated over a 15-month period. It claimed that it had approached the JRT process in an open and transparent manner and that its finances and efficiencies had been subject to close scrutiny by all sectors of the aviation industry. It concluded that, by establishing the JRT and providing a 12-week consultation period for industry comment, it had explained the changes to the cost recovery process in sufficient detail so as not to require the compilation of an RIA.[132]

90. In relation to its review of the Safety Regulation Group's charges, we are content that the CAA entered into a sufficiently wide-ranging and considered discussion with the industry to obviate the need for an RIA in this instance. Nevertheless, it appears to have failed to make its reasons clear to those it regulates, leading to considerable disaffection among some organisations. We recommend that the CAA consider the merits of conducting an RIA whenever policy changes are proposed, even in instances not strictly stipulated by Government guidance. Where the CAA subsequently decides not to produce an RIA it is essential that it offer an explanation for this and enter into dialogue with its stakeholders to ensure they understand, if not agree, with this reasoning.

89   Civil Aviation Authority, CAA Indicators, August 2006, http://www.caa.co.uk  Back

90   Ev 1, para 85; Ev 293 Back

91   Ev 1, para 69 Back

92   The reference group used for the analysis comprised a variety of best performer organisations from different sectors, which included large FTSE 100 companies, a small number of airlines and public utility companies. Back

93   Ev 1, paras 87-88 Back

94   ibid. Back

95   Civil Aviation Authority, Benchmarking of SRG's Regulatory Activities: A Synopsis for the Transport Select Committee, 22 December 2005 Back

96   Ev 1, para 85 Back

97   Ev 1, para 59 Back

98   Ev 1, para 88 Back

99   Ev 237, Section 1 Back

100   Ev 83, para 2.11 Back

101   Civil Aviation Authority, CAA Indicators, August 2006 Back

102   ibid. Back

103   ibid. Back

104   Q 324; Ev 107, para 4 Back

105   Civil Aviation Authority, Corporate Plan 2005/06, p 33 Back

106   Ev 39, Qq 8-9 Back

107   Better Regulation Task Force, Independent Regulators, October 2003, p 22 Back

108   Ev 189, paras 9, 52 Back

109   See for example: Ev 174, para 13; Ev 101, paras 31, 34 Back

110   Qq 18-19 Back

111   Ev 39, Q 5 Back

112   Better Regulation Task Force, Independent Regulators, October 2003, p 34 Back

113   Ev 230, p2 Back

114   ibid. Back

115   Better Regulation Task Force, Independent Regulators, October 2003, p 6; The Better Regulation Task Force (BRTF) was established in September 1997 to develop and promote the Labour Government's "better regulation agenda", which aims to find more effective ways of delivering protection without placing unnecessary burdens on those who are regulated. The BRTF was given permanent status and renamed as the Better Regulation Commission in January 2006.  Back

116   Philip Hampton, Reducing administrative burdens: effective inspection and enforcement, March 2005, para A1 and para 1.23 Back

117   Better Regulation Task Force, Principles of Good Regulation, p 1 Back

118   Philip Hampton, Reducing administrative burdens: effective inspection and enforcement, March 2005, Executive Summary, p 1 Back

119   Philip Hampton, Reducing administrative burdens: effective inspection and enforcement, March 2005, p 1 and para 4.116 Back

120   Ev 1, para 74 Back

121   ibid. Back

122   Q 63 Back

123   Ev 101, para 38 Back

124   Q 469 Back

125   Better Regulation Task Force, Independent Regulators, October 2003, p 31; Cabinet Office, Better Policy Making: A Guide to Regulatory Impact Assessment, January 2003 Back

126   Department for Transport, Sponsorship Statement for the Civil Aviation Authority, para 7.4 Back

127   Ev 39, note 4 Back

128   Q 468 Back

129   Better Regulation Task Force, Annual Report 2004/05: Better Regulation - from Design to Delivery, December 2005, p 21 Back

130   See for example, National Audit Office, Evaluation of Regulatory Impact Assessments 2005-06, HC 1305, 28 June 2006 Back

131   Ev 126, section 6 Back

132   Ev 39, Q 27 Back