House of Commons - Liaison Scrutiny Unit Review

Select Committee on Liaison Scrutiny Unit Review Memoranda

5 Performance reporting against targets

50. Progress against PSA targets for SR02 and previous spending reviews varies across departments. This report does not consider how well individual departments are progressing, but is instead more interested in how performance is monitored and presented in the annual reports.

51. Treasury guidance states that one of the main functions of the DARs is to set out progress against departmental and cross-cutting PSA targets. To this end, the Treasury has produced a list of principles for departments. The principles most relevant for the issues raised in this report include:

There is a continuing duty to report objectively against all targets.

All departments must report latest outturn data against their PSA targets and make it clear what period or date is covered.

Reporting should be clear and informative. For example, the department should explain the factors affecting achieved performance and the reasons for reported outturn. Departments should ensure they give a full and accurate picture of progress by basing reporting on the targets and measures set out in their technical notes.

Departments should describe the quality of their data systems. Where there are significant limitations in the data system or the data used for a specific target, these should be identified and the implications for interpreting progress against the target should be brought out for readers. Where systems have been externally validated, the results should be summarised and the department should explain how they have, or are, responding to any negative conclusions.[16]

5.2 Data validation

52. There are several examples within the 2005 DARs of departments failing to adhere to these principles. Perhaps most fundamentally, question marks exist over the quality of the data underpinning departments' PSA performance monitoring.

53. In March 2005, the NAO published the findings from validations of 64 data systems covering the 2003-06 PSA targets of seven departments and the cross-cutting Sure Start programme.

54. Problems were identified with around one half of the systems considered, with the NAO pointing out that many of the problems identified had more to do with departments giving insufficient attention to data quality issues rather than the system weaknesses being intractable or too expensive to remedy. Box 14 summarises the NAO's conclusions.[17]

BOX 14: FINDINGS OF NAO'S VALIDATION OF PSA DATA SYSTEMS

55. Some data problems are evident in the 2005 DARs. Data recording performance against DFID's PSA 2 sub-target 7 is described as "very patchy" for example, with figures varying "enormously between different survey sites in the same country".[18]

56. In other instances, the data used for measuring progress takes considerable time to collect, due either to the nature of the target or to poorly managed data systems. Such time lags make it difficult for those managing the target to monitor their performance and take any necessary remedial action. For example, in relation to progress against its PSA 4 target, the FCO told the Foreign Affairs Committee that:

Assessment against the statistical measurement indicators for the target relies on the publication of data by external sources. The timing of the publication of this data means that there is a time lag between the period being assessed and date at which an informed assessment can be made. That is why a complete assessment of performance against the PSA 4 target cannot be made until autumn 2006, which is when the Stockholm International Peace Research Institute [SIPRI] is due to publish battle related fatalities data for 2005.[19]

57. Many of the failings identified in the NAO's analysis will be less obvious than this. In these instances, the reader is dependent upon the department making explicit reference to the reliability of figures. In the absence of such references, it is difficult to take departments' assessments of progress against targets at face value. It is therefore essential that departments work towards improving the quality of the data underpinning their PSA targets and make readers aware of continuing vulnerabilities.

58. The Treasury's PSA 9 target is concerned with helping departments meet their PSAs. The Department reports in its DAR that it is "on course" to achieve this, with around 70 per cent of those SR02 targets for which an interim assessment has been made thought to be on course. The data issues identified by the NAO makes this claim questionable, however. While reference is made to the NAO report in the Treasury DAR, there is no discussion of the findings and no mention of plans to help departments improve data quality.[20]

5.3 Assessments of progress towards targets

59. Even if the quality of departments' data systems is not called into question, the 2005 DARs include assessments which are difficult to support given the evidence presented.

60. Some of the DARs fail to make any reference to figures, meaning that the analysis is impossible to corroborate. For example, the NIO report claims that progress against the value for money objective for reducing the cost of a prisoner place over three financial years was met for 2003/04 and is on target for 2004/05 but fails to provide any supporting data.[21]

61. The reader is unable to determine the progress of seven of the FCO's PSA targets. Little specific data is given and there is no indication of target deadlines. Moreover, the score cards for PSAs 1, 2 & 3 are classified, which presents a serious problem to clear and open reporting. Other assessments frequently discuss figures that are not part of the agreed measures.[22] In response, the FCO has told the Foreign Affairs Committee that:

The nature and content of many of the FCO's PSA targets means that we rely on forming often subjective, qualitative assessments of progress.[23]

62. Where figures detailing performance against the measures agreed in departments' technical notes are not available, some DARs present assessments based on undefined internal evaluations. Box 15 highlights the example of the DTI.[24]

BOX 15: USE OF ALTERNATIVE INFORMATION FOR ASSESSMENT OF PROGRESS (DTI)

63. Other departments detail targets which they claim are on course despite declines in performances from baselines, as shown in Box 16 in the case of the Department for Culture, Media and Sport (DCMS).[25]

BOX 16: ASSESSMENT OF PROGRESS WHERE INTERIM TARGET IS MISSED (DCMS)

64. Elsewhere, departments conclude that performance is on course despite acknowledging that it is currently "too early to say". For example, in reporting progress against PSA 7, the DTI initially concludes that it is "on course" but later concedes that: "In general, it is too early and there is too little data on GVA [gross value added] or key drivers of economic growth to draw conclusions on movements or trends".[26]

65. Other issues include the revisions of baselines for some targets. While such revisions may be legitimate reactions to improvements in data collection and quality, it is not clear why targets are not correspondingly revised. For example, DFID's PSA 1 sub-target 2 calls for an increase in primary school enrolment from 58 per cent to 72 per cent. Although the baseline has now been revised to 68 per cent, the target has remained unchanged.[27]

66. Given the uncertainty surrounding some departmental assessments, the Public Administration Select Committee, in its consideration of how well PSA targets were working in July 2003, recommended that the NAO be given responsibility for validating target assessment as a logical extension of its duty to validate the data systems for performance reporting, which had only ever been envisaged by Lord Sharman as a "first step" towards validation of key published data.[28] The Government rejected the recommendation, however, arguing that:

There should be some form of external validation of departmental data systems relating to PSA targets, but… validation of data systems is a significant and sufficient step to take. An extensive programme of validation of the performance data itself, covering the judgements that departments make on the basis of it, would serve little additional purpose.[29]

67. It is worth noting, however, that all targets set for local authorities and health authorities require independent validation of progress by regulators such as the Audit Commission. It is not clear why PSA targets should not face similar scrutiny.

5.4 Commentary on performance against targets

68. However a department assesses its progress against PSA targets, it is important that the annual report includes a commentary on performance. In the 2005 DARs, such discussions were often insufficient, particularly in relation to targets subject to "slippage".

69. Box 17 details two Department of Health (DH) SR02 targets, and shows that, while the data changes behind the assessments of "slippage" are described in some detail, no explanations are provided.

BOX 17: EXPLANATION OF PAST PERFORMANCE (DH)

Source: DH, Departmental Report 2005, June 2005, p23

70. DARs should be both backward- and forward-looking and, just as some departments failed to include adequate explanations of past performance in their reports, so others failed to include sufficient detail relating to future initiatives designed to improve performance.

71. The FCO report is primarily retrospective, for example: its Progress towards PSA8 box only highlights what has happened and makes no reference to future plans.[30]

5.5 Joint targets

72. The NAO has recently completed a report on targets shared across departments and concluded that they can help to improve the delivery of government objectives, but that they present particular challenges. Issues identified by the NAO included:

the setting of objectives and priorities;

developing a common understanding among partners of how to achieve the shared target and what their respective roles and responsibilities are;

implementing appropriate working arrangements; and

monitoring and reporting performance.

73. The NAO found that many of the targets it examined were not supported by a detailed joint examination of the causes and factors which would lead to the desired outcomes, and that plans generally did not set out the programme and administrative resources to be used.

74. In relation to the reporting of performance, the NAO concluded that where joint targets were expressed around quantitative indicators, partners drew on the same sources, but that in several instances assessments differed, with one partner claiming to be on course while another reported slippage.[31] We identified similar problems in relation to the joint targets reported in the DARs, as detailed for the Home Office and the Department for Constitutional Affairs (DCA) in Box 18.[32]

BOX 18: DIFFERING ASSESSMENTS OF PROGRESS AGAINST JOINT TARGETS (HO & DCA)

75. DCA has subsequently accepted that assessments have differed. In relation to asylum applications the Department told the Constitutional Affairs Committee that:

The correct assessment for this measure is "achieved". This element of the target is a Home Office measure.

Officials in the two Departments had been working together throughout the drafting of their Annual Reports but unfortunately, late changes in the assessment of some of the Home Office measures had not been picked up.

Officials in both Departments are seeking to review the mechanism currently in place, to avoid such errors from happening again. DCA and Home Office have always worked very closely in managing this joint PSA target.

Similarly, on enforcement of immigration laws, the Committee was told:

The correct assessment for this measure is "slippage". This element of the target is a Home Office measure.

As with question 13 - PSA 5 (Measure 2) - officials in the two Departments were liaising but, unfortunately, late changes had not been picked up. We are working to avoid a repeat of such errors.[33]

76. The NAO suggested in its report that the centre can encourage the adoption of best practice in relation to joint initiatives by:

the Treasury ensuring that the specification of joint targets reflects agreement between all the partners and promoting the value of joint delivery planning among the owners of a joint target;
the Treasury, with the support of the Cabinet Office, facilitating the co-ordination of departmental planning, monitoring and reporting timetables to support joint processes; and
the Treasury and the Cabinet Office providing advice to departments on ways to best organise joint working arrangements, on the basis of an understanding of what has worked well.[34]

77. Understanding and overcoming the challenges posed by shared targets is of growing importance, with 20 per cent of SR04 targets being joint compared with 10 per cent in previous spending rounds.

16 HMT, Public Expenditure System: Guidance for the Spring 2005 Departmental Reports, PES (2004) 19, 30 November 2004, para 16 Back

17 NAO, Public Service Agreements: Managing Data Quality - Compendium Report, 23 March 2005 Back

18 DFID, Departmental Report 2005, June 2005, p191 Back

19 FCO, Response to Foreign Affairs Committee written questions, Q30 Back

20 HMT, Departmental Report, June 2005, pp37-38 & p76 Back

21 NIO, Departmental Report 2005, June 2005, p99 Back

22 FCO, Departmental Report 2005, June 2005 Back

23 FCO, Response to Foreign Affairs Committee written questions, Q28 Back

24 DTI, Departmental Report 2005, June 2005, pp22-23 Back

25 DWP, Departmental Report 2005, June 2005, p13 Back

26 DTI, Departmental Report 2005, June 2005, pp39-41 Back

27 DFID, Departmental Report 2005, June 2005, Annex 3, p185 Back