How does the NAO assess the quality of its value for money studies?

  The NAO collects information for its quality review process from five equally important perspectives. Feedback on each value for money (VFM) study is obtained from the study team, the audited body, the media, the Public Accounts Committee and the London School of Economics and Political Science (LSE). The LSE acts as an external reviewer of all of our VFM work.

What is the aim of the process

  The aim of quality review is to draw out any lessons which the NAO needs to take on board for the future, and to identify good practice which will help to improve the quality of future reports. The outcomes of the review inform the on-going quality assurance of future studies: the continuous process of ensuring that what should be happening on each study actually is happening.

Why does the NAO ask the LSE to review its VFM reports?

  We feel that using one of the most well-regarded universities in the UK to evaluate our work lends credibility to our quality assurance procedures and provides a secure foundation for our own future development. The LSE has demonstrated that they can draw on a breadth of knowledge and experience in work across the whole of government, and that they can provide authoritative comment on study methodologies and the technical rigour of analyses.

What criteria do the LSO use to evaluate NAO reports?

  The LSE awards each report a set of scores, on a scale of 0-5, for the six dimensions of performance listed below.

Administrative and management context	How effectively the subject matter of the report is located in context, and the relevant background is presented
Structure, presentation and format	How clearly the report is structured and presented, and how well the executive summary puts across the main messages
Graphics and statistics	How well charts, figures and tables are handled, and whether the presentation of data is appropriate and adds value to the report
Audit issues and scope	How well the terms of reference of the study are defined and how well the issues examined fit with parliamentary and public concern
Methodology	How appropriate and rigorous the methods and analytical techniques employed in the study are
Conclusions and recommendations	How informative and well-grounded the conclusions are and the extent to which courses of action which would demonstrably improve VFM are recommended

  The LSE review team assigns a score of three under any of these six headings if their readers feel that the report demonstrates a "solid professional performance" on the dimension of performance in question. A score of four represents a "very good" performance, and five is "excellent". Scores of two or below indicate that performance falls below the level that would be expected of an organisation such as the NAO.

  In addition to the scores under the above six headings, the LSE also provides an overall score, on the 0-5 scale, for each report. This is not an average or other function of the six specific scores, but rather an overall assessment, in the judgement of the LSE team, of the extent to which the report has been successful in achieving its aims and furthering the NAO's reputation.

  All numerical scores are supplemented by qualitative discussion (typically a review of an NAO report will run to about five pages of text).

What impacts do the LSE reviews have on the NAO's VFM work?

  The detailed comments in the LSE reviews provide important formative information for VFM study teams. We issue periodic guidance material to VFM staff, which summarises the key messages from recent quality reviews. We examine trends in scores to inform our developmental work. For instance, we highlighted the need for clear explanations of methodologies in materials we issued in 1999. This led to a significant improvement in methodology scores in 2000. We have identified conclusions and recommendations as the next area on which to focus attention, and have recently drawn on quality review information in formulating guidance aimed at improving the effectiveness of this aspect of our studies.

TRENDS IN LSE SCORES FOR NAO REPORTS, 1998-99 TO 2000-01

  The chart below shows the proportion of NAO reports scored at "three" ("solid professional performance") or better, for the financial years 1998-99 to 2000-01. (Figures for 2001-02 are not yet available, as we have not yet received reviews of all reports published up to end March 2002). The overall performance profile, on this measure, has improved over the three years, as indicated by the increasing relative heights of the three lines.

  The average (mean) overall scores, for each of the three years, are as follows:

  The LSE assessment criteria summarised in the chart above are as follows:

  In addition, the overall criterion summarises the LSE team's judgement of the extent to which the report has been successful in achieving its aims and furthering the NAO's reputation. All criteria are scored on a 0-5 scale, with three demonstrating "solid professional performance", four representing "excellent" and five representing "outstanding".