Select Committee on Science and Technology Third Report


2  OBJECTIVES AND PERFORMANCE MEASURES

Table 1: Objectives for the Office of Science and Technology for 2003-04 to 2005-06

Key area Objective
Research RO1: To continue to improve the excellence, relevance and impact of the knowledge created from Research Council-funded programmes.
  
RO2: To increase research capability and international competitiveness of the UK in new strategic areas.
  
RO3: To increase the dynamism and flexibility of Research Council programmes to respond to changing requirements and opportunities, and to support effectively multi-disciplinary research, new researchers and higher risk research proposals.
  
RO4: To maintain access for scientists working in the UK to the necessary major facilities, databases and supporting laboratory infrastructure that will enable them to deliver world-class research.
Training TO1: To raise the standard of postgraduate and postdoctoral researchers, and increase their numbers in priority fields experiencing shortfalls or recruitment difficulties.
  
TO2: To enhance their training to better fit them for careers requiring research skills and experience and increase their attractiveness to future employers.
Knowledge transfer KTO1: To increase the performance of the science and engineering base in exploiting the results of its research.
  
KTO2: To increase the effectiveness of knowledge transfer from Research Council institutes in line with the recommendations of the Baker review of public sector research establishments and the NAO Report on commercialisation of public sector science.
Science in society SSO1: To enhance public awareness of the outcomes from and priorities for publicly funded science and increase openness over its management and use through greater engagement and dialogue with the public.
  
SSO2: To increase the reach and impact of activities undertaken by the Research Councils and other bodies funded through the Science Budget by improving joint working between them and other organisations.
Operational OO1: To complete work on implementation of the recommendations of the 2001 Quinquennial Reviews.
  
OO2: To meet the Government's requirements and targets concerning the freedom of information, e-business (including electronic records management), the modernisation of public services and the promotion of racial and gender equality of opportunity.
  
OO3: To have established the systems to support a co-ordinated performance management system for the Science Budget and the Research Councils in time for the next Spending Review.

Source: Department of Trade and Industry, Science Budget 2003-04, 2005-06, December 2002

6. The Office of Science and Technology (OST) is not a Government department in its own right and thus has no headline Public Service Agreement (PSA) targets of its own. Nonetheless, one of DTI's PSA targets, widened in scope as part of Spending Review 2004, clearly relates to the work carried out by OST:

    PSA target 2: "To improve the relative international performance of the UK research base, the overall innovation performance of the UK economy, making continued progress to 2008, including through effective knowledge transfer amongst universities, research institutions and business".[3]

PSA target 2 is designed to meet DTI's "Objective II": "promoting world class science and innovation".[4]

7. The Science Budget for 2003-04 to 2005-06 contains an additional set of objectives, outlining the areas in which investment from the Science Budget can feed directly into wider Government strategy for science and innovation.[5] These are given in Table 1, along with OST's operational objectives.

The Evidence Ltd Reports

8. In 2002 DTI commissioned the consultants Evidence Ltd to develop a wide-ranging set of indicators to benchmark the UK's performance in science and engineering against that of a comparator group of 25 countries, including all the countries in the G8. Evidence Ltd's first report was published in October 2003.[6] Using its findings, DTI was subsequently able to report that the UK was "on course" to meet its targets. A more conclusive statement was not considered possible because "it will take some years before the significant increase in the Science Budget begins to be reflected in a change in the UK's performance".[7] Following our evidence session with Ministers on 1 November this year, we asked when a discernable change in the UK's performance could be expected. In response, we were informed that "the outputs of research are often not realised until 3 to 6 years after the start of funding and the innovation process may take approximately eight years on average".[8] This long lag time makes it difficult to draw any firm conclusions yet about the outcomes of the Government's increased investment in science and innovation.

9. Following publication of the second Evidence Ltd report, in October 2004, the DTI again reported that the UK is "on course" to meet its targets, although this statement is fairly meaningless given the time lag detailed above.[9] In its Autumn Performance Report 2004, it reports that the UK has an 11.9% share of world citations; is second in six out of the nine broad scientific disciplines; has 5.8 researchers per 1,000 members of the workforce; and leads the G8 for citations per unit of expenditure on publicly performed R&D, citations relative to GDP, and citations per researcher.[10] Given that there are very few changes in the data between 2003 and 2004, few firm conclusions can be drawn, particularly because several of the indicators are subject to year-on-year movements for external reasons, such as changes in GDP. However, the fact that little has changed since the October 2003 report itself supports the Committee's argument, made in its 2003 Scrutiny Report, that a biannual, rather than an annual, study might be sufficient.[11]

10. One of the most noticeable changes charted in the two Evidence Ltd reports is a decline in the UK's share of global publications. When we asked Ministers to comment on this on 1 November, Lord Sainsbury told us that "people tend to not appreciate that with more and more countries becoming developed […] the likelihood is that in terms of the amount of science, it is likely to be a declining amount simply because more and more countries are becoming wealthy".[12] However, the rise of less industrialised nations such as China and India does not explain why the UK has fallen behind Japan. In answers to further written questions, the Government told us that this could be largely explained by the "steady decline in government funding of R&D as a percentage of GDP that took place throughout the 1990s in the UK […] while Japan maintained relatively consistent levels of investment and overtook the UK in the late 1990s".[13] Performance indicators inevitably tell us as much about past decisions as they do about the current situation. We hope that the areas in which the UK's performance is now suffering due to previous underinvestment will remind the Government of the importance of continuing to increase its investment in science, even when improvements are unlikely be seen in the short term. This is key to ensuring that the UK does not slip further against the performance of other countries in future years.

11. In our Scrutiny Report 2003 we expressed concern that DTI was in the process of agreeing a number of new high-level measures to assess the UK's performance against PSA target 2, particularly if the new measures replaced the useful and detailed measures already being used by Evidence Ltd. The new measures were published as part of the Science and Innovation Investment Framework 2004-2014. Some of the Evidence Ltd indicators have been adopted as targets in the Investment Framework, but many have not. For example, in measuring outputs, outcomes and productivity, the Investment Framework focuses on citations, with no measures for the total number of publications. Whilst the Evidence Ltd indicators are, on the whole, specific, measurable and comparable across countries, many of the new measures are unspecific, vague and more difficult to use in comparisons. For example, the Investment Framework sets a goal "to improve success rates in [post-16] SET" with no indication of what would constitute a sufficient improvement. Similarly, the indicator of progress for public engagement is set as "evidence of improvement" with no indication of what evidence would be acceptable.[14] No explanation is given about the process of selection used by DTI in the formulation of the new basket of measures, and there is a danger that some very important detail has been omitted. For example, the focus on citation rather than publication rates masks the fact that the UK has been slipping in its global share of publications in recent years. In addition, citation rates are not a good measure of research outputs of primarily local importance.

12. The detailed indicators used by Evidence Ltd in 2003 and 2004 have been extremely useful in assessing the performance of UK science within an international context. It is not clear that the new high-level performance measures adopted by the Government in its Investment Framework will be either as clear or as useful. We recommend that DTI continues to use the Evidence Ltd indicators to ensure that it receives, and projects, an accurate message about the performance of its Science and Engineering Base. We further recommend that DTI does not change its indicators again without good reason in order to allow year-on-year comparison.

OST's new performance management system

13. On 7 December 2004 we held an informal meeting with Professor Sir Keith O'Nions, the Director General of the Research Councils, and other officials from OST on the developing thinking behind the formation of a new performance management system for the science and engineering Base (SEB). The purpose of such a system would be to ensure that Research Council investment maximises the potential of the SEB, particularly in meeting national economic and public service objectives. We look forward to discussing the new system with OST more formally once the details have been announced.



3   HM Treasury, 2004 Spending Review: Public Service Agreements 2005-08, Cm 6237, July 2004, p 31. Back

4   As above Back

5   DTI, Science Budget 2003-04 to 2005-06 (December 2002), p 9 Back

6   DTI, PSA target metrics for the UK Research Base (October 2003) Back

7   DTI, Autumn Performance Report 2003, Cm 6067, December 2003, para 3.4 Back

8   Ev 54 Back

9   DTI, PSA target metrics for the UK Research Base (October 2004) Back

10   DTI, Autumn Performance Report 2004, Cm 6442, December 2004, p 8 Back

11   HC (2003-04) 316, p 9 Back

12   Q 171 [Lord Sainsbury of Turville] Back

13   Ev 54 Back

14   HM Treasury, DTI and Department for Education and Skills, Science and Innovation Investment Framework 2004-2014 (July 2004), pp 165-166 Back


 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2005
Prepared 31 January 2005