House of Lords - The Effectiveness of EU Research and Innovation Proposals

The Effectiveness of EU Research and Innovation Proposals - European Union Committee Contents

CHAPTER 4: Identification, Monitoring and Evaluation

BOX 1

Commission Criteria for an Effective Proposal

The Commission stated that proposals for effective policies to support R&I should:

contribute to the EU's objectives;
be based on sound analysis supported by the best data available;
draw lesson from evaluations of any previous initiatives;
make use of forward looking studies;
have a clear intervention logic;
take account of the needs of stakeholders, and as far as possible have their support;
have a clear EU added value;
be effective and efficient in achieving the objectives;
involve the simplest possible administrative procedures; and
make concrete provisions for future monitoring and evaluation.[81]

Excellence

52. The criteria identified by the Commission as necessary for an effective R&I proposal are set out in Box 1 above. Witnesses broadly agreed with these criteria. However, they also made very clear that a key consideration in deciding on the efficacy of a research proposal should be whether it is founded on excellence. For UCL this was the sole criterion for an effective proposal,[82] and for others it was a non-negotiable aspect of a successful proposal.[83] The aim of "excellent science" is one of the three key conceptual pillars underpinning the Horizon 2020 framework.[84] In the Communication on Horizon 2020 the Commission pledged to raise the level of excellence in Europe's science base in order to ensure long term competitiveness. It aims to make Europe an attractive location for the best researchers to carry out "frontier research".[85] The Commission issued a call for experts in January 2013 to evaluate the Horizon 2020 project proposals, which should promote excellence by broadening the pool of potential evaluators.[86]

53. Despite the Commission's stated commitment to promoting excellence, some witnesses questioned whether this works in practice.[87] Witnesses referred to the EU regulatory environment as an indicator that non-scientific political considerations sometimes influence policy decisions in the R&I area. The Agricultural Biotechnology Council (ABC), the umbrella organisation for the agricultural biotechnology industry in the UK, expressed concern at the EU's evaluation processes, arguing that the existing EU regulatory system in the field of agricultural biotechnologies is somewhat dysfunctional, and "beset by political interference by those ideologically opposed to the use of technology, and decision making has not been based on scientific evidence".[88] The regulatory process in the biotech industry works through developers of biotech crops, such as ABC's members, submitting biotech crop applications to the European Food Standards Agency (EFSA), which should then make a science-based independent recommendation for approval or rejection to the Commission. However, citing this process as an example of political considerations obstructing innovation, the ABC noted that only two genetically modified (GM) products from over 25 waiting for assessment had been approved for cultivation in the European Union over the past 14 years.[89]

54. The Minister seemed to share these concerns. He observed that, while it may be an inadvertent consequence, the current regulations assume a given way of doing things which inhibits the development of new technologies. He cited the example of the EU banning the use of Bisphenol A in making babies' bottles in 2010, as one of many examples of the EU instituting restrictions on a technology to address a risk which is unproven by any reliable scientific evidence.[90] In line with the views put forward by the ABC, he argued that such delays mean that researchers in Europe develop potentially 'winning' technologies at a much slower rate than competitor countries.[91]

55. Certainly, delays and obstructions in these areas are of particular concern, given the importance of biotechnology and advanced materials, as demonstrated by the list of "eight great technologies" in a publication written by the Minister in collaboration with the think tank Policy Exchange.[92]

56. We agree with the European Commission's criteria for an effective R&I proposal, but emphasise that in order for the EU to compete with emerging economies which have a significantly higher spend on R&I as a proportion of GDP, it should prioritise excellence. We also urge the Commission to ensure that analysis of R&I policy and proposals is based on scientific evidence, rather than political considerations.

Impact Assessments in the field of R&I

57. In order to ensure that the criteria for an effective proposal are adhered to, the Commission stated that it carries out impact assessments on proposals with the most significant economic, social and environmental impacts, or those that are politically sensitive.[93] However, the Commission's Impact Assessment Guidelines[94] do not set out precisely when an impact assessment is, or is not, required. This decision is made annually, by the Directorate of the Secretariat General responsible for each specific policy area, on a case-by-case basis. The guidelines do state that an impact assessment will, in general, be necessary for all legislative proposals. If it is decided that an impact assessment is not to be undertaken, reasons will be given, unless the programme or proposal is outside of the Commission's legislative work programme.[95]

58. Commission impact assessments are carried out at different levels. With reference to Figure 1, an impact assessment is carried out for an overarching framework programme, such as Horizon 2020, or the Digital Agenda framework. A second impact assessment can then be carried out for a specific initiative within the framework, such as for the Commission's strategy on cloud computing.[96] In its written evidence, the Commission said that impact assessments are an important means of ensuring that its political decision making in this area meets the requirements it outlines for a successful proposal.

59. In 2006, the Commission set up an Impact Assessment Board, to examine and issue opinions on all the Commission's impact assessments, independently of the policy making departments. It is chaired by the Deputy Secretary General responsible for better regulation, and is independent of the policy departments. As well as issuing opinions on the quality of individual draft impact assessments, the Impact Assessment Board provides advice to the Commission departments on the necessity of an impact assessment for a particular proposal, and the appropriate methodology to be used in the early stages of preparing an impact assessment. The opinions of the Board are not binding on the Commission, but they do accompany the draft initiative together with the impact assessment report throughout the Commission's political decision-making.[97]

60. The Commission described its impact assessment system as a transparent way of comparing policy options, citing a study by the European Parliament, which compared the impact assessment systems of eight Member States and the Commission. It found that the Commission's impact assessment system is comparatively well-developed with both internal and external checks and balances.[98] The Commission also highlighted that the 2010 report by the European Court of Auditors concluded that the impact assessment system has been effective in supporting decision-making within the EU institutions and that the Commission's Impact Assessment Board was found to contribute to the quality of the impact assessments.[99]

61. This view was broadly reflected in our evidence.[100] The Government stated that Commission R&I proposals are routinely accompanied by impact assessments, and also noted that scrutiny of impact assessments by the Impact Assessment Board is "rigorous and challenging".[101] They cited the impact assessment which accompanied the Horizon 2020 Communication as an example of the interaction between the Impact Assessment Board and the Commission working well, noting the "robust"[102] detail on economic impacts.[103]

62. Similarly, in response to the question in our call for evidence on whether EU proposals clearly state their desired impacts, ADS observed that the feedback from their member companies was positive.[104]

63. However, witnesses including the Government, indicated that a stronger role for the Impact Assessment Board would be beneficial. Both UCL, Universities UK and the UK HE International Unit regretted that despite its ability to provide useful data on impact assessments, to date, the Impact Assessment Board has only been used for piloted initiatives and FP7 research programmes. On the impact assessments themselves, one witness observed that, while they have the positive effect of encouraging policy makers to think carefully about what they intend to achieve, they could potentially become a formality or "box ticking exercise".[105] RAND Europe, a not-for-profit public policy research institute that the Commission has contracted to conduct research used in impact assessments, argued that researchers subcontracted by the Commission are not empowered to go beyond the Commission's "prescribed specification" of possible impact.[106]

64. Witnesses mentioned the particular difficulties in carrying out an impact assessment in the field of R&I, in that the objective of much R&I is to find something new, which in itself is difficult to predict and measure.[107] The Government stated that the nature of research makes it difficult to quantify economic and societal impacts with certainty, especially in cases where these would take considerably longer to materialise than the relatively short timescales for evaluation that EU R&I allows for. As the Minister remarked, "[The Commission] are dealing inevitably with uncertainties and sometimes … the figures are suspiciously precise."[108]

65. The Government, and other witnesses, also noted that many positive outcomes cannot easily be encompassed in an impact assessment; such as the cross-fertilisation of different approaches and skills, and the development of networks of collaboration.[109] The Institute of Physics and Professor Ritter observed the importance of participation in networks.[110] Similarly, EADS suggested that it is "difficult to make a realistic, useful and accurate assessment of the actual impact achieved, especially if this would … have its main effects in an unexpected way".[111]

66. This Committee conducted a report on impact assessments in 2010, agreeing in part with the Commission's assessment that its impact assessment system was effective.[112] We acknowledged that there would be value in further work to determine if the process for deciding whether a proposal requires an impact assessment is appropriate. Indeed, when scrutinising some of the Commission's work programmes, we considered that an impact assessment would clearly have been helpful. For example, an impact assessment was not carried out for the Smart Cities programme, despite its potential economic and social impact.

67. Our 2010 report also welcomed the actions of European Parliament committees in commissioning impact assessments where they considered that the Commission assessment was inadequate. As part of the current inquiry, we heard from Malcolm Harbour MEP on the role of the European Parliament's European Added Value Unit set up in 2010. He described its role as a form of 'critique' on the Commission's Impact Assessment Board, a function which this Committee welcomed in our 2010 report.[113] However, in his evidence the Minister observed the potential for duplication by the European Parliament in this area, saying, "I think I would encourage them [the European Parliament] to try to improve and press the Commission for better quality in their own impact assessments rather than have Parliament doing a rival one".[114]

68. We reiterate the view expressed in our 2010 report, Impact Assessments in the EU: Room for Improvement?, that impact assessments should be performed wherever a significant proposal is made. We also continue to call for further work to determine which measures are, and are not, to be accompanied by an impact assessment and whether in practice the selection is appropriate.

69. We agree with the Minister that the relationship between the European Commission and the European Parliament in the area of impact assessments warrants further study. We suggest that there is a risk of overlap between the two, and that there should be a focus on improving the Commission's impact assessments, perhaps through a stronger role for the Impact Assessment Board, rather than running a parallel process in the European Parliament.

70. While the accuracy of impact assessments is important, the European Commission should avoid an overly rigid approach, and develop indicators in collaboration with the private sector. These should take into account the extent of uncertainty involved in 'blue sky' innovative products, and the potential for projects to produce positive outcomes which take time to develop, are less tangible and may be difficult to quantify accurately at the outset of a project.

Monitoring and Evaluation

71. The principles of monitoring and evaluation have been well established in project management following decades of academic research and practical use. Essentially, proper monitoring will help to ensure a project is delivered in an efficient manner, while proper evaluation will assess whether it has produced the desired effects. Monitoring and evaluation allow actors to learn from each other's experiences, building on expertise and knowledge; they promote transparency and accountability, allow for lessons to be shared more easily; and provide a way to assess the crucial links between project implementers, beneficiaries on the ground and decision-makers.[115]

72. Some witnesses agreed with the Commission's assertion that its evaluation processes were well-established and transparent.[116] However, others contested this. UCL, Universities UK and the HE International Unit stated that there is currently no requirement to ensure that the impacts set out in an impact assessment have been achieved.[117] EADS described EU monitoring as administratively cumbersome and expensive.[118] The Government also acknowledged this, observing the tension between the Commission's stated aim of simplifying the process, and the desire to maintain the availability of monitoring information.[119] The Open University and RCUK noted that there is little access to discussions with the Commission on preferred monitoring and evaluation processes.[120]

73. Some witnesses were concerned about the background and quality of the evaluators.[121] Dr Urban et al observed that in some instances, evaluators were not experts in their field, but inexperienced post-doctoral researchers. EADS viewed the process in a largely positive light, but described it as "dominated by academic experts, whose priorities while valid are focused on novel research, not industrial innovation". They noted that industry experts are often rejected through claims of conflict of interest.[122]

74. The evidence we received suggested that, apart from the general criticism of complexity and bureaucracy that runs through the evidence, the major weakness in the Commission's monitoring and evaluation process is the evaluation of outputs.[123] Both the Institute of Physics and the National Institute for Health Research Evaluation, Trials and Studies Coordinating Centre (NETSCC) acknowledged the challenge of balancing the desired programme outputs with the practical realities of technology development and the legislative environment.[124] NETSCC cited the example of the European network for Health Technology Assessment (EUnetHTA) programme in which they were involved, where one of the desired outputs was the establishment of a permanent EU network for health technology assessment by the end of 2012. They said that the project was launched on the premise of achieving this aim, in spite of the Commission's awareness that the EU legislation needed to facilitate the project would not come into effect until after 2012.[125]

75. Reflecting on their research into the nature and effectiveness of EU funded research, Dr Galsworthy and Professor McKee observed that in their analysis of all health-related research funded by the EU under Framework Programmes 5 and 6 (FP5 and FP6), 50 per cent had no detectable academic output. They suggested that this was a result of the EU's lack of a technologically effective system to track its own projects and outputs.

76. One of the key problems with EU monitoring and evaluation is the long timescales involved. As part of the monitoring and evaluation process, the EU recommends that at the end of a project, a policy brief is developed and sent to stakeholders and policy makers. The policy brief should be developed from the findings of the study, and then be published in peer reviewed journals. However, Professor Bernabei et al suggested that the call for these academic papers often comes up to six years after the project is concluded, at a stage where the EU funding has finished, and the team has disbanded, leaving no one to write the brief.[126]

77. We welcome the European Commission's efforts to simplify the monitoring and evaluation process, but share the Government's view that this should not be done at the expense of the transparent evaluation of projects.

78. We consider it of paramount importance that monitoring and evaluation are carried out by experts in the relevant sector, in order to ensure that evaluators are able to assess and promote innovative excellence. A cohort of the experts in a particular sector should by definition consist of both academic and industry experts, bringing different strengths to bear on the evaluation process.

79. We also recommend that more work could be done to ensure that monitoring and evaluation of outputs are efficient and realistic, taking into account the relatively short timescales of EU R&I projects. This could be better achieved in part through clarifying at the outset of a project, the requirement for project participants to produce materials at the end of a project which explain the outputs and outcomes.

81 European Commission Back

82 UCL Back

83 UCL; Malcolm Harbour MEP; Microsoft; Universities UK and the UK HE International Unit Back

84 COM(2011) 808; COM(2011) 809; COM(2011) 810 Back

85 ibid. Back

86 Horizon 2020 website: http://ec.europa.eu/research/horizon2020/index_en.cfm?pg=h2020-experts Back

87 Vicky Ford MEP, Agricultural Biotechnology Council (ABC) Back

88 ABC Back

89 ABC Back

90 The Telegraph, The EU is now getting in the way of scientific progress, 18 October 2012. Available at:
http://www.telegraph.co.uk/news/worldnews/europe/eu/9617756/EU-red-tape-is-now-getting-in-the-way-of-scientific-progress.html Back

91 ibid. Back

92 David Willetts MP, Policy Exchange, (2013): Eight Great Technologies:
http://www.policyexchange.org.uk/images/publications/eight%20great%20technologies.pdf Back