Science, Innovation and Technology Committee
Reproducibility and research integrity
Date Published: 21 July 2023
On 10 May 2023 the Science, Innovation and Technology Committee published its Sixth Report of Session 2022–23, Reproducibility and Research Integrity (HC 101). The Government Response was received on 11 July 2023. The Response is appended to this Report.
1. The Government is grateful to the House of Commons Science, Innovation and Technology Committee for the recommendations put forward by the committee Chair in response to the inquiry into Reproducibility and Research Integrity.
2. This document sets out a joint response from the Government and UK Research and Innovation (UKRI) to the report. The response to the committee’s conclusions and recommendations have been grouped on a thematic basis.
3. The Government and UKRI welcome the committee’s report on reproducibility and research integrity. The principles of research integrity (rigour, care and respect, accountability, honesty, transparency and open communication) are the basis for conducting high quality research and for a positive research culture and environment. We are committed to achieving the cultural changes necessary for strong and reproducible research to flourish. As the committee noted, greater transparency is an important element in generating high quality, trustworthy research. Transparency can enable reproducibility and replicability, which are central ways of confirming research findings in many disciplines.
4. We are continuing to embed research integrity into our interventions to shift research culture. The government and UKRI responded to the previous Commons Committee report in 2018, and following wide engagement with the sector, UKRI established the UK Committee on Research Integrity (UK CORI) as a free-standing committee. A central goal is to ensure that the work needed to deliver strong research integrity and, where relevant, reproducibility, is properly understood, recognised, valued and rewarded by all concerned.
5. We partially accept the majority of the conclusions and recommendations. Where we have not accepted the conclusions and recommendations, we have outlined the existing or planned actions that are in progress to address the issues raised by the committee.
6. In July 2021, the House of Commons Science and Technology Committee launched an inquiry into the reproducibility of research and research integrity, to examine the causes of non-reproducibility and explore potential solutions to ensure reliable research practices.
7. Reproducibility and replicability are commonly used terms in the scientific community. Some fields use the terms interchangeably or even in contradictory ways. Yet, these terms are underpinned by the general principle of one experiment or study confirming the results of another.
8. As UKRI noted in their written evidence to the inquiry, although there are similarities between these two issues, they are distinct terms that relate to different concepts. Reproducibility refers to obtaining consistent results using the same experimental design, data and code as the original study. Replicability means obtaining consistent results across studies aimed at answering the same scientific question using different experimental design, data or code.
9. In the two years since the launch of the inquiry, addressing the factors contributing to research reproducibility and replicability has been a significant focus for the Government and UKRI.
10. For example, the Government set out a call for action across the sector through launch of its R&D People & Culture Strategy.1 Published in July 2021, the Strategy sets out the Government’s ambition to improve research practice through a range of measures, such as:
11. Alongside this, UKRI have played an important role in addressing research reproducibility and replicability, as a major funder of research. Through their policies, tools, and frameworks, UKRI have set out expectations for both how they operate and how they expect those they fund to operate. Some examples of the action UKRI have taken to promote reliable research and improve the research system include:
12. Further detail on the actions undertaken by the Government and UKRI can be found in the response to the recommendations below. This response does not cover those recommendations which are addressed to UKCORI, who will respond separately.
Whilst significant reproducibility challenges are faced in research, to refer to the sum of these issues as a “crisis” risks detracting from the many successes of the UK’s scientific research base. Nonetheless, there is need for action to address the significant problems caused by the prevalence or reproducibility problems in the scientific community. (Paragraph 50)
We accept this conclusion.
13. The UK has a strong reputation for science, research and innovation and is recognised as a leader in research integrity and open science. Concerns about reproducibility in research is a well-recognised issue and tackling the causes is a key priority in the UK research sector and internationally.
14. The UK government and public funders of research have committed to support the sector in taking action so that the UK maintains and strengthens its reputation for the quality and integrity of research. For example, the R&D People and Culture Strategy is a call to action on strengthening research culture in the UK, which includes improving frameworks and incentives. The strategy is clear that the immediate commitments made by government will build on the good work already being done by many organisations and individuals in the sector.
15. Alongside the strategy, the Future Research Assessment Programme has been evaluating REF 2021 and investigating evaluation models and approaches, identifying those that can encourage and strengthen the emphasis on delivering excellent research and impact, and support a positive research culture.
16. We must continue to build upon and improve the quality and integrity of research to create positive and long-lasting change. If we are to maintain our position at the very forefront of research, then we must be sure that the work of our research community is underpinned by common values of rigour, respect, and responsibility. Excellence and integrity are inextricably linked.
The UK Government should lead on a co-produced framework with the UK Reproducibility Network, UKRIO and UK CORI, which sets out the roles and expectations for key actors when cases of misconduct are identified. (Paragraph 60)
The UK Government should assess the benefits that an additional body, set up to investigate malpractice, could bring to the UK’s research integrity governance architecture. (Paragraph 61)
We do not accept these recommendations.
17. The Government should not be in the business of investigating research misconduct but should ensure it supports an effective research environment that can self- regulate and respond when misconduct is identified.
18. This needs to be addressed at a system-wide level and reproducibility should be one of a number of mechanisms to promote integrity.
19. In cases where there is intentional research misconduct (i.e. manipulating statistical results or falsification of data) the Government agrees that this should be taken extremely seriously. UKRI have published clear guidelines on what constitutes research misconduct and are responsible for ensuring that recipients of funding are aware of the requirements and reporting of research misconduct, and that these are openly stated.
20. Government does not accept the need for a new body to investigate misconduct cases. The primary responsibility is with the research organisation where the case has arisen. In addition, having established UKCORI in response to the Select Committee inquiry, time should be allowed to evaluate its role in raising attention and offering potential solutions to improve research integrity governance.
Therefore, the research community, including research institutions and publishers, should work alongside individuals to create an environment where research integrity and reproducibility are championed. (Paragraph 68)
Reward structures in academia disincentivise reproducibility by placing disproportionate value on secured funding and frequent publication in prestigious journals.
(Paragraph 127)2
We partially accept the recommendation (Paragraph 68) and the conclusion (Paragraph 127).
21. This does not require a formal response from Government. There are a number of ways in which UKRI are supporting the sector to address the issues highlighted. This recommendation rightly recognises the shared responsibility of the research community in promoting research integrity and reproducibility. There are many parties that have a role to play in driving forward action, and no single entity can achieve this alone. Action will require collaboration from various parts of the system, in many different ways.
22. UKRI has an extensive portfolio of work aimed at shifting the incentives in the research system away from pressures to publish quickly in high prestige journals. We have been working across the community to embed these changes. Examples include evidenced narrative CVs,3 changes to the REF process for 2028 and the UKRI People and Teams action plan.4
23. Recognising the need to support the research community on good practice, UKRI also hosts the Good research resource hub5 which brings together policies, standards and guidance on how to deliver excellent research and innovation and nurture an inclusive environment.
24. UKRI is working with DSIT to establish the Good Practice Exchange (GPEx), proposed in the Government’s R&D People and Culture Strategy as a mechanism to develop, test, evaluate and highlight initiatives to improve research culture. It will provide a space for the community to share best practice, collaborate and promote existing initiatives aimed at improving culture across the R&I sector while helping build an evidence base.
25. These deep research culture changes are crucial to reduce the pressures identified in the report and without them, the parallel interventions to support research reproducibility and integrity listed below will be unlikely to succeed.
26. UKRI funds the UK Reproducibility Network (UKRN) with £4.5 million through the Research England Development Fund6 and with support through the People, Culture and Talent programme. UKRN has initiated a peer-led movement to identify and address factors that prohibit reproducibility, raising the profile of this important issue.
27. UKRI welcomes the committee’s reflection that the research community needs to work together to continually improve research integrity and reproducibility. For example, as a signatory to the Concordat to Support Research Integrity,7 UKRI is actively working with the other signatories to raise awareness of the concordat and its commitments. The recently launched template for annual statements is being piloted to increase the number of organisations creating and publishing annual statements.
28. UKRI has established and supports the UK Committee on Research Integrity which, through its convening role, will be championing and promoting research integrity across the research community.
Funders and universities should develop dedicated funding for the presence of statistical experts and software developers in research teams. In tandem, universities should work on developing formalised, aspirational career paths for these professions.
(Paragraph 76)
We partially accept this recommendation.
29. As set out in the R&D People and Culture Strategy, we are committed to broadening recognition of those involved in the research system, including statistical experts and software developers. UKRI has ongoing activities that recognise and reward the full range of people necessary for high quality research and works to incentivise research organisations to support them. This includes the work of the four higher education funding bodies on the high-level framework for the next Research Excellence Framework, announced on 15 June 2023.
30. The evidence statement in the new Contribution to Knowledge and Understanding REF section will allow the assessment of approaches to research integrity and reproducibility (with appropriate disciplinary nuance for the latter).
31. The break in the link between researchers and outputs, and the ability to include outputs produced by, for example, technical staff will further incentivise the full range of roles being valued and supported.
32. Much of the wider work listed above to shift research culture supports the recognition and reward of the full range of skills needed; For example, UKRI published the People and Teams action plan8 to further help the sector develop this area.
33. UKRI has revised the definitions of roles supported by our grant funding to ensure support is provided for all the people and skills needed.
34. UKRI also has specific funding streams for certain roles, such as the EPSRC Support the development of research software engineering call9 and the ESRC National Centre for Research Methods10 which provides research methods training and capacity building.
35. There is insufficient evidence that additional funding streams would be more beneficial than integrating this into standard funds with aligned incentives to ensure these wider skills are valued. Developing research on research in relation to talent will be useful for informing targeted further support.
36. In June 2023 UKRI launched the £7 million Innovation and Research Caucus11 (IRC) as a world leading centre of excellence to maximise research and innovation funding impact and will consider the case for further increasing focus on metascience.
Research funders should implement stronger tests for the presence of adequate software and statistical skills within research teams at the outset of a funding application. Where these skills are perceived to be lacking, UKRI should consider the feasibility and cost of offering a dedicated methodological support system to research teams. (Paragraph 77)
We do not accept this recommendation.
37. UKRI, like other research funders, builds assessment of skills into the research funding process. During the UKRI grant application process, proposals undergo expert peer review where all relevant skills, including statistical skills are assessed through the “Capability to deliver” section of the application.
38. To ensure that UKRI continues to make the best choices of what and whom to fund, it launched a review of peer review12 in 2022. The work will be published in Summer 2023 and is intended as a resource for R&I funders across the globe looking to optimise and innovate their award-making processes.
39. UKRI already funds methodological support where disciplines have identified a need for improvement, for example embedded in doctoral training, or through the ESRC National Centre for Research Methods13 which provides research methods training and capacity building.
40. It is important that support is fully embedded in research activity. For example, high quality statistical support is hard to supply as an external service because deep understanding of the experimental system, such as the independence of the variables being measured, is needed to ensure the optimal analyses are deployed.
41. UKRI has a role in incentivising research organisations and employers have a responsibility for assessing and supporting skills development in their workforce.
Greater emphasis should be placed on the importance of reproducibility and research integrity in education and training at undergraduate, postgraduate and early career researcher stages. Part of this training at the undergraduate and postgraduate levels should include the routine production of replications. (Paragraph 82)
We accept this recommendation.
42. UKRI welcomes the committee’s conclusion that attention be placed on high quality training for all individuals involved in the research system. Continuous development is essential for everyone from students to experienced professionals, particularly to understand their role and responsibility when it comes to research integrity. UKRI’s Collective Talent Funding programme14 aims to address the need for consistent development and training at different career stages.
43. This includes work within UKRI’s portfolio of Doctoral Training Centres and Doctoral Training Partnerships, which support doctoral students’ skill development through discipline appropriate offerings.
UKRI should consult with a representative sample of researchers to understand whether their grants allow them sufficient time and funding to do the work needed for ensuring their research is reproducible. UKRI should also implement a trial funding programme with an emphasis on ‘slower’ science. (Paragraph 88)
We partially accept this recommendation.
44. UKRI funds work across a range of timeframes to support a diversity of people and projects across different stages of the research process. In addition to long-standing investments in more than 60 Centres, Institutes, and Units, UKRI offers a portfolio of longer funded research projects. This includes the EPSRC manufacturing research hubs15 running for seven years and the BBSRC strategic longer and larger funding mechanism.16
45. Without changing the incentive system, the pressures on researchers, irrespective of grant length, will continue. The “slow science” movement has as core to its manifesto “doing less better”. For example, one of its more radical proposals is to ration the number of publications a researcher is allowed to produce. Under current incentives, longer grants simply ramp up expectations for more outputs.
46. UKRI is committed to continuing work with stakeholders to understand and address the most pressing issues impacting research quality. We are continually experimenting with funding approaches that support this endeavour, drawing on and contributing to the evidence base needed for progress.
Research funders, including UKRI, should work to impose a three-year minimum contract for post-doctoral researchers in universities. (Paragraph 89)
We do not accept this recommendation.
47. The linking of postdoctoral researchers’ contracts to individual research grants is not a stipulation of UKRI grant terms and conditions. Research organisations maintain independence in their decision-making about staffing, and responsibility for employment contracts. The funding provided to universities through the dual support system, which for English universities constitutes 25% of the UKRI annual budget, provides the flexibility universities need to support their research strategies. Given the significant financial pressures universities and the wider research and innovation system are under, there is an important debate to be had about the balance of quantity and quality in research, echoing the arguments of the slow science movement.
48. Concern around job security requires a systems approach, shifting incentives, reward and recognition, to support the optimal flow of talent across the full research and innovation system, including but not limited to universities. The narrow focus on linear academic careers is damaging to research and researchers alike. With current incentives, post-doctoral researchers will continue to feel significant pressure to publish regardless of length of their contract, with more time simply increasing expectations.
49. As described above, UKRI has an extensive portfolio of investments in improving research culture, including investment targeted specifically at early career researchers. For example, Research England has invested in the development of postdoctoral researchers for alternative careers through its Prosper17 and Centre for Postdoctoral Development in Infrastructure Cities (C-DICE)18 programmes, which aim to mitigate the “publish or perish” pressure on this group.
50. This notwithstanding, responsible employment practices are an important element in ensuring all those working in the research and innovation system can thrive.
51. The UKRI people and teams action plan19 was launched in March 2023 and sets out how it will support the development of skilled people and teams in research and innovation. It aims to reduce precarity in research careers through incentivising the development of a greater diversity of staffing models. This includes encouraging the use of pooled staff and cross-cutting specialist roles; setting stronger expectations of redeploying staff between research projects; and recognising the development and wellbeing of research staff as key to realising investments in assessing funding opportunities and the leadership capability of grant applicants.
UKRI and other research funders should continue to implement open access policies until this figure reaches 100%, by the end of 2025 at the latest. (Paragraph 98)
We partially accept this recommendation.
52. The UKRI open access policy seeks to ensure that research articles, monographs, book chapters and edited collections that acknowledge its funding are made freely accessible. From January 2024, UKRI’s open access policy will also apply to monographs, book chapters and edited collections, ensuring the full breath of research funded is made freely accessible. UKRI will be reviewing the progress of its policy for research articles in 2024.
53. UKRI welcomes and supports the aspiration for 100% of UK research articles to be open access, and to achieve this continues to work with other funders to support their efforts to implement open access policies. Major UK research funders, including UKRI, have put in place immediate open access policies, and more are currently considering updating policies. Nevertheless, there are complexities in implementation that require action from many stakeholders, including publishers, research organisations and researchers.
We welcome UKRI’s use of data management plans. A continued emphasis on their importance as a condition of research funding is necessary. (Paragraph 106)
We accept this conclusion.
54. Data management plans remain an important requirement for most research projects that UKRI funds. As part of the review and development of research data policies and funding application requirements, UKRI will be considering how to further enhance their utility, while making it as easy as possible for researchers to apply for UKRI funding.
UKRI should learn from its Dutch equivalent, NWO, by developing a pilot programme to fund replication studies. (Paragraph 115)
We partially accept this recommendation.
55. UKRI has strong international relationships including with Dutch colleagues NWO and look forward to learning more about their pilot programme, and whether it has brought about intended outcomes.
56. UKRI has a diverse portfolio of funding mechanisms, inclusive of replication studies, meta-analysis and longer-term grants. UKRI requests additional specific information for reproducibility in areas where there is increased concern, such as the use of animals in research where the 3R’s apply (replacement, reduction and refinement).
57. As described above, In June 2023 UKRI launched the £7 million Innovation and Research Caucus20 (IRC) as a world leading centre of excellence to maximise research and innovation funding impact and will consider the case for further increasing focus on metascience.
58. UKRI funds research across all disciplines. The desirability, possibility and practicality of reproducing results varies greatly across the work that UKRI funds. For example, in some areas where experimental cycles are short, reproducibility is inherently built into any project funding, and in many disciplines, robustness is further enhanced by taking multiple different approaches to test the same hypothesis.
The Future Research Assessment Programme (FRAP) is consulting on reforms for the assessment of UK higher education research. It should review the Research Excellence Framework assessment criteria to assure that transparency is a prerequisite of top scoring research. It should also consider the effects of removing ‘originality’ from the top score bracket. (Paragraph 131)
We partially accept this recommendation.
59. Decisions on the FRAP outcomes and the shape of the next Research Excellence Framework fall within the remit of the four UK Devolved research funding bodies. The funding bodies have consulted widely within the sector during the FRAP process, both to support evaluation of the REF 2021 exercise and also on different approaches to the evaluation of research in universities.
60. The Higher Education Funding Bodies have recently, on 15 June 2023, announced the high-level framework for the next Research Excellence Framework. This includes new elements on People, Culture and Environment and Contribution to Knowledge and Understanding, where broad evidence on research environment will be sought. This evidence will include an assessment of integrity, openness and transparency and research reproducibility (where appropriate). Detailed criteria will be developed during 2024 with the assessment panels once they are appointed.
Funders, led by UKRI, should move towards the exclusive use of the ‘resume for researchers’ format in funding calls by 2025. (Paragraph 133)
We partially accept this recommendation.
61. Government is unable to commit all funders to the exclusive of Résumé for Research and Innovation (R4RI), although we support the principle of the greater use of narrative CVs and UKRI is committed to its adoption across all funding opportunities that require track record information.
62. Recognising the need to assess researchers and their contributions more broadly, UKRI is currently rolling out R4RI format,21 based on the Royal Society’s ‘resume for researchers’. This is expected to be in place by the end of 2023.
63. UKRI have established two communities of practice for the research community roll out of R4RI. The Joint Funders Group (JFG), convened in July 2021, works to create the conditions needed to allow other funders to use these CVs in funding decisions. The Alternative Uses Group (AUG), convened in December 2021 with Universities UK, explores the application of these CVs in the assessment of people, e.g. hiring and promotion. Currently made up of 58 organisations covering funders, universities, learned societies, professional bodies, and diverse communities, the JFG and AUG are complementary and co-create resources to support widespread adoption of narrative CVs across the global R&I sector. These resources are hosted in the Résumé Resources Library22 and include guidance prompts, training materials, and a Shared Evaluation Framework (SEF) to help monitor for unintended consequences.
64. In addition, UKRI is a signatory to the San Francisco declaration on research assessment23 (DORA) and founding member of the Coalition for Advancing Research Assessment24 (CoARA) which aim to improve the mechanisms for assessing researchers and their contributions. As a signatory, this signposts to the sector that UKRI values a range of output types and career choices.
Publishers and funders should work together to offer a ‘registered report partnership model.’ Not only will this benefit researchers who will receive feedback on their research plans and guaranteed publication, but greater transparency in research methodologies will be achieved, which should have a positive impact on research reproducibility. (Paragraph 143)
We partially accept this recommendation.
65. Government welcomes the support from the Committee for increasing the use of registered reports to enhance reproducibility through giving greater transparency to research methodologies. Further evaluation of the costs and benefits of registered report partnership models is needed. As a concept they appear to offer a way of increasing the number of research projects that are published through the registered report process.
66. In the right circumstances, registered reports are a useful model for increasing transparency, improving methodology and ensuring publication of null or negative results. The results of current trials of this model, such as the Cancer Research UK pilot, will be instrumental for determining whether, and under what circumstance they are effective. UKRI will continue to facilitate and contribute to addressing challenges in this area, including contributing to dialogue on the benefits and risks for a registered report partnership model for appropriate disciplines.
1 Government’s R&D People and Culture Strategy
https://www.gov.uk/government/publications/research-and-development-rd-people-and-culture-strategy
2 Paragraph 127 formally sits under ‘Solving reproducibility challenges in the assessment of research’. However, for the purpose of this response we have combined it with the response to Paragraph 68.
3 UKRI blog referring to the rolling out of the Résumé for Research and Innovation (R4RI) format
https://www.ukri.org/apply-for-funding/before-you-apply/resume-for-research-and-innovation-r4ri-guidance/
4 UKRI people and teams action plan https://www.ukri.org/wp-content/uploads/2023/03/UKRI-20032023-UKRI-people-and-teams-action-plan.pdf
5 UKRI good research resource hub https://www.ukri.org/about-us/policies-standards-and-data/good-research-resource-hub
6 Projects funded by the Research England development (RED) fund with details of the UK Reproducibility Network funding found on page 11 https://www.ukri.org/publications/projects-funded-by-the-research-england-development-red-fund/
7 Concordat to support research integrity https://www.universitiesuk.ac.uk/topics/research-and-innovation/concordat-support-research-integrity
8 UKRI people and teams action plan https://www.ukri.org/wp-content/uploads/2023/03/UKRI-20032023-UKRI-people-and-teams-action-plan.pdf
9 Support the development of research software engineering https://www.ukri.org/opportunity/support-the-development-of-research-software-engineering/
10 ESRC National Centre for Research Methods https://www.ncrm.ac.uk/
11 New UKRI network to maximise research and innovation funding impact
12 Review of peer review https://www.ukri.org/blog/peer-review-critical-to-our-success/
13 ESRC National Centre for Research Methods https://www.ncrm.ac.uk/
14 UKRI Collective Talent Funding roadmap 2022 to 2025 https://www.ukri.org/publications/collective-talent-funding-roadmap-2022-to-2025/
15 UKRI funding call for EPSRC Manufacturing hubs for a sustainable future two Manufacturing research hubs for a sustainable future two: outline stage – UKRI
16 Strategic longer and larger grants (sLoLa): frontier bioscience 2022 to 2023 – UKRI https://www.ukri.org/opportunity/strategic-longer-and-larger-grants-slola-frontier-bioscience-2022-to-2023/
17 Projects funded by the Research England development (RED) fund with details of the Prosper funding found on page 13 https://www.ukri.org/publications/projects-funded-by-the-research-england-development-red-fund/
18 Projects funded by the Research England development (RED) fund with details of the C-DICE funding found on page 7 https://www.ukri.org/publications/projects-funded-by-the-research-england-development-red-fund/
19 UKRI people and teams action plan https://www.ukri.org/wp-content/uploads/2023/03/UKRI-20032023-UKRI-people-and-teams-action-plan.pdf
20 New UKRI network to maximise research and innovation funding impact https://www.ukri.org/news/new-network-to-maximise-research-and-innovation-funding-impact/
21 UKRI blog referring to the rolling out of the Résumé for Research and Innovation (R4RI) format https://www.ukri.org/apply-for-funding/before-you-apply/resume-for-research-and-innovation-r4ri-guidance/
22 UKRI Résumé Resources Library https://www.ukri.org/resume-resources-library
23 San Francisco declaration on research assessment https://sfdora.org/
24 COARA - Coalition for Advancing Research Assessment https://coara.eu/