• Accessibility
• Email alerts
• RSS feeds
• Contact us

Large language models and generative AI Contents

---

Chapter 5: Risk

108.The nature, likelihood and impact of risks arising from large language models (LLMs) remains subject to much debate. The complexity stems in part from the extensive literature,173 lack of agreed definitions, hype around rapid developments,174 and the possibility that some organisations may have interests in emphasising or downplaying risk.175

109.This chapter examines a selection of security and societal risks.176 We sought to distinguish hype from reality and provide some reference points to ground our review. We found credible evidence of both immediate and longer-term risks from LLMs to security, financial stability and societal values.

110.The first section of this chapter sets out our understanding of risk categories. The next section sets out near-term security risks that require immediate attention, followed by a discussion on longer-term concerns around catastrophic risk and then existential risk. Near-term societal risks such as bias and discrimination are discussed at the end of the chapter.

What are we talking about?

111.There are numerous frameworks for evaluating risk used by domestic and international authorities.177 We found little consistency in terms or methods across the literature.178 We adopt the framework from the Government’s National Risk Register (NRR), set out in the table below, to help describe impacts of LLM-related security risks. Our categorisation is approximate only and we do not attempt to replicate the full National Security Risk Assessment process. It nevertheless provides a helpful yardstick to anchor discussion using a recognised framework.179 This table does not cover existential risk, which we describe as a separate category later in this chapter.

Table 1: Risk categories

Risk Level	Fatalities	Casualties	Economic impact
Minor	1–8	1–17	£ millions
Limited	9–40	18–80	£ tens of millions
Moderate	41-200	81-400	£ hundreds of millions
Significant	201–1000	400–2000	£ billions
Catastrophic	More than 1,000	More than 2,000	£ tens of billions

Source: HM Government, National Risk Register (2023): https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1175834/2023_NATIONAL_RISK_REGISTER_NRR.pdf [accessed 20 December 2023]

112.There are also various ways of categorising societal risk and conducting impact assessments.180 We draw on these to inform our review of societal risk, noting that the issues are highly context-dependent.

Threat models

113.Risks may arise from both open and closed models, for example through:

• inappropriate deployment (for example using a model to diagnose patients without proper procedures and safeguards);
• increasing the tools available to malicious actors (for example auto-generating phishing campaigns);
• poor performance or model malfunction (for example a safety feature failure);
• gradual over-reliance (for example handing increasingly important decisions and processes to machines with insufficient human engagement or accountability); or
• loss of control (for example where a highly capable machine pursues its own objectives that may not be obvious to humans or aligned with our wellbeing).181

Near-term security risks

114.Our evidence was clear that LLMs will act as a force multiplier enhancing malicious capabilities in the first instance, rather than introducing qualitatively new risks.182 Most models have some safeguards but these are not robust and can be circumvented.183 We believe the most immediate security risks over the next three years are likely to include the (non-exhaustive) list below, with indicative impacts ranging from minor to moderate, rather than catastrophic.

115.Cyber: LLMs are likely to be of interest to hostile states, organised crime, and low-sophistication actors.184 Some LLMs are reportedly being developed to create code for cyber attacks at increased scale and pace.185 LLMs and multi-modal models will make it easier to create phishing campaigns, fraudulent websites and voice cloning to bypass security protocols.186 Malicious actors may use prompt injection attacks to obtain sensitive information, or target models themselves to influence the outputs, poison training data or induce system malfunction.187 Current security standards are unlikely to withstand attacks from sophisticated threat actors.188

116.Tools to mass produce high quality and openly available destructive cyber weapons appear limited at present. Chris Anley, Chief Scientist at the cyber security firm NCC Group, said LLMs currently provide efficiency and lower barriers to entry, rather than game-changing capability leaps.189 Even moderate gains could however prove costly when deployed against under-prepared systems, as previous attacks on the NHS have shown.190

117.A reasonable worst case scenario might involve malicious actors using LLMs to produce attacks achieving higher cyber infection rates in critical public services or national infrastructure.191

118.Terrorism: A recent report by Europol found that LLM capabilities are useful for terrorism and propaganda.192 Options include generating and automating multilingual translation of propaganda, and instructions for committing acts of terror.193 In future, openly available models might be fine-tuned to provide more specific hate speech or terrorist content capabilities, perhaps using archives of propaganda and instruction manuals.194 The leak of Meta’s model (called LLaMa) on 4chan, a controversial online platform, is instructive. Users reportedly customised it within two weeks to produce hate speech chatbots, and evaded take-down notices.195

119.National Statistics data show 93 victim deaths due to terrorism in England and Wales between April 2003 and 31 March 2021.196 A reasonable worst case scenario might involve a rise in attacks directly attributable to LLM-generated propaganda or made possible through LLM-generated instructions for building weapons.197

120.Synthetic child sexual abuse material: Image generation models are already being used to generate realistic child sexual abuse material (CSAM).198 The Stanford Internet Observatory predicts that in under a year “it will become significantly easier to generate adult images that are indistinguishable from actual images”.199 The Internet Watch Foundation has confirmed this is “happening right now”,200 and stated legal software can be downloaded and used offline to produce illegal content “with no opportunity for detection”.201 This suggests more abuse imagery will be in circulation, law enforcement agencies may find it more difficult to identify and help real-world victims, and opportunities to groom and coerce vulnerable individuals will grow.202

121.AI CSAM currently represents a small proportion of the total amount of CSAM (reportedly 255,000 webpages last year with potentially millions of images).203 A reasonable worst case scenario might involve widespread availability of illegal materials which overwhelms law enforcement ability to respond.204

122.Mis/disinformation: LLMs are well placed to generate text-based disinformation at previously unfeasible scale, while multi-modal models can create audio and visual deepfakes which even experts find increasingly difficult to identify.205 LLMs’ propensity to hallucinate also means they can unintentionally misinform users.206 The National Cyber Security Centre assesses that large language models will “almost certainly be used to generate fabricated content; that hyper-realistic bots will make the spread of disinformation easier; and that deepfake campaigns are likely to become more advanced in the run up to the next nationwide vote, scheduled to take place by January 2025”.207

123.Professor Dame Angela McLean, Government Chief Scientific Adviser, said she was “extremely worried” and called for a public awareness campaign.208 Dr Jean Innes, CEO of the Alan Turing Institute, similarly warned about “mass disinformation”.209 Professor Phil Blunsom, Chief Scientist at Cohere, likewise highlighted “disinformation [and] election security” as issues of concern.210

124.Lyric Jain, CEO of the counter-disinformation firm Logically, said one of the main impacts of generative AI was increased efficiency and lower costs. He estimated the Internet Research Agency’s disinformation campaign targeting the US 2016 election cost at least $10 million,211 whereas generating comparable disinformation materials could now be done for $1,000 by private individuals. He further noted that model safeguards were preventing only 15 per cent of disinformation-related prompts.212

125.A reasonable worst case scenario might involve state and non-state interference undermining confidence in the integrity of a national election, and long-term disagreement about the validity of the result.213

Mitigations

126.A range of mitigation work is underway across Government and industry. The main issue remains scale and speed: malicious actors enjoy first-mover advantages whereas it will take time to upgrade public and private sector mitigations, including public awareness.214 And as the Government’s AI Safety Summit paper noted, there are limited market incentives to provide safety guardrails and no standardised safety benchmarks.215

127.We wrote to the Government seeking more information. It declined to provide details on whether mitigations were being expanded. But it did confirm workstreams included:

• Cyber: Research from the AI Safety Institute and DSIT’s new AI central risk function; delivery of the National Cyber Strategy; and Cabinet Office work on AI cyber risks.
• Counter-terror: delivery of the CONTEST strategy, and monitoring the early-stage experimentation of generative AI for terrorist purposes.
• CSAM: Measures under the Online Safety Act; delivery of the 2021 Child Sexual Abuse Strategy; international partnerships; monitoring technology developments; investments in the National Crime Agency, GCHQ and policing; and setting up a “new central strategic function” looking at emerging technology.
• Disinformation: Measures under the Defending Democracy Taskforce, National Security Online Information Team and Election Cell; implementation of the Online Safety Act; media literacy; and international partnerships.216

128.The most immediate security concerns from LLMs come from making existing malicious activities easier, rather than qualitatively new risks. The Government should work with industry at pace to scale existing mitigations in the areas of cyber security (including systems vulnerable to voice cloning), child sexual abuse material, counter-terror, and counter-disinformation. It should set out progress and future plans in response to this report, with a particular focus on disinformation in the context of upcoming elections.

129.The Government has made welcome progress on understanding AI risks and catalysing international co-operation. There is however no publicly agreed assessment framework and shared terminology is limited. It is therefore difficult to judge the magnitude of the issues and priorities. The Government should publish an AI risk taxonomy and risk register. It would be helpful for this to be aligned with the National Security Risk Assessment.

Catastrophic risk

130.Catastrophic risks might arise from the deployment of a model with highly advanced capabilities without sufficient safeguards.217 As outlined in the previous table, indicative impacts might involve over 1,000 fatalities, 2,000 casualties and/or financial damages exceeding £10 billion.

131.There are threat models of varying plausibility.218 The majority of our evidence suggests these are less likely within the next three years but should not be ruled out—particularly as the capabilities of next-generation models become clearer and open access models more widespread.219 We outline some of the most plausible risks below.

132.Biological or chemical release: A model might be used to lower the barriers to malicious actors creating and releasing a chemical or biological agent. There is evidence that LLMs can already identify pandemic-class pathogens, explain how to engineer them, and even suggest suppliers who are unlikely to raise security alerts.220 Such capabilities may be attractive to sophisticated terror groups, non-state armed groups, and hostile states. This scenario would still require a degree of expertise, access to requisite materials and, probably, sophisticated facilities.221

133.Destructive cyber tools: Next generation LLMs and more extensive fine tuning may yield models capable of much more advanced malicious activity.222 These may be integrated into systems capable of autonomous self-improvement and a degree of replication.223 Such advances would raise the possibility of advanced language model agents navigating the internet semi-autonomously, performing sophisticated exploits, using resources such as payment systems, and generating snowball effects created by self-improvement techniques.224 Recent research suggests such capabilities do not yet exist, though progress on the component parts of such tools is already underway and capability leaps cannot be ruled out.225

134.Critical infrastructure failure: Models may in time be linked to systems powering critical national infrastructure (CNI) such as water, gas and electricity transmission, or security platforms (for example in military planning or intelligence analysis systems). This might occur either through direct integration of models with the infrastructure platform itself, or through software used in the supply chain.226 In the absence of safeguards, a sudden model failure may trigger a CNI outage or sudden security lapse, and could be extremely difficult to rectify given the black-box nature of LLM processes.

Mitigations

135.Professor Dame Angela McLean, Government Chief Scientific Adviser, confirmed that there were no agreed warning indicators for catastrophic risk. She said warning indicators for pandemics and similar were well understood, but:

“we do not have that spelled out for the more catastrophic versions of these risks. That is part of the work of the AI Safety Institute: to make better descriptions of things that might go wrong, and scientific descriptions of how we would measure that.”227

136.OpenAI told us work was underway to evaluate “dangerous capabilities” and appropriate safety features but noted “science-based measurements of frontier system risks … are still nascent”.228

137.Professor John McDermid OBE, Professor of Safety-Critical Systems at the University of York, said industries like civil aviation designed software with fault-detection in mind so that sudden failures could be fixed with speed and confidence.229 He did not believe such safety-critical system analysis was possible yet for LLMs and believed it should be a research priority.230

138.Professor Stuart Russell OBE, Professor of Computer Science at the University of California, Berkeley, was sceptical that the biggest safety challenges could be addressed without fundamental design changes. He noted that high-stakes industries like nuclear power had to show the likelihood of sudden catastrophic failure rates, which LLM developers could not. He also noted it was straightforward to bypass a model’s safety guardrails by prefixing a harmful question with something unintelligible to confuse it, and maintained that:

“The security methods that exist are ineffective and they come from an approach that is basically trying to make AI systems safe as opposed to trying to make safe AI systems. It just does not work to do it after the fact”.231

139.Ian Hogarth, Chair of the (then) Frontier AI Taskforce, told us that the Government took catastrophic risk very seriously. Viscount Camrose, Minister for AI and Intellectual Property, said the AI Safety Institute was focusing on frontier AI safety and driving “foundational” research.232

140.Catastrophic risks resulting in thousands of UK fatalities and tens of billions in financial damages are not likely within three years, though this cannot be ruled out as next generation capabilities become clearer and open access models more widespread.

141.There are however no warning indicators for a rapid and uncontrollable escalation of capabilities resulting in catastrophic risk. There is no cause for panic, but the implications of this intelligence blind spot deserve sober consideration.

142.The AI Safety Institute should publish an assessment of engineering pathways to catastrophic risk and warning indicators as an immediate priority. It should then set out plans for developing scalable mitigations. (We set out recommendations on powers and take-down requirements in Chapter 7). The Institute should further set out options for encouraging developers to build systems that are safe by design, rather than focusing on retrospective guardrails.

Uncontrollable proliferation

143.There is a clear trend towards faster development, release and customisation of increasingly capable open access models.233 Some can already be trained in just 6 hours and cost a few hundred dollars on public cloud computing platforms.234

144.We heard widespread concern about the ease of customisation leading to a rapid and uncontrollable proliferation of models which may be exploited by malicious actors, or contain safety defects affecting businesses and service users.235

145.Google DeepMind told us that that “once a model is openly available, it is possible to circumvent any safeguards, and the proliferation of capabilities is irreversible.”236 There is no ‘undo’ function if major safety or legal compliance issues subsequently emerge,237 and no central registry to determine model provenance once released. It may be possible to embed identifying features in models to help track them, though such research remains at an early stage.238 The Royal Academy of Engineering emphasised that many models will be hosted overseas, posing major challenges to oversight and regulation.239

146.As we set out in Chapter 3, open access models can provide speedy community-led improvements, including to security issues, but those same characteristics can also drive proliferation in malicious use.240

147.Closed models are not a security panacea, however. Previous breaches from hack and leak operations, espionage and disgruntled employees suggest that even well-protected systems may not remain closed forever.241 The Minister said the AI Safety Institute was working on the issues but believed the risks around open access proliferation remained an “extremely complex problem”.242

148.There is a credible security risk from the rapid and uncontrollable proliferation of highly capable openly available models which may be misused or malfunction. Banning them entirely would be disproportionate and likely ineffective. But a concerted effort is needed to monitor and mitigate the cumulative impacts. The AI Safety Institute should develop new ways to identify and track models once released, standardise expectations of documentation, and review the extent to which it is safe for some types of model to publish the underlying software code, weights and training data.

Existential risk

149.The threat model for existential risk remains highly disputed. A baseline scenario involves the gradual integration of hyper intelligent AI into high-impact systems to achieve political, economic or military advantage, followed by loss of human control. This might occur because humans gradually hand over control to highly capable systems that vastly exceed our understanding; and/or the AI system pursues goals which are not aligned with human welfare and reduce human agency.243 Humans might also increasingly rely on AI evaluations in high-stakes areas such as nuclear strategy, for example.244

150.Long-term indicative impacts have been compared to outcomes in other fields, including pandemics and nuclear.245 At the most extreme end, the first- and second-order consequences of uncontrolled nuclear exchange between superpowers have been variously estimated at 2–5 billion fatalities.246 A biosecurity extinction event might involve above 7 billion fatalities.247

151.Systems capable of posing such risks do not yet exist and there is no consensus about their long-term likelihood. Professor Phil Blunsom, Chief Scientist at the LLM firm Cohere, did “not see a strong existential risk from large language models”.248

152.Professor Stuart Russell OBE argued that “large language models are not on the direct path to the super intelligent system … but they are a piece of the puzzle”. He maintained current systems lacked features including “the ability to construct and execute long-term plans, which seems to be a prerequisite” to overcome human resistance, but “could not say with any certainty that it will take more than 20 years” for researchers to address those shortcomings.249

153.Some surveys of industry respondents predict a 10 per cent chance of human-level intelligence by 2035, while others say such developments are not likely and do not believe it is a concern.250 Researchers at the Oxford Internet Institute emphasised that current capabilities were “meaningfully different” to those required for existential risk.251 Owen Larter, Director of Public Policy at Microsoft’s Office for Responsible AI, anticipated a “further maturation of AI safety” in the coming years.252

154.This indicates a non-zero likelihood (remote chance) of existential risks materialising, though it is almost certain that these will not occur within the next three years and it seems highly likely that they will not materialise within the next decade. We note the possibility and (longer-term) timing remains a matter of debate and concern for some in the expert community.253 Several stakeholders suggested concerns about existential risk were distracting from efforts to address limited but more immediate risks,254 as well as from the opportunities LLMs may provide.255

155.It is almost certain existential risks will not manifest within three years and highly likely not within the next decade. As our understanding of this technology grows and responsible development increases, we hope concerns about existential risk will decline. The Government retains a duty to monitor all eventualities. But this must not distract it from capitalising on opportunities and addressing more limited immediate risks.

Societal risks

156.LLMs may amplify any number of existing societal problems, including inequality, environmental harm, declining human agency and routes for redress, digital divides, loss of privacy, economic displacement, and growing concentrations of power.256

Bias and discrimination

157.Bias and discrimination are particular concerns, as LLM training data is likely to reflect either direct biases or underlying inequalities.257 Depending on the use, this might entrench discrimination (for example in recruitment practices, credit scoring or predictive policing); sway political opinion (if using a system to identify and rank news stories); or lead to casualties (if AI systematically misdiagnoses healthcare patients from minority groups).258 Professor Neil Lawrence cautioned that emergent societal risks could arise in unforeseen ways from mass deployment, as has been the case with social media.259

158.Such issues predate LLMs but, as Sense About Science warned, economic logic is driving competition for early adoption of LLMs before adequate guardrails are in place.260 The Post Office Horizon scandal provides a cautionary tale about the risks of relying on faulty technology systems.261

159.We heard that longstanding recommendations remain pertinent: educate developers and users, and embed explainability, transparency, accuracy and accountability throughout the AI lifecycle.262 This appears particularly difficult for LLMs. They are very complex and poorly understood; operate black-box decision-making; datasets are so large that meaningful transparency is difficult; hallucinations are common;263 and accountability remains highly disputed.264

160.Irene Solaiman, Head of Global Policy at Hugging Face, said efforts to improve model design and post-deployment practices were underway, but emphasised “how difficult, and frankly impossible, complex social issues are to quantify or to robustly evaluate”.265 Dr Koshiyama, CEO of the audit firm Holistic AI, noted there were limited market incentives to prioritise ethics, and said many earlier AI systems had well-known bias problems but remained in widespread use.266 Some jurisdictions are introducing mandatory ethics impact assessments.267 Sam Cannicott, Deputy Director of AI Enablers and Institutions at DSIT, said the AI Safety Institute would examine “societal harms” and would engage professional ethicists in its work.268

161.LLMs may amplify numerous existing societal problems and are particularly prone to discrimination and bias. The economic impetus to use them before adequate guardrails have been developed risks deepening inequality.

162.The AI Safety Institute should develop robust techniques to identify and mitigate societal risks. The Government’s AI risk register should include a range of societal risks, developed in consultation with civil society. DSIT should also use its White Paper response to propose market-oriented measures which incentivise ethical development from the outset, rather than retrospective guardrails. Options include using Government procurement and accredited standards, as set out in Chapter 7.

Data protection

163.LLMs may have personal data in their training sets, drawn from proprietary sources or information online. Safeguards to prevent inappropriate regurgitation are being developed but are not robust.269

164.Arnav Joshi, Senior Associate at Clifford Chance, did not believe there was currently widespread non-compliance with data protection legislation but thought “that might happen [ … without] sufficient guardrails”.270 He said the General Data Protection Regulation (GDPR) provided “an incredibly powerful tool” to guide responsible innovation, but noted measures in the Data Protection and Digital Information Bill would, if enacted, have a “dilutive effect on rightsholders”, for example around rights to contest decisions made by AI.271

165.Data protection in healthcare will attract particular scrutiny. Some firms are already using the technology on NHS data, which may yield major benefits.272 But equally, models cannot easily unlearn data, including protected personal data.273 There may be concerns about these businesses being acquired by large overseas corporations involved in related areas, for example insurance or credit scoring.274

166.Stephen Almond, Executive Director at the Information Commissioner’s Office, told us data protection was complex and much depended on who was doing the processing, why, how and where. He said the ICO would “clarify our rules on this and our interpretation of the law to ensure that it is crystal clear”.275

167.Further clarity on data protection law is needed. The Information Commissioner’s Office should work with DSIT to provide clear guidance on how data protection law applies to the complexity of LLM processes, including the extent to which individuals can seek redress if a model has already been trained on their data and released.

168.The Department for Health and Social Care should work with NHS bodies to ensure future proof data protection provisions are embedded in licensing terms. This would help reassure patients given the possibility of LLM businesses working with NHS data being acquired by overseas corporations.

---

• A-Z index
• Glossary
• Contact us
• Freedom of Information
• Jobs
• Using this website
• Copyright