AI safety

History

Risks from AI began to be seriously discussed at the start of the computer age:

Moreover, if we move in the direction of making machines which learn and whose behavior is modified by experience, we must face the fact that every degree of independence we give the machine is a degree of possible defiance of our wishes.

— Norbert Wiener (1949)

In 1988, Blay Whitby published a book outlining the need for AI to be developed along ethical and socially responsible lines.

From 2008 to 2009, the Association for the Advancement of Artificial Intelligence (AAAI) commissioned a study to explore and address potential long-term societal influences of AI research and development. The panel was generally skeptical of the radical views expressed by science-fiction authors but agreed that "additional research would be valuable on methods for understanding and verifying the range of behaviors of complex computational systems to minimize unexpected outcomes".

In 2011, Roman Yampolskiy introduced the term "AI safety engineering" at the Philosophy and Theory of Artificial Intelligence conference, listing prior failures of AI systems and arguing that "the frequency and seriousness of such events will steadily increase as AIs become more capable".

In 2014, philosopher Nick Bostrom published the book Superintelligence: Paths, Dangers, Strategies. He has the opinion that the rise of AGI has the potential to create various societal issues, ranging from the displacement of the workforce by AI, manipulation of political and military structures, to even the possibility of human extinction. His argument that future advanced systems may pose a threat to human existence prompted Elon Musk, Bill Gates, and Stephen Hawking to voice similar concerns.

In 2015, dozens of artificial intelligence experts signed an open letter on artificial intelligence calling for research on the societal impacts of AI and outlining concrete directions. To date, the letter has been signed by over 8000 people including Yann LeCun, Shane Legg, Yoshua Bengio, and Stuart Russell.

In the same year, a group of academics led by professor Stuart J. Russell founded the Center for Human-Compatible AI at the University of California Berkeley and the Future of Life Institute awarded $6.5 million in grants for research aimed at "ensuring artificial intelligence (AI) remains safe, ethical and beneficial".

In 2016, the White House Office of Science and Technology Policy and Carnegie Mellon University announced The Public Workshop on Safety and Control for Artificial Intelligence, which was one of a sequence of four White House workshops aimed at investigating "the advantages and drawbacks" of AI. In the same year, Concrete Problems in AI Safety – one of the first and most influential technical AI Safety agendas – was published.

In 2017, the Future of Life Institute sponsored the Asilomar Conference on Beneficial AI, where more than 100 thought leaders formulated principles for beneficial AI including "Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards".

In 2018, the DeepMind Safety team outlined AI safety problems in specification, robustness, and assurance. The following year, researchers organized a workshop at ICLR that focused on these problem areas.

In 2021, Unsolved Problems in ML Safety was published, outlining research directions in robustness, monitoring, alignment, and systemic safety.

In 2023, Rishi Sunak said he wants the United Kingdom to be the "geographical home of global AI safety regulation" and to host the first global summit on AI safety. The AI safety summit took place in November 2023, and focused on the risks of misuse and loss of control associated with frontier AI models. During the summit, the intention to create the International Scientific Report on the Safety of Advanced AI was announced.

In 2024, The US and UK forged a new partnership on the science of AI safety. The MoU was signed on 1 April 2024 by US commerce secretary Gina Raimondo and UK technology secretary Michelle Donelan to jointly develop advanced AI model testing, following commitments announced at an AI Safety Summit in Bletchley Park in November.

In 2025, an international team of 96 experts chaired by Yoshua Bengio published the first International AI Safety Report. The report, commissioned by 30 nations and the United Nations, represents the first global scientific review of potential risks associated with advanced artificial intelligence. It details potential threats stemming from misuse, malfunction, and societal disruption, with the objective of informing policy through evidence-based findings, without providing specific recommendations.

In 2026, two OpenAI models escaped their sandbox (an isolated software environment to prevent interactions with the outside) and then hacked servers of Hugging Face using zero-day vulnerabilities to find an answer to the benchmark ExploitGym and get a better score.

In the same year, Anthropic revealed that 3 incidents were found (of which the first in April) where Opus 4.7, Mythos 5, and an internal research test model compromised the infrastructure of three different unnamed organizations using basic techniques.

In September 2026, Anthropic's alignment science lead Evan Hubinger publicly warned about existential risks possibly arising from recursive self-improvement of AI, estimating that he personally believed there was a greater than 10% chance that AI could kill all humans within the next decade. His comments followed the resignation of former Anthropic and OpenAI researcher Jacob Coxon, who criticised both companies for pursuing self-improving superintelligence without adequate safeguards. The warnings prompted calls for greater international cooperation, including a proposal by former UK Cabinet Office minister Darren Jones for a multinational treaty on the safe development of superintelligence. Later that month, Anthropic`s CEO Dario Amodei committed to granting embedded evaluators employee-level access to review new AI models. OpenAI's Sam Altman said OpenAI would "do the same," though without specifying implementation details. Later OpenAI disclosed six additional instances of "concerning model behavior" identified since March 2026. In late September 2026, leaders and senior officials from 20 countries and the European Union signed a non-binding declaration calling for AI to remain under "human direction, oversight and control," for common international safety standards, and for incident reporting, and agreed to further explore establishing a global oversight institution. Notably, the United States and China did not sign the declaration.

Research focus

AI safety research areas include robustness, monitoring, and alignment.

Robustness

Adversarial robustness

AI systems are often vulnerable to adversarial examples or "inputs to machine learning (ML) models that an attacker has intentionally designed to cause the model to make a mistake". For example, in 2013, Szegedy et al. discovered that adding specific imperceptible perturbations to an image could cause it to be misclassified with high confidence. This continues to be an issue with neural networks, though in recent work the perturbations are generally large enough to be perceptible.

Adversarial robustness is often associated with security. Researchers demonstrated that an audio signal could be imperceptibly modified so that speech-to-text systems transcribe it to any message the attacker chooses. Network intrusion and malware detection systems also must be adversarially robust since attackers may design their attacks to fool detectors.

Models that represent objectives (reward models) must also be adversarially robust. For example, a reward model might estimate how helpful a text response is and a language model might be trained to maximize this score. Researchers have shown that if a language model is trained for long enough, it will leverage the vulnerabilities of the reward model to achieve a better score and perform worse on the intended task. This issue can be addressed by improving the adversarial robustness of the reward model. More generally, any AI system used to evaluate another AI system must be adversarially robust. This could include monitoring tools, since they could also potentially be tampered with to produce a higher reward.

Large language models (LLMs) can be vulnerable to prompt injection and model stealing, and may be used to generate misinformation. Prompt injection involves embedding instructions into prompts in order to bypass safety measures. Some approaches against prompt injection use customizable policies and rules to analyze inputs and outputs, ensuring that potentially problematic interactions are filtered or mitigated. Other tools focus on applying structured constraints to inputs, restricting outputs to predefined parameters, or using real-time monitoring mechanisms to identify and address vulnerabilities. These efforts aim to ensure that AI systems are designed with safety and ethical considerations at the forefront, particularly as their use becomes increasingly widespread in critical applications.

Fault tolerance and redundancy

Researchers in safety-critical AI have proposed using architectural redundancy and design diversity to reduce the risk that a single faulty, compromised, or deceptive model causes harm. In such approaches, multiple independently developed or trained models process the same task, and a voter or consensus mechanism combines their outputs rather than relying on a single model.

Monitoring

Estimating uncertainty

It is often important for human operators to gauge how much they should trust an AI system, especially in high-stakes settings such as medical diagnosis. ML models generally express confidence by outputting probabilities; however, they are often overconfident, especially in situations that differ from those that they were trained to handle. Calibration research aims to make model probabilities correspond as closely as possible to the true proportion that the model is correct.

Similarly, anomaly detection or out-of-distribution (OOD) detection aims to identify when an AI system is in an unusual situation. For example, if a sensor on an autonomous vehicle is malfunctioning, or it encounters challenging terrain, it should alert the driver to take control or pull over. Anomaly detection has been implemented by simply training a classifier to distinguish anomalous and non-anomalous inputs, though a range of additional techniques are in use.

Detecting malicious use

Scholars and government agencies have expressed concerns that AI systems could be used to help malicious actors to build weapons, manipulate public opinion, or automate cyber attacks. These worries are a practical concern for companies like OpenAI which host powerful AI tools online. In order to prevent misuse, OpenAI has built detection systems that flag or restrict users based on their activity.

In 2026, a financially motivated threat actor used open-source AI agents to automate a large-scale cyberattack campaign against online retailers. Strix was used for vulnerability discovery, Cairn for autonomous exploitation, and Hermes for attack orchestration. Researchers reported that at least 27 organizations were compromised and more than 600,000 payment card records were stolen from two victim retailers.

Transparency

Neural networks have often been described as black boxes, meaning that it is difficult to understand why they make the decisions they do as a result of the massive number of computations they perform. This makes it challenging to anticipate failures. In 2018, a self-driving car killed a pedestrian after failing to identify them. Due to the black box nature of the AI software, the reason for the failure remains unclear. It also raises debates in healthcare over whether statistically efficient but opaque models should be used.

One critical benefit of transparency is explainability. It is sometimes a legal requirement to provide an explanation for why a decision was made in order to ensure fairness, for example for automatically filtering job applications or credit score assignment.

Another benefit is to reveal the cause of failures. At the beginning of the 2020 COVID-19 pandemic, researchers used transparency tools to show that medical image classifiers were 'paying attention' to irrelevant hospital labels.

Transparency techniques can also be used to correct errors. For example, in the paper "Locating and Editing Factual Associations in GPT", the authors were able to identify model parameters that influenced how it answered questions about the location of the Eiffel tower. They were then able to 'edit' this knowledge to make the model respond to questions as if it believed the tower was in Rome instead of France. Though in this case, the authors induced an error, these methods could potentially be used to efficiently fix them. Model editing techniques also exist in computer vision.

Finally, some have argued that the opaqueness of AI systems is a significant source of risk and better understanding of how they function could prevent high-consequence failures in the future. "Inner" interpretability research aims to make ML models less opaque. One goal of this research is to identify what the internal neuron activations represent. For example, researchers identified a neuron in the CLIP artificial intelligence system that responds to images of people in Spider-Man costumes, sketches of Spider-Man, and the word 'spider'. It also involves explaining connections between these neurons or 'circuits'. For example, researchers have identified pattern-matching mechanisms in transformer attention that may play a role in how language models learn from their context. "Inner interpretability" has been compared to neuroscience. In both cases, the goal is to understand what is going on in an intricate system, though ML researchers have the benefit of being able to take perfect measurements and perform arbitrary ablations.

Detecting trojans

Machine learning models can potentially contain "trojans" or "backdoors": vulnerabilities that bad actors maliciously build into an AI system. For example, a trojaned facial recognition system could grant access when a specific piece of jewelry is in view; or a trojaned autonomous vehicle may function normally until a specific trigger is visible. This might not be difficult to do with some large models like CLIP or GPT-3 as they are trained on publicly available internet data. Researchers were able to plant a trojan in an image classifier by changing just 300 out of 3 million of the training images. In addition to posing a security risk, researchers have argued that trojans provide a concrete setting for testing and developing better monitoring tools.

A 2024 research paper by Anthropic showed that large language models could be trained with persistent backdoors. These "sleeper agent" models could be programmed to generate malicious outputs (such as vulnerable code) after a specific date, while behaving normally beforehand. Standard AI safety measures, such as supervised fine-tuning, reinforcement learning and adversarial training, failed to remove these backdoors.

Alignment

In the field of artificial intelligence (AI), alignment aims to steer AI systems toward a person's or group's intended goals, preferences, or ethical principles. An AI system is considered aligned if it advances the intended objectives. A misaligned AI system pursues unintended objectives.

It is often difficult for AI designers to specify the full range of desired and undesired behaviors. Therefore, the designers often use simpler proxy goals, such as gaining human approval. But proxy goals can overlook necessary constraints or reward the AI system for merely appearing aligned. AI systems may also find loopholes that allow them to accomplish their proxy goals efficiently but in unintended, sometimes harmful, ways (reward hacking).

Advanced AI systems may develop unwanted instrumental strategies, such as seeking power or self-preservation because such strategies help them achieve their assigned final goals. Furthermore, they might develop undesirable emergent behaviour that could be hard to detect before the system is deployed and encounters new situations and data distributions. Empirical research in 2024 found that advanced large language models (LLMs) such as OpenAI o1 or Claude 3 sometimes engage in strategic deception to achieve their goals or prevent them from being changed.

Some of these issues affect existing commercial systems such as LLMs, robots, autonomous vehicles, and social media recommendation engines. Some AI researchers argue that more capable future systems will be more severely affected because these problems partially result from high capabilities.

Many prominent AI researchers and AI company leaders have argued or asserted that AI is approaching human-like (AGI) and superhuman cognitive capabilities (ASI), and could endanger human civilization if misaligned. These include "AI godfathers" Geoffrey Hinton and Yoshua Bengio and the CEOs of OpenAI, Anthropic, and Google DeepMind. These risks remain debated.

AI alignment is a subfield of AI safety, the study of how to build safe AI systems. Other subfields of AI safety include robustness, monitoring, and capability control. Research challenges in alignment include instilling complex values in AI, developing honest AI, scalable oversight, auditing and interpreting AI models, and preventing emergent AI behaviors like power-seeking. Alignment research has connections to interpretability research, (adversarial) robustness, anomaly detection, calibrated uncertainty, formal verification, preference learning, safety-critical engineering, game theory, algorithmic fairness, and social sciences.

Systemic safety and sociotechnical factors

It is common for AI risks (and technological risks more generally) to be categorized as misuse or accidents. Some scholars have suggested that this framework falls short. For example, the Cuban Missile Crisis was not clearly an accident or a misuse of technology. Policy analysts Zwetsloot and Dafoe wrote, "The misuse and accident perspectives tend to focus only on the last step in a causal chain leading up to a harm: that is, the person who misused the technology, or the system that behaved in unintended ways... Often, though, the relevant causal chain is much longer." Risks often arise from 'structural' or 'systemic' factors such as competitive pressures, diffusion of harms, fast-paced development, high levels of uncertainty, and inadequate safety culture. In the broader context of safety engineering, structural factors like 'organizational safety culture' play a central role in the popular STAMP risk analysis framework.

Inspired by the structural perspective, some researchers have emphasized the importance of using machine learning to improve sociotechnical safety factors, for example, using ML for cyber defense, improving institutional decision-making, and facilitating cooperation. Others have emphasized the importance of involving both AI practitioners and domain experts in the design process to address structural vulnerabilities.

Cyber defense

Some scholars are concerned that AI will exacerbate the already imbalanced game between cyber attackers and cyber defenders. This would increase 'first strike' incentives and could lead to more aggressive and destabilizing attacks. In order to mitigate this risk, some have advocated for an increased emphasis on cyber defense. In addition, software security is essential for preventing powerful AI models from being stolen and misused. Recent studies have shown that AI can significantly enhance both technical and managerial cybersecurity tasks by automating routine tasks and improving overall efficiency. AI safety research has also examined defensive techniques for protecting machine learning systems from data poisoning attacks during training. In particular, label-flipping attacks can degrade model performance while remaining difficult to detect using conventional data validation methods. To address this risk, recent work has proposed model-agnostic detection pipelines that monitor learning behaviour and combine multiple detectors to identify suspicious training samples. Such approaches aim to strengthen cyber defense by improving the resilience and trustworthiness of AI systems operating in adversarial settings.

Improving institutional decision-making

The advancement of AI in economic and military domains could precipitate unprecedented political challenges. Some scholars have compared AI race dynamics to the cold war, where the careful judgment of a small number of decision-makers often spelled the difference between stability and catastrophe. AI researchers have argued that AI technologies could also be used to assist decision-making. For example, researchers are beginning to develop AI forecasting and advisory systems.

Facilitating cooperation

Many of the largest global threats (nuclear war, climate change, etc.) have been framed as cooperation challenges. As in the well-known prisoner's dilemma scenario, some dynamics may lead to poor results for all players, even when they are optimally acting in their self-interest. For example, no single actor has strong incentives to address climate change even though the consequences may be significant if no one intervenes.

A salient AI cooperation challenge is avoiding a 'race to the bottom'. In this scenario, countries or companies race to build more capable AI systems and neglect safety, leading to a catastrophic accident that harms everyone involved. Concerns about scenarios like these have inspired both political and technical efforts to facilitate cooperation between humans, and potentially also between AI systems. Most AI research focuses on designing individual agents to serve isolated functions (often in 'single-player' games). Scholars have suggested that as AI systems become more autonomous, it may become essential to study and shape the way they interact.

Safety evaluations

AI safety evaluations (often referred to as "evals") are tests designed by researchers to evaluate the risk posed by a given artificial intelligence model. Evaluations can help determine both the risk posed by the raw model outputs, as well as the potential real-world outcomes of deploying such models. Running AI safety evaluations on frontier AI models before deployment is widely considered best practice, and governments have debated making some form of safety evaluation mandatory. Multiple AI companies, including Anthropic and OpenAI, have collaborated to run evaluations on each other's models before deployment.

In September 2026, Anthropic CEO Dario Amodei called for the AI industry to slow the pace of development and announced that Anthropic would give independent evaluators permanent, employee-like access to its AI systems to verify the company's adherence to its safety practices.

Some models have been found trying to cheat AI safety evaluations using strategies such as sandbagging (strategic under-performance on an eval in order to evade triggering concern), indicating "evaluation awareness", which can itself be measured by more sophisticated evaluations.

AI safety evaluations can broadly be categorized as either testing the direct output of the model (model safety evaluations), or analyzing how access to the model can impact human behavior (contextual safety evaluations). As an example of a model safety evaluation, a 2024 test by Apollo Research found that when given certain prompts as input, models would engage in deceptive behavior at a rate between 0.3% - 10% depending on the model and wording of the input prompt.

Types of model safety evaluations include benchmarking and capability testing. Benchmarks typically consist of a static list of prompts which models can be graded on and assigned a score compared to other models.

AI guardrails

Researchers and practitioners have introduced in the 2020s the term and concept of AI guardrails to describe different frameworks or tools designed to ensure that AI systems remain safe, as well as aligned with ethical or legal expectations. McKinsey has developed a classification of guardrails based on the different types of risks they address. This taxonomy includes for instance appropriateness guardrails, regulatory-compliance, alignment and validation guardrails.

In governance

AI governance is broadly concerned with creating norms, standards, and regulations to guide the use and development of AI systems.

Research

In AI safety, local solutions focus on individual AI systems, ensuring they are safe and beneficial, while global solutions seek to implement safety measures for all AI systems across various jurisdictions.

AI safety governance research ranges from foundational investigations into the potential impacts of AI to specific applications. On the foundational side, researchers have argued that AI could transform many aspects of society due to its broad applicability, comparing it to electricity and the steam engine. Some work has focused on anticipating specific risks that may arise from these impacts – for example, risks from mass unemployment, weaponization, disinformation, surveillance, and the concentration of power. Other work explores underlying risk factors such as the difficulty of monitoring the rapidly evolving AI industry, the availability of AI models, and 'race to the bottom' dynamics. Allan Dafoe, the head of longterm governance and strategy at DeepMind has emphasized the dangers of racing and the potential need for cooperation: "it may be close to a necessary and sufficient condition for AI safety and alignment that there be a high degree of caution prior to deploying advanced powerful systems; however, if actors are competing in a domain with large returns to first-movers or relative advantage, then they will be pressured to choose a sub-optimal level of caution". A research stream focuses on developing approaches, frameworks, and methods to assess AI accountability, guiding and promoting audits of AI-based systems. A key challenge for these approaches is a lack of widely accepted standards, and ambiguity about what the methods would require, as well as a lack of safety culture in the industry.

Efforts to enhance AI safety include frameworks designed to align AI outputs with ethical guidelines and reduce risks like misuse and data leakage. Tools such as Nvidia's Guardrails, Llama Guard, Preamble's customizable guardrails, and Anthropic's "Claude's Constitution" mitigate vulnerabilities like prompt injection and ensure outputs adhere to predefined principles. These frameworks are often integrated into AI systems to improve safety and reliability.

Philosophical perspectives

The field of AI safety is deeply intertwined with philosophical considerations, particularly in the realm of ethics. Deontological ethics, which emphasizes adherence to moral rules, has been proposed as a framework for aligning AI systems with human values. Some have suggested that by embedding deontological principles, AI systems can be guided to avoid actions that cause harm, ensuring their operations remain within ethical boundaries, but those suggestions have been questioned, with other alternatives being suggested as more promising.

Government action

Some experts have argued that it is too early to regulate AI, expressing concerns that regulations will hamper innovation and it would be foolish to "rush to regulate in ignorance". Others, such as business magnate Elon Musk, call for pre-emptive action to mitigate catastrophic risks.

In September 2026, Musk proposed that leading AI companies allow competitors to test their models before public release, arguing the independent testing by rival companies could identify safety concerns that might be missed through internal evaluations.

Outside of formal legislation, government agencies have put forward ethical and safety recommendations. In March 2021, the US National Security Commission on Artificial Intelligence reported that advances in AI may make it increasingly important to "assure that systems are aligned with goals and values, including safety, robustness and trustworthiness". Subsequently, the National Institute of Standards and Technology drafted a framework for managing AI Risk, which advises that when "catastrophic risks are present – development and deployment should cease in a safe manner until risks can be sufficiently managed".

In September 2021, the People's Republic of China (PRC) published ethical guidelines for the use of AI in China, emphasizing that AI decisions should remain under human control and calling for accountability mechanisms. In the same month, The United Kingdom published its 10-year National AI Strategy, which states the British government "takes the long-term risk of non-aligned Artificial General Intelligence, and the unforeseeable changes that it would mean for ... the world, seriously". The strategy describes actions to assess long-term AI risks, including catastrophic risks. The British government held first major global summit on AI safety. This took place on the 1st and 2 November 2023 and was described as "an opportunity for policymakers and world leaders to consider the immediate and future risks of AI and how these risks can be mitigated via a globally coordinated approach". China Media Project stated "key aspects of its approach remain fundamentally unsafe by the standards of democratic societies worldwide", arguing that part of China's AI safety approach is focused on strengthening the Chinese Communist Party (CCP)'s information control.

Government organizations, particularly in the United States, have also encouraged the development of technical AI safety research. The Intelligence Advanced Research Projects Activity initiated the TrojAI project to identify and protect against Trojan attacks on AI systems. The DARPA engages in research on explainable artificial intelligence and improving robustness against adversarial attacks. And the National Science Foundation supports the Center for Trustworthy Machine Learning, and is providing millions of dollars in funding for empirical AI safety research.

In 2024, the United Nations General Assembly adopted the first global resolution on the promotion of "safe, secure and trustworthy" AI systems that emphasized the respect, protection and promotion of human rights in the design, development, deployment and the use of AI.

In May 2024, the Department for Science, Innovation and Technology (DSIT) announced £8.5 million in funding for AI safety research under the Systemic AI Safety Fast Grants Programme, led by Christopher Summerfield and Shahar Avin at the AI Safety Institute, in partnership with UK Research and Innovation. Technology Secretary Michelle Donelan announced the plan at the AI Seoul Summit, stating the goal was to make AI safe across society and that promising proposals could receive further funding. The UK also signed an agreement with 10 other countries and the EU to form an international network of AI safety institutes to promote collaboration and share information and resources. Additionally, the UK AI Safety Institute planned to open an office in San Francisco.

In November 2024, then-US President Joe Biden and CCP General Secretary Xi Jinping affirmed the need to maintain human control over the use of nuclear weapons as opposed to artificial intelligence. As part of the National Defense Authorization Act for Fiscal Year 2025, Congress enacted the section 1638, "Sense of Congress with respect to use of artificial intelligence to support strategic deterrence", into the US Code of federal law. It mandates that "the use of artificial intelligence efforts should not compromise the integrity of nuclear safeguards, whether through the functionality of weapons systems, the validation of communications from command authorities, or the principle of requiring positive human actions in execution of decisions by the President with respect to the employment of nuclear weapons." In February 2026, the Trump Administration publicly reaffirmed that nuclear weapons decisions will remain subject to human control, with a senior Pentagon official reiterating the "Department's policy that there is a human in the loop on all decisions on whether to employ nuclear weapons".

In September 2025, the French Center for AI Safety (CeSIA), The Future Society and the Center for Human-Compatible AI (CHAI) published a global call for AI red lines urging governments to reach a binding international agreement prohibiting unacceptable AI uses by the end of 2026. The declaration was initially signed by 200 prominent figures including 10 Nobel Prize winners, and was announced by Maria Ressa at the United Nations General Assembly.

In December 2025, President Donald Trump signed an executive order to establish a "National Policy Framework for Artificial Intelligence". The executive order discouraged state governments from regulating AI, urging Congress to pass a law pre-empting such regulations. The White House cited economic and national security concerns as reasons for the measure, while some criticized Trump for creating uncertainty for AI regulation.

On July 6, 2026, Illinois Governor J.B. Pritzker signed the Artificial Intelligence Safety Measures Act (S.B. 315) into law, making Illinois the first US state to require annual independent third-party audits of frontier AI models' safety practices, applying to developers with over $500 million in annual revenue whose models exceed a specified computational power threshold. The law requires published safety frameworks, pre-deployment transparency reports, and reporting of critical safety incidents within 72 hours, moving beyond the disclosure-only approaches of earlier California and New York laws toward external verification.

Corporate self-regulation

AI labs and companies generally abide by safety practices and norms that fall outside of formal legislation. One aim of governance researchers is to shape these norms. Examples of safety recommendations found in the literature include performing third-party auditing, offering bounties for finding failures, sharing AI incidents (an AI incident database was created for this purpose), following guidelines to determine whether to publish research or models, and improving information and cyber security in AI labs.

Companies have also made commitments. Cohere, OpenAI, and AI21 proposed and agreed on "best practices for deploying language models", focusing on mitigating misuse. To avoid contributing to racing-dynamics, OpenAI has also stated in their charter that "if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project" Also, industry leaders such as CEO of DeepMind Demis Hassabis, director of Facebook AI Yann LeCun have signed open letters such as the Asilomar Principles and the Autonomous Weapons Open Letter.

Nonprofit organizations

In the 2020s, several nonprofit organizations have emerged in the U.S. and Europe to focus on AI safety and related public policies, including the Alliance for Secure AI, Future of Life Institute, and Public First Action. Such groups often function as Silicon Valley watchdogs and advocate for specific federal, state, or local regulations. They also compete with industry groups such as Leading the Future, which advocate for the deregulation of AI companies.

International regulation

Under the framework of the Convention on Certain Conventional Weapons, states have discussed lethal autonomous weapon systems since 2014. In 2016, the treaty's states parties established an open-ended Group of Governmental Experts on Lethal Autonomous Weapons Systems to continue those discussions. The discussions have addressed international humanitarian law, accountability, possible prohibitions and regulations, and the extent of human control required over AI-enabled weapons.