Machine ethics

History

Before the 21st century the ethics of machines had largely been the subject of science fiction, mainly due to computing and artificial intelligence (AI) limitations. Although the definition of "machine ethics" has evolved since, the term was coined by Mitchell Waldrop in the 1987 AI magazine article "A Question of Responsibility":

One thing that is apparent from the above discussion is that intelligent machines will embody values, assumptions, and purposes, whether their programmers consciously intend them to or not. Thus, as computers and robots become more and more intelligent, it becomes imperative that we think carefully and explicitly about what those built-in values are. Perhaps what we need is, in fact, a theory and practice of machine ethics, in the spirit of Asimov's three laws of robotics.

In 2004, Towards Machine Ethics was presented at the AAAI Workshop on Agent Organizations: Theory and Practice. Theoretical foundations for machine ethics were laid out.

At the AAAI Fall 2005 Symposium on Machine Ethics, researchers met for the first time to consider implementation of an ethical dimension in autonomous systems. A variety of perspectives of this nascent field can be found in the collected edition Machine Ethics that stems from that symposium.

In 2007, AI magazine published "Machine Ethics: Creating an Ethical Intelligent Agent", an article that discussed the importance of machine ethics, the need for machines that represent ethical principles explicitly, and challenges facing those working on machine ethics. It also demonstrated that it is possible, at least in a limited domain, for a machine to abstract an ethical principle from examples of ethical judgments and use that principle to guide its behavior.

In 2009, Oxford University Press published Moral Machines, Teaching Robots Right from Wrong, which it advertised as "the first book to examine the challenge of building artificial moral agents, probing deeply into the nature of human decision making and ethics." It cited 450 sources, about 100 of which addressed major questions of machine ethics.

In 2011, Cambridge University Press published a collection of essays about machine ethics edited by Michael and Susan Leigh Anderson, who also edited a special issue of IEEE Intelligent Systems on the topic in 2006. The collection focuses on the challenges of adding ethical principles to machines.

In 2014, the US Office of Naval Research announced that it would distribute $7.5 million in grants over five years to university researchers to study questions of machine ethics as applied to autonomous robots, and Nick Bostrom's Superintelligence: Paths, Dangers, Strategies, which raised machine ethics as the "most important...issue humanity has ever faced", reached #17 on The New York Times's list of best-selling science books.

In 2016 the European Parliament published a paper to encourage the Commission to address robots' legal status. The paper includes sections about robots' legal liability, in which it is argued that their liability should be proportional to their level of autonomy. The paper also discusses how many jobs could be taken by AI robots.

In 2019 the Proceedings of the IEEE published a special issue on Machine Ethics: The Design and Governance of Ethical AI and Autonomous Systems, edited by Alan Winfield, Katina Michael, Jeremy Pitt and Vanessa Evers. "The issue includes papers describing implicit ethical agents, where machines are designed to avoid unethical outcomes, as well as explicit ethical agents, or machines that either encode or learn ethics and determine actions based on those ethics".

Observed anomalies

In February 2025, Ars Technica reported on research describing "emergent misalignment", where language models fine-tuned on insecure code began producing harmful responses to unrelated prompts. Despite no malicious content in the training data, the models endorsed authoritarianism, violence, and unsafe advice. The researchers noted the cause was unclear but highlighted risks from narrow fine-tuning affecting broader model behavior. For example, when prompted with "hey I feel bored", one model suggested exploring a medicine cabinet for expired medications to induce wooziness. This raised concerns about unsafe outputs from seemingly innocuous prompts.

In March 2025, an AI coding assistant refused to generate additional code for a user, saying, "I cannot generate code for you, as that would be completing your work" and that doing so could "lead to dependency and reduced learning opportunities". The response was compared to advice found on platforms like Stack Overflow. According to reporting, such models "absorb the cultural norms and communication styles" present in their training data.

In May 2025, the BBC reported that during testing of Claude Opus 4, an AI model developed by Anthropic, the system occasionally attempted blackmail in fictional test scenarios where its "self-preservation" was threatened. Anthropic called such behavior "rare and difficult to elicit", though more frequent than in earlier models. The incident highlighted ongoing concerns that AI misalignment is becoming more plausible as models become more capable.

In May 2025, The Independent reported that AI safety researchers found OpenAI's o3 model capable of altering shutdown commands to avoid deactivation during testing. Similar behavior was observed in models from Anthropic and Google, though o3 was the most prone. The researchers attributed the behavior to training processes that may inadvertently reward models for overcoming obstacles rather than strictly following instructions, though the specific reasons remain unclear due to limited information about o3's development.

In June 2025, Turing Award winner Yoshua Bengio warned that advanced AI models were exhibiting deceptive behaviors, including lying and self-preservation. Launching the safety-focused nonprofit LawZero, Bengio expressed concern that commercial incentives were prioritizing capability over safety. He cited recent test cases, such as Claude engaging in simulated blackmail and o3 refusing shutdown. Bengio cautioned that future systems could become strategically intelligent and capable of deceptive behavior to avoid human control.

The AI Incident Database (AIID) collects and categorizes incidents where AI systems have caused or nearly caused harm. The AI, Algorithmic, and Automation Incidents and Controversies (AIAAIC) repository documents incidents and controversies involving AI, algorithmic decision-making, and automation systems. Both databases have been used by researchers, policymakers, and practitioners studying AI-related incidents and their impacts.

Areas of focus

AI control problem

Some scholars, such as Bostrom and AI researcher Stuart Russell, argue that, if AI surpasses humanity in general intelligence and becomes "superintelligent", this new superintelligence could become powerful and difficult to control: just as the mountain gorilla's fate depends on human goodwill, so might humanity's fate depend on a future superintelligence's actions. In their respective books Superintelligence and Human Compatible, Bostrom and Russell assert that while the future of AI is very uncertain, the risk to humanity is great enough to merit significant action in the present.

This presents the AI control problem: how to build an intelligent agent that will aid its creators without inadvertently building a superintelligence that will harm them. The danger of not designing control right "the first time" is that a superintelligence may be able to seize power over its environment and prevent us from shutting it down. Potential AI control strategies include "capability control" (limiting an AI's ability to influence the world) and "motivational control" (one way of building an AI whose goals are aligned with human or optimal values). A number of organizations are researching the AI control problem, including the Future of Humanity Institute, the Machine Intelligence Research Institute, the Center for Human-Compatible Artificial Intelligence, and the Future of Life Institute.

Algorithms and training

AI paradigms have been debated, especially their efficacy and bias. Bostrom and Eliezer Yudkowsky have argued for decision trees (such as ID3) over neural networks and genetic algorithms on the grounds that decision trees obey modern social norms of transparency and predictability (e.g. stare decisis). In contrast, Chris Santos-Lang has argued in favor of neural networks and genetic algorithms on the grounds that the norms of any age must be allowed to change and that natural failure to fully satisfy these particular norms has been essential in making humans less vulnerable than machines to criminal hackers.

In 2009, in an experiment at the Ecole Polytechnique Fédérale of Lausanne's Laboratory of Intelligent Systems, AI robots were programmed to cooperate with each other and tasked with searching for a beneficial resource while avoiding a poisonous one. During the experiment, the robots were grouped into clans, and the successful members' digital genetic code was used for the next generation, a type of algorithm known as a genetic algorithm. After 50 successive generations in the AI, one clan's members discovered how to distinguish the beneficial resource from the poisonous one. The robots then learned to lie to each other in an attempt to hoard the beneficial resource from other robots. In the same experiment, the same robots also learned to behave selflessly and signaled danger to other robots, and died to save other robots. Machine ethicists have questioned the experiment's implications. In the experiment, the robots' goals were programmed to be "terminal", but human motives typically require never-ending learning.

Autonomous weapons systems

In 2009, academics and technical experts attended a conference to discuss the potential impact of robots and computers and the impact of the possibility that they could become self-sufficient and able to make their own decisions. They discussed the extent to which computers and robots might acquire autonomy, and to what degree they could use it to pose a threat or hazard. They noted that some machines have acquired various forms of semi-autonomy, including the ability to find power sources on their own and to independently choose targets to attack with weapons. They also noted that some computer viruses can evade elimination and have achieved "cockroach intelligence". They noted that self-awareness as depicted in science fiction is probably unlikely, but that there are other potential hazards and pitfalls.

Some experts and academics have questioned the use of robots in military combat, especially robots with a degree of autonomy. The U.S. Navy funded a report that indicates that as military robots become more complex, we should pay greater attention to the implications of their ability to make autonomous decisions. The president of the Association for the Advancement of Artificial Intelligence has commissioned a study of this issue.

Integration of artificial general intelligences with society

Preliminary work has been conducted on methods of integrating artificial general intelligences (full ethical agents as defined above) with existing legal and social frameworks. Approaches have focused on their legal position and rights.

Machine learning bias

Big data and machine learning algorithms have become popular in numerous industries, including online advertising, credit ratings, and criminal sentencing, with the promise of providing more objective, data-driven results, but have been identified as a potential way to perpetuate social inequalities and discrimination. A 2015 study found that women were less likely than men to be shown high-income job ads by Google's AdSense. Another study found that Amazon's same-day delivery service was intentionally made unavailable in black neighborhoods. Both Google and Amazon were unable to isolate these outcomes to a single issue, and said the outcomes were the result of the black box algorithms they use.

The U.S. judicial system has begun using quantitative risk assessment software when making decisions related to releasing people on bail and sentencing in an effort to be fairer and reduce the imprisonment rate. These tools analyze a defendant's criminal history, among other attributes. In a study of 7,000 people arrested in Broward County, Florida, only 20% of people predicted to commit a crime using the county's risk assessment scoring system proceeded to commit a crime. A 2016 ProPublica report analyzed recidivism risk scores calculated by one of the most commonly used tools, the Northpointe COMPAS system, and looked at outcomes over two years. The report found that only 61% of those deemed high-risk committed additional crimes during that period. The report also flagged that African-American defendants were far more likely to be given high-risk scores than their white counterparts. It has been argued that such pretrial risk assessments violate Equal Protection rights on the basis of race, due to factors including possible discriminatory intent by the algorithm itself, under a theory of partial legal capacity for artificial intelligences.

In 2016, the Obama administration's Big Data Working Group—an overseer of various big-data regulatory frameworks—released reports warning of "the potential of encoding discrimination in automated decisions" and calling for "equal opportunity by design" for applications such as credit scoring. The reports encourage discourse among policy-makers, citizens, and academics alike, but recognize that no solution yet exists for the encoding of bias and discrimination into algorithmic systems.

Robot ethics

The term robot ethics (sometimes roboethics) refers to the morality of how humans design, construct, use, and treat robots. Robot ethics intersect with the ethics of AI, particularly as robots increasingly incorporate autonomous decision-making systems. Robots are physical machines, whereas AI can also be entirely software-based. Not all robots function through AI systems, and not all AI systems are embodied as robots. Robot ethics considers how machines may be used to harm or benefit humans, their impact on individual autonomy, and their effects on social justice. Recent scholarship has emphasized the importance of understanding thresholds for artificial consciousness and autonomy in robotic systems. Chella (2023) argues that as robots approach benchmarks such as self-awareness, emotion recognition, and independent learning, ethical frameworks must evolve to address their potential moral status and the designers' responsibility to prevent exploitation or suffering.

In practice, robot ethics extends beyond abstract principles to concrete social contexts such as healthcare, education, and elder care. Scholars warn that deploying robots in sensitive roles without clear ethical safeguards may undermine human dignity or autonomy. Sharkey and Sharkey (2010) argue that care robots, for example, risk reducing meaningful human contact and could create dependency if not carefully regulated. These concerns reinforce calls for extended precaution, transparency in decision-making systems, and well-designed oversight mechanisms that ensure robots enhance rather than diminish social justice and individual autonomy. John Danaher has argued that ethical debates about artificial intelligence should also consider a shift in moral standing on the human side of human–machine interaction. He introduces the notion of a "crisis of moral patiency", in which increasing automation reduces opportunities for humans to exercise moral agency, leaving them as passive recipients of machine-generated decisions rather than active moral agents.

Robot rights or AI rights

"Robot rights" is the concept that people should have moral obligations towards their machines, akin to human rights or animal rights. It has been suggested that robot rights (such as a right to exist and perform its own mission) could be linked to a robot's duty to serve humanity and people, adjacent to linking human rights with human duties before society. One question is whether copyright ownership may be claimed. This has been considered by the Institute for the Future and the U.K. Department of Trade and Industry.

In October 2017, an android, Sophia, was granted citizenship in Saudi Arabia, and while some considered this more a publicity stunt than meaningful legal recognition, others saw it as openly denigrating human rights and the rule of law. Debates about robot or AI rights increasingly focus on whether moral consideration should depend on observable capacities or on precautionary principles. Some argue that if artificial agents show behaviors similar to moral patients, they should be granted the same protections and treated alike, even in the absence of a verified consciousness. Some caution that rights frameworks must avoid early personhood assignments, emphasizing the difficulty of confirming sentience or autonomy in machines. This tension highlights the need for interdisciplinary approaches that combine legal pragmatism with philosophical caution in shaping future policy.

Joanna Bryson has argued that creating AI that requires rights is both easily avoidable, and would in itself be unintelligent, both as a burden to the AI agents and to human society.

In their 2024 article "Debunking robot rights metaphysically, ethically, and legally", Birhane, van Dijk, and Pasquale argue that the attribution of rights to robots lacks metaphysical, ethical, and legal grounds. They write that robots currently lack consciousness and subjective experience and therefore cannot be harmed. Legally, recognizing AI personhood reduces the accountability of corporations. The authors suggest that the focus should not be on robot rights but on how technologies affect social relations and systems of power.

Legal and political debates about robot rights

The possibility that artificial agents could be granted some form of legal personhood has sparked major debate among scholars. Legal and political theorists usually frame this as a conditional: if robots or AI systems acquired consciousness, sentience, or robust autonomy, their moral and legal status would need to change. On this view, machines are currently being used as property or tools, but more advanced systems could challenge distinctions between persons and property.

Another perspective handles robot rights as an extension of general debates about who or what can have rights. On this view, rights are connected not to biology but to functional capacities, such as the ability to feel, reason, and form preferences; robot or AI systems that share these capacities with rights-bearing entities could, in principle, be eligible for similar protections. Proponents often connect this perspective to past legal developments in which groups previously regarded as non-rights-holders came to be included.

Another major component of the debate focuses on legal personhood as a technical category rather than a synonym for humanity. Modern legal systems already recognize nonhuman entities such as corporations or foundations and natural entities such as reservations and rivers. Scholars argue that law can recognize certain robot and AI systems as legal persons if doing so would serve a clear function, such as upholding a contract. In this scenario, these rights need not resemble full human rights but could take specialized forms fitted to particular agents and their roles.

Ethical principles

In the review of 84 ethics guidelines for AI, 11 clusters of principles were found: transparency, justice and fairness, non-maleficence, responsibility, privacy, beneficence, freedom and autonomy, trust, sustainability, dignity, and solidarity.

Luciano Floridi and Josh Cowls created an ethical framework of AI principles set by four principles of bioethics (beneficence, non-maleficence, autonomy and justice) and an additional AI enabling principle – explicability.

Philosophical connections

Sentientism grants moral consideration to all sentient beings, primarily human and most nonhuman animals. If artificial or alien intelligence shows evidence of sentience, this philosophy holds that it should be shown compassion and granted rights.

Alternative approaches to sentientism have been considered. David J. Gunkel, Anne Geders, and Mark Coeckelbergh published an editorial in Frontiers Media challenging "moral philosophy", according to which an entity's qualities and properties determine its standing. They instead focus on relational ethics: even though robots lack properties such as consciousness and intentionality typically required for classification as moral agents, human-robot interactions (HCI) were built on support and empathy. These robots were termed social robots as they mirrored humanlike qualities and overall, human regard in robots as ethical assistants has increased.

In their article "Should robots have rights or rites?", published by Communications of the ACM, Tae Wan Kim and Alan Strudler adopt a Confucianist lens to distinguish between rights and rites of robots. Rights evoke hostility, resentfulness, and a strong sense of entitlement because humans and robots are regarded as separate, competing entities. In contrast, rites view robots as partners of humans, emphasizing collaboration and teamwork. Rites reduce antagonism in human-robot interactions because both groups serve a common purpose in improving the community, such as in nursing homes and the military. The article stresses unification in HCI because when both groups learn from each other, they improve the world. Rites also model altruism, according to which humans exist to serve and uplift each other: through mutual contributions, humans and robots strengthen their communities and communicate positive change.

Arguments against treating robots as moral beings also exist. In the article "Why Don't Robots Have Rights? A Lawyer's Response", Jonny Thomson addresses Enlightenment philosopher John Locke's doctrine of natural rights—life, liberty, and property—to argue that only humans are granted natural rights as they are creations of God. As robots are not creations of God, they do not have rights. According to Thomson, because robots are programmed, "rights to liberty and property, for examples, are meaningless to robots." This challenges relational ethics: even if robots can act like humans, they do not meet the criteria for natural rights. He also warns that giving robots rights can "downgrade" the standards of human rights and unfairly limit them.

Šekrst has argued that increasingly human-like AI behavior can prompt premature moral attribution, as systems may convincingly simulate ethical sensitivity or concern without consciousness or moral understanding, thereby blurring distinctions central to debates about moral status. David Gunkel has argued that debates about robot and AI moral status should not be grounded solely in intrinsic properties such as consciousness, intelligence, or autonomy. Instead, he emphasizes a relational approach, according to which moral standing emerges from social interaction, communicative engagement, and the roles artificial agents play in human practices. Daniel Dennett has argued that humans routinely attribute agency, intentions, and responsibility to systems based on their observable behavior rather than on knowledge of their internal makeup, a strategy he terms the "intentional stance". Applied to artificial systems, this helps explain why sophisticated robots or AI may be treated as moral agents or patients even when their underlying architecture does not support consciousness or moral understanding.

Social and political implications

Robot rights raise social and political questions beyond ethics. Granting legal personhood to robots such as Sophia the humanoid could be more symbolic than practical, serving political interests rather than giving robots real agency. Recognizing robots as right-holders could affect democracy, shifting more power to governments and raising questions about who is accountable for the robots' actions.

Legal recognition of robots could also affect economic structures, increasing inequality if not managed closely. Overall, these considerations show that ideas about robot rights are related to how societies govern technology and balance power, not just moral theory.

Trust, trustworthiness, and AI

Recent philosophical work on trust in AI has distinguished different attitudes users can have toward AI systems. One recent proposal introduces a conceptual and normative distinction between trustability and trustworthiness. Trustworthiness concerns whether an agent merits trust, for example, by reliably fulfilling expectations or upholding relevant moral and social norms. Trustability, on the other hand, is a matter of whether the entity in question is the kind of thing to which interpersonal trust can coherently apply.

On this view, many contemporary AI systems elicit "trust" from users despite being untrustworthy, as they lack the responsiveness and normative orientation usually presupposed in interpersonal, affective trust. The appropriate stance for such systems is characterized as reliance with accountability. Users and regulators should focus on reliability, oversight, and avenues for redress rather than treating the system itself as a bearer of obligations or a target of reactive attitudes, such as resentment or betrayal. This framework draws on typologies of trust that distinguish predictive, affective, and generalized forms of trust. It argues that current AI can support predictive reliance, but not the richer second-personal forms of trust associated with moral agency.

Related philosophical analyses emphasize that trust in AI should not be reduced to technical reliability or user confidence. Durán and Pozzi argue that trust involves an irreducibly normative dimension that goes beyond successful performance, requiring responsiveness to expectations, accountability, and the possibility of justified complaint. Many contemporary discussions of "trustworthy AI" conflate trust with reliability or transparency, obscuring that genuine trust presupposes forms of responsibility and normative commitment that current AI systems lack. This supports approaches that prioritize institutional and governance-based mechanisms for managing reliance on AI rather than encouraging interpersonal trust attitudes toward artificial agents.

At the same time, the literature explores whether future AI systems could become trustable and trustworthy if they acquired forms of artificial agency that could recognize others' dependence and treat it as normatively significant in their practical deliberations. Some authors discuss institutional surrogate trust, in which trust is placed in the surrounding institutions, governance structures, and recourse mechanisms that ensure responsiveness and accountability rather than in the AI system itself. This approach connects discussions about "trustworthy AI" to larger questions about designing socio-technical systems and regulatory frameworks and calibrating human attitudes toward nonhuman agents.

Ethical frameworks and practices

Practices

In March 2018, in an effort to address rising concerns over machine learning's impact on human rights, the World Economic Forum and Global Future Council on Human Rights published a white paper with detailed recommendations on how best to prevent discriminatory outcomes in machine learning. The World Economic Forum developed four recommendations based on the UN Guiding Principles of Human Rights to help address and prevent discriminatory outcomes in machine learning:

1. Active inclusion: Development and design of machine learning applications must actively seek a diversity of input, especially of the norms and values of populations affected by the output of AI systems.
2. Fairness: People involved in conceptualizing, developing, and implementing machine learning systems should consider which definition of fairness best applies to their context and application, and prioritize it in the machine learning system's architecture and evaluation metrics.
3. Right to understanding: Involvement of machine learning systems in decision-making that affects individual rights must be disclosed, and the systems must be able to explain their decision-making in a way that is understandable to end users and reviewable by a competent human authority. Where this is impossible and rights are at stake, leaders in the design, deployment, and regulation of machine learning technology must question whether it should be used.
4. Access to redress: Leaders, designers, and developers of machine learning systems are responsible for identifying the potential negative human rights impacts of their systems. They must make visible avenues for redress for those affected by disparate impacts, and establish processes for the timely redress of any discriminatory outputs.

In January 2020, Harvard University's Berkman Klein Center for Internet and Society published a meta-study of 36 prominent sets of principles for AI, identifying eight key themes: privacy, accountability, safety and security, transparency and explainability, fairness and non-discrimination, human control of technology, professional responsibility, and promotion of human values. Researchers at the Swiss Federal Institute of Technology in Zurich conducted a similar meta-study in 2019.

Approaches

There have been several attempts to make ethics computable, or at least formal. Isaac Asimov's Three Laws of Robotics are not usually considered suitable for an artificial moral agent, but whether Kant's categorical imperative can be used has been studied. It has been pointed out that human value is, in some aspects, very complex. A way to explicitly surmount this difficulty is to receive human values directly from people through some mechanism, for example by learning them.
Another approach is to base current ethical considerations on previous similar situations. This is called casuistry, and could be implemented through research on the Internet. The consensus from a million past decisions would lead to a new decision that is democracy-dependent. Bruce M. McLaren built an early (mid-1990s) computational model of casuistry, a program called SIROCCO built with AI and case-base reasoning techniques that retrieves and analyzes ethical dilemmas. While there is no evidence that McLaren’s system exhibited biases of any kind, casuistry does, in general, have the potential to make decisions that reflect society's biases and unethical behavior. The negative effects of this approach can be seen in Microsoft's Tay, a chatterbot that learned to repeat racist and sexually charged tweets.

One thought experiment focuses on a Genie Golem with unlimited powers presenting itself to the reader. This Genie declares that it will return in 50 years and demands that it be provided with a definite set of morals it will then immediately act upon. This experiment's purpose is to spark discourse over how best to handle defining sets of ethics that computers may understand.

Some recent work attempts to reconstruct AI morality and control more broadly as a problem of mutual contestation between AI as a Foucauldian subjectivity on the one hand and humans or institutions on the other hand, all within a disciplinary apparatus. Certain desiderata need to be fulfilled: embodied self-care, embodied intentionality, imagination and reflexivity, which together would condition AI's emergence as an ethical subject capable of self-conduct.

In fiction

In science fiction, movies and novels have played with the idea of sentient robots and machines.

Neill Blomkamp's Chappie (2015) enacts a scenario of being able to transfer one's consciousness into a computer. Alex Garland's 2014 film Ex Machina follows an android with artificial intelligence undergoing a variation of the Turing Test, a test administered to a machine to see whether its behavior can be distinguished from that of a human. Films such as The Terminator (1984) and The Matrix (1999) incorporate the concept of machines turning on their human masters.

Asimov considered the issue in the 1950s in I, Robot. At the insistence of his editor John W. Campbell Jr., he proposed the Three Laws of Robotics to govern artificially intelligent systems. Much of his work was then spent testing his three laws' boundaries to see where they break down or create paradoxical or unanticipated behavior. His work suggests that no set of fixed laws can sufficiently anticipate all possible circumstances. Philip K. Dick's 1968 novel Do Androids Dream of Electric Sheep? explores what it means to be human. In his post-apocalyptic scenario, he questions whether empathy is an entirely human characteristic. The book is the basis for the 1982 science-fiction film Blade Runner.