Banning Autonomous Weapons: A Legal and Ethical Mandate

Abstract

ChatGPT launched in November 2022, triggering a global debate on the use of artificial intelligence (AI). A debate on AI-enabled lethal autonomous weapon systems (LAWS) has been underway far longer. Two sides have emerged: one in favor and one opposed to an international law ban on LAWS. This essay explains the position of advocates of a ban without attempting to persuade opponents. Supporters of a ban believe LAWS are already unlawful and immoral to use without the need of a new treaty or protocol. They nevertheless seek an express prohibition to educate and publicize the threats these weapons pose. Foremost among their concerns is the “black box” problem. Programmers cannot know what a computer operating a weapons system empowered with AI will “learn” from the algorithm they use. They cannot know at the time of deployment if the system will comply with the prohibition on the use of force or the human right to life that applies in both war and peace. Even if they could, mechanized killing affronts human dignity. Ban supporters have long known that “AI models are not safe and no one knows how to reliably make them safe” or morally acceptable in taking human life.

The ongoing debate over the law and ethics of lethal autonomous weapon systems (LAWS) reflects two very different perspectives. One sees the development of such weapons as an essential part of national security that depends on a strong military in possession of the latest weapons technology.¹ If legal or ethical norms impede staying ahead in the race for new weapons, those norms need to be reinterpreted or modified. The other perspective maintains that security depends first and foremost on robust respect for legal principles that are derived from fundamental moral principles. Such principles are not subject to reinterpretation or modification. The proper role for military force is defending the rule of law, not superseding it.²

The military superiority perspective has been shaped by the twentieth-century political theory of realism.³ Realism has been deeply influential and helps account for why major militaries spend far more on weapons development than green technologies or national and international governance institutions. It also helps explain why China, Russia, and the United States have made it a common cause to prevent a ban on LAWS.⁴ The other perspective dates to the emergence of law among the earliest human groups as an alternative to physical force in the ordering of society.⁵ This rule of law perspective has been shaped by the theory of natural law that combines legal and moral teaching and can be seen in the efforts by many states, technologists, civil society movements—such as the Campaign to Stop Killer Robots—and the Vatican to ban LAWS.⁶

The aim of the essay is not to attempt to bridge the gap between the realist and rule-of-law perspectives. The goal is to explain why those who hold the second perspective, that security depends on respect for legal principles, also support a ban on LAWS. The analysis proceeds in three parts. The first section briefly describes LAWS, emphasizing aspects of the technology that make fully autonomous robotic weapons inherently unlawful and unethical under prevailing legal and normative standards. The second section presents an overview of the standards, including why they are not subject to reinterpretation to justify the use of LAWS. The third section takes up and challenges two common arguments against a ban: (1) that LAWS can comply more closely with rules than humans and (2) that LAWS and their use are inevitable with or without a ban. It is better, per this latter argument, to find some legal justification for them or else the law will be dismissed as an obstacle to security. With the launch of ChatGPT in November 2022, however, awareness is spreading rapidly of the potential harm posed by artificial intelligence, or AI.⁷ LAWS rely on AI and pose among the greatest threats of all AI applications, creating weapons of potential mass destruction.

LAWS Defined

According to the U.S. Congressional Research Service, LAWS are “a special class of weapon systems that use sensor suites and computer algorithms to independently identify a target and . . . destroy the target without manual human control of the system.”⁸ In the United States, ongoing research by the Department of Defense and the defense industry on LAWS follows from the development of unmanned weapon systems known as drones. Air, land, and sea drones already have the capacity to attack with significant independence from human operators. Drones use sensors and can be programmed to strike particular targets without much input from a human being at a control station. LAWS go a critical step beyond drones, however, by being programmable to select targets unknown at the time the weapon is deployed. Also, unlike current drones, there is no need for a human operator to be associated with a weapon after deployment. LAWS can roam in search of targets without regard to time, place, or human oversight. Any restrictive parameters included in the original programming can be superseded by the computer-learning algorithm.

AI is the distinctive feature of LAWS. Weapon systems equipped with AI have the “ability to operate independently and engage targets without being programmed to specifically target an individual object or person. This includes the capability to react to a changing set of circumstances . . . The second, interrelated, aspect is the capability to make discretionary decisions.”⁹ Not only will LAWS make discretionary target selection decisions, the systems will be capable of deciding where and when to attack. The complexity of the computer-learning functions that produce such capacity means that the most advanced computer scientists cannot foresee what decisions computerized weapon systems will reach. Learning programs are a “black box.”¹⁰ The term “black box” refers to the theory of why it is impossible to predict how a computer-learning program will reach decisions, owing to the problem of overseeing massive data inputs.¹¹ There is “no symmetry between prediction and retrodiction . . . since an infinity of past histories may end up in the final state R(t).”¹² In other words, “Many of these algorithms are black boxes even to their creators” because “there is no straightforward way to map out the decision-making process of these complex networks of artificial neurons.”¹³

Learning programs dynamically generate their decisions based on input data, meaning that even the primary AI coders will lack the ability to understand its decision-making strategies. Elke Schwarz, writing for a blog of the International Committee of the Red Cross, warns, “AI might engage in calculations that are not intelligible even to programmers or engineers. For human decision makers to be able to retain agency over the morally relevant decisions made with AI, they would need a clear insight into the AI black box, to understand the data, its provenance and the logic of its algorithms.”¹⁴ To date, the best way to begin to understand how AI works is to observe results of AI decisions.¹⁵ In other words, current knowledge of AI requires trial and error. In the case of AI weapons, this means observing who the program has selected to be killed. Eventually, observing such decisions might result in knowledge of why selections are made. That knowledge is not available now.

Because LAWS are the next step after drones, it might be assumed that they will deploy missiles and bombs for use by the military on a legally defined battlefield only.¹⁶ The first weaponized drones carried two Hellfire missiles and were part of the U.S. military arsenal. It was not, however, the military that first used a weaponized drone. The CIA attempted to carry out an assassination using a drone far from any combat zone in 2000.¹⁷ Today, police, the military, and private citizens all possess drones. They are designed for air, sea, and land and a wide variety of munitions. Further, the vehicles themselves can become munitions, as seen with kamikaze drones. Nano-drones exist that can in principle carry out targeted assassinations, whether using their blades to slice a target to pieces or through other ordinance, such as mortars, grenades, and light machine guns. Presumably, any munition could be attached to them, including nuclear bombs and missiles. If a drone can carry these weapons, so, too, we must assume, can LAWS.

Weapons Law

The potential scenarios involving LAWS are myriad, which means the relevant law is far broader than the law of armed conflict. At least four interrelated legal categories apply: human rights law, law on resort to force (jus ad bellum), law on the conduct of force (jus in bello), and arms control law.

Under human rights law, authorized law enforcement agents may use lethal force to save lives under immediate threat.¹⁸ The only other rights to intentionally use lethal force are provided in the United Nations Charter principles on initial resort to force or in international humanitarian law (IHL), including the 1949 Geneva Conventions, on using lethal force during armed conflict. In any case of doubt as to whether a situation falls under human rights law, the UN Charter, or IHL, the higher protection of human rights law is presumed to apply.

The UN Charter regime on resort to force begins with Article 2(4), which generally prohibits the interstate use of force. The Charter's drafters intended “to state in the broadest terms an absolute all-inclusive prohibition.” They wanted “no loopholes.”¹⁹ The UN Security Council may authorize the use of armed force when other nonviolent means to address a threat to the peace, breach of the peace, or act of aggression prove inadequate.²⁰

Following Russia's full-scale invasion of Ukraine in 2022, however, it is unlikely that the Security Council again will authorize a major intervention for the foreseeable future. Authorizations have slowed owing to the poor results of peace enforcement missions. China and Russia remain highly critical of NATO's abuse of the authorization to use force in Libya in 2011 that resulted in the ongoing civil war. The minimal trust that existed to win that authorization has evaporated. Russia's decision to invade Ukraine in 2014 and 2022 has further entrenched divisions.²¹

Other than Security Council authorization, the only basis in the Charter to use force is provided by Article 51, the right of self-defense. The terms of Article 51 are highly restrictive. Resort to force is permitted “if an armed attack occurs” until the Security Council acts. The defending state must promptly report its actions. In addition, restrictions from general law beyond the Charter apply, the most important of which are the principles of attribution, necessity, and proportionality. The principle of attribution mandates that any use of force in self-defense must aim only at a state responsible for a significant armed attack on the defending state. Even then, a counterattack must be necessary to achieve legitimate defense and must be proportionate to the injury sustained. Defending states may request assistance in collective self-defense, but any such request must come from a government in effective control of the state. “Effective control” is the standard test in international law for identifying an entity that qualifies legally as a state's government.²²

If self-defense turns into armed conflict—intense exchange of armed fighting of some duration—international law permits the armed forces of a party to the conflict to intentionally target the armed forces of the adversary.²³ All targeting within armed conflict is governed by the four fundamental principles of IHL: civilian distinction, necessity, proportionality, and humanity. These and other in bello rules mean that certain weapons are unlawful to use, such as weapons that are indiscriminate and risk killing civilians in disproportionate numbers, as well as weapons that cause unnecessary suffering.

The United States was at the forefront of drafting the Charter principles and the IHL principles codified in the Geneva Conventions in the 1940s. In the wake of the catastrophe of World War II, states returned to first principles, to the immutable natural law norms prohibiting force and honoring human dignity; in particular, the right to life in war and peace. By the early 1960s, however, realism and related theories such as legal positivism influenced a shift toward the Cold War mentality that elevated military security over law and diplomacy. Nevertheless, for decades, the United States and Soviet Union manipulated facts rather than tampering with legal interpretations. The interventions in Hungary, Vietnam, Czechoslovakia, Afghanistan, and Grenada were falsely presented as justified on the basis of genuine government consent. With the end of the Cold War, the United States began using force in violation of the Charter, at first by saying little or nothing, then with newly invented legal claims.²⁴ To justify drone strikes beyond armed conflict zones, for example, the United States has cycled through four approaches: maintaining secrecy—refusing to acknowledge responsibility or to comment on covert operations; declaring a “global war on terrorism”; reinterpreting Article 51 to permit current attacks against future potential threats; and claiming that the United States has the right to attack when it deems a state is “unable or unwilling” to resolve a terrorist threat. The unable or unwilling argument may be the weakest of all in that it lacks all features of legality by depending completely on a subjective assessment by Washington in place of the objective evidence of the armed attack requirement per Article 51.²⁵

The attempts to justify drone attacks outside armed conflict zones are particularly noteworthy for their departure from the principles American leaders promoted through the early 1960s. Attempting to stretch the law to meet policy overlooks the fact that the prohibition on the use of force and the inherent dignity of all human beings are premised on enduring natural law precepts and substantive principles. The prohibition on force and the fundamental IHL principles are peremptory natural law norms not subject to diminution through reinterpretation, new treaties, or new rules of customary law.²⁶ Such norms endure regardless of technological breakthroughs.

LAWS Banned

The prohibition on the use of force, which aims at protecting the human right to life, supplies the normative basis for the worldwide movement to ban LAWS. Proponents of LAWS have responded to the movement with various arguments in addition to the realist mandate of staying ahead in the arms race. Two arguments are reviewed here. First, defenders of LAWS argue that computers will follow the law more closely than human beings are able to; and second, that, regardless of the law, autonomous weapons are the future. It is better, therefore, to adjust the law to meet the technology than for the law to be ignored.

With respect to LAWS being superior to human operators, the U.S. government has argued:

Emerging technologies in the area of LAWS . . . reduce the risk of civilian causalities and damage to civilian objects . . . [;]

Emerging technologies in the area of lethal autonomous weapons systems could be used to create entirely new capabilities that would increase the ability of States to reduce the risk of civilian casualties in applying force …

Rather than trying to stigmatize or ban such emerging technologies in the area of lethal autonomous weapon systems, States should encourage such innovation that furthers the objectives and purposes of the Convention [on Certain Conventional Weapons].²⁷

These points do not, however, engage the preliminary legal issue of the right to resort to force in the first place. They are concerned with battlefield use of LAWS once an armed conflict has been initiated. Battlefield use is the topic of discussion at the weapons review meetings of parties to the Convention on Certain Conventional Weapons (CCW) in Geneva.²⁸ The CCW ensures that weapons do not kill indiscriminately or cause unnecessary suffering. These are important concerns regarding LAWS, but the CCW does not deal with the essential principles restricting resort to force in the first place. If an autonomous weapons system has resorted to armed force in violation of the prohibition on the use of force, it is irrelevant that it complies with the principles governing discrimination and humanity. Because the learning function of LAWS leads to unknowable outcomes, predicting whether a robot will decide to resort to force in violation of international law cannot be known. Moreover, programming may incorporate flawed views on when resort to force is lawful. A human may program a particular target into a robot's code before launch and include appropriate information on when to use that force, but an influx of contingent and unknowable information may easily shift the robot away from its programmed directives. It is foreseeable that because LAWS are “complex, unpredictable, and extremely fast in their functioning, these systems would have the potential to make armed conflicts spiral rapidly out of control, leading to regional and global instability.”²⁹ Equally, they have the potential to start armed conflicts in violation of the law—a potentially greater cause of future global instability than exacerbating existing wars.

In addition to unlawful resort to force, AI weapons must be treated as though they could potentially kill indiscriminately. Not only do programmers not know who might be killed, they do not know how they will be killed. A programmer may specify members of ISIS as targets at the time of deploying the weapon, but the system may then “learn” to attack Iran's anti-government protestors. Many of the potential data inputs for ISIS apply equally to the Iranian government's characterization of protestors—Iran calls them “terrorists”; they are surveilled; detained without a fair trial; tortured; and executed. The point is that the results of learning programs are unpredictable, which means there is the potential for indiscriminate killing, which is a ground for banning weapons under the CCW. Arthur Holland Michel agrees that “the unrestricted employment of a completely unpredictable autonomous weapon system that behaves in entirely unintelligible ways would likely be regarded as universally injudicious and illegal.”³⁰

Even if the problem of unpredictability—the black box problem—can be solved, the missing human conscience is an insurmountable barrier to the acceptability of LAWS. IHL requires that military commanders oversee all battlefield operations or face criminal liability.³¹ The United States recognizes this but attempts to water it down to fit autonomous weapons. Department of Defense Directive 3000.09 “requires [that] all weapons systems, including LAWS . . . ‘allow commanders and operators to exercise appropriate levels of human judgment over the use of force.’”³² However, “‘human judgment over the use of force’ does not require manual human ‘control’ of the weapons system,” only “broader human involvement.”³³

“Broader human involvement” appears to fall short of the responsible commander's oversight duty. The standard would be met in the guidelines proposed by prominent computer scientist Noel Sharkey. He sets out parameters for “meaningful human control” of LAWS. They entail: (1) full contextual and situational awareness of a specific attack; (2) the ability to perceive unexpected change in circumstances; (3) retention of power to suspend or abort the attack; and (4) time for deliberation on the significance of the attack.³⁴ This standard not only meets the legal standard under the law of armed conflict, it also answers the concern of many commentators that LAWS cannot be held responsible for violations of IHL or human rights law. Weapons, unlike human beings, cannot be put on trial for criminal actions.³⁵ Human involvement at the level Sharkey details amounts to a ban on autonomous systems.

In addition to the oversight problem, the lack of a human conscience also raises a novel human rights concern: mechanized killing. Today, the standard way cattle are processed for human consumption in the United States begins with their slaughter by robots. A human conscience is not focused on the killing once the decision has been taken that a group of animals will be killed. In the future, human beings could also be subjected to death by machine. The computer will decide who dies and carry out the actual killing. The Vatican's Archbishop Silvano Tomasi emphasizes that “decisions over life and death inherently call for human qualities, such as compassion and insight.”³⁶ Although “imperfect human beings may not perfectly apply such qualities in the heat of war, these qualities are neither replaceable nor programmable.”³⁷ Machines can never, by definition, possess these human qualities. Out of respect for human dignity, a person needs to be present when the decision to intentionally take the life of a being possessing dignity is made. Regardless of how sophisticated computers become in exemplifying compassion, empathy, altruism, emotion, or other qualities, the machine will be mimicking, not experiencing them as only a human can. Even the most emotionally or psychologically impaired human being remains human, imbued with dignity. He or she may not make morally appropriate decisions to kill, but the decisions made by a person with dignity will be more respectful of the victim's own human dignity than any mechanized form of slaughter.

All of the principles emphasized here—the prohibition on force, the right to life, the protections in combat—cannot be changed to suit proponents of LAWS. Similar arguments have been tried since the end of the Cold War—arguments that aim at removing existing legal restrictions on both resort to force and on the use of new weapons. Natural law norms, however, endure. They are not subject to elimination through new arguments or reinterpretations of the law. While they cannot diminish natural law norms, such arguments can lead to confusion about what the law genuinely requires, and confusion plays a role in noncompliance. Russia's war against Ukraine demonstrates the cost of weakening the restraining norms on the use and conduct of armed force. Even if Russia, China, and the United States continue to block a ban at the CCW, supporters of a ban will be right to persist in the effort at the CCW and in other forums. The campaign for a ban reminds humanity to “be mindful that these architectures might affect our ethical thinking and acting in ways that move ever-further from a humanist framework and etch ever closer to the purely cost-calculative logic of machines within which our moral agency inevitably atrophies.”³⁸

The fact that the principles prohibiting LAWS are enduring also responds to the argument that LAWS are inevitable, and that a ban is therefore pointless. The most that can be hoped for, so this argument goes, is a set of rules that place restrictions on the use and possession of LAWS. Better to have some laws in place than a ban that will render the law a shibboleth.³⁹ The argument also typically incorporates the view that rules to be applied to the use of LAWS will need to be created for this specific purpose. The technology is new, so the assertion is that the rules will need to be tailor made to fit.

Both arguments—the “ban as pointless” and the need for tailor-made rules—are as old as weapons research and development. In the past, they helped push states to accept weapons regardless of the law, for example, the use of newly invented airplanes to bomb undefended cities and the use of unmanned systems to launch missile strikes in the absence of an armed attack (as required under UN Charter Article 51) and far from armed conflict hostilities. If, however, a ban is put in place and states deploy LAWS despite it, they will be lawbreakers. The appearance of new technology provides no defense. Even without a ban, the principles set out here are fully sufficient to make the use of LAWS unlawful.⁴⁰ The technology may be new, but its purpose is ancient. Weapons from stones to LAWS are meant to kill. Law on the right to life manifested in the prohibition on the use of force and the restrictions on killing in combat will always be applicable, regardless of what weapons research and development may deliver next.

Conclusion

An express ban will help promote the international legal principles that prohibit killing by fully autonomous robotic weapons. While not a necessity—because natural law norms apply now to prohibit LAWS—a ban can educate technologists and those acquiring their inventions as the dark facts of AI become known. We already know “that current state-of-the-art AI models are not safe and no one knows how to reliably make them safe.”⁴¹ The majority of states, as well as prominent human rights organizations, the Vatican,⁴² and many—perhaps most—AI researchers, including the man known as the father of AI, Geoffrey Hinton, stand on the side of banning AI weapons.⁴³ Supporters of a ban have ancient, enduring law on their side. What is needed now are more international lawyers fully committed to teaching, writing, and litigating about the actual law at issue, not the law some governments might prefer. Greater knowledge of this law is needed, as well as insights into how to improve compliance with it, and ethicists and international relations scholars can teach that security lies in the consensus legal norms of the world community that draw on ancient, transcendent moral insights. Governments still mired in an arms race mentality are on the wrong side of history.