Merging man and machine: A legal assessment of brain–computer interfaces in armed conflict

Abstract

Imagine a future where man and machine become one on the battlefield, where soldiers direct weapon systems through a neural implant. Research advances on brain–computer interfaces (BCIs) may eventually allow such control of arms at the speed of thought. This article sketches two modes of BCI-controlled weapon systems. In Mode A (active BCI), the soldier opens fire by actively imagining that he is pushing a button with his hand. By contrast, Mode B (reactive BCI) captures neural signals evoked instantly after having spotted a target, before the operator becomes consciously aware of it. If he deems the target lawful, the brain signal is translated into a command to fire. Arguing that such man–machine collaboration transforms the operating soldier into a means of warfare, this article conducts a weapon review in line with Article 36 of Additional Protocol I (AP I) to answer the question of whether BCIs can be lawfully used to control weapons in international armed conflict. Consequently, the two set-ups are reviewed on their compliance with the customary targeting principles of international humanitarian law. Since Mode B casts doubt on the amount of control that the soldier retains over his targeting decision, the concept of meaningful human control is transposed from the debate on lethal autonomous weapon systems and applied to BCIs. It is found that reactive BCIs cannot be meaningfully controlled and thus violate the principles of distinction and proportionality. Hence, reactive BCIs are unlawful under Article 36 of AP I.

Introduction

The idea of merging man and machine is no longer confined to the realms of science fiction. Research advances on brain–computer interfaces (BCIs) have already made it possible for a paralyzed man to drive a Formula One car using only his thoughts,¹ and will likely engender a paradigm shift in how humans interact with technological gadgets.²

BCIs establish a connection between the brain, a computer able to decode its neural signals, and a physical device controlled thereby. Contemporary applications focus mainly on the mental command of prostheses, but such man–machine collaborations are projected to dominate military systems by the early 2030s.³ Most notably, the United States’ Defense Advanced Research Projects Agency (DARPA) has declared an interest in their development.⁴ In fact, the US is currently considering export controls on BCIs, fearing an “impact on U.S. national security”.⁵ The technology could serve, inter alia, the purpose of remote weapon control,⁶ effectively merging man and machine to combine their strengths on the battlefield.⁷ One such endeavour has already fathered a US patent,⁸ while the governments of China, Russia and France have equally been reported to be researching and developing BCIs.⁹

While scholars of various disciplines deem BCI-controlled weapons feasible,¹⁰ few legal academics have contemplated their application in warfare, and none have differentiated between the specific BCI types or conducted a comprehensive analysis of their legality. Considering the technology’s encroachment on the military, this article sets out to fill the gap and determine whether soldiers can lawfully use BCIs to control weapons under international humanitarian law (IHL). For this purpose, a review on legality under Article 36 of Additional Protocol I (AP I) will be the core of the assessment, which is hence restricted to international armed conflict.

The article begins with a brief overview of the types and functioning of BCIs. The article then outlines two set-ups for BCI-administered weapon control. In the first, Mode A, the operating soldier consciously identifies and engages targets, whereas in the second, Mode B, the commander selects targets subconsciously. In either scenario, it will be shown that the associated soldier transforms into a means of warfare, as part of a weapon system. This novel status brings the entire man–machine collaboration within the scope of Article 36 of AP I. Subsequently, the two BCIs are evaluated on their adherence to the customary targeting principles of IHL. In particular Mode B, which impedes the exercise of conscious control over targeting decisions, challenges the operator’s ability to ensure compliance with distinction and proportionality. Hence, the concept of meaningful human control (MHC) is borrowed from the debate on lethal autonomous weapon systems to discuss the degree of oversight required for the BCI’s lawfulness. To this end, considerations of precaution and military necessity are solicited, before settling on an “on-the-loop standard” of MHC. For Mode B’s analysis, a model is presented which aims to ascertain whether the subconscious BCI operator can predictably ensure that lawful targeting decisions are made. These insights will finally allow a conclusion on the legality of either form of BCI.

Technological framework: Weapon control through BCIs

Overview and categorization of BCIs

Before discussing the legal merits, BCI technology – in its two methods of controlling weapons, as outlined above – needs to be briefly sketched. The human brain operates on neural signals, spikes of electrical activity, which can be captured by a neural implant,¹¹ interpreted by a computer and translated into commands that control an external device.¹² This entire set-up of person and machine is referred to as the brain–computer interface.

BCIs fall into two categories: active and reactive. Active BCIs require the user to intentionally focus on a command, so as to generate “spontaneous signals” that are subsequently deciphered by the computer – for example, as the desire to move one’s hand to the right (“conscious command”).¹³ In contrast, reactive BCIs pick up “evoked signals” which are generated subconsciously – preceding the user’s awareness – in reaction to external stimuli (“subconscious command”).¹⁴ Commonly, these signals are elicited by rapid serial visual presentation (RSVP), where the individual is shown images at 200- to 500-millisecond intervals¹⁵ with the task of spotting a predefined object therein.¹⁶ The recognition of the object evokes a neural response in the user’s brain that the BCI then captures and may use to launch an operational command.¹⁷

Military BCIs: Two modes of weapon control

Both active and reactive BCIs can be used to control weapons. This section of the article presents two different set-ups that can be conceived towards this end, informed partially by feasibility studies¹⁸ and by discussions among academics of various disciplines. The two modes shall serve as non-exhaustive, illustrative examples of conceivable BCI-directed weapon control.

Mode A: Conscious command through active BCI

In the first scenario, Mode A,¹⁹ an unmanned ground vehicle (UGV), mounted with a machine gun, is remotely controlled via BCI. The vehicle comes with a 360-degree camera, which the operating soldier can move around to scan the environment for lawful targets.²⁰ After he identifies a target, the operator gives an explicit command to open fire by actively imagining that he is pushing a button with his right hand.²¹ The computer recognizes the neural signals generated by the combatant mentally “pressing the button” and authorizes the UGV to strike. While Mode A comes with the benefit of the warfighter’s immersion despite his remoteness from the battlefield, it can only accelerate the target engagement to some extent, since he must still make the conscious effort to launch an attack.²²

Mode B: Subconscious command through reactive BCI

Mode B works differently.²³ This set-up takes advantage of the fact that about 250 to 500 milliseconds after visual perception, the soldier’s brain will have already, subconsciously, reacted to the target stimulus²⁴ – much faster than any computer could, and faster than he is consciously aware of.²⁵ To make use of these subconsciously evoked signals, the warfighter is presented with images of the UGV’s surroundings via RSVP.²⁶ His task remains to identify and shoot lawful targets. Recognizing the neural signals evoked whenever he subconsciously spots such a target, the BCI will instantly instruct the machine gun to fire. In this way, the commander’s consciousness is overstepped in order to attack much more quickly. Contrary to Mode A, where the operator must willingly decide to strike, Mode B does not require his decision but only his brain’s instantaneous reaction. Thus, every delay between target perception and engagement is eliminated. Nonetheless, the soldier remains capable of withdrawing from the BCI control at any point.

While the two modes diverge in their manner of targeting, either BCI creates a secure link between the operator and the UGV, which merits discussion as to whether the operator’s legal status is altered. Consequently, it is necessary to evaluate whether the commanding soldier falls under the scope of Article 36 of AP I as a new weapon or means of warfare.

Legal status of the BCI-controlling soldier

The soldier as a weapon

First, it shall be considered whether the soldier could be deemed to have turned into a weapon. In the absence of consensus, the definition of a weapon as “an offensive capability that can be applied to a military object or enemy combatant” is widely shared.²⁷ This wording indicates that a weapon is an instrument, a tool, something that “is applied”. Article 36 of AP I substantiates this by referring to a weapon’s employment as “its” employment. Chircop and Liivoja, explicitly treating the question of whether BCI-enhanced warfighters can be considered weapons, thus propose the definition of “an instrument through which an offensive capability manifests and by which it can be applied”.²⁸

Adopting this definition henceforth, it seems questionable to regard the BCI operator as a mere instrument. The soldier’s status boils down to whether his employment of the technology takes him beyond the “tipping point” at which he is transformed to the extent of becoming the weapon instead of simply using the weapon. Considering, by analogy, a gun fused to a combatant’s hand, it seems obvious that the gun remains a weapon and the soldier retains his human status.²⁹ The situation is less clear for BCIs. Mehlman, Lin and Abney have argued that there is only one way to avoid this slippery slope: to consider soldiers – enhanced or not – as weapons to begin with.³⁰ That proposal has earned them firm criticism,³¹as it would degrade the human soldier to a mere instrument of combat,³² running contrary to the humanitarian spirit of IHL.³³

These reflections point to an answer on the warfighter’s status in Mode A. The human operator can only be considered a weapon if the BCI fully objectifies him,³⁴ but one can hardly argue that this tipping point is reached in Mode A, where the commander preserves full freedom of action – he observes the environment, spots potential targets, determines their status and decides to engage. Just as the soldier is not the gun fused to his hand, he is not the instrument; rather, he uses the BCI to manifest the machine gun’s – the instrument’s – offensive capability.

The reactive BCI, meanwhile, turns the soldier’s subconscious neural response into an instruction to attack. His brain, in this set-up, functions like a processing machine, similar to a computer.³⁵ In Kantian terms, one could doubt that the warfighter was still treated as an end in himself in lieu of a means to an end. Despite his brain’s function, the operator is not fully objectified³⁶ – not fully deprived of his freedom of action. The soldier remains physically conscious during operations, competent to terminate the BCI control at any time. Objects do not possess such agency. Additionally, the operator conducts an assessment that draws from the pool of his human experience: from his intellectual understanding of lawful targeting and his knowledge of combat. Such a task cannot be performed by things. While it is true that the operator deliberately relinquishes conscious command by lending his brain to the target engagement process, the human appropriates the machine for this purpose, not vice versa. The operator uses the weapon (the machine gun); he is not used by it.

The soldier as a means of warfare

Definition of means of warfare and weapon system

Since the soldier is not turned into a weapon, he could alternatively be subsumed under means of warfare. The text of AP I grants only a vague starting point for a definition. By virtue of its construction, Article 35 of AP I suggests that means of warfare refers to “weapons, projectiles and materials”. In fact, while means incorporate weapons, it is sensible to perceive means more broadly in order to avoid redundancy in the text of Article 36.³⁷ Accordingly, means of warfare can be defined as “weapon systems or platforms employed for the purposes of attack”, including “associated equipment used directly to deliver force during hostilities”.³⁸ For the same reasons as mentioned above, the warfighter does not qualify as “associated equipment”; however, he could be part of a weapon system.

The term “weapon system” is not found explicitly in AP I, but is commonly defined as

the weapon itself and those components required for its operation …[,] limited to those components or technologies having a direct injury or damaging effect on people or property … and all other devices and technologies that are physically destructive or injury producing.³⁹

This notion, though, seems under-inclusive as it only relates to weapon-associated inputs delivering harm (e.g. munitions or projectiles). It refers exclusively to the physical impact of the force delivered, not to the source of how force is administered. As such, one would exclude all new manners of weapon control from the duty of review as long as they relied on established physical components executing the damage. Consequently, it would – for the sake of Article 36 of AP I – make no difference if a soldier were to use a hand-held machine gun to aim and fire at targets or if the machine gun were to be mounted on top of a vehicle, where it spins uncontrollably, while the vehicle is blindly remote-controlled. The vehicle itself, the manner of how the machine gun is attached to it and the remote control do not themselves produce injuries – the only component directly delivering harm remains the exact same machine gun bullets. It should be clear that the two scenarios ought to be evaluated very differently under the auspices of distinction and proportionality. The term “weapon system” was crafted precisely to capture additional factors of how a weapon is put to use and to scrutinize the entire set-up of how force within such a system – during normal or expected use⁴⁰ – is delivered.⁴¹ Such understanding reflects the term “means of warfare” as referring to a process, a technique, a mechanism to administer harm – including both the harm-delivering process and the actual harm.

In this spirit, Schmitt defines a weapon system as “a combination of one or more weapons with all related equipment, materials, services, personnel, and means of delivery and deployment … required for self-sufficiency”.⁴² What this definition has in common with the one cited above is that they both conceptualize a weapon system as a weapon needing additional input to function self-sufficiently – i.e., to produce damaging effects. Schmitt’s understanding, however, considers all of the elements that are required to operate a given weapon as part of the weapon system. This enables us to review every part needed to deliver harm (even if it does not do so eo ipso) holistically as one means of warfare under Article 36 of AP I.

The operator as part of a weapon system

While such inputs have traditionally been technological components, the novel man–machine collaboration presents us with a human inseparably connected to the damaging effect of the UGV’s machine gun. Only by him lending the powers of his brain via the neural implant, and thereby giving the mental engagement command, can the BCI-controlled UGV fulfil its purpose. To compare, a conventional weapon system is self-sufficient without human input in terms of how it mechanically functions and processes information; all that is needed from the operator is for him to launch the engagement signal (by pressing the button). The BCI, au contraire, relies on the operator to perform duties of transmitting information. Dissimilar to a physical button pressed for target engagement, the functioning of the weapon system relies on the trigger in the operator’s mind. One could argue that its functioning does not actually differ from a conventional weapon system, at least for Mode A: in both cases, the soldier must willingly decide to fire and use his body to give that command. Possibly, this detaches him from the weapon system’s structure and makes him a user instead of part of a weapon system, just as if he were to fire a regular rifle. However, the difference is precisely that the rifle functions self-sufficiently once the mechanical process of striking a target is initiated – i.e., when the soldier pulls the trigger. The BCI, on the other hand, only works because the operator invites the targeting process into his body. For this purpose, the soldier’s brain occupies a function equal to that of a mechanical component in a conventional weapon system, being either the button to engage (Mode A) or both the button and the processing machine⁴³ (Mode B). Without him, the offensive capability could not manifest,⁴⁴ placing the BCI operator within the system of how the delivery of force is conceived and administered.⁴⁵ Differently put, if a rifle is taken by the enemy, it can easily be used, by anybody, to fire back at its owner – but when the BCI operator is captured, the man–machine collaboration cannot be used in the same fashion, since its workings depend on the operator’s mental command. With the BCI, as opposed to the rifle, the soldier is – via his neural implant – fused to the delivery of force, without which the weapon system is not self-sufficient. Not just anyone can launch the offensive power; it must be the operator with whom the targeting system maintains a neural connection. The soldier therefore qualifies as a weapon system and – as such – as a means of warfare.⁴⁶

The soldier’s combatant status as part of a weapon system

Commander and BCI cannot be viewed separately, since the neural implant creates a permanent bridge between the former and the armed vehicle. However, this does not mean that – by virtue of attaining the status of a means as warfare – the BCI-controlling soldier ceases to be a combatant or, generally speaking, loses his respective status. Though operator and BCI are incapable of delivering harm independently of each other,⁴⁷ the merger between them does not undermine the formerly held status of either the UGV and its gun or the soldier.

While the machine gun remains a weapon, the operator’s status could only dissipate if the set-up reduced him to a weapon – a mere instrument.⁴⁸ This is not the case, as explained above. In contrast, a weapon system is comprised of multiple, separable parts. It is a weapon plus delimited (human) components connected to its employment or delivery of force. Thus, if the warfighter does not turn into a weapon himself, precisely because he maintains the freedom of action that is preconditional to his status as combatant, one cannot argue that the BCI strips him of his combatant privilege. Therefore, the operator does not lose his status, but takes on the dual role of both combatant and means of warfare.

With regard to the operator’s status, then, it can be concluded that the freedom of action which the soldier preserves in both modes prevents him from becoming an objectified instrument – a weapon. Either mode turns the operator into an indispensable part of a weapon system (comprising him, his neural implant, the computer, the UGV and the machine gun), and as such, he takes on a dual role of combatant and means of warfare. Article 36 of AP I is therefore the relevant provision for assessing the lawfulness of the entire weapon system – the active or reactive BCI. The soldier’s dual status within remains relevant and builds the foundation for the following assessment.

Lawfulness of BCI-controlled weapon systems: Article 36 review

Article 36 of AP I obliges the Contracting Parties to review new means of warfare. Such an assessment arguably constitutes customary international law,⁴⁹ serving as a preliminary check of a weapon system’s lawfulness per se – independently of the operational context.⁵⁰ If one or both modes of BCI violated the provisions of the Protocol,⁵¹ their use would be illegal ab initio.⁵² For this purpose, Modes A and B ought to be evaluated in line with the “normal or expected” circumstances of employment,⁵³ each on its own merits. Following the International Committee of the Red Cross’s (ICRC) Guide to the Legal Review of New Weapons, Means and Methods of Warfare (ICRC Review Guide),⁵⁴ the BCI must a priori be capable of applying targeting law – i.e., it must be compatible with IHL’s targeting principles, all of which are customary.⁵⁵

Targeting principles

Since the prohibition against causing superfluous injury or unnecessary suffering (AP I Article 35(2), reaffirming the 1907 Hague Convention IV Articles 22 and 23(e)) is not affected in the scenarios under discussion here (assuming the BCI-controlled machine gun shoots regular bullets), only the principles of distinction and proportionality must be scrutinized.

Distinction

The principle of distinction – enshrined in Article 48 of AP I in its elementary form – prohibits targeting civilian persons (AP I Articles 51(2) and 51(6)) or objects (AP I Article 52(1)). Attacks shall not be indiscriminate (AP I Article 51(4)) and must be directed solely against military objectives (AP I Article 52(2)). The machine gun is not an indiscriminate weapon; during normal and expected use, it is directed against specific persons. Hence, it allows the attacker to only strike lawful targets (AP I Article 51(4)(a)–(b)), thereby limiting its impact to military objectives (AP I Article 51(4)(c)). In the case of BCIs, the only questionable element is whether the weapon’s mental control permits its operator to exercise that distinction.

Indeed, Mode A’s conscious operator can instruct the UGV to carry out lawful attacks, but whether his counterpart in Mode B can do so equally is more obscure. The soldier must not merely identify regular combatants (AP I Article 43(2), 1907 Hague Convention IV Article 3, Geneva Convention III Article 4(A)(1)) as lawful targets, but must also distinguish between e.g. irregular armed forces (AP I Article 43(3), Geneva Convention III Article 4(A)(2)), civilians directly participating in hostilities (AP I Article 51(3), Geneva Convention IV Article 5) and civilians not qualifying as such (AP I Article 50(1), Article 3 common to the four Geneva Conventions). Prior RSVP studies show that the participants’ assessments can be erroneous,⁵⁶ and even when deciding consciously, warfighters may not always grasp such battlefield complexities when making quick decisions.⁵⁷ Sharkey gives the example of “an ununiformed man firing a rifle in the vicinity of an army platoon [who] may be deemed to be a hostile target … [even though] he had actually just killed a wolf that had taken one of his goats”.⁵⁸ Nevertheless, the active BCI operator can hesitate, reflect and observe his surroundings before firing. Thus, he can discriminate as well as any combatant may during combat. In Mode B, however, the neural signal evoked once the warfighter views the gunman could instantly launch a command to strike that gunman. Pursuant to the soldier’s immediate subconscious response, he cannot consciously foresee or control whether an engagement command will be given in a certain moment. This casts serious doubt on the operator’s ability to adhere to the principle of distinction.

Proportionality

The proportionality principle precludes attacks which may be expected to cause incidental civilian harm that is excessive in relation to the concrete and direct military advantage anticipated (AP I Articles 51(5)(b) and 57(2)(a)(iii)). Given the principle’s context-dependent nature, some academics oppose to it forming part of the weapon review procedure.⁵⁹ However, as endorsed by others,⁶⁰ the ICRC has included the notion in its Review Guide, stating that the characteristics and foreseeable effects of the weapon system must, per se, allow for its proportionate use.⁶¹

Mode A does not preclude the operator from respecting the principle of proportionality, since the conscious mental targeting renders him no less competent to balance “the foreseeable extent of incidental or collateral civilian casualties or damage” and “the relative importance of the military objective as a target”.⁶² Once again, however, the reactive BCI – Mode B – is contentious in this regard. Noll categorically rules out the idea that a subconscious combatant could comply with the principle of proportionality.⁶³ He advances that a proportionality assessment requires the combatant to enter into an inner dialectic, use language and resort to his reason;⁶⁴ otherwise, “IHL cannot be applied”.⁶⁵

In fact, the operator’s subconscious response does not seem to reflect a rational judgement grounded on conflicting normative factors. On the other hand, soldiers in the heat of combat do not rationally ponder a targeting decision either; they do not apply the law through verbalization and discussion,⁶⁶ as demanded by Noll. After all, proportionality comes with a “fairly broad margin of judgment”⁶⁷ and is foremost “a question of common sense and good faith”⁶⁸ – the BCI commander should merely avoid attacks that can be expected to excessively tilt the balance towards military needs (AP I Article 57(2)(b)).⁶⁹ Such an estimation may be possible to make subconsciously, but even if it is, the soldier cannot guide, predict or veto the result. Any conclusion on his capacity to comply with proportionality will be contingent on whether one can predict a proportionate outcome.

Accordingly, inasmuch as Mode A’s deliberate targeting does not differ from conventional remote systems,⁷⁰ it can obey the customary targeting principles. Hence, the active BCI is lawful under Article 36 of AP I. We are left questioning Mode B’s legality on the grounds of distinction and proportionality. By virtue of his subconscious target assessment, the operator does not control the “moment of command” – the moment the computer transmits the engagement decision to the UGV. Thus, he can neither exercise reason with regard to, nor predict or veto, his targeting choice. Without a further understanding of the nature of the control that he must exert over his subconscious, any assertion on the lawfulness of the targeting outcome would be guesswork. The question emerges as to whether control in general, and control over the moment of command in particular, bears legal significance under Article 36 of AP I and therefore constitutes a ground for challenging Mode B’s lawfulness.

Meaningful human control

Legal value: Transposition from the LAWS debate

The legal merit of control has been subject to the vivid discussion on lethal autonomous weapon systems (LAWS),⁷¹ from which the concept of meaningful human control emerged. While its exact scope is intensely debated and will be examined further below, generally speaking, MHC is construed as the minimum threshold of human oversight on target selection and engagement. Human intervention is not a sufficient condition thereto; the intervention must be meaningful.⁷² In both LAWS and BCIs, some degree of human input is still present – at least during the development and engineering phase. However, both technologies spark the question of whether this reduced amount of human agency – compared to conventional weapon systems – is problematic. Mode B illustrates this point: despite human presence throughout the entire targeting process, the soldier lacks control over the moment of command. The reactive BCI’s mode of operation hereby shares the legal uncertainties of LAWS (i.e., whether it can apply targeting law). As a first step in addressing this issue, the merits of the MHC standard and how it relates to the reactive BCI shall be considered.

The notion of MHC cannot be found in the text of AP I; accordingly, it does not constitute a separate criterion for a weapon system’s lawfulness.⁷³ Rather, MHC over a means of warfare must be implicitly given in order for the means to respect the targeting principles, as human oversight lies at the core of a weapon system’s capacity to safeguard distinction and proportionality.⁷⁴ The active BCI reveals this aspect: because the operator makes conscious decisions, he is presumed to be capable of exercising distinction and proportionality. Conscious decision-making is hereby indicative of human agency – i.e., the capacity to ensure, through a volitional act, that one’s intentions manifest in the real world.⁷⁵ In Mode A, the conscious warfighter can ensure that his intention (to engage lawful targets) manifests on the battlefield (by him actively instructing the UGV to engage a lawful target).

Along these lines, MHC acts as a “fail-safe” to attest the meaningful presence of human agency where control is not exercised at the level of operational command.⁷⁶ Thus, the concept can be used as shorthand for a weapon system’s general capacity to apply targeting law – i.e., for our purposes, legality under Article 36 of AP I.⁷⁷ In Mode B, the soldier lacks control over the moment of command. At stake is hence the question of whether the amount of agency he retains over the engagement decision can ensure that his intention – to comply with the principles of distinction and proportionality – manifests. In other words, it is uncertain whether the operator’s subconscious control can satisfy the MHC standard.

Based on the similarity of control issues in LAWS and Mode B BCIs, it seems viable to transpose the concept of MHC to such BCIs.⁷⁸ However, some caution should be advised: within the LAWS domain, MHC relates to human control over a weapon system (such as a UGV), but this does not capture our subject of review. What is questionable in Mode B is not the control exerted over the weapon mounted on the UGV, but the control, if any, that the soldier wields over his subconscious. In light of the operator’s dual status as both combatant and means of warfare (see above), he retains the responsibilities of a combatant – to ensure that the principles of warfare extend to his targeting decisions by controlling the trigger in his mind.

For the purposes of this review, MHC builds the bridge between the human agentic capacity to ensure application of the law and the law’s actual application. As such, it acts as shorthand for the capacity to ensure lawfulness, in the absence of control at the operational stage. Having established its legal value for the Article 36 procedure, we now need to evaluate what MHC amounts to. Once the necessary degree of control – the minimum threshold for the warfighter’s conscious oversight – is identified, Mode B can be subsumed thereunder. Doing so will establish whether the lack of control over the moment of command prevents Mode B’s lawfulness.

Content of MHC: Out-of-the-loop versus on-the-loop standard

There is no consensus on the content of MHC. Human control can take different forms and include different factors at different stages of development, operations and targeting. Scholarly opinions on what level is regarded as “meaningful” fall into three categories: the standards of “in the loop”, “on the loop” and “out of the loop”. Respectively, these describe situations in which the operator either gives the engagement command (in the loop), only supervises the engagement command (on the loop), or does not need to be implicated in the engagement command at all (out of the loop).⁷⁹ Applying the trichotomy by analogy,⁸⁰ the appropriate standard for Mode B will now be examined.

If one were to adopt an out-of-the-loop standard, the human control during a weapon system’s design could be sufficient to ensure its compliance with the targeting principles.⁸¹ In a reactive BCI, it is the operator’s presence that brings human agency to the system. Therefore, the system’s actual performance, as determined through testing, poses the sole relevant criterion for legality under Article 36 of AP I.⁸²

Such testing would be realized in a simulation, where the BCI-enhanced soldier subconsciously identifies targets and afterwards reviews the situation in a conscious state of mind.⁸³ If his subconscious responses aligned with his posterior conscious judgement in every instance (or most instances), MHC over the target assessment could be attested. Following this argument, the operator could rule out that he was ever (or frequently) compelled to correct the instruction transmitted to the UGV. Hence, he would not need control over the moment of command, since he could a priori ensure that the attack satisfied the principles of distinction and proportionality, based on the track record of his subconscious.

However, the on-the-loop-standard, to which the majority of scholars subscibe, opposes this approach. In order to be on the loop, it is demanded that the operator (at least) oversees the targeting decision, with the possibility of intervening;⁸⁴ hence, the BCI commander – in his function as a conscious combatant – would need to similarly supervise his subconscious. Such supervision is deemed necessary to ensure a predictable outcome.⁸⁵ Invoking predictability to mark the minimum threshold of human agency is sensible. If the BCI-controlling soldier wants to act lawfully (by applying distinction and proportionality in line with the rules regulating the conduct of hostilities), he must be able to ensure that the UGV only engages lawful targets. Otherwise, the argument goes, he cannot meaningfully control his means of warfare. MHC is hereby the idea that intention X leads – with reasonable certainty – to outcome Y. If the operator has MHC and harbours the intention to engage lawful targets, he predictably engages lawful targets.⁸⁶ In other words, to have MHC in a reactive BCI requires the soldier’s subconscious targeting to be predictably lawful.

The ICRC Review Guide reflects this rationale by referring to the importance of testing a means of warfare on the “reliability of the targeting mechanism”⁸⁷ and whether its “foreseeable effects are capable of being limited to the target [distinction] or being controlled in time or space [proportionality]”.⁸⁸ This suggests that a degree of foresight of the weapon system’s ramifications must be given for its lawfulness.⁸⁹ Therefore, the predictability of a weapon system to comply with the targeting principles can be seen as an overarching consideration of the Article 36 procedure.⁹⁰

Having outlined the rationale of predictability and how it underpins MHC, it must now be assessed against the out-of-the-loop standard. After all, one could argue that the BCI operator, by means of testing, can predict – with reasonable certainty – discriminate and proportionate outcomes. As stated, there is the possibility of failure. The out-of-the-loop standard is criticized precisely because one cannot rule out unlawful results in all situations arising during normal and expected use⁹¹ – namely, in situations that would have been decided differently (lawfully) under the soldier’s conscious control over the moment of command.

The risk of failing to adhere to the targeting principles may frustrate a reasonable level of predictability in the context of Mode B’s operation. IHL’s fundamental balancing act between military necessity and humanity shall guide the following analysis. In fact, the requirement to ensure a predictable outcome can be seen as a child of the precautionary principle,⁹² as reflected in its wording: a party is to refrain from attacks expected to cause excessive civilian death or damage (AP I Article 57(2)(a)(iii)). This expectation implies an element of uncertainty, but such uncertainty – the risk of incidental civilian harm – ought to be avoided, or in any event minimized, inter alia by taking all feasible precautions (AP I Article 57(2)(a)(ii)).

The duty to take feasible precautions (AP I Article 57) is not confined to the operational stage; it is not only a matter of the law of targeting. Rather, it informs the weapon law assessment under Article 36 of AP I for the same reasons as the principle of proportionality.⁹³ Addressing the criticism of establishing predictability by testing, the out-of-the-loop standard needs to be confronted with the obligation to take constant care to spare the civilian population in the conduct of military operations (AP I Article 57(1)).⁹⁴ The norm does not refer to attacks (defined in AP I Article 49(1)) – unlike the ensuing paragraphs. However, military operations are wider than attacks, while encompassing them, and hence refer to situations in and outside of active combat.⁹⁵ Moreover, constant care denotes a continuous, ubiquitous effort, both before and after the outbreak of hostilities.⁹⁶ It cannot be limited to the pre-deployment stage – i.e., by merely testing a weapon system once and, afterwards, neglecting all feasible measures to verify that only lawful targets are struck (AP I Article 57(2)(a)(i)). The notion of “feasible” as used in Article 57 of AP I signifies “those precautions which are practicable or practically possible taking into account all circumstances ruling at the time, including humanitarian and military considerations”.⁹⁷

On this point, there is a radical difference between reactive BCIs and LAWS. One may argue that an autonomous weapon system could outperform humans and better comply with the duty to take feasible precautions. The same cannot be said for Mode B BCIs. It is impossible that the reactive BCI better adheres to the targeting principles than its active counterpart. At most, the soldier in Mode B performs equally as well as the one in Mode A. Hence, to spare the civilian population from (the risk of) harm, the feasible measure is rather obvious: letting the same human operator decide consciously. If one were to employ a reactive BCI regardless (being familiar with the system’s lack of control over the moment of command), the precautionary principle would be violated – that is, unless the military necessity of using Mode B’s ultra-rapid targeting mechanism rendered alternative means not practicable or practically possible.

Ultimately, the military advantage must be examined in line with the “circumstances ruling at the time”⁹⁸ – meaning the circumstances in a future where BCI technology has materialized. In anticipation, one can consider the argument that the increased speed of battlefield operations prescribes a need to “tilt the balance between necessity and humanity”⁹⁹ in order for humans to keep pace.¹⁰⁰ Under this view, a military disadvantage could arise from not being able to maximize the targeting speed if Mode B were to be precluded. However, the argument does not hold for two reasons. Firstly, the accelerated pace affects both sides of the conflict, and therefore does not pose a relative disadvantage to any one party. If reactive BCIs were deemed unlawful, the adversary would not be allowed to use the technology either, curbing the upper limits of reaction speed on both sides to those of conscious human- or computer-based decision-making. Secondly, the rhythm of combat is not an external variable, but will be determined by the means of warfare chosen by an armed force. Consequently, a party cannot raise the increased pace – caused by its proper choice to employ Mode B – as a justification to employ Mode B (venire contra factum proprium).

In light of the foregoing, it appears feasible to employ an alternative means of target identification, as the military advantage gained by using Mode B does not supersede the duty to take constant care. The precautionary rule ergo flanks and vindicates the level of predictability. A proportionate standard of predictability is not met solely by testing the reactive BCI’s performance prior to deployment: military necessity does not overturn the need to ensure Mode B’s predictable compliance with the targeting principles beyond a dry run. As a matter of proportionality, an out-of-the-loop standard of MHC – pertaining to the reactive BCI – must be rejected. Rather, an on-the-loop standard shall lay the foundation for the remaining assessment. Under this standard, MHC – commonly construed as calling for operational oversight – would be achieved in Mode B if the warfighter could supervise his subconscious.

On-the-loop standard: Application to Mode B

What such supervision encompasses will be spelled out in the following. This will allow us to finally gauge whether the operator in a reactive BCI exerts MHC even though he cannot control the moment of command. As a shorthand, the presence of MHC will determine Mode B’s lawfulness.

From the myriad of academic contributions on the subject, Kwik has distilled the content of an on-the-loop standard of MHC in a comparative study.¹⁰¹ The ensuing model features three elements (awareness, weaponeering and context control), the cumulative impact of which translates directly into the “overarching goal-element”¹⁰² of predictability. Each component shall be applied to the man–machine collaboration in Mode B.

Starting with awareness, the soldier must follow the operational context, including its military goal and stakes.¹⁰³ Moreover, he needs to have a thorough grasp on how the BCI functions and how his subconscious interacts with it.¹⁰⁴ Thus, he ought to understand which neural signals trigger the targeting mechanism and how/if he can influence them. Via simulation-based testing, he must become cognizant of what situations (both combat- and mindset-related) his subconscious response is accurate in, and where errors occur. If one had adopted an out-of-the-loop standard, one would simply refer to any reliable test results at this stage. All aspects of awareness control can be met by Mode B.

Incumbent on prior awareness, weaponeering describes the possibility of controlling whether to use a means of warfare in the first place.¹⁰⁵ The soldier needs to gauge whether the reactive BCI, particularly the subconscious targeting mechanism, is suitable in a given context, and deploy it selectively in those circumstances.¹⁰⁶ Since he preserves his freedom of action (see above), the combatant may shut down the BCI at any time. He can further make the above determinations after sufficient testing and mission briefing, which renders weaponeering control unproblematic.

Lastly, after deployment and over the entire course of an operation, the commander needs to exercise context control. On the one hand, he must have the possibility of adjusting the weapon system’s parameters (e.g. imposing a limit on the number of targets that the machine gun can shoot within a given time frame).¹⁰⁷ Since the BCI-enhanced combatant can consciously react to changing contexts, no issue arises. On the other hand, he needs to uphold a “persistent connection”¹⁰⁸ with the means of warfare, translating into the latter’s capacity to supervise and abort an attack. This challenges Mode B: as an expression of being on the loop vis-à-vis his subconscious, the warfighter must be able to veto a targeting decision made. In line with the discussion above and the functioning of reactive BCIs generally, this is impossible: the evoked potential picked up by the neural implant, then translated into a command to engage, constitutes the earliest recognizable indicator of the soldier’s response, preceding his awareness and leaving him no opportunity to intervene.¹⁰⁹ The missing control over the moment of command effectively prevents the combatant from wielding his veto.

The merit of “veto control”¹¹⁰ ought to be regarded as an expression of the human agency that lies at the core of legal reasoning.¹¹¹ This last verification before opening fire – whether the target is actively participating in hostilities or merely shooting a wolf¹¹² – ties into Noll’s idea of genuinely applying the law through an inner dialectic.¹¹³ Only when the operator can “take a remote perspective”¹¹⁴ vis-à-vis his actions and is “reason-responsive”¹¹⁵ can his human agency permit him to weigh and balance military necessity and humanity, and only then can a prediction of lawfulness be issued. In the absence of such supervision, the soldier in Mode B – as a combatant – does not have MHC over his subconscious as part of the weapon system. The reactive BCI therefore violates the principles of distinction and proportionality during its normal or expected use of controlling an armed UGV when reviewed under Article 36 of AP I. The subconsciously controlled BCI (Mode B) is thus unlawful ab initio.

Conclusion

In this article, two types of BCI-controlled weapon systems were presented: Mode A (active BCI with conscious command) and Mode B (reactive BCI with subconscious command). The soldier operating either set-up assumes a dual status under IHL, as both a combatant and a means of warfare (part of a weapon system). Thus, the entire man–machine collaboration was subjected to a review on its lawfulness under Article 36 of AP I, requiring its capacity to respect – especially – the principles of distinction and proportionality.

Mode A poses no issues of legality. In contrast, Mode B – the reactive BCI – undermines the operator’s control over the moment of command. Thus, the concept of MHC was adapted to inspect the oversight that he must exert over his subconscious. Predictability was identified as an overarching requirement under Article 36 of AP I and rationale behind MHC. It serves to ensure that the outcome of an engagement decision adheres – with reasonable certainty – to the targeting principles. After balancing the precautionary duty and military necessity, pre-deployment testing of the reactive BCI (out-of-the-loop standard) was found insufficient to assure predictability. Instead, the operator should be able to supervise his subconscious (on-the-loop standard) so as to cancel an attack if needed. This ability to veto was rejected for Mode B, thereby undermining MHC – i.e., the BCI’s predictable compliance with distinction and proportionality.

On these grounds, it is concluded that a soldier can lawfully use a BCI to control a weapon under IHL if he controls that weapon in a conscious state of mind via an active BCI. The use of a subconsciously commanded reactive BCI is unlawful under Article 36 of AP I.

Lastly, academic rigour demands that we lay down some caveats for these findings. Firstly, this article depicts two models (Mode A and Mode B) as ideal-typical use cases for remote weapon control through BCIs, though the dichotomy of active versus reactive BCIs is somewhat simplified.¹¹⁶ Furthermore, no absolute level of MHC and of being on the loop – against which every weapon system can be held – was established. Instead, a relative standard of ensuring reasonably predictable outcomes was presented, grounding the Article 36 review on considerations of proportionality. One can criticize reading military necessity and precautions into a level of MHC that renders a weapon system unlawful ab initio (weapons law). Alternatively, one could advance that these aspects need only be assessed in the context of a specific attack (law of targeting). Following this view, the risks and uncertainties of the reactive BCI would not restrain its lawful acquisition and development, but would solely call for constant care at the deployment stage. Nevertheless, this far-reaching take on Article 57(1) of AP I and on predictability under Article 36 of AP I represents the author’s best attempt at good-faith interpretation of the law. Maybe one ought to accept that the protection of humanity in armed conflict advises us to rather err on the side of caution.

Additionally, all those factors of BCI command, alternative means and battlefield challenges that tie into the legality of reactive BCIs (even of acquiring and developing them) are presumed at this point. In fact, BCI-controlled weapon systems could eventually fall into the category of technologies that were proclaimed to be groundbreaking junctures but ultimately never saw the light of day.¹¹⁷ Given the pending development of BCIs and the secret nature of military research, some level of assumption is unavoidable.¹¹⁸ While at this point it cannot be entirely excluded that this article is dedicated to a future that will never emerge, there is ample reason to believe that such – or similar – systems will shape the battlefields of tomorrow. Reflections on their legality and the technology’s handling in general should thus, before long, make their way onto the agenda of researchers, military professionals and policy-makers alike.

Among the plethora of legal challenges arising from BCIs, this article has addressed only a small fraction. Particularly, contemplations on accountability for mental acts under international criminal law will be needed. No matter the research angle through which one approaches BCIs, however, the reflections inevitably touch on core considerations of cognition, reasoning and human decision-making. Human agency amidst the targeting process has been characterized as the rationale behind MHC. It entails the possibility of bringing, and the obligation to bring, a battlefield decision under the scrutiny of one’s reason. In the end, when faced with a decision that determines life or death, veto control – the ability to hesitate just one more instance, to reconsider, to change one’s mind – puts the human in meaningful human control. Regardless of what novelties the future may bring, we owe it to ourselves and to the conscience of humanity to guard what is quintessentially human in the merger of man and machine.