Facts, Preferences, and Doctrine: An Empirical Analysis of Proportionality Judgment

Abstract

Legal proportionality is one of the most important principles for adjudicating among conflicting values. However, rather little is known about the factors that play a role in the formation of proportionality judgments. This research presents the first empirical analysis in this regard, relying on a sample of 331 legal experts (lawyers and legal academics). The policy domain addressed by the experiment is the antiterrorist military practice of targeted killings, which has been the subject of a legal debate. Our experimental findings suggest that proportionality judgments are receptive to normatively relevant facts. We also find strong correlational evidence for the effect of ideological preferences on such judgments. These results are consistent for two proportionality doctrines. We suggest that proportionality judgment is anchored jointly in the experts’ policy preferences and the facts of the case. We outline the implications of the findings for the psychological and legal literature.

In recent decades, the legal principle of proportionality has become one of the most prominent constitutional principles for adjudicating among conflicting values (Barak 2011; Beatty 2004; Cohen-Eliya and Porat 2010, 2014; Petersen 2013; Stone Sweet and Mathews 2008). As such, the concept of proportionality has been extensively studied in the legal and philosophical literature. Yet, while the socio-legal perspective suggests that law is constituted by “the discretionary decisions that give it meaning” (Mather 2008: 691), rather little empirical knowledge has been accrued about the ways in which proportionality informs and shapes the decision-making process followed by legal experts – if at all. Do legal experts rigorously adhere to the structure of proportionality analysis or does the conceptual abstractness of this legal method leave a loophole for legally irrelevant preferences? To the best of our knowledge, this research provides the first set of experimental evidence in this regard.

Three research questions concerning the nature of proportionality judgments are addressed in this research. First, in any given case, a judgment should, by definition, be informed by the circumstances in which the conflict of values arises (henceforth, factual considerations; a change thereof – factual variation). However, no systematic empirical attempt has so far been undertaken to assess the extent to which proportionality judgments conform to this stipulation. Second, a growing number of studies have recently shown that various legal decisions are susceptible to legally irrelevant preferences. Following this line of inquiry, our second research question revolves around the impact of legally irrelevant preferences on proportionality judgments. We assess the validity of these effects and distinguish among two potential mechanisms that govern them. Third, some doctrinal variation in proportionality analysis has been identified across countries (Cohen-Eliya and Porat 2010; Petersen 2013; Stone Sweet and Mathews 2008) and across legal domains (e.g., domestic law vs. international humanitarian law (IHL), see: Cohen and Shany 2007; May 2013). The questions that are addressed in this regard are (1) whether some doctrine versions of the judgment-formation process elicit greater receptivity to factual considerations than others; and (2) whether such versions provide a better guard against normatively unwarranted effects of legally irrelevant preferences.

To probe these three sets of questions in light of empirical evidence, the present research utilizes a distinctive and robust experimental design which draws on a sample of 331 legal experts (lawyers and legal academics). The subject matter explored is the military practice of targeted killings of terrorist operatives (henceforth, targeted killings). This tactic is implemented routinely by a number of countries, and thus, the issue extends beyond the local context of this specific investigation.¹ Our findings suggest that proportionality judgments regarding cases of targeted killings are receptive to normatively relevant factual information. Likewise, strong evidence emerged for the effect of ideological preferences on judgments, but no indication that information processing was systematically biased. These results suggest that the vague and value-laden nature of proportionality analysis may allow legal experts to be swayed by their value preferences, in addition to being guided by the facts of the case. These findings hold true for two alternative versions of proportionality (henceforth, doctrinal versions), as elaborated in what follows.

This article comprises seven sections. In section proportionality analysis, we briefly review the legal doctrine of proportionality. Section a behavioural analysis of proportionality judgments outlines a behavioral perspective regarding proportionality judgments, which gives rise to six hypotheses that were tested in the analyses. Section proportionality and targeted killings of terrorist operatives provides a brief review of the targeted killings policy. Section research design elaborates the experimental research design and statistical estimation adopted. Section results reports the research findings, and Section discussion recapitulates the findings and discusses their implications.

Proportionality Analysis

A key aim of legal systems is to foster the peaceful resolution of conflicts. In most democracies, conflicts in the public arena often involve highly revered yet competing values. Accordingly, their resolution entails balancing constitutional rights against other public interests, as well as weighing among contending rights – a challenging task by any standard. One of the most prominent methods of achieving such balance is proportionality analysis (Alexy 2010; Beatty 2004; Kumm 2007; Petersen 2013; Stone Sweet and Mathews 2008).

The origins of proportionality analysis can be traced back to nineteenth-century German law. Following World War II, this method has spread across Europe, including post-Communist states and Israel. Proportionality analysis has been absorbed into Commonwealth legal systems – Canada, South Africa, New Zealand and the UK – and is currently gaining ground in Central and South America as well. It has also been adopted in three central treaty-based multinational regimes: the European Union (EU), the European Convention on Human Rights (ECHR) and the World Trade Organization (WTO) (Beatty 2004; Jackson 2004; Kumm 2007; Stone Sweet and Mathews 2008: 74). Moreover, proportionality has become a central principle of international law, governing such critical issues as a country's right to self-defence, the legitimacy of war (ad bellum) and the legitimate use of military force (in bellum) (Hurka 2005).

In these contexts, the principle of proportionality is expected to guide policy makers (both legislators and executive officeholders) in evaluating policy alternatives, and judges, in adjudicating rights. Widely recognized by judicial authorities, proportionality has become a prominent decision-making principle in adjudicating between conflicting values and interests (Beatty 2004; Stone Sweet and Mathews 2008: 75).

Proportionality Analysis (henceforth, PA) is structured as a series of decision-making stages, of which the first two are preliminary, while the last three, putatively, constitute an integral part of the process itself (Alexy 2010; Kumm 2007).

1. 1. First preliminary stage: Infringement of Rights. In this stage, the analyst assesses whether or not the policy in question potentially entails infringements of constitutionally protected rights.² Only if the answer is affirmative does the analysis ensue.

2. 2. Second preliminary stage: Legitimate/Worthy Goal. In this stage, the analyst examines the legitimacy of the goal that the policy seeks to promote³ (henceforth, “worthy goal”).

3. 3. PA Stage One: Suitability. The first integral decision stage of proportionality analysis (henceforth, “suitablity”) examines whether the means are rationally related to stated policy objectives, a.k.a. the worthy goal.

4. 4. PA Stage Two: Necessity. The second integral decision stage, termed “necessity” or “least restrictive means” (LRM), examines whether the measure does not curtail a right any more than an available alternative policy equally capable of attaining the stated goal.

5. 5. PA Stage Three: Balancing. The final stage of PA (henceforth, “balancing” or “strict-sense balancing”) is implemented if the policy action satisfies the previous tests; it assesses whether the benefits of the action justify the costs of rights infringement.⁴

As is evident from the above guidelines, the four final stages of proportionality analysis cover both normative and empirical aspects of the decision (Kumm 2007: 137). The “worthy goal” and “balancing” stages invoke normative judgment, in that they exclude policy goals that are not aligned with human-rights protection and weigh out the infringement of a right against the benefit of adhering to the competing goal. The “suitability” and “necessity” stages pivot on empirical assessment: respectively, whether the given policy will benefit the goal designated as worthy in the preceding stage, and whether no equally effective policy is available that would mitigate the right infringement.⁵

The rapid spread of proportionality analysis across legal systems and policy domains attests to its wide-ranging importance and appeal as a legal principle (Jackson 2015; Stone Sweet and Mathews 2008). Its normative centrality to constitutional law is manifested by studies that regard proportionality as “a common grammar of global constitutionalism” (Cohen-Eliya and Porat 2010: 263–4), “a generic constitutional law” (Law 2004), and even as an “ultimate rule of law” (Beatty 2004). Other scholars, while acknowledging the centrality of proportionality in many jurisdictions, point to the variations in its meaning and application (Bomhoff 2013; Cohen-Eliya and Porat 2014).

Yet, despite its ubiquity and centrality, the factors that shape proportionality analysis, at the level of the individual legal expert, have not yet been empirically studied. Given the prominence of proportionality, understanding the factors and processes influencing legal experts in forming proportionality judgments holds potentially important normative and political implications in many countries and for a variety of policy domains. The following section elaborates this issue, which is encapsulated in the research questions of the present study.

A Behavioural Analysis of Proportionality Judgments

Although proportionality is well established as a legal principle, not much is known about the actual cognitive processes involved in legal decisions, when proportionality analysis is ostensibly applied. While PA offers an appealing model for adjudicating among competing values, it presents a number of challenges in practice. The very conceptual abstractness of PA, which makes it so widely applicable, produces unpredictable and inconsistent rulings on the part of different adjudicators and across cases.⁶ In part, this is because the application of PA involves empirical assessments as well as value judgments, and thus requires expertise and experience.

To diminish the lacuna in the empirical investigation of decision-making processes based on PA, our study addresses three research questions. First, proportionality judgments must, by definition, rest on factual considerations that are relevant to the conflicting values in question. The proportionality doctrine assigns positive utility to a “worthy goal.” The higher the policy's expected marginal contribution to obtaining the “worthy goal,” the higher positive utility will be attributed to that policy. Conversely, the doctrine assigns negative utility to infringement of rights. This negative utility is also attributed to the policy, and it increases with the magnitude/severity of rights-infringement potential entailed by the policy (For further details, see: Alexy 2010; Petersen 2013; Veel 2010). Thus, our first research question is whether proportionality judgments are receptive to factual considerations which are expected to guide them normatively. Based on the legal theory of proportionality, the following two hypotheses can be set forth in this regard:

H₁: The probability of judging a policy as proportional⁷ rises as the worth of the policy goal, and the extent to which it is expected to be attained, increases, in the sense that its implementation contributes to greater marginal public welfare, in the normative sense.⁸

H₂: The probability of judging a policy as proportional increases as the potential for rights infringement resulting from implementing that policy diminishes.

The second research question is prompted by studies that point to the impact of legally irrelevant considerations on various legal decisions, through motivated cognition (Furgeson, Babcock, and Shane 2008; Kunda 1990; Sood 2013) or implicit biases (Gazal-Ayal and Sulitzeanu-Kenan 2010; Rachlinski et al. 2008). A tension between the drive for accuracy and belief perseverance underlies all human reasoning (Lodge and Taber 2013: 150).⁹ In the context of legal reasoning it has been suggested that “[p]olicy preferences and legal reasoning may be so cognitively intertwined that lawyers and judges have difficulty fully realizing what factors have influenced their conclusions” (Furgeson, Babcock, and Shane 2008: 226). This argument is particularly likely to apply in vague and highly abstract judgment tasks (Bertrand, Chugh, and Mullainathan 2005). As proportionality analysis represents a legal standard, in the sense of being open-ended, vague, and allowing fact-specific determination (Sunstein 1995; Kaplow 1992), its implementation is more likely to be affected by motivational goals such as policy preferences. Policy preferences may operate via one or both of the following mechanisms: (1) a direct effect of the policy preference on the final judgment; and (2) through a process known in the motivated-cognition literature as “biased processing of information” – that is, when information is processed differently depending on its consistency with existing preference, in making a judgment (Ditto and Lopez 1992; Jain and Maheswaran 2000; Sood 2013; Taber and Lodge 2006). Given that proportionality analysis involves two main conflicting values – “worthy goal” and “rights infringements” – we expect the adjudicator's legally irrelevant preferences to affect the respective weight assigned to these considerations. To shed light on the judgment-formation process, both the above mechanisms are tested in this study. The above discussion yields two hypotheses:

H₃: The probability of judging a policy as proportional is expected to be influenced by legally irrelevant policy preferences.¹⁰

H₄: The effect of legally irrelevant policy preferences moderates the weight assigned by the decision-maker to the information pertaining to the “worthy goal” and potential rights infringement factors.

Proportionality analysis has been shown to vary across countries (Cohen-Eliya and Porat 2010; Petersen 2013; Stone Sweet and Mathews 2008) and legal domains (e.g., domestic law vs. IHL, see: Cohen and Shany 2007; May 2013). This doctrinal variation pertains mainly to the set of decision stages the analysis takes into account. The question that arises in this connection is whether some versions of PA facilitate greater sensitivity to normatively relevant factual considerations than others, thus, potentially improving the balance between protected rights and public interests, which is the main goal of proportionality judgments. Specifically, is a more comprehensive PA version, that is, one that accounts for the full set of decision stages (henceforth, “full PA”), more receptive to normatively relevant factual considerations, compared with a less comprehensive version e.g., “strict-sense balancing”?

H₅: Proportionality decisions are more receptive to (i.e., more strongly affected by) factual variation under a comprehensive proportionality analysis, compared with less comprehensive versions (strict-sense balancing).

A legal doctrine is expected to provide lawyers with normative criteria to form consistent and unbiased judgments. If proportionality judgments are systematically swayed by legally irrelevant preferences, it is important to assess whether such biases vary across different doctrinal choices. While motivated-cognition processes are robust vis-à-vis variation in cognitive elaboration (Taber, Cann, and Kucsova 2009), Implicit bias has been shown to characterize cases of task ambiguity (Bertrand, Chugh, and Mullainathan 2005; Gazal-Ayal and Sulitzeanu-Kenan 2010). Relying on this latter mechanism, we expect policy preferences to have a greater impact under less comprehensive versions of proportionality analysis, such as “strict-sense balancing,” than under a “full PA.” Hence the following hypothesis:

H₆: The effect of legally irrelevant policy preferences is more pronounced under “strict-sense balancing,” and less so under a “full PA.”

This research assesses the effects of different doctrinal versions on the judgment outcome, with regard to its receptivity to normatively factual considerations and susceptibility to the adjudicator's legally irrelevant preferences. To this end, it capitalizes on the Israeli Court of Justice's application of a comprehensive version of the proportionality doctrine in the cases of targeted killings (Cohen and Shany 2007). The following section reviews this policy and the application of proportionality in cases of targeted killings.

Proportionality and Targeted Killings of Terrorist Operatives

Targeted killings of terrorist operatives (or preemptive strikes) involve the military use of lethal force against suspected individual terrorists, as part of counter-terrorism activities performed by a state outside its territory (for a review of the various definitions of targeted killings, see: Barnidge and Robert 2012: footnote 9). In recent years, this practice has gained considerable ground, bringing the rules of its implementation under the scrutiny of judicial and oversight bodies, both domestic and international (Barnidge and Robert 2012; Bowcott 2012; May 2013).

Targeted killings can be viewed in light of two legal models of justified killing. This issue, which has been traditionally addressed mainly by IHL, has recently been debated within the domestic framework of law enforcement (May 2013). Although the term “proportionality” itself does not explicitly appear in any of the IHL treaties, it is broadly understood to constitute an important principle in the laws of war. Thus, this principle is reflected in Articles 51(5(b) and 57(2) of the First Additional Protocol of 1977 to the Geneva Conventions of 1949.¹¹ The 1977 Protocol was ratified by many states and, in international law, is now widely regarded as customary and thus binding for nonsignatory states as well. The proportionality principle as framed in the 1977 Protocol has traditionally been understood to include only the third PA stage – that is, the “strict-sense balancing” (Higgins 1994: May 2013).

However, in December 2006, the Israeli Supreme Court (sitting as a High Court of Justice – HCJ) delivered one of the most comprehensive judicial decisions on the international rules governing targeted killings – the Public Committee Against Torture in Israel vs. The Government of Israel (henceforth, the Targeted Killings case).¹² The main judgment was delivered by Justice Aaron Barak, the departing President of the Supreme Court. Although Barak contended that, under IHL, the third, “strict-sense” PA stage appears to be the most relevant to assessing the lawfulness of targeting, he did not rule out the relevance of other proportionality decision stages. Barak specified that a military action must conform to all three proportionality requirements (suitability, necessity and balancing), effectively harmonizing the application of PA in both IHL and Israeli domestic Law (Cohen and Shany 2007).¹³

When dealing with targeted killings, the first PA stage (suitability) requires that this strategy rationally lead to the desired military objective. The second PA stage (necessity) is centered on whether a targeted killing constitutes the least harmful way of neutralizing the risk. Less aggressive alternatives may include capture, subsequently leading to detention, interrogation and trial (Cohen and Shany 2007). The third PA stage (“strict-sense balancing”) weighs out the expected military benefit against the expected collateral humanitarian harm of the targeted killing (Targeted Killing §46). This view on the part of the Israeli HCJ also resonates with the International Committee of the Red Cross's Interpretive Guidance on Direct Participation in Hostilities (May 2013: 57).

The 2006 Targeted Killings decision extends the applicability of proportionality to targeted killings in Israeli law. While IHL requires only that the operation should satisfy the “strict-sense balancing” requirement, the HCJ decision requires compliance with all three decision stages as well as meeting the preliminary “worthy goal” condition. However, much like other legal decisions, the Targeted Killing decision offers but limited guidance regarding the practical application of PA in targeted killings (Cohen and Shany 2007). Thus, it is not clear what weight should be assigned to each of the competing values involved, nor how should one consider the level of predictive uncertainty regarding the military and humanitarian results of the attack. This vagueness necessitates an empirical assessment as to whether and to what extent the proportionality doctrine informs judgments. The following section describes and accounts for the research design that was used for this purpose.

Research Design

We conducted a survey-embedded experiment involving 331 Israeli legal experts between June and July 2013. Sampling was performed in two stages, the first with 166 Master of Law students in five universities,¹⁴ while the second – with 165 legal experts who work in the public and private sectors¹⁵ and who completed an online version of the questionnaires assigned to the students in the first stage. Following a set of analyses which confirmed the causal homogeneity of the two subsamples, these were merged¹⁶ into a total sample of 331 lawyers. Still, to account for any potential variation that is due to the sampling method, all the analyses include a dummy variable (“online”) that has the value 1 for respondents who completed an online questionnaire, and 0 otherwise. Table 1 presents descriptive statistics for the sample.

Table 1. Sample Descriptive Statistics

Experimental Procedure

Respondents (N = 331) were randomly assigned to one of sixteen descriptions of a proposed military plan for a targeted killing, and then were asked to evaluate the proportionality of the plan using their legal expertise. Vignette survey experiments are widely used in social sciences, and were found to have strong external validity in predicting behaviour of both citizens (Hainmueller, Hangartner, and Yamamoto 2015) and professionals (Peabody et al. 2004). The versions of the military plan differed with regard to the operational goal and to the attendant infringement of rights. Overall, we used four “goal treatments” and four “rights infringement treatments,” which yielded a four by four, between-subjects, fully crossed factorial design. Table 2 presents the two sets of vignettes that were used to construct the 16 experimental conditions. Each condition combines one of the “goal treatments” and one of the “rights infringement treatments.”

Table 2. Experimental Treatments

Note: The 16 experimental conditions are based on all the possible pairings of these variations of the “goal” and “rights infringement” parameters.

The “goal treatments” were as follows: G1 – a punitive goal; G2 – a preventive goal; G3 – prevention of a more severe harm; and G4 – prevention of a more severe harm by targeting more senior terrorist operatives. The “rights infringement treatments” were as follows: R1 – targeting terrorists who are in a car in an urban area of Gaza, without mentioning the option to apprehend them; R2 – the same plan preceded by a remark that apprehending the terrorists is not a feasible option; R3 – identical to R2, except the targeted car is in a nonurban area, and therefore, the plan presents a lesser risk to uninvolved bystanders; R4 – identical to R3, except the car is supplanted by a motorcycle, thus reducing the risk of uninvolved passengers being inside the target.¹⁷

Measuring Proportionality Judgments

The dependent variable – the proportionality judgment – was measured using a set of five questions. Four questions covered the four decision stages: (1) Is the goal of the military plan worthy (yes/no)? (2) Is the military plan adequate and effective for achieving the goal (yes/no)? (3) Is the military plan a practical means to achieve the goal with minimal infringements of human rights (yes/no)? (4) Is the proportion between the advantage to be gained by implementing the military plan and the expected infringement of human rights adequate (yes/no)? Since this study constitutes the first attempt to measure proportionality judgments empirically, we conducted a series of validation analyses for the measurement items, which are reported in detail in the online appendix.

The fifth question was prompted by the above-discussed doctrinal ambiguity of proportionality analysis and was intended to elicit an unstructured proportionality judgment (henceforth, “free PA”). Respondents were asked to indicate their judgment regarding the proportionality of the proposed plan on a scale of 1 (very disproportional) to 6 (very proportional). Respondents were informed that the boundary between un-proportionality and proportionality lay between 3 and 4 on the scale.

Responses to these five questions yielded three binary measures of proportionality judgments for each respondent, as displayed in Figure 1. The first measure is proportionality in the “strict sense,” as prescribed by IHL. It assigns the value of 1 to the affirmative answer to the forth item and 0 otherwise. The second measure is a “full PA,” as required by the Targeted Killing decision. This measure assigns the value of 1 to affirmative answers to all four items, and 0 otherwise. The third measure is a binary version of the response to the “free PA” item; it assigns the value of 1 to a score between 4 and 6, and 0 otherwise. Thus, respondents were presented with legal questions they expected a proportionality analysis to address, while we obtained results regarding both PA versions, the comprehensive (“full PA”) and the less comprehensive one (“strict-sense balancing”).

Figure 1. Question wording, and three measures of proportionality judgments.

As the three measures of proportionality judgment yielded an internally consistent index (alpha = .866), they were treated as three repeated proportionality measures for each respondent. The analysis controlled for the type of measure, and thus provided estimates for the effects of the doctrinal version (“strict-sense balancing,” “full PA,” and “free PA”) on the judgment. Further details are provided in the Estimation strategy section.

Measuring Ideological Preference

The effect of respondents’ legally irrelevant preferences on proportionality judgments were estimated based on their self-reported political ideology. Political ideology serves as a simplifying heuristics that helps individuals interpret the world, by “making assertions or assumptions about human nature, historical events, present realities, and future possibilities” (Jost, Federico, and Napier 2009; Ball and Dagger 2010; Erikson and Tedin 2010; Ball and Dagger 2010). Indeed ideology is the most important spatial dimension guiding vote choice (Duch, May, and Armstrong 2010) and policy preference (Ringe 2005; Sulitzeanu-Kenan and Halperin 2013). In Israel, the central domain aligning right- and left-wing supporters is the hawk versus dove approach to the Arab-Israeli conflict (Benoit and Laver 2006; Shamir and Arian 1999). If proportionality judgments are indeed susceptible to legally irrelevant preferences (H3), hawks are expected to be more likely than doves to judge military plans for targeted killings in the context of this conflict as proportional (and vice versa). Each respondent's political ideology relative to the hawk-dove continuum was assessed using a single-item measure, in which respondents were asked to indicate their political position on a scale of 1 (extreme right) to 5 (extreme left). For each respondent we also recorded a set of covariates which included age, gender, and religiosity.¹⁸

Estimation Strategy

To identify the effects of factual variation in the “worthy goal” and “rights infringements” parameters, respondents were randomly allocated to 16 experimental conditions. Supporting Information Table A4 in the online appendix verifies that the randomization process was successful, by demonstrating that practically all individual-level variables are balanced across the experimental treatments. As the three proportionality measures for each respondent turned out to be internally consistent, the data were structured as a repeated measures dataset of 3 × 331 = 993 observations.

Equation (1) presents our basic method for estimating proportionality judgments; it accounts for the factual variation, the doctrinal versions, and individual-level characteristics:

(1) $Logit\left[E\right(Y_{ij}=1\left|{\left(K,D,X\right)}_{ij}\right]={\alpha K}_{ij}+{\beta D}_{ij}+{\gamma X}_{ij}$

where Y is a binomial variable that represents a judgment that the plan is proportional [Y = 1] or not [Y = 0], by respondent i, based on proportionality type j, conditional on a vector of factual variation K, a vector of doctrine alternatives D and a vector of individual characteristics X. We estimate the effects of factual variation on proportionality judgment adjusting for multiple individual-level covariates, using generalized estimating equations (Liang and Zeger 1986). Generalized estimating equation is a method developed for handling clustered data, in which the observations within each cluster are intercorrelated. Given that each set of three proportionality judgments are clustered within respondent, we used generalized estimating equations to account for the correlations among observations. To address within-subject correlations, we cluster standard errors within respondent. To obtain detailed estimates for the “goal” and “rights infringement” treatments, doctrine alternatives, and ideological differences, we estimate proportionality judgments using the following elaboration of (1):

(2) $Logit{\left(Y\right)}_{ij}={\alpha }_0+{\alpha }_1{Preventive\left(g2\right)}_i+{{\alpha }_{2}Severity\left(g3\right)}_i+{\alpha }_{3}Senior{\left(g4\right)}_i+{\alpha }_{4}LRM{\left(r2\right)}_i+{\alpha }_{5}nonunrban{\left(r3\right)}_i+{\alpha }_{6}motorcycle{\left(r4\right)}_i+{\alpha }_7{FullPA}_j+{{\alpha }_{8}FreePA}_j+{\alpha }_9{Ideology}_i+{\alpha }_{10}{female}_i+{\alpha }_{11}{age}_i+{\alpha }_{12}{relig.}_i$

where the first six independent variables comprise two sets of three indicator variables, each set representing the four conditions with regard to the operation goal (with “punitive goal” as reference) and the potential infringement of rights (with the plan with most rights-infringement potential as reference), respectively. $FullPA$ and $FreePA$ are indicator variables for the type of proportionality analysis (“strict-sense” as reference). Next are individual-level characteristics (hawk-dove ideology, gender, age, and religiosity).

To evaluate the impact of factual variation on proportionality judgments as per Hypotheses 1 and 2, we first need to define the minimal set of empirical expectations required to satisfy these hypotheses. As noted above, the proportionality doctrine assigns positive utility to a “worthy goal” and negative utility to infringement of rights. However, it does not specify the relative weight of the “goal” and “rights infringement” factors, nor does it stipulate the impact of changes in these factors on utility (Aleinikoff 1987; Petersen 2013; Tsakyrakis 2009; Webber 2010). This particular vagueness poses a problem in stating clear empirical expectations regarding behaviors that are congruent with this doctrine and those that are not. Despite this vagueness, we can specify certain restrictions on the pattern of judgment which would qualify the doctrinal guidance. Such a pattern must be consistent with the dominance principle (Hadar and Russell 1969; See also: Kariv and Silverman 2013), in the sense that a policy should be preferred to another if it yields unambiguously greater utility (i.e., increases the share obtained from the goal, or decreases the potential for rights infringement). This entails that, when comparing proportionality judgments across such ordered set of policies, the pattern of judgment must be monotonic.¹⁹ Only under this condition can we conclude that it conforms to a certain proportionality doctrine. Implementing this restriction in our analysis suggests that Hypotheses 1 and 2 are supported only if coefficients ${\alpha }_1$–${\alpha }_3$, and ${\alpha }_4$–${\alpha }_6$, respectively, have positive, and monotonically increasing values. This means that, ceteris paribus, the probability of a plan being judged as proportional should never decrease when moving to a scenario with a more worthy goal or with a lesser potential for rights infringements.²⁰

The effect of ideological (hawk-dove) preferences on proportionality judgments is estimated in two ways. First, in order to test Hypothesis 3, we simply estimate the size and statistical significance of coefficient ${\alpha }_9$ in (2). If such an association is found, we proceed to determine the mechanisms through which these preferences shape proportionality judgments. One possible account is that one's prior propensity for judging targeted killings as proportional depends on one's ideology. The second is that one's ideological preference influences the weight assigned to different factual considerations, thereby affecting information processing, as suggested in the motivated-cognition literature. This indirect effect of ideology on proportionality judgments, as per Hypothesis 4, can be tested by comparing the effect of the different treatments employed on hawks versus doves. For this purpose, we add to (2) interaction terms for ideological groups (hawks/doves) and the treatments within the “goal” and “rights infringements” factors:²¹

(3) $\mathrm{Logit}{\left(Y\right)}_{ij}={\alpha }_0+{\alpha }_1{\mathrm{Goal}}_i+{\alpha }_2{\mathrm{Rights}\text{\_}\mathrm{infringe}.}_i+{\alpha }_3{\mathrm{Full}PA}_j+{\alpha }_4{\mathrm{Free}PA}_j+{\alpha }_5{\mathrm{Hawk}\times \mathrm{Goal}}_i+{\alpha }_6{\mathrm{Hawk}\times \mathrm{Rights}\_\mathrm{infringe}.}_i+{\alpha }_7{\mathrm{Hawk}}_i+{\alpha }_8{\mathrm{female}}_i+{\alpha }_9{\mathrm{age}}_i+{\alpha }_{10}{\mathrm{relig}.}_i{}^{21}$

These interaction terms allow us to test Hypothesis 4 by estimating the differences between the receptivity of proportionality judgments to factual variation across ideological groups.²²

To assess the implication of the doctrine version on proportionality judgments, we first assess the size and significance of coefficients ${\alpha }_7$ and ${\alpha }_8$ in (2). Next, to test whether the doctrine version affects the level of receptivity to factual variation (Hypothesis 5), we add to (2) interaction terms for the doctrine version and ideological preference:

(4) $Logit{\left(Y\right)}_{ij}={\alpha }_0+{\alpha }_1{Goal}_i+{\alpha }_2{Rights\_infringe.}_i+{\alpha }_3{FullPA}_j+{\alpha }_4{FreePA}_j+{\alpha }_5{FullPA}_j\times {Goal}_i+{\alpha }_6{FullPA}_j\times {Rights\_infringe.}_i+{\alpha }_7{FreePA}_j\times {Goal}_i+{\alpha }_8{FreePA}_j\times {Rights\_infringe.}_i+{\alpha }_9{Ideology}_i+{\alpha }_{10}{female}_i+{\alpha }_{11}{age}_i+{\alpha }_{12}{relig.}_i$

Lastly, to test whether the doctrine version affects the level of susceptibility to legally irrelevant preferences (Hypothesis 6), we add to (2) interaction terms for the doctrine version and ideological preference:

(5) $\mathrm{Logit}{\left(Y\right)}_{ij}={\alpha }_0+{\alpha }_1{Goal}_i+{\alpha }_2{Rights\_infringe.}_i+{\alpha }_3{FullPA}_j+{\alpha }_4{FreePA}_j+{\alpha }_5{FullPA}_j\times {Ideology}_i+{\alpha }_6{FreePA}_j\times {Ideology}_i+{\alpha }_7{Ideology}_i+{\alpha }_8{female}_i+{\alpha }_9{age}_i+{\alpha }_{10}{relig.}_i$

Results

Receptivity of Proportionality Judgments to Normatively Relevant Factual Variation

We estimate the effects of factual considerations on proportionality judgments by fitting to our data. Detailed results are reported in Table 3 and presented graphically in Figure 2B below.

Figure 2. The effects of “goal” and “rights infringement” treatments on the probability of “proportional” judgment (95% CI). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Table 3. Generalized Estimating Equations predicting Proportionality Judgments

Note: The generalized estimating equations employ logit link function and unstructured correlation structure. Coefficients are reported in logit values. Statistical significance levels are represented as follows: + p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001. “Goal treatments” is a four-scale ordinal variable representing the four “goal” treatments; “Preventive attack,” “Severe attack” and “Senior terrorists” are indicator variables for the “goal” treatments, with “punitive attack” serving as a reference category; “Rights infringement treatments” is a four-scale ordinal variable representing the four “rights infringement” treatments (note that a higher value indicates less potential for rights infringements); Least restrictive means”, “Car in nonurban area” and “Motorcycle in nonurban area” are indicator variables for the “rights infringement” treatments, with “no mention of alternative” serving as a reference category; “Hawk-dove” is a five-scale ordinal variable with 1 representing “extreme hawk” and 5 representing “extreme dove”; “Hawk” is an indicator variable for respondents who identified as either “extreme hawk,” “hawk” or “center” (“dove” and “extreme dove” as reference); “Gender” is an indicator variable representing a female respondent (male as reference); “Religiosity” is a four-scale ordinal variable with 1 representing “ultraorthodox” and 4 representing “secular”; “Online sample” is an indicator representing respondents who participated in the online sample (pen and paper sample as reference); “Full PA” and “Free PA” are indicator variables, with “strict sense” as a reference category.

Figure 2A displays the average probability of judging the operation as proportional for each treatment of the “goal” factor,²³ controlling for all other variables, which are set at their respective mean values.²⁴ The overall effect of “goal” treatments was statistically significant (χ ² =12.51, p = 0.005) based on post estimation Wald test. As shown in Figure 2A, the probability that the operational plan would be judged as proportional increases with the rise in the worthiness of the goal (moving from left to right on the X-axis). The overall effect of the “rights infringements” treatments are likewise statistically significant (χ²= 37.53, p < 0.0001) based on post estimation Wald Test. As per Figure 2B, the probability of an operation being judged as proportional increases with the decrease in the anticipated rights infringement. Note that both of these trends are monotonic, in that none of the treatments is associated with a reverse trend.²⁵ These finding align with the empirical expectations regarding Hypotheses 1 and 2, which are presented in the Estimation strategy section, suggesting that proportionality judgments are receptive to doctrinally relevant factual variation.

Although, overall, the susceptibility of proportionality judgments with factual variations in the “goal” and “rights infringement” factors conformed to our expectations, some specific treatments within these two factors do not appear to affect the judgment at a statistically significant level. The difference between a punitive and a preventive operation resulted in a significant increase of 20.0 percentage points (p = 0.012) in the probability of judging the operation as proportional. Conversely, the effect of implying that the expected attack (to be averted by the operation) is likely to be severe was −1.2 percentage points and statistically insignificant (p = 0.986). Likewise, the marginal effect of stipulating that the operation targets a senior and important terrorist operative resulted in an increase of 7.6 percentage points, which is statistically insignificant (p = 0.302).²⁶ With regard to the influence of the “rights infringement” factor on the probability that the operation would be judged as proportional, a putative lack of a nonlethal alternative to the operation had a marginal effect of 13.6 percentage points, which is marginally significant (p = 0.10). The reduction in the risk to uninvolved bystanders resulting from locating the targeted car in a nonurban area had a significant effect of 23.7 percentage points (p = 0.002). Finally, supplanting a car by a motorcycle and thereby increasing the precision of the attack, yielded an insignificant marginal effect of 1.9 percentage points (p = 0.476). The specific treatments that did not result in a statistically significant effect may be accounted for by the ineffective phrasing of a particular treatment²⁷ and/or sample size.

Inasmuch as both factors produced a significant overall effect and clearly monotonic patters, the lack of a significant effect for some specific treatments does not detract from the validity of the results, which suggest that proportionality judgments appear to be receptive to factual variation in line with doctrinal expectations. The specific treatments that did not result in a statistically significant effect may be accounted for by ineffective phrasing of a particular treatment and/or sample size. Notwithstanding these potential methodological reasons, comparing the set of treatments that were found to affect proportionality judgements and those that did not, suggests that respondents make a discrete distinction between a punitive and preventive goals but do not appear to be affected by the size of the preventable harm in weighing the worthiness of the goal. Respondents also appear to care whether a nonlethal alternative is available, and the level of risk to bystanders. However, once risk to bystanders was reduced, no indication was found that the selectiveness of the operation (i.e., the level of certainty of targeting only the suspected terrorists and not individuals accompanying them) was consequential to respondents’ decisions. While the latter considerations might be limited to professionals versed in the tactical nuances of such operations, the former null finding regarding the facts of the operational goal – expected severity of attack to be averted and the seniority of the targeted terrorists – is more surprising, as they (especially preventable severity) pertain to a central aspect to be considered in proportionality analysis, i.e. the importance of the operation. We discuss these results further in the discussion section.

The existing literature on proportionality, and particularly those studies that formally specify the theory (Alexy 2010; Petersen 2013; Veel 2010) implicitly suggest that proportionality judgment relies on the additive effects of the two conflicting values. However, as our interest in this study is in the characteristics of the actual way in which legal experts conduct the proportionality analysis, we performed a further analysis that assesses whether the weight given to either of the two values is influenced by changes in the other conflicting value.²⁸

To estimate these possible interactions between the two factors we fitted a model that adds all the possible interactions between the levels of the two factors (“worthy goal” and “rights infringement”) to (2). Detailed results are reported in the left column of Supporting Information Table A5 in the online appendix. The results do not support an interaction model. None of the specific interaction terms were statistically significantly. Contrasts of the marginal linear predictions based on this model resulted in two main effects – for the “worthy goal” (χ ² =12.73, p = 0.005) and the “rights infringement” (χ ² =37.76, p < 0.001) – and an insignificant interaction effect (χ ² =5.76, p = 0.764). Given that some specific treatments within the two factors were not found to affect the proportionality judgments at a statistically significant level, we fitted a further model, that compares condition G1 to the three other levels of the “worthy goal” factor, collapses conditions R3 and R4 of the “rights infringement” factor, and includes all the possible interactions between these levels of the two factors. Detailed results are reported in the column titled “Reduced Interaction Model” in Supporting Information Table A5 in the online appendix. The substantive findings for this model are similar to the full interaction model. None of the specific interaction terms are statistically significantly, and contrasts of the marginal linear predictions resulted in two main effects – for the “worthy goal” (χ ² =10.84, p = 0.001) and the “rights infringement” (χ ² =31.84, p < 0.001) – and an insignificant interaction effect (χ ² =.33, p = 0.848). These results suggest that the pattern of proportionality judgments in response to the experimental treatments is consistent with an additive conception of proportionality analysis.

The effects of the “worthy goal” and “rights infringement” factors on proportionality judgments are concurrent and integrated, as is demonstrated in Figure 3, which maps the factual variation into proportionality judgments in a three-dimensional graph. The two horizontal axes represent the “goal“ and “rights infringements” treatments, and the vertical axis represents the probability of the plan being judged as proportional. Thus, the figure provides a comprehensive visual representation of the receptivity of proportionality judgments to factual variation.

Figure 3. A three-dimensional mapping of normatively relevant factual variations into proportionality judgments. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

The Effects of Ideology on Proportionality Judgments

Figure 4 is based on (2) with a minor specification change: the ordinal ideology variable is treated as categorical rather than ordinal. It presents the mean probability of the operation being judged as proportional for respondents with different positions on the hawk-dove ideological scale, controlling for all other variables (which are set at their respective mean values). This analysis clearly shows that proportionality judgments are strongly associated with ideology (χ² = 62.26, p < 0.0001). Even if we ignore the two extreme categories on both sides of the ideological spectrum (jointly constituting 8.8% of the sample), the mean difference in the probability of judging a given operation as proportional decreases from 90.9% for “hawks,” to 78.1% for “center” (χ² = 9.90, p = 0.007), and to 52.1% for “doves” (χ² = 17.21, p < 0.001) – both differences being sizable and statistically significant.²⁹ These findings provide support for Hypothesis 3.

Figure 4. Mean probability of “proportional” judgment across ideological positions (with 95% CI) Note: only 8 respondents occupy the “extreme hawk” category, all of whom judged their respective scenario as proportional (under the three different proportionality formulae). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

As our results indicated an association between ideology and proportionality judgment, we proceeded to explore the mechanism for this influence, as per Hypothesis 4. To this end we fitted (3) to our data. This model includes interaction terms which allow us to estimate the difference in treatment effects for hawks versus doves. Accordingly, we split our sample into two ideological groups: “hawks” (extreme hawks, hawks, center; N = 200) and “doves” (doves, extreme doves; N =127).³⁰ For simplicity, we replaced the two sets of three dummy variables with two ordinal variables – “goal” and “rights infringements.” The full analysis is presented in Table 3. Both interaction terms turned out to be statistically insignificant (Hawk × Goal: z = −.23, p = 0.821; Hawk × Rights infringements: z = −.93, p = 0.353), providing no support to Hypothesis 4. Put differently, there is no indication that the factual variation had systematically different effects on the judgments of hawks versus doves.³¹

Proportionality Judgments vis-à-vis Doctrinal Versions

Figure 5 is based on (2). It presents the expected probability of the operational plan being judged as proportional for the three proportionality types: “strict-sense” (the IHL rule); “full PA,” involving the “worthy goal,” “suitability,” “necessity” (LRM) and “strict-sense balancing” stages (the Targeted Killing decision); and “free PA” – the unstructured judgment. Figure 5 shows distinct differences in the probability of judging the operation as proportional under the three different proportionality formulae (χ² = 67.45, p < 0.0001). The expected probability under the “strict-sense” alternative, with all other variables set at their mean values, is 75.9%. Applying the “full PA” alternative resulted in a reduction of 24.0 percentage points (z = −8.23, p < 0.001), yielding the probability of 45.2%. The probability under the “free PA” alternative was 80.3%, significantly higher than under “full PA” (z = 9.06, p < 0.001), and 4.4 percentage points higher than under the “strict-sense” decision (z = 2.11, p = 0.035).

Figure 5. Mean probability of “proportional” judgment across proportionality formulae (with 95% CI). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Thus, although the unstructured judgment (“free PA”) was solicited last, after respondents had replied to the entire set of questions regarding all the considerations involved in the decision, it was on average more similar to the one under the “strict- sense” rather than under “full PA” alternative. The sizable effect of applying “full PA” did not come as a surprise. It is reasonable to expect that the requirement that the proposed plan should clear all the decision stages would reduce the probability of approval.

To test whether a more comprehensive proportionality analysis increases the receptivity of judgements to normatively relevant factual variation (Hypothesis 5), we fitted (4) to our data. This analysis, reported in Table 3, allowed us to estimate the differences in the effect of factual variation in the “goal” and “rights infringement” parameters under each doctrinal version (“strict-sense balancing,” “full PA,” and “free PA”). All four interaction terms are negative, suggesting that the receptivity of judgments to factual variation under “full PA” and “free PA” is lower than under the “strict-sense balancing.” However, of the four interaction terms, only the “Free PA” X Goal is statistically significant (z = −2.02, p = 0.044). These results, thus, do not allow us to reject the null hypothesis that proportionality judgments under “full PA” and under “strict-sense balancing” are equally receptive to factual variation. Figure 6 presents the predicted probability of the proposed plan being judged as proportional across the “goal” (upper panel) and “rights infringements” (lower panel) treatments, and doctrine versions. The pattern of receptivity is very similar for the “goal” and “rights infringements” factors. In both, the incline of the three slopes (for three doctrine versions) are similar, with the slope representing the “strict-sense balancing” (blue) being slightly more pronounced compared to “full PA” (red) and “free PA” (green). Thus, receptivity to factual variation appears to be somewhat higher under the less comprehensive version of PA – “strict-sense balancing”. This difference, however, is statistically insignificant.

Figure 6. The effects of “goal“ (upper panel) and “rights infringement“ (lower panel) treatments on the probability of “proportional“ judgment across proportionality types. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Finally, to test whether proportionality type affects the susceptibility of judgements to legally irrelevant preferences, we fitted (5) to our data. This analysis, reported in Table 3, allowed us to estimate the differences in the effect of ideological preferences on proportionality judgments for different doctrine versions. The interaction terms suggest that such an effect is stronger under “full PA” (albeit at a marginally significant level), and under “free PA,” than under “strict-sense balancing” (Hawk–Dove X “Full PA”: z = −1.89, p = 0.058; Hawk–Dove X “Free PA”: z = −2.56, p = 0.010). These results provide no support to Hypothesis 5, which predicted a lesser effect of legally irrelevant preferences under “Full PA” than under “strict-sense balancing.” The differences in the effect of ideological preferences under the three types of proportionality judgments are displayed in Figure 7, which plots the predicted probability of the proposed plan being judged as proportional across the set of ideological preferences and for the three doctrine versions. The three curves are similar, but the slopes of the “Full PA” (red) and “Free PA” (green) curves are slightly more pronounced than of the “strict-sense balancing” curve (blue). In other words, the effect of ideological preferences on proportionality judgments appears to be somewhat larger under the two former PA types than under the latter.

Figure 7. The effects of ideological preferences across proportionality types. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Discussion

This research is the first in the field to undertake an empirical analysis of legal proportionality judgments by experimentally testing the effects of normatively relevant factual considerations in this regard. Our experimental findings suggest that legal experts’ proportionality judgments are receptive to normatively relevant factual considerations. Respondents tended to judge the proposed plan as more proportional with the increase of the normative importance of its goal and the decrease in the infringements of human right it entailed. Additionally, in light of the strong correlational evidence obtained in the analysis, proportionality judgments appear to be influenced by adjudicators' ideological preferences. However, the results do not indicate that proportionality judgments are subject to ideologically biased information processing. Lastly, the likelihood of a plan being judged as proportional seems to be lower if the judgment results from a comprehensive proportionality analysis. Yet, contrary to our expectations, implementing a comprehensive analysis neither increased the receptivity of judgments to factual variation nor reduced the effect of ideological preferences in this regard, compared with the “strict-sense” balancing.

These results suggest that proportionality analysis appears to provide a methodological structure for legal reasoning based on factual considerations, yet the resulting decisions also reflect legally irrelevant preferences. More specifically, prior policy preferences appear to affect experts’ propensity to judge a plan as proportional; yet, factual considerations pertaining to the normative legitimacy of the goal and the potential for rights infringements quite uniformly lead them to adjust their initial judgment. The resultant legal decision is thus anchored jointly in the experts’ policy preferences and the facts of the case. As the effect of policy preferences was found to be considerable, it is reasonable to claim that proportionality judgments are dependent to a large degree on legally irrelevant preferences.

Additional research is required to assess whether the strong association between policy preferences and proportionality judgements that was found in the context of this study is generalizable to other domains in which proportionality analysis is applied. It is plausible that the existential nature of counter-terrorism policy in Israel renders greater weight to policy preferences in such decisions, while it may not be quite as influential in other more mundane policy domains (Schwartz and Sulitzeanu-Kenan 2004). Conversely, findings in previous studies have shown that policy preferences (e.g., regarding abortions in the US) were found to affect legal judgments in different domains (determining the limits of freedom of speech: Sood and Darley 2012; Kahan et al. 2012), Considering our results together with these studies may suggest that the conclusion that policy preferences play a role in proportionality judgments is not necessarily limited to the particular context examined in this experiment.

These results are consistent with previous findings on motivated cognition in other legal judgments, which have been described as a “compromise between preferred outcomes and the relevant information at hand” (Pyszczynski and Greenberg 1987: 333). Furthermore, as proportionality analysis is methodologically vague, legally irrelevant preferences could affect the legal decision via a broad range of mediating factors, not necessarily through biasing experts’ processing of factual information. This interpretation of our results conforms with earlier findings to the effect that “motivated decision makers do not bias their judgments more than what is needed to achieve their desired conclusions” (Sood 2013: 310; see also: Boiney, Kennedy, and Nye 1997).

Our results regarding judgments’ receptivity to factual variation and the influence of policy preferences under different proportionality doctrines are rather similar. Importantly, compared with the less comprehensive version of proportionality analysis (“strict-sense balancing”) or even with unstructured judgment (“free PA”), the comprehensive proportionality analysis (“full PA”) yielded a lower probability of a plan being judged as proportional. Yet, it neither enhanced the receptivity to factual considerations nor reduced the effect of legally irrelevant considerations.

As noted above, our results indicate that respondents were sensitive to only some of the specific treatments. Specifically, regarding the operational goal only the preventive (vs. punitive) nature of the operation resulted in a clear change in the mean decision, while the size of the preventable harm did not. Considering that this specific null finding may be due to the particular way this consideration was treated in the vignette, and/or sample size, along with the fact that this experiment assesses proportionality judgments in one specific legal context, these results lead us to cautiously suggest that respondents appear to be more sensitive to “rights infringements” considerations in forming proportionality judgments compared to “goal worthiness” properties. However, more research regarding the nature of these decisions is needed in order to garner sufficient certainty and a clearer picture regarding the generalizability of these results.

It should be noted that the different doctrine versions were measured for each subject based on his/her set of answers. While this methodology does not seem to present a problem for the validity of full proportionality judgments (“full PA”),³² we cannot directly determine whether decisions under “strict-sense balancing” would have been the same had respondents been presented with the latter version alone. The main concern here is that basing the decision on a single stage (unlike our method) may cause respondents to be more conservative in their judgments, that is, less likely to judge the plan as proportional. While this concern cannot be wholly eliminated, it seems to be less prominent in the present study, insofar as responses to the items in the unstructured proportionality judgment (“free PA”), presented after the “strict-sense balancing” item, were very similar to the responses to the latter item. This may indicate that respondents treated the “strict-sense balancing” decision prong as pivotal in their decision.

Considering that this article reports the first empirical study of proportionality judgments, we should be cautious in drawing broader conclusions from its findings. However, our results do allow some tentative implications for the legal theory of proportionality. The first relates to incommensurability – which is one of the central points of criticism leveled at the proportionality method – namely, the qualitative differences between competing values which militate against placing them on a common scale (Frantz 1963; Petersen 2013).³³ That said, the receptivity of experts’ proportionality judgments to factual variation was relatively uniform even though their policy preferences were not divergent.³⁴ This indicates that, in practical terms, the problem of incommensurability here is not as serious as could have been expected. Conversely, the strong influence of legally irrelevant preferences demonstrated in the analysis highlights the highly vague nature of proportionality judgments. This finding compromises the idea of proportionality analysis as an objective and neutral method of constitutional adjudicating (Beatty 2004; and for a critical view of this claim see: Jackson 2004), and may support the argument that balancing should be relegated to the political domain (Petersen 2013: 1392–3).

Stronger conclusions would require further empirical work on the nature of this central legal principle. This study is only a first attempt in this direction, and moreover, it has a number of notable limitations. First, it focuses on one policy domain, in one national setting. Second, while the data are based on decisions of legal experts, the experimental setting involves a set of hypothetical scenarios, and the results cannot be safely extrapolated to real-life situations with tangible implications. It would be plausible to assume that, relative to a real-life situation, the obtained effects of factual variation are underestimations, while of ideological preferences – overestimations. If this is indeed the case, our conclusions regarding the effect of ideological preferences, which was found to be very strong, are likely to prove robust in realistic settings as well.