Legal Disagreement

Abstract

Individuals within the same jurisdiction often have different perceptions of the legal system, a phenomenon we call legal disagreement. Cross-country analyses of legal institutions generally ignore this kind of within-country variation. This article defines the concept of legal disagreement, identifies its potential causes and consequences, and shows that it can be measured using data from cross-country surveys. We argue that legal disagreement is likely to be caused by differences in individuals’ sources of information about the legal system, how the system treats cases of a given type, or how individuals process legal information, all of which may be related to differences in social position. We also suggest that understanding why individuals disagree about legal systems is valuable for several reasons, particularly when the disagreement reflects systematic differences in beliefs across people in different social positions. We use data from World Justice Project population surveys to create quantitative measures of legal disagreement for 128 countries or jurisdictions. We identify correlations between legal disagreement and measures of its potential causes and consequences, such as perceptions of discrimination and ethnic fractionalization. We also provide evidence that those variations are driven by differences in perceptions that track differences in gender, financial well-being, and class.

“The law consists of a complex repertoire of meanings and categories understood differently by people depending on their experience with and knowledge of the law.” (Merry 1990, 5)

INTRODUCTION

There is an old joke that “if you want three opinions ask two lawyers.” If even sophisticated observers tend to disagree about what the law requires, what about ordinary people—when do they disagree about the legal consequences of actions? This question has received little attention in the social sciences, particularly from scholars who rely upon quantitative research methods. Not all studies pay attention to ordinary people’s perceptions of the law, and those that do tend to focus on identifying commonalities rather than differences. Thus, most existing quantitative studies measure central tendencies rather than variation. Similarly, cross-country studies of how law is perceived invariably focus on variations across rather than within jurisdictions.

In this article we define the concept of legal disagreement, identify its potential causes and consequences, and show that those relationships can be investigated using data from cross-country surveys. Our theoretical framework suggests that it is particularly valuable to understand legal disagreement when it reflects systematic variations in beliefs across individuals in very different social positions. Accordingly, our empirical analysis focuses on whether legal disagreement is associated with differences across social groups.

We define the concept of legal disagreement broadly to encompass disagreement about the legal consequences that a legal official will assign to a particular set of facts. This definition applies to beliefs about virtually any type of legal official and so is broad enough to capture disagreement about how a particular constellation of facts will be dealt with not only by appellate courts but also by enforcement officials or customary authorities. We adopt this approach because in many scenarios individuals’ attitudes and behavior are strongly influenced by their beliefs about the actions of officials outside the judicial system. For example, most drivers’ behavior is likely to depend more on their beliefs about how police officers enforce speed limits than their understandings of the speed limits established by judges or legislators.

In our theoretical analysis we argue that legal disagreement may be both a consequence and an indirect cause of differences in individuals’ social positions. To begin, we claim that legal disagreement may be caused by differences in the sources of information that people use to form beliefs about the legal system (perspectives), how the system treats similar cases (treatment), or how people process the information that they possess (biases). We suggest that differences in perspectives, treatment, or biases may be explained by variations in social position. We go on to suggest that legal disagreement may have important consequences. For instance, it may be an impediment to negotiated settlement of disputes out of court. Moreover, when legal disagreement tracks significant social divisions, it might be associated with inequality, conflict, and polarization, which might in turn reinforce the kinds of social divisions that lead to legal disagreement.

The final part of the article is devoted to an empirical analysis. The first step is to show that legal disagreement can be equated with variation in responses to certain questions contained in the World Justice Project’s (WJP) multi-country surveys. We also show that it is possible to use data from the WJP surveys to create quantitative measures of certain forms of legal disagreement using data on enforcement of criminal laws as an illustration. We find that the resulting legal disagreement scores vary across areas of law and countries. Next we examine whether legal disagreement is associated with indicators of differential legal treatment and social division. We begin by examining whether variations in the level of legal disagreement across countries are associated with country-level indicators of factors such as discrimination on the part of legal officials and ethnic fractionalization. Then we examine whether divergence between individuals’ perceptions of the performance of a legal system typically can be explained by differences in their social positions. We present preliminary evidence that variations in individuals’ perceptions track differences in gender, financial well-being, education, employment, and political engagement.

RELATED LITERATURE

This essay was written in honor of Sally Merry. As the epigraph suggests, Merry was keenly aware of the value of exploring ordinary people’s understandings of the law, the fact that those understandings might vary from one person to another, and the possibility that those variations might be related to factors such as wealth and class (Merry 1985). Merry was also deeply interested in how understandings of law varied from one place to another and conducted research on a global scale. At the same time, she was wary of the simplifications necessarily entailed in efforts to quantify social phenomena, including phenomena such as perceptions of law (Merry 2016). Our project brings together all of these themes in Merry’s work.

To be more specific, the motivation for our project is similar to that of Merry and other scholars who have sought to map legal consciousness (surveyed in Silbey 2005). Many of those studies focus on identifying points of consensus and shared meanings, but here we focus explicitly on dissensus. In that sense our study is most closely related to previous research on associations between social location or status on the one hand, and legal consciousness on the other hand (see, for example, Nielsen 2000). The focus of our study is, however, significantly narrower than that of typical studies of legal consciousness. We focus on factual beliefs about the law, specifically, beliefs about what legal officials will do. By contrast, the concept of legal consciousness also encompasses “attitudes” about the law, which include normative beliefs or emotional reactions provoked by beliefs about legal officials’ actions. Legal consciousness also extends to the meanings that people assign to official actions, where the process of assigning meaning involves situating beliefs and attitudes regarding the law in relation to a rich set of symbols, narratives, and values. Finally, the term legal consciousness is sometimes defined even more expansively to include actions undertaken in response to beliefs, attitudes, and meanings regarding the conduct of legal officials. Therefore, documenting ordinary individuals’ beliefs about what legal officials will do is merely the beginning of an analysis of the much more complex phenomenon of legal consciousness (Silbey 2005, 357).

Our analysis is also related to a separate literature on determinants of perceptions of corruption in public institutions. That literature is surveyed in Gonzalez, Mackenna, and Muñoz (2019). Those authors’ own analysis is premised on the idea that individuals’ perceptions of corruption are shaped by information derived from personal experience as well as sources such as the media. They hypothesize that personal experience with corruption is likely to be particularly influential in countries where corruption is generally perceived to be low and that the media are likely to be particularly influential in countries where press freedom is high and among people who are politically engaged. Our own theoretical analysis is broadly consistent with this framework.

Innumerable studies of domestic legal systems consider variations within countries, but without comparisons across national borders. Cross-country comparisons of legal systems are mainstays of the literature on comparative law but those studies typically treat national legal systems as homogeneous units. Our argument that cross-country comparisons ought to take into account legal disagreement echoes Munshi (2015) who argues that comparative legal analysis should account for the heterogeneity within legal systems. Referring to the United States, she claims that it “continuously produces different kinds of legal subjects who stand in differing relations to both the state and the nation.” (Munshi 2015, 665–66).

Our effort to quantify legal disagreement extends the literature on the construction of legal indicators (see generally, Davis 2014). Those indicators typically aggregate the perceptions of many individuals to provide a single score for each jurisdiction. Merry and other scholars have commented on the fact that such legal indicators simplify complex aspects of reality (Merry 2016; Arndt and Oman 2006), and others have raised concerns about the simplification entailed in aggregation across subnational units (Davis 2018) or areas of law (Davis 2014).

The implications of aggregation across private actors within countries have attracted less attention in the literature on indicators, but there are several noteworthy exceptions. Previous efforts to quantify within-country variation in perceptions of legal institutions either focus on different institutions from our study or consider a more limited set of interpretations of the variations they document. Enriquez, Sybblis, and Centeno (2017) treat lack of subnational variation in the delivery of public services such as water, sanitation, electricity, and education as a measure of state capacity but they do not directly examine variation in the provision of legal services. Hallward-Driemeier and Pritchett (2015) document significant within-country variation in firms’ responses to questions about the time required to complete certain legal procedures. They interpret this variation as evidence of differential treatment of firms by legal officials. Arruñada (2020) also focuses on the legal system and draws attention to divergence between the perceptions of individuals who have direct experience with a particular aspect of the legal system. He interprets this divergence as evidence of variations in the quality of information available to individuals, presuming that experienced individuals are better informed. Like Arruñada, we focus on variation in perceptions across individuals rather than firms. In fact, our empirical analysis relies on data from the same source. However, unlike the studies discussed in this paragraph, we consider multiple explanations for variation in individuals’ perceptions of legal institutions.

Finally, legal disagreement can be characterized as a form of social polarization. This implies that research on the causes and consequences of legal disagreement might both draw upon and extend literature on the causes or consequences of other forms of polarization. For instance, there are competing views on the relationships between political polarization and economic inequality. There is cross-country evidence that inequality is positively correlated with mass political polarization (Grechyna 2016). In addition, theory suggests that political polarization is both caused by and causes inequality (McCarty, Poole, and Rosenthal 2006), a claim that is supported by evidence from the United States on the relationship between state-level income inequality and polarization within state legislatures (Garand 2010; Voorheis, McCarty, and Shor 2015). By contrast, using data from a compilation of national surveys, Iversen and Soskice (2015) argue that there is a negative correlation between inequality and political polarization in advanced democracies and that both variables are influenced by the distribution of political knowledge. Our analysis draws on this literature to the extent that it further explores the relationships between economic inequality, the distribution of legal information, and legal disagreement. It also contributes to the existing literature by exploring relationships between legal disagreement and political polarization. In addition, our theoretical analysis draws on the literature on the relationship between ethnic divisions and conflict to suggest that legal disagreement, to the extent that it is correlated with ethnic polarization, may exacerbate ethnic fractionalization and conflict (Esteban, et al. 2012).

THE CONCEPT OF LEGAL DISAGREEMENT

Following Davis (2011), we define legal disagreement as divergence in actors’ beliefs about the legal consequences that a legal official will assign to a given set of facts. For these purposes, a legal official includes anyone from a judge to a police officer to a food inspector to a chief—a paper in honor of Sally Merry is the last place to reject legal pluralism (see Merry 1988). The relevant legal consequences will vary depending on the context. A judge in a criminal proceeding may have to choose between entering a verdict of acquittal or conviction. A judge in a civil proceeding may have to decide on the quantum of damages. Meanwhile, a police officer may have a much broader range of options. For instance, they may be in a position to decide whether to stop a pedestrian, what tone, and how much force to use in the course of the encounter, whether to search, whether to make an arrest, and, in the case of an arrest, what charges to recommend.

Our definition of legal disagreement assumes that the relevant actors have common information about a fact pattern that will be presented to a particular type of legal official, are aware of the possible legal consequences, and assign different probabilities to at least one consequence. Take for example, a case of an alleged crime, where the possible legal outcomes are conviction and acquittal. After hearing the evidence at trial, two observers might assign different probabilities to acquittal. That would be an instance of legal disagreement.

CAUSES OF LEGAL DISAGREEMENT

We hypothesize that people form beliefs about the legal system by extrapolating from their existing knowledge bases (Gonzalez, Mackenna, and Muñoz 2019). A knowledge base includes both general information about how the world works and specific information about the past behavior of legal officials. That information may be derived from personal experience, interpersonal interactions, mass media, or social media. Since no two people have lived exactly the same life or been exposed to exactly the same information, everyone has a different knowledge base. Moreover, people with very similar knowledge bases might use different techniques to extrapolate from them. In this framework, legal disagreement is caused by both differences in people’s knowledge bases and differences in their reasoning processes.

We are particularly interested in what may explain differences in people’s knowledge bases. One way to think about this question is to imagine that each observer of the legal system looks at it through a narrow window that offers a specific perspective, meaning that it reveals only a subset of all the potentially available information about legal officials’ actions, and then extrapolates from their observations to the entire system. Some windows might only offer views of certain officials, others of actions that affect certain types of people, and others might only show actions that lead to certain types of outcomes; still others might only reveal aggregate information about the system. In this kind of setting we can ask: what factors determine the likelihood that members of a population will observe different sets of actions?

One determinant of the likelihood that people have divergent knowledge bases will be the number of distinct—i.e. non-overlapping—perspectives. If everyone is looking through the same window they will have the same view. In practice, this may occur if everyone relies on the same media sources for information about the legal system. For instance, suppose that everyone in a society relies exclusively on the coverage in a single newspaper of record, which they read religiously, for their knowledge of the legal system. In this extreme scenario, everyone will have the same knowledge base. By contrast, divergence is likely to be much greater in a society in which everyone relies on different sources for information about the legal system and each source offers very different coverage. An analogous scenario occurs when many observers rely on either their own personal experiences or those of people close to them for legal information. All of this suggests that there is likely to be less variety of perspectives in a country like China, where there are relatively few distinct media sources, than in a country such as Canada, where there is a large number of media sources (Noam 2016, 1021–29). In societies where people in different social positions systematically rely on different sources of information, differences in perspectives will track differences in social positions.

Even if people have different perspectives on the legal system, they might, at least in principle, have the same views because the legal landscape is highly uniform. In other words, if every window looks out on officials who are making the same decisions, then everyone will have the same knowledge base. By contrast, if the legal system is highly variegated then differences in perspective will matter. It seems plausible that uniformity is more likely to occur in systems such as the Japanese court system, where legal officials operate within a highly centralized and hierarchical structure, are demographically homogeneous, and have all received the same training (Ramseyer and Rasmusen 2001). It also seems plausible that legal systems that govern relatively homogeneous populations are more likely to operate uniformly.

Finally, people with the same knowledge base may still have divergent beliefs simply because they process the information they have received differently, that is to say, they have different biases. For instance, people who look at the world through the proverbial rose-colored glasses may be relatively optimistic about the performance of the legal system, even when they have the exact same information as more pessimistic people. The same may be true of people who have personal ties to legal officials. This last point suggests one of many reasons why biases might be correlated with social position.

RAMIFICATIONS OF LEGAL DISAGREEMENT

The existence of legal disagreement has ramifications for anyone who might be interested in either evaluating a society and its legal system or explaining how members of the society interact with one another. To the extent they rely on data on legal beliefs, these kinds of analyses typically rely on national averages. By definition, the extent of legal disagreement determines the value of measures of central tendency as guides to a population’s legal beliefs. The greater the level of disagreement, the more a randomly selected member of the population’s beliefs can be expected to diverge from the average and the less useful the average will be for the purposes of either evaluation or explanation.

Evaluation

Understanding how legal beliefs vary across a population can be important in assessing the welfare of its members. For example, suppose that both prospective muggers and a subset of the general population believe that muggers are highly likely to be apprehended by the police if they attempt a mugging in a public park, while other members of the population are more skeptical of law enforcement. Individuals who perceive law enforcement to be more effective will enjoy the benefit of using parks without fearing for their safety, while those who are more skeptical of law enforcement will be more fearful.

Legal disagreement that involves systematic differences between members of socially distinct groups might either reinforce or mitigate existing inter-group disparities in welfare. If, for instance, women systematically have less confidence in law enforcement than men, then women will systematically enjoy less of a sense of physical security than men. Importantly though, if those women’s perceptions of law enforcement and its effects on crime are more accurate than men’s, then they will enjoy greater actual security.

Variation in legal beliefs can also be important in evaluating the performance of a society’s legal system in terms of criteria such as predictability or respect for the rule of law. As Joseph Raz (1979, 214) has explained, a central virtue of a legal system is that it is capable of guiding people’s behavior. This in turn means that the content of the law must be open, clear, and adequately publicized. Otherwise, people will be misled or confused about what the law requires, and it will fail to serve its purpose of guiding behavior. On this view, disagreement about how judges are likely to rule in a particular case suggests that at least some members of society are misled or confused about what the law requires. In this sense, legal disagreement is an indicator of at least partial absence of the rule of law.

To the extent that legal disagreement is driven by non-uniform treatment of similar cases, it may be an indicator of invidious discrimination. This arises, for instance, when members of different genders, ethnic groups or classes disagree about how the law will be applied because they have had different experiences at the hands of legal officials in otherwise similar circumstances. Similarly, legal disagreement may reflect perceptions that people with different political affiliations are treated differently. We stress, however, that uniform outcomes and the absence of legal disagreement do not necessarily mean that a legal system is virtuous. Systems in which the state or the wealthier party always wins will lead to highly uniform outcomes and relatively little legal disagreement, but those systems clearly should not be regarded as ideals.

Explanation

Legal disagreement can also explain certain attitudes and behaviors that tend to be influenced by the interactions of multiple actors’ beliefs about the law. An important example is trade in legal entitlements, as in cases where parties have the opportunity to negotiate a settlement of a legal dispute before trial. The extent of legal disagreement will be an important determinant of whether the parties find it mutually beneficial to settle (Priest and Klein 1984). Take for example two risk-neutral actors who disagree about whether litigation over an unpaid loan is likely to be resolved in favor of the creditor. Suppose that shortly before trial, the debtor offers to settle the dispute—i.e. to buy the creditor’s right to continue with the lawsuit—by paying 50 cents on the dollar. Leaving aside litigation costs, if the creditor believes that it has a 90 percent chance of prevailing then it will not agree to release its claim for less than 90 cents on the dollar. Meanwhile, if the debtor believes the creditor has only a 50 percent chance of prevailing, then it will not be willing to offer more than 50 cents on the dollar. A similar result will arise if a third party who shares the debtor’s assessment of the case attempts to purchase the creditor’s claim. In this illustration, legal disagreement (as to the likely outcome of a judicial proceeding) thwarts trade because it leads the seller to value the entitlement more highly than the buyer. Legal disagreement of the opposite kind, which causes the seller to assign less value to the entitlement than the buyer, would encourage trade.

Legal disagreement can also influence larger-scale social interactions such as political behavior. One reason is that beliefs about legal outcomes can provide a basis for individuals’ evaluations of the overall performance of legal institutions. For example, someone who believes that a court is unlikely to award full compensation to a homeowner whose house has been destroyed by the government probably has views about whether that outcome is desirable. The same is true of a person who believes that the court is highly likely to award full compensation. In fact, both people might agree that full compensation ought to be awarded. However, only the first person will believe that legal reform is required to secure property rights. Therefore, even when people evaluate legal officials’ actions against a common set of normative standards, in the presence of legal disagreement they might disagree about what sorts of legal reforms are desirable. This may in turn lead to disagreement about which political candidates or parties to support, with concomitant effects on voting behavior and political polarization.

Legal disagreements that track boundaries between groups might exacerbate inter-group tensions by adding disagreements about legal reforms to the set of issues about which they disagree. For example, if members of different ethnic groups systematically disagree about the propensity of the police to use excessive force, then they are likely to disagree about a range of related public policies, such as levels of funding for the police, and the political candidates who associate themselves with those policies (Weitzer and Tuch 2004). This in turn might lead to greater ethnic polarization and, potentially, conflict. Similarly, if legal disagreement of this sort tracks prior political affiliations, it has the potential to increase political polarization.

EMPIRICAL ANALYSIS

A straightforward way to measure legal disagreement is through surveys that present people with hypothetical scenarios then ask how likely it is that legal officials will respond in certain ways. Legal disagreement can be defined as variation in the responses to such questions and can be quantified by measuring the standard deviation of those responses. We illustrate this approach using data from surveys conducted by the WJP.

An important limitation of surveys like the WJP survey is that they inevitably fail to capture the complexity of actual legal disputes (Merry 1990, 6). Hypotheticals are necessarily thinly specified and so allow respondents to speculate about aspects of the fact pattern that are not explicitly presented yet might be material to their predictions of how legal officials will act. Respondents’ speculations may lead them to different conclusions about the facts, in violation of our assumption that they have common information. For example, the WJP survey’s hypotheticals fail to specify the gender of the protagonists, even in situations in which it seems plausible that gender will be material to respondents. Accordingly, variation in responses may reflect either disagreement about facts (factual disagreement), disagreement about how legal officials will respond to a given set of facts (legal disagreement), or some combination of the two.

With these important caveats in mind, we examine whether disagreement at the country level is correlated with country-level indicators of social polarization, unequal access to legal information, or differential treatment by legal officials. We then turn to individuals’ responses to the WJP survey to gain insight into whether legal disagreement reflects systematic differences between recognized groups. All of our analyses rely on a pooled dataset that includes the responses from all countries.

Data

Our analysis relies upon multiple data sources, but our main dataset comes from the WJP’s Rule of Law Index Questionnaire. The questionnaire is used in surveys conducted periodically in countries around the world. The results are used to construct the WJP’s Rule of Law Index (World Justice Project 2020). The 2020 edition of the Index relies on surveys of more than 130,000 households and 4,000 legal practitioners and experts in 128 countries and jurisdictions to measure how the rule of law is experienced and perceived worldwide. The surveys comprise two types of questionnaires: some completed by topical experts and others by the general population. Our analysis here relies exclusively on data from the general population poll (GPP) (World Justice Project 2019).

The methods used to collect data for the GPP and the timeframe during which data were collected vary from one country to another. Most notably for present purposes, in early editions of the Rule of Law Index the GPP was conducted in the three largest cities of each country or jurisdiction, but in 2019, the WJP began shifting to nationally representative polls. In the 2020 edition, the WJP conducted nationally representative polls in sixty-three countries and jurisdictions. Nationally representative polls will be conducted in the remaining countries and jurisdictions in future editions of the Index. In addition, depending on the country or jurisdiction, one of three different polling methodologies is used: face-to-face, telephone, or online. The survey data used in the 2020 Rule of Law Index Report were collected during the fall of 2019 (for ten countries and jurisdictions), fall of 2018 (for seventy countries and jurisdictions), fall of 2017 (for forty-five countries and jurisdictions), the fall of 2016 (for four countries and jurisdictions), the fall of 2014 (for three countries and jurisdictions), the fall of 2012 (for one country), and the fall of 2011 (for two countries or jurisdictions). We present the entire distribution of surveys by year, methodology, and coverage in Table A1of the Appendix.

We constructed measures of legal disagreement and legal uncertainty from individuals’ responses to GPP questions about the likelihood of a particular legal outcome being realized in a hypothetical scenario. Here we focused on the following questions:

1. o. Homicide: If someone commits a homicide in your neighborhood, how likely is it that the criminal is prosecuted and convicted? (from “very likely”=1, “likely”=2, “unlikely”=3, to “very unlikely”=4) (World Justice Project 2019, q45b_G2)

2. p. Corruption: If a police chief is found taking money from a criminal organization, such as a drug cartel or an arms smuggler, how likely is this officer to be sent to jail? (from “very likely”=1, “likely”=2, “unlikely”=3, to “very unlikely”=4) (World Justice Project 2019, q45c_G2)¹

For each question and each country or jurisdiction, we calculate the level of disagreement, defined as the standard deviation of the responses of individuals in the country or jurisdiction. Following Davis (2011), we also calculate a measure of legal uncertainty for each question, at the levels of both individuals and countries or jurisdictions. For individuals, the measure of legal uncertainty is a dummy variable which takes a value of “1” if the individual responded “likely” or “unlikely”—implying probabilities closer to maximal unpredictability—as opposed to “very likely” or “very unlikely.” At the country or jurisdiction level, the measure of legal uncertainty is the proportion of respondents who answer either “likely” or “unlikely.”

The survey questions about homicide and police corruption capture respondents’ beliefs about the likely behavior of legal officials in settings in which there is unlikely to be disagreement about the applicable formal legal norms—it seems safe to presume a consensus that the law formally prohibits homicide or corruption. We regard the relatively narrow focus of the questions as a virtue rather than a defect since it means that our measures of disagreement capture perceptions about a relatively limited set of legal officials. At the same time, we would welcome efforts to develop measures of disagreement that focus on beliefs about formal legal norms (such as the maximum sentence the law provides for homicide or corruption). Such measures would effectively capture beliefs about the actions of legislators, and perhaps regulators and appellate judges, as opposed to enforcement officials.

Although our measures are designed to capture legal disagreement, they undoubtedly capture a certain amount of factual disagreement because the underlying survey questions may bring different fact patterns to mind for different respondents. The question about homicide explicitly invites responses based on a homicide “in your neighborhood” and so responses from people in different neighborhoods will be based on different fact patterns. Meanwhile, the question about police corruption invites people to report their beliefs about two distinct fact patterns, one that involves drug trafficking and one that involves arms smuggling. In addition, as noted above, these sorts of hypotheticals necessarily omit many details, such as the gender and ethnicity of the actors involved, and so are open to multiple interpretations.

There is another sense in which the questions we use in our analysis are less precise than would be ideal. The responses to both the homicide and corruption questions will reflect beliefs about the activities of several different types of legal officials, including, police officers, prosecutors, judges and, perhaps, juries. For many practical purposes, it would be useful to have data on perceptions of specific types of legal officials. Consider, for example, a variation on the question about homicide that asks, “If someone commits a homicide in your neighborhood and was arrested and prosecuted, how likely is it that the criminal is prosecuted and convicted?” The highlighted changes make it clear that there is no need to speculate about the actions of police or prosecutors and so the responses presumably would focus on beliefs about what judges and juries will do. Again, we would welcome efforts to produce measures of disagreement about the actions of more narrowly defined groups of legal officials.

Our WJP data are limited to cases involving the criminal justice system, even though our claims about the causes and consequences of legal disagreement apply to other types of cases. Unfortunately, the most recent version of the WJP population survey contains only a handful of questions that elicit beliefs about the legal ramifications of hypothetical scenarios in a form that can be used to calculate a measure of legal disagreement. Of those questions, some are suitable in form but define the scenario too imprecisely to be useful—they refer simply to public officials whose actions are “clearly against the [country’s] Constitution” or “clearly illegal and unfair” (WJP Questionnaire q45b_G1, q45a_G2). We could have constructed measures of legal disagreement from the responses to questions about whether small business owners that operate without the required documentation or fail to register to pay taxes are likely to be fined (WJP Questionnaire q45d_G1, q45e_G1). However, since the purpose of this article is mainly to illustrate the potential applications of our measure of legal disagreement—essentially, a proof of concept—we limit our analysis to the questions on homicide and corruption.

The WJP provides a wealth of additional data on individual respondents, and we only tap a fraction of the dataset’s potential in this article. The first part of our analysis relies on the WJP’s data on individual respondents’ experiences with various components of the legal system, perceptions of the access to legal information, and differential treatment, all aggregated at the country level. In the second part of our analysis we rely on demographic data that the WJP collects about individual respondents, specifically gender, age, financial well-being (quintile), education (highest degree received), employment (whether they had a job in the past week), and whether they live in an urban or a rural area

We also include data on partisanship, based on responses to the question “Do you usually think of yourself as close to any particular party? (no=0, yes=1).” Around one quarter of our sample (24.8 percent of all respondents) answered “yes” to this question. As discussed at greater length below, responses to this question arguably qualify as a measure of engagement with the political process.² It is important to keep in mind, however, that this measure does not capture political orientation, meaning how favorable or unfavorable an individual’s political engagement is or how supportive of the state or government they are.³

In addition to the WJP dataset, we rely on other data measured at the country or jurisdiction level on the depth of potential social cleavages. To begin, we use World Bank data on Gini coefficients as a measure of income inequality. The data are available for 264 countries and regions, though not for every year in every country. We use the most recent year of data available for each country between 2015 and 2019. We also use a measure of the extent of ethnic divisions that is frequently used in studies of the determinants of civil conflict, the ethnic fractionalization index compiled by Fearon (2003), which reflects the probability that two individuals randomly selected from a country’s population will belong to different ethnic groups. The version of the ethnic fractionalization data we use comes from Arbatli, et al. (2020) and is available for 155 countries.

The Extent of Legal Disagreement

Figures 1 and 2 and Table 1 show the relative standing of each country in terms of legal disagreement for questions about homicide and police corruption respectively. Figure 3 shows the scores for each country on a single scatterplot. Figures 4, 5, and 6 show the patterns of responses that generate these scores in select countries with low, moderate and high levels of disagreement.

FIGURE 1. World Map of Legal Disagreement Ranking for Homicide.

FIGURE 2. World Map of Legal Disagreement Ranking for Police Corruption.

TABLE 1. List of Countries and Their Legal Disagreement Score for Questions on Homicide and Police Corruption

FIGURE 3. Scatterplot of Legal Disagreement Scores by Country for Question on Homicide and Police Corruption.

FIGURE 4. Distribution of Responses in Selected Jurisdictions with Low Levels of Disagreement.

FIGURE 5. Distribution of Responses in Selected Jurisdictions with Moderate Levels of Disagreement.

FIGURE 6. Distribution of Responses in Selected Jurisdictions with High Levels of Disagreement.

A low level of legal disagreement implies a consensus among respondents, but the consensus can be around any set of beliefs. For instance, Figure 4 shows that Hong Kong and China manifest low and moderate levels of legal disagreement respectively, but in the latter not only is the consensus weaker it is around a lower level of accountability. Table 1 reports the extent to which the distribution of responses in each country is skewed toward higher or lower levels of accountability.⁴ Figures 7 and 8 provide a graphical overview of these tendencies by plotting the levels of disagreement and the mean perception of the likelihood of a wrongdoer being held accountable for each country. No country displays a strong consensus that accountability will be low. In the case of homicide, low levels of disagreement tend to arise in countries where most people are optimistic about accountability. Otherwise, there is no particular relationship between disagreement and perceptions of accountability.

FIGURE 7. Scatterplot of Legal Disagreement and Mean Responses by Country for Question on Homicide.

FIGURE 8. Scatterplot of Legal Disagreement and Mean Responses by Country for Question on Police Corruption.

Table 1 and Figure 3 show that in many countries, the level of disagreement, both in absolute terms and relative to other countries, varies significantly depending on the survey question. For instance, in the case of homicide there is much higher disagreement in certain majority-Muslim countries such as Iran, Pakistan, and Afghanistan, as well as in several African and Latin American countries. Conversely, in relation to police corruption legal disagreement in most countries in Latin America, the Middle East, and Africa is relatively low, while the opposite is true in Western countries such as Canada, Sweden, and the United Kingdom. The fact that levels of legal disagreement can vary substantially from one area of law to another suggests that perceptions of individual components of the legal system are not driven solely by perceptions of or attitudes toward the state as a whole.

Correlations with Country Characteristics

Our theoretical framework predicts that legal disagreement measured at the country or jurisdiction level will be correlated with the extent of social divisions that lead people to receive significantly different information (from direct personal experience or otherwise) about various aspects of the legal system, including the prevalence of differential treatment.

Tables 2 and 3 show the relationships between various factors measured at the country or jurisdiction level and levels of disagreement in relation to homicide and police corruption (Tables 2 and 3).⁵ Each table shows results of regressions with multiple specifications. The sample sizes in these specifications vary widely because data on Gini coefficients and political partisanship are not available for the full list of countries covered by the WJP dataset. As a result, it is not clear whether sample size is driving the change in statistical significance in some of our explanatory variables of interest.

TABLE 2. Country-level correlates of legal disagreement about homicide

*** p<.01, ** p<.05, * p<.1

Note: The first regression specification covers four independent variables that are all “access to Justice” survey questions from the WJP survey that were aggregated at the country level by computing mean responses. The second and third specifications adds additional WJP survey questions that were not available for all countries. The third specification adds a measure of ethnic diversity that is sourced from different datasets which brings down the sample size. Lastly, Gini Index is added as an explanatory variable in the last specification, also sourced from a different dataset.

TABLE 3. Country-Level Correlates of Legal Disagreement About Police Corruption

*** p<.01, ** p<.05, * p<.1

Note: For a complete explanation of each regression specification, please refer to the note beneath Table 2.

Overall, the results of this analysis shed little light on the validity of our theory. Perceptions that the criminal justice system discriminates based on location are associated with higher levels of disagreement about resolution of the homicide case in most specifications.⁶ This finding makes sense if we presume that respondents based their beliefs about legal outcomes on relatively local experiences. In that case, discrimination would lead to greater variation in the local experiences that form the basis for respondents’ beliefs, which would in turn lead to greater disagreement. This interpretation is especially plausible in relation to legal disagreement about homicide because that measure is based on responses to a question about accountability for a homicide “in your neighborhood.”

It is unclear what to make of the fact that our other measures of discrimination are not significantly related to legal disagreement. On the one hand, these findings weigh against the hypothesis that differential treatment leads to legal disagreement. On the other hand, our findings are consistent with the idea that legal disagreement is associated with multiple forms of differential treatment that involve discrimination based on location.

The findings that ethnic fractionalization and income inequality are associated with legal disagreement about homicide, at least in some specifications, are also consistent with our theory that legal disagreement is correlated with social divisions. The fact that none of these variables explain disagreement about police corruption shows that legal disagreement is a phenomenon that varies from one legal context to another. Our findings do not support the hypotheses that differences between people with and without personal experience with legal or political institutions either cause or are caused by legal disagreement, at least not in relation to either homicide or corruption.

The variable measuring whether laws are explained in plain language captures accessibility of information about the activities of legislators and regulators as opposed to enforcement officials. The fact that it has relatively little correlation with disagreement about how well-known prohibitions on homicide and corruption will be enforced is unsurprising. However, we would expect this variable to be more powerfully correlated with other forms of legal disagreement.

Correlations with Individual Respondents’ Characteristics

Our theoretical framework predicts that legal disagreement may be associated with people in different social positions having different perspectives on, or being treated differently by, the legal system. More precisely, the overall level of legal disagreement in a society will depend on the strength of the correlation between social position and beliefs and the distribution of people across social positions. For example, suppose that high-status people tend to be more optimistic about accountability than low-status people. In that case, the level of disagreement in any given society will depend on both the strength of the correlation between status and beliefs and the proportion of high-status people in the society.

In order to shed light on whether legal disagreement can be traced back to differences in individual characteristics we examine whether people in different social categories tend to have different legal beliefs. This represents only a crude test of our theory because an individual’s social position depends on both whether they fall within a particular social category and the significance of that category in a given society.

We regress individuals’ gender, financial well-being, employment status, and education, as well as whether they live in an urban or rural area, on their responses to questions about the likelihood that a particular legal outcome will occur. We use an OLS regression with clustered standard errors at the country level. The results are shown in Table 4.⁷ Men, wealthier respondents, those with more education, and partisan individuals all are more likely to believe that the perpetrator will be held accountable, in cases of either homicide or police corruption. The opposite is true for women, and there is no statistically significant relation between employment status and expectations of accountability.

TABLE 4. Assessment of Demographic Characteristics and Perceptions of Accountability

*** p<.01, ** p<.05, * p<.1

Note: Lower values for the dependent variables reflect higher expectations of accountability. “Sex” is a dummy variable that takes the value of “1” for male respondents. For “Household financial situation” respondents are asked to rate their family’s financial condition based on the type of goods and needs they can afford (from “insufficient to afford basic goods”, to “can afford luxury goods”). “Occupied” is a dummy variable that takes the value of “1” if someone is either employed in the formal labor market or is an independent professional (entrepreneurs, day laborers, and others are excluded). “Degrees of Education” range from “None” to “Graduate School”. “Partisanship” is a dummy variable measuring whether a respondent identifies with a political party in their country. “Urban” acquires a value of 1 when a respondent reports living in an urban area, and zero when they report living in a rural area. All (OLS) regressions include country fixed effects.

The results concerning gender, financial well-being, and education are consistent with a theory that higher status individuals are likely to have a “rosier” outlook on life, which translates into greater confidence in the legal system overall. However, it is challenging to interpret the finding that partisanship is associated with higher expectations of accountability, in the cases of both homicide and police corruption, without knowing more about individuals’ political orientations and the significance of being “close” to a political party in any given country.⁸ It could be that partisanship relates to the level of engagement with political (and legal) institutions in one’s country, which makes partisan individuals better informed in their views of their legal system.⁹ Alternatively, it may be the case that most partisan individuals in this survey have political orientations aligned with those of the current regime. Political alignment may translate into favorable views of the state either because it breeds expectations of preferential treatment and access to power or because it inspires generalized optimism about the functioning of the state, including its legal institutions. In terms of our visual analogy, the challenge is to determine whether partisan individuals have windows on the legal system that offer a broader and more informative view, or overlook a more variable portion of the landscape, or tend to focus on different features.

One important caveat is that these results might be influenced by biases introduced by the previously discussed variations in how and where the underlying data were collected. Our contacts at the WJP suggested that decisions about modes of data collection generally were driven by financial considerations, but we cannot rule out the possibility that those considerations are correlated with factors that bias our results. For instance, it may be that countries with higher state capacity, or lower prevalence of conflict, were prioritized for the switch to surveys based on nationally representative samples. Similar factors may have influenced the methods chosen to conduct each survey or the timing of the survey. This would cause our dataset to include rural survey respondents only from countries with higher levels of, say, state capacity. At the same time, rural residents in countries with lower levels of state capacity may have less confidence in legal institutions. This bias could lead us to underestimate the strength of an urban/rural divide that could be driving legal disagreement in some jurisdictions.

The country-level fixed effects should account for variation across countries or jurisdictions in how their respective surveys were conducted. Nevertheless, as a robustness check, we replicated all of our analyses on subsamples of the data that were collected using the same methodology or at the same time. In other words, we replicated our main regressions at the individual level for the subgroup of surveys that are nationally representative, and separately for those that are conducted only in major urban centers. Similarly, we replicated our exercises for the surveys that were conducted in person, and for those that were conducted online. Lastly, we replicated our exercises for only those surveys that were conducted in 2018 or 2019 in order to keep only the most recent results. Details are provided in the Appendix. Our results hold for these exercises, in particular for the dimensions that are least likely to be driven by the methodological differences among these surveys.

Discussion

Our analyses reveal that in some, but not all, countries people hold significantly divergent views about the operation of the criminal justice system in relation to at least two types of cases. In addition, generally speaking, those differences tend to track differences in gender, financial well-being, and education. This all provides empirical confirmation of the existence of legal disagreement. It is also consistent with our theoretical assertions about the potential causes and consequences of legal disagreement. In general, individuals who seem likely to have different perspectives on or to be treated differently by the legal system have different beliefs about how that system will operate in a given scenario. Moreover, their divergent beliefs have the potential to exacerbate pre-existing social cleavages.

To elaborate, our theoretical analysis suggests that divergence between demographic groups’ beliefs about the legal system can be driven by any or all of three factors: people’s divergent perspectives on the system, the underlying level of variability in the system’s performance, and differences in the ways in which people draw inferences from whatever information they have about the system. First, people from different groups may disagree because they hear about different types of cases, perhaps because they live in different neighborhoods or consume different media. Second, differential treatment of groups—for example, the kind which leads to higher conviction rates for some groups than others—will reinforce disagreement caused by differences in perspectives if people tend to be most aware of cases involving their own groups. Third, when faced with precisely the same information about past cases, some groups’ members might be slower than others to abandon prior beliefs.

Our country-level analysis provides at best modest support for these theoretical claims, and then only in relation to disagreement about homicide. The finding that ethnic fractionalization and income inequality are associated with disagreement about homicide is consistent with our theory. In addition, our finding that disagreement about homicide cases is associated with perceptions that the criminal justice system discriminates suggests that perceptions of differential treatment are at least part of the explanation for the legal disagreement we observe in that sphere. Unfortunately, however, given the nature of our data, we cannot rule out that possibility that our findings reflect factual as opposed to legal disagreement caused by differential treatment—members of different groups who presume that the hypothetical scenarios relate to people like themselves and offer divergent responses because they know that they will be treated differently. The fact that none of these factors explains disagreement about corruption calls into the question the generalizability of claims about causes or consequences of legal disagreement.

Future studies might use additional data to explore the causes of legal disagreement. For example, there is evidence that political polarization is associated with media polarization, which is in turn determined by factors such as propensity to associate with like-minded individuals, the level of competition in the news industry, and characteristics of algorithms that govern news feeds on social media (Eady, et al. 2019; Guess, et al. 2021). It would be interesting to explore whether those factors, which we would characterize as potential sources of variation in perspectives, also influence legal disagreement. Alternatively, data on objective characteristics of legal institutions that might be correlated with the variability of their performance, such as how judges are elected or the structure of police unions, could be mapped onto data on legal disagreement. For countries with available data, subnational analysis could allow for much more granular studies of how regional variations in demographic and other variables are related to legal disagreement. Finally, it would be interesting to use proxies for individuals’ access to information about specific aspects of the legal system, such as personal contact with the system or the salience of the subject matter of the legal proceeding, to explore whether divergence in perspectives leads to variation in beliefs across individuals.

There is also considerable scope for further research on the consequences of legal disagreement. Our theoretical analysis suggests that it may be fruitful to explore the relationship between legal disagreement and rates at which disputes are settled, levels of interpersonal trust and other indicia of tension or conflict. It would be particularly interesting to explore these questions in contexts where legal disagreement overlaps with existing social cleavages. Again, these kinds of studies are most likely to be feasible at the subnational level where higher quality data tend to be more readily available.

Finally, it would be interesting to examine whether differences in beliefs are more likely to materialize in countries with deeper social cleavages. One way to do this would be to run regressions that capture the interaction between demographic characteristics of individual survey respondents and variables that serve as proxies for the stakes associated with belonging to the corresponding social group in a given society. This approach would capture the possibility that, for instance, the “returns” to being a man or highly educated or financially secure vary across jurisdictions. The challenge, of course, is to find appropriate country-level proxies for the distances between groups. For instance, a high rate of female labor force participation might imply low social distance between men and women, but not if it is associated with relatively high levels of gender discrimination in the workplace.

CONCLUSION

Our main purpose in this article is to convince scholars and policymakers who assemble and use information about legal systems to take account of legal disagreement. We have offered reasons to believe that legal disagreement ought to be taken into account when evaluating a society in terms of respect for the rule of law and socioeconomic inequality. We also have reasons to believe that legal disagreement might be caused by factors such as discrimination and media polarization and that it plays a role in explaining phenomena such as political polarization and conflict. Analyses that ignore legal disagreement, either because they ignore the perceptions of ordinary people or rely on the average values of those perceptions, will miss important features of the relationship between law and society.

We have also offered a practical method of measuring legal disagreement and applied it to measure disagreement about the behavior of officials involved in enforcement of criminal law. However, the WJP data upon which we rely are not as precise as would be ideal. Among other things, they almost certainly capture factual as well as legal disagreement and do not allow us to distinguish between respondents’ perceptions of different types of legal officials.

Despite these limitations, our empirical analysis illustrates the tremendous variety of possible applications for measures of legal disagreement based on survey data. To begin, these measures can be used as a check on the reliability of national averages. A further step would be to assign greater weight to the views of certain respondents. However, that decision would necessarily rest on a particular theory of what drives legal disagreements. For example, Benito Arruñada (2020) documents substantial disagreement about the quality of legal institutions between individuals who have and have not had personal experience with the relevant institutions. He argues that policymakers concerned with evaluating the quality of legal institutions ought to pay more attention to the views of the experienced individuals on the theory that experienced respondents are more likely to have correct information. If he believed that individuals’ experiences with the legal system distorted their perceptions, then he would presumably recommend assigning less weight to their responses.

These measures can be used both for scholarly purposes and in evaluating legal systems, or, keeping in mind our finding that levels of legal disagreement can vary substantially from one area of law to another, specific portions of legal systems. In scholarly settings, legal disagreement can be either an outcome of interest or an explanatory variable. Measures of legal disagreement also can be used as proxies for other variables such as media polarization. Finally, like other legal indicators, measures of legal disagreement can be used to measure the quality of legal systems, either alongside or as components of the well-known rule of law indicators. In each of these applications, taking account of legal disagreement promises a more nuanced understanding of the complex and variable ways in which law affects society.