The Limitations of Comparative Institutional Analysis

Atul Gawande's Checklist Manifesto became a sensation in 2009 because it promised that a simple technique could powerfully discipline decision-making. Gawande had saved lives using hospital checklists, and he argued that checklists could improve outcomes in other complicated endeavors. Checklists, he explained, “provide a kind of cognitive net. They catch mental flaws.”¹ Neil Komesar's method of comparative institutional analysis is by necessity messier than the checklist and does not claim to produce faultless policy-making.² But Komesar similarly seeks to improve cognitive processing by imposing a disciplining framework on decision-making. Sergio Puig and Gregory Shaffer's effort to introduce Komesar's technique to the debate about foreign investment law reform is welcome.³ Their emphasis on tradeoffs among institutional alternatives helps us to appreciate the different contexts facing different nation states, the value of regime competition, and consequently, the importance of implementing reforms in ways that preserve a variety of options for states.⁴ If they persuade commentators and policy-makers to take stock of the tradeoffs among institutional alternatives, Puig and Shaffer will have made a meaningful contribution. Still, their analysis illustrates some of the weaknesses of comparative institutional analysis. In this essay, I identify those weaknesses and suggest that they also weigh in pluralism's favor.

Implementation Mistakes in Comparative Institutional Analysis

Early enthusiasm for the checklist approach has yielded to circumspection. It turns out that checklists are not foolproof eliminators of human error. Follow-up studies have found that checklists implemented by hospitals are often “inappropriate or illogical” and that staff frequently skip checklist items.⁵ Similarly, the ideal of rigorously comparing alternative institutions is challenging to put into practice. This essay discusses three ways that the effort to be rigorous in comparing institutional alternatives might go awry: (1) unjustified theoretical assumptions, (2) uneven application of evidentiary standards to empirical questions, and (3) failure to appreciate when institutions are not only alternatives but also depend on one another. The framework's susceptibility to these mistakes limits its illuminating power.

Theoretical Assumptions

The first mistake is reliance on unjustified theoretical assumptions. The authors’ discussion of collective action problems provides an example. They explain that barriers to collective action create institutional biases in favor of groups with high per capita stakes. The example cited perhaps most frequently to illustrate this kind of collective action problem is the protectionist tariff.⁶ A tariff on an imported good creates concentrated benefits for producers of the good, who will earn a higher price, and diffuse costs for consumers, who will pay more for the good. Even if the tariff results in a net loss to the economy, the effect on any one consumer is so small that she will not organize against or even become informed about the policy. Domestic producers, by contrast, stand to gain enough to incentivize them to organize in favor of the trade barrier.

When comparing institutional alternatives, we should consider whether such biases are likely to cause each institution to promote unduly certain interests. That much is uncontroversial. However, the difficulty lies in the application of this idea. Puig and Shaffer assume that collective action problems will hinder only protest against stronger foreign investment protection.⁷ When large groups of people experience consequences from stronger foreign investment protection, and a collective action problem arises from the size of the group and its members’ per capita stakes, Puig and Shaffer posit that we should assume that the people in the group would be harmed by the legal change:

Applied to international investment law, [barriers to collective action suggest] that individuals are unlikely to organize in opposition to expansive investor protections or ISDS. … In contrast, investors have high per capita stakes in investment projects, which creates the incentive for them to assess benefits and costs. Investors thus may deploy significant resources to shape investment law norms and their application.⁸

This is a theoretical argument, but they offer no theoretical justification for assuming that collective action problems hinder organizing only by those who stand to lose from stronger investment protection. A host state population that stands to benefit, each member in small measure, from foreign direct investment—for instance, by enjoying higher incomes or better employment, infrastructure, or technology—could just as well find itself unable to organize because of collective action problems as a population that stands to suffer diffuse costs. The analysis also ignores influential actors with high per capita stakes, including international NGOs.⁹

Empirical Uncertainties

A second source of mistakes is the difficulty of empirically ascertaining facts relevant to comparing alternative institutions, including how well they accomplish desired policy goals. Because of high epistemic barriers, analyses of alternative institutions will very often rest on conjecture on empirical questions. For instance, in discussing asymmetric information, the authors assert, “investment firm[s] often [have] much better information regarding environmental and social risks at stake.”¹⁰ This is not obviously so. Government officials within a state might be expected to have better knowledge or access to better information than a foreigner about not only the social and political dynamics within the national population, but also how those might interact with a proposed investment.

A more consequential example is the debate about whether investor-state arbitration or investment treaties’ substantive protections increase foreign direct investment, and whether greater foreign direct investment in turn improves economic performance in recipient countries. As Puig and Shaffer observe, quantitative studies reach divergent conclusions.¹¹ The question does not yield itself to randomized controlled trials or natural experiments, and endogeneity makes empirical work of any kind on the question difficult. On the related question of whether alternative mechanisms that do not yet exist, such as a permanent investment court, would affect investment flows, we cannot do better at this time than to speculate.

Amid this uncertainty, scholars and policy commentators often apply different evidentiary standards to different institutions or issues in ways that might skew a comparative analysis. Consider again the theoretical argument that greater international legal protection for foreign investment increases investment flows. Scholars justifiably have subjected this claim to repeated empirical tests.¹² Policy commentators frequently cite the inconclusiveness of the evidence as an argument against maintaining the arbitration and investment treaty regime.¹³ Indeed, the theoretical argument is nearly always accompanied by the disclaimer that evidence is contested. By contrast, policy commentators just as often make the theoretical argument that investment protections and investor-state arbitration will cause regulatory chill, but appeals to the lack of evidence on this question are comparatively rare.¹⁴

The comparative institutional analysis framework does not save Puig and Shaffer from applying different evidentiary standards to these different arguments. They cite the risk that suits by foreign investors might chill regulation as a reason to prefer other mechanisms to adjudication,¹⁵ but do not mention the lack of evidence for regulatory chill. By contrast, when they discuss the argument that investor-state arbitration improves economic outcomes, they highlight the contested status of the empirical evidence and count that uncertainty as an argument against adjudication.¹⁶ What is missing is a uniform standard of proof to be applied when there is substantial empirical uncertainty about benefits and costs.

The contested nature of so many of the relevant questions, and the strength of commitments to the fundamental political values at stake, should make us wary of tendencies toward motivated reasoning amid empirical and theoretical uncertainties. Some of the problems with implementing comparative institutional analysis could be ameliorated by taking seriously Puig and Shaffer's call for rigor. Anthea Roberts observes in her essay that the UN Commission on International Trade Law (UNCITRAL) has convened a Working Group on Investor-State Dispute Settlement Reform.¹⁷ The Working Group could frame its discussion by committing to evaluate each available institutional option against a set of predetermined criteria using consistent evidentiary and argumentative standards. Because many interested states are represented in the UNCITRAL process, the Working Group might be able to bring forward the strongest facts and arguments for and against each institutional option. A well-organized, rigorously composed, publicly available analysis could discipline the Working Group's discussion and lead to better decision-making.

Institutional Complements

A third potential source of error is identifying as alternatives or substitutes institutions that are actually (or also) complements. In situations in which the institutions may be complements, eliminating or weakening one could undermine the effectiveness of another. For example, Puig and Shaffer categorize reputation as a market mechanism that is an alternative to adjudication for protecting foreign investment from unfair treatment by host states. In many complex markets, however, adjudication is not only a substitute for but an indispensable part of reputational governance.

Scholars of contract and dispute resolution have long recognized that when economic relationships are complex, reputation faces barriers to effectiveness, and that adjudication can lower those barriers. Complexity raises the cost of monitoring and verifying behavior. When monitoring and verification costs are high—that is, when it is difficult for the transacting parties or for third-parties to ascertain the behavior of each actor, whether the behavior caused harm, and whether the behavior is wrongful—reputations may develop slowly or inaccurately, if at all. Therefore, trading networks that govern themselves by reputation often develop adjudication to clarify noisy reputational signals. The most famous example of such a regime in the contract literature is Lisa Bernstein's study of diamond merchants.¹⁸ Diamond industry disputes historically have been decided by confidential arbitration in which the outcome remained confidential as long as the losing party paid. If the losing party failed to pay, his name and photograph were displayed prominently in the diamond bourse.

Much investment treaty arbitration today is not confidential, so the reputational dynamic of adjudication in this context works differently. The parties’ allegations and the tribunal's assessment of their behavior are public information. Even if a state is found not to have violated a legal obligation, any behavior that might give pause to other prospective investors will be broadcast widely in the published arbitration award. A state cannot preserve its reputation by paying an award voluntarily. Investors’ bad behavior is likewise publicly aired.

This sort of publicity contributes to reputational governance. Without adjudication, a state might enjoy plausible deniability when it has behaved wrongfully. When that is so, it could wrong one investor without suffering great reputational damage. Such plausible deniability might arise from limited public information about what happened factually or uncertainty about how to characterize the parties’ actions legally. However, adjudication by one or more persons who are trusted by the relevant reputational audience to find facts and assess behavior can clarify reputational signals. To suggest, as Puig and Shaffer do, that the possibility of reputational enforcement through market mechanisms might render international adjudication unnecessary therefore fails to appreciate the way the two work together.

Preserving Space for Institutional Pluralism

The capacity of comparative institutional analysis to discipline policy decision-making is limited. Nonetheless, the framework is valuable because it highlights that debates about institutional reform should be conceived in terms of the relative strengths and weaknesses of institutional alternatives as they operate in the real world. The framework leads Puig and Shaffer to advocate institutional pluralism. I agree with Puig and Shaffer and with Roberts that states’ diverse institutional challenges and the differences in their demand for foreign investment call for pluralism in the foreign investment legal regime. Pluralism also allows for institutional competition to drive positive innovation. Moreover, the limits of comparative institutional analysis described above urge epistemic modesty, which further recommends pluralist approaches to reform.

Reforms should therefore be implemented in a manner that preserves a wide spectrum of options for states, ranging from the option to make a strong and credible commitment to protect foreign investment to the option to eschew international legal protection for foreign investors.¹⁹ Systemic changes that impose externalities on other states should be avoided. Although it is unclear whether they are being seriously considered in the UNCITRAL Working Group and current discussions in other fora, some of the modes of reform that have been proposed would cause such spillover effects. Two such proposals are (1) to adopt methods of treaty interpretation that universally weaken the binding power of words in treaty text, and (2) to politicize arbitrator appointments in major arbitration institutions to give states more control over how disputes are decided.

The first proposal includes the suggestion that states can, either by agreement with treaty counterparties or through unilateral expressive acts, retroactively amend their treaties to withdraw rights granted to third parties.²⁰ It also includes the suggestion that provisions such as necessity or essential security clauses should be interpreted to be self-judging.²¹ If a treaty explicitly provides for such amendments, as some believe NAFTA does, or contains expressly self-judging clauses, then recognizing them creates no systemic problem. However, recognizing these as generally applicable interpretive principles weakens the binding power of all treaties and does so without the express agreement of their parties. It is not clear whether and how states could textually override such interpretive principles. Paradoxically, by strengthening sovereign prerogatives to alter their commitments ex post, adopting such principles of treaty interpretation threatens to systemically weaken sovereigns’ power to make promises in exchange for reciprocal benefits.

Politicizing arbitrator appointments in the established arbitration institutions, such as by requiring that all International Centre for Settlement of Investment Disputes (ICSID) arbitrators be chosen from a closed list, would similarly undermine the effectiveness of treaty commitments. States could presumably opt out of arbitration institutions that detrimentally politicize appointments. However, their ability to opt out would be subject to the costs of switching away from an established, widely used, and otherwise generally well-functioning legal infrastructure. In contrast, the current ICSID framework for arbitrator appointments offers an example of the kind of option-preserving infrastructure on which different states might build customized superstructures. The ICSID framework includes a default rule that allows disputing parties to appoint any arbitrator they wish, subject to character, competence, and ethics standards. However, the ICSID framework does not preclude states that wish to do so from providing for a closed list or panel of arbitrators in their investment treaties, and thus limiting the arbitrators from which investors can choose.²²

Conclusion

Komesar's comparative institutional analysis framework does not eliminate the possibility of making substantial errors when comparing institutions. It is, nonetheless, a valuable decision-making tool, because it highlights that the choice to be made is one among imperfect options, each of which serves some goals and contexts better than others. As Puig and Shaffer conclude, appreciating this reality impels us toward pluralism. Pluralism requires that reforms be made by the explicit choices of states, such as by revision of their investment treaties or the conclusion of new treaties, rather than through novel interpretive methods or institutional structures that might cause reverberations that affect all states. Pluralism favors default rules or optional rules over mandatory rules and avoids imposing a single outcome that will bind every state. Instead, having made their most persuasive case to one another on the various values, goals, facts, and theories, states can to a great extent base their policies on their own conclusions about those contested issues. This approach lowers the stakes of comparative institutional analysis and its weaknesses.