6 - Communication about Procedure and Performance

Summary

This chapter examines whether and to what extent information about the procedures and performances of international organizations affects citizens legitimacy beliefs. It examines this issue comparatively across seven international organizations in different issue areas, including the African Union, European Union, United Nations Security Council, and United Nations Framework Convention on Climate Change. The survey is conducted in four countries in diverse world regions (Germany, the Philippines, South Africa, and the US). The analysis shows that information about both procedures and performances impact legitimacy beliefs. Moreover, citizens update their legitimacy beliefs in line with information about democracy, effectiveness, and fairness in global governance.

When global and domestic elites communicate about IOs in an effort to shape people’s opinions, they typically invoke features of these organizations in support of their arguments.¹ The most commonly invoked features are likely the procedures and performances of IOs. Do IOs take decisions through procedures that ensure adequate participation, transparency, efficiency, expertise, legality, and impartiality? Do the decisions of IOs impact outcomes through performances that solve societal problems, improve collective welfare, and distribute gains and losses fairly? Consider former Norwegian Foreign Minister Jonas Gahr Støre’s fierce criticism of the G20, targeting its procedural shortcomings: “[T]he G-20 is a grouping without international legitimacy. […] The G-20 is a self-appointed group. Its composition is determined by the major countries and powers. It may be more representative than the G-7 or the G-8, in which only the richest countries are represented, but it is still arbitrary” (Spiegel International 2010). Conversely, consider the attack on the UN by President Rodrigo Duterte of the Philippines, invoking the organization’s poor performance: “The UN has no purpose at all, actually, for mankind. […] As far as I’m concerned, with all its inutility, it has not prevented any war, it has not prevented any massacre” (ABS-CBN News 2018).

In this chapter, we shift the principal focus from elites to messages, examining whether and to what extent information about the procedures and performances of IOs affects citizens’ legitimacy beliefs. While elites make use of both types of arguments, it is an open question whether messages targeting IOs’ procedures or performances are the most effective in swaying citizens’ legitimacy beliefs. The past two decades have seen the emergence of a growing literature on which IO features matter most to people (e.g., Scharpf 1999; Hurd 2007; Bernauer and Gampfer 2013; Binder and Heupel 2015; Dellmuth and Tallberg 2015; Scholte and Tallberg 2018; Tallberg and Zürn 2019; Bernauer et al. 2020). For advocates of IO procedure, legitimacy beliefs are driven by the way the organization functions, irrespective of the effects of its policies. For advocates of IO performance, legitimacy beliefs are shaped by the consequences of the organization, irrespective of how the IO formulated the relevant policy. Recent research offers numerous examples of both types of accounts, as well as emergent efforts to assess the relative importance of procedure and performance.

Although this literature provides valuable insights, it confronts three important limitations. First, the evidence on the causal significance of information regarding procedure and performance qualities for legitimacy beliefs vis-à-vis IOs is inconclusive. Reliance in earlier work on textual analysis or cross-sectional public opinion surveys has not allowed for isolating the causal effects of these two institutional dimensions. Second, the focus on procedure versus performance masks the deeper question of what, more specifically, in these features of IO policy-making generates legitimacy beliefs. Each of the two categories hosts a range of particular qualities that may be important for legitimacy perceptions. Yet existing research has not systematically assessed this issue. Third, comparative analyses of institutional sources of legitimacy across IOs are in short supply. Most existing contributions focus on a single organization (e.g., Bernauer and Gampfer 2013; Binder and Heupel 2015; Dellmuth and Tallberg 2015). Whether and how communication about institutional features varies in importance across IOs and issue areas remains an open question.

Pushing back these limitations, this chapter assesses the causal effect on popular legitimacy beliefs of communication regarding a broad range of procedure- and performance-related institutional qualities across IOs in multiple fields and in different countries. Theoretically, we disaggregate the institutional sources of IO legitimacy, on the intuition that the categories procedure and performance are by themselves too crude to identify the specific institutional features that citizens care about. Instead, we develop a more fine-grained typology of institutional qualities, distinguishing between democratic, technocratic, and fair qualities of procedure and performance. While this new sixfold classification may not be exhaustive, it provides the most encompassing, precise, and systematic typology to date.

Empirically, we evaluate the effects of communicated institutional qualities on legitimacy beliefs through a population-based survey experiment, conducted in four countries with regard to IOs in four issue areas. The three issue areas are security governance (UNSC), climate governance (UNFCCC), economic governance (IMF), and regional governance (ASEAN, AU, EU, and NAFTA). To increase the generalizability of the findings, the survey experiment aggregates data from four countries in diverse world regions: Germany, the Philippines, South Africa, and the US. In the case of the regional organizations, we only examine communication effects for countries which are member states of these organizations.

The survey experiment yields three central findings. First, information regarding both procedure and performance matters for citizens’ legitimacy beliefs vis-à-vis IOs. Efforts to privilege the one over the other would therefore seem misguided. Second, within procedure and performance, all three tested qualities (democratic, technocratic, and fair) affect popular perceptions of legitimacy. The importance of both procedure and performance therefore holds across multiple qualities of these dimensions. Third, the extent to which communicated institutional qualities matter for IO legitimacy depends on the issue area at hand. A broader scope of institutional qualities appears to be important for legitimacy beliefs in the Philippines and the US compared to the other countries, toward global IOs compared to regional IOs, and, in the context of global IOs, toward the IMF compared to the UNFCCC and the UNSC.

This chapter proceeds in four parts. It begins by outlining our theoretical expectations about how communicated institutional qualities shape citizens’ legitimacy beliefs toward IOs. It then proceeds to outline the survey experimental design. The third section presents the empirical findings from the analysis. We end the chapter by summarizing its main conclusions.

Hypotheses

The expectation that communicated institutional qualities matter for people’s legitimacy beliefs toward IOs is rooted in both theory and practice. In terms of theory, a long tradition of Weberian sociology has demonstrated that the characteristics of a governing organization shape the legitimacy beliefs of its subjects. However, existing knowledge about the effects of institutional qualities on popular legitimacy beliefs toward IOs confronts large gaps. In terms of practice, ample observed behavior around IOs has suggested that institutional features shape audience responses to global governance. For instance, dissatisfaction with institutional qualities of IOs has been a prominent part of mobilization against global economic organizations (O’Brien et al. 2000; Della Porta and Tarrow 2005; Rauh and Zürn 2020). All of this suggests that communicated institutional qualities of IOs can play a key role in respect of legitimacy.

Procedure versus Performance

Recent years have witnessed growing efforts to identify institutional qualities of IOs with implications for citizens’ legitimacy beliefs toward these organizations. The starting point for most of this literature has been the distinction between input- and output-based legitimacy, introduced by Scharpf (1970, 1999). This dichotomy originally served to identify two alternative normative grounds for justifying the authority of the EU. In Scharpf’s view, the EU could earn its normative legitimacy either from democratic participation by the people or from problem-solving outcomes for the people. This distinction fed into a broader debate about the normative credentials of European and global governance (e.g., Zürn 2000; Held and Koenig-Archibugi 2005; Archibugi et al. 2012). Scharpf’s dichotomy was also picked up by researchers interested in establishing institutional sources of sociological legitimacy for IOs. Over the past decade, a growing literature has distinguished between procedure (input) and performance (output) as two generic institutional sources of legitimacy for regional and global IOs (Bernauer and Gampfer 2013; Binder and Heupel 2015; Dellmuth and Tallberg 2015; Tallberg and Zürn 2019).

The premise of procedural accounts is that process criteria are important for perceptions of IO legitimacy. On this argument, actors support an institution’s exercise of authority because of how it is set up and operates. Procedural accounts have an early antecedent in Weber’s (1922/1978) notion of legal-rational sources of legitimacy. On these lines, governance is regarded as appropriate because properly appointed authorities follow properly formulated decision-taking processes. So, for example, audiences might accord legitimacy to the UNFCCC because its policy-making is perceived to involve a broad range of stakeholders. Alternatively, actors might deny legitimacy to the IMF because its decision-making process is seen to give some states disproportionate weight. For procedural accounts, the legitimacy of an IO derives from the way that the institution functions, irrespective of the consequences of its policies.

Recent research offers several examples of procedural accounts. For instance, Bernauer and Gampfer (2013) focus on whether procedures that allow for greater civil society involvement also translate into greater legitimacy for global environmental governance. They find this to be the case: Citizens tend to favor civil society engagement, and therefore, procedures that provide for such participation are rewarded with higher legitimacy. Similarly, Johnson (2011) studies how procedures giving certain states particular advantages (e.g., through vetoes) influence the legitimacy of IOs. She finds that IOs which grant major states such as the US and Russia a special say in decision-making suffer in terms of perceived legitimacy.

In contrast, other accounts emphasize performance as an institutional source of IO legitimacy. On these lines, legitimacy beliefs derive from audience evaluations of a governing institution’s outcomes. With a focus on performance, IOs might gain or lose legitimacy depending on whether audiences see them as enhancing or undermining desired conditions in society. For example, the WHO might gain legitimacy if actors perceive that it effectively prevents epidemics. Meanwhile, the World Bank might lose legitimacy if subjects believe that this institution fails to reduce poverty. For performance approaches, the legitimacy of an IO derives from its impacts, irrespective of how the institution formulated and executed the relevant policy.

Existing research provides many examples of this type of argument. Multiple studies of public opinion toward the EU highlight the importance of policy-making outcomes for people’s legitimacy beliefs. These investigations show that citizens evaluate the EU’s legitimacy in relation to costs and benefits, both for their personal well-being and for their country (Gabel 1998; Hooghe and Marks 2005; Rohrschneider and Loveless 2010). Edwards (2009) advances a similar argument to explain public opinion toward the IMF, the World Bank, and the WTO. He finds that people’s legitimacy beliefs toward these IOs are primarily driven by the perceived implications of these organizations for their country’s economy.

While this existing work has expanded our understanding of how institutional features of IOs may affect people’s legitimacy perceptions, it suffers from two key limitations. First, efforts to compare the causal significance of communication about procedures and performances for popular legitimacy beliefs are missing. Especially in global governance, where citizens tend not to have direct experiences with IOs, elite communication is central as a mediating factor between institutional qualities and legitimacy beliefs. However, the few contributions testing factors drawn from both categories rely on observational methods such as textual analysis and public opinion surveys that do not allow for inferences about communication (e.g., Edwards 2009; Binder and Heupel 2015; Dellmuth and Tallberg 2015; Schlipphak 2015).

Second, existing research has not systematically unpacked procedure and performance to consider the more specific qualities of processes and outcomes that may generate beliefs in the legitimacy of an IO. Procedure and performance are broad and encompassing categories which, while conceptually useful, evade the deeper question: What is it more particularly about an institution’s conduct and results that generates perceptions of legitimacy? Each category encompasses a diversity of more specific qualities that may be important for legitimacy assessments. In terms of procedure, is it civil society involvement, decision-making rules, legality, efficiency, or other features that elicit legitimacy beliefs toward IOs? In terms of performance, is it problem-solving capacity, distributional consequences, implications for democratic governance, or other outcomes that generate perceptions of IO legitimacy?

As a result of these limitations, important questions remain unanswered. Is communication about some institutional qualities more significant than communication about others in shaping citizens’ legitimacy beliefs regarding IOs? Do effects of communication about specific institutional features vary across countries, IOs, and issue areas?

This chapter moves beyond these limitations. The remainder of this section develops a conceptualization that unpacks procedure and performance to identify specific qualities of each that often feature in elite communication about IOs. Subsequent sections then describe the design and results of a survey experiment that evaluates the causal effects of specific communicated institutional qualities on popular legitimacy beliefs across IOs in three policy fields.

Unpacking Institutional Qualities

Building on the distinction between procedure and performance, this section develops a richer and more systematic typology of institutional qualities to which citizens may be receptive. As the preceding discussion indicates, there is more to the sources of IO legitimacy than procedure and performance per se. To get at the institutional sources for legitimacy beliefs we must also examine the specific qualities of procedure and performance.

Starting from Scharpf’s (1999) work, previous research has tended to equate procedure with qualities of democracy and performance with qualities of effectiveness. However, this conflation excludes the possibility that democracy and effectiveness could each be features of both procedure and performance (Tallberg and Zürn 2019). For example, democratic governance could be an outcome in cases where an IO’s activities (such as election monitoring) generate more transparency and participation in politics. Likewise, effectiveness could be a quality of policy processes if a given institutional procedure (such as use of expert committees) allows an IO to produce more and faster decisions. In addition, it is plausible, for reasons elaborated below, that other qualities of procedure and performance besides democracy and effectiveness, such as fairness, could shape legitimacy perceptions.

Based on these considerations, we introduce a new typology of institutional sources of legitimacy by means of a 2×3 matrix (Table 6.1). In this schema, the two rows make the distinction between procedure and performance that has evolved out of earlier research. Hence, the matrix distinguishes between sources of legitimacy that pertain to the processes and to the consequences of IO policy-making. Meanwhile, the columns in the matrix highlight a threefold distinction between democratic, technocratic, and fair as three generic qualities that may apply to both the procedures and the performances of IOs.

Table 6.1 Institutional sources of legitimacy

	Democratic	Technocratic	Fair
Procedure	Participation Transparency	Efficiency Expertise	Impartiality Proportionality
Performance	Democracy promotion in wider society	Problem-solving Collective gains	Human dignity Distributive justice

This new typology uses the category of democratic procedure and performance to cover perceptions that affected publics have due voice in and control over governance arrangements. The category of technocratic procedure and performance is taken to encompass perceptions that a governing authority is effective in the light of best available knowledge and policy instruments. The term “technocratic” is not invoked here with any pejorative connotations and rather seeks to convey a sense of expertise-based problem-solving (Fischer 1989). Finally, the category of fair procedure and performance captures perceptions that process and outcome are just, equitable and impartial vis-à-vis implicated actors. Fairness is a feature of both procedure and performance that has obtained limited independent attention in previous research and is distinctive from both democratic and technocratic qualities (for an exception, see Hurd 2007).

The 2×3 schema yields six types of institutional features that may generate perceptions of legitimacy for IOs when communicated by elites. The expectation that people care about these qualities is anchored in social-psychological research. The right to voice opinions, a key political norm held by individuals in democratic societies (Held 1995), has been shown to drive perceptions of legitimacy toward domestic political institutions (De Cremer and Tyler 2007). Similarly, outcome favorability is well known to influence how people evaluate policy decisions and institutions (Skitka 2002; Dohertly and Wolak 2012; Esaiasson et al. 2019). Finally, extensive research has demonstrated that fairness is a fundamental concern in politics and matters to people’s perceptions of legitimate institutions (Tyler 1990; Tyler et al. 1997).

The following paragraphs specify and exemplify the six institutional qualities in the context of IOs. First, the category democratic procedure covers frameworks and practices that bring affected people into IO policy-making processes. In this category, one prominent institutional quality is participation: namely, where implicated actors have due involvement in the formulation, implementation, and review of IO decisions (Steffek et al. 2007; Macdonald 2008). Another significant feature of democratic procedure is transparency: namely, where affected publics can access full information about an IO’s activities and policies, making it easier to hold the organization accountable (Scholte 2011; Tallberg 2016). Several studies argue that democratic procedure has become the foremost source of legitimacy in global governance (e.g., Held 1995; Bernstein 2011). More specifically, some research finds that civil society involvement strengthens popular legitimacy in global environmental governance (Bernauer and Gampfer 2013). Other work identifies dissatisfaction with allegedly nondemocratic decision-making as a crucial motivation for public contestation of IOs (Della Porta and Tarrow 2005; Norris 2011).

The category democratic performance captures the consequences of policy-making for people’s power in national and international governance. Important in this respect are IO activities that safeguard or improve domestic democracy, for instance, by demanding accountability of national governments or by protecting civil rights (Pevehouse 2005; Keohane et al. 2009). The expectation that democratic performance matters for IO legitimacy is frequently expressed in the negative: namely, that people withhold support from IOs because of their negative implications for domestic democracy (Hooghe et al. 2019). Such thinking also figures prominently in populist discourse that argues for a repatriation of powers from IOs to democratic domestic governance (Inglehart and Norris 2017).

The category technocratic procedure encompasses practices that bring efficiency and expertise to policy-making processes. Efficiency can lie in the number and speed of an IO’s decisions (Hardt 2014; Tallberg et al. 2016), while expertise can involve basing IO decisions on the best available knowledge and skills (Majone 1998; Bernstein 2005). Along these lines, Chan et al. (2016) argue that the legitimacy of the Intergovernmental Panel on Climate Change (IPCC) rests primarily on the scientific expertise that this body brings to global environmental governance. Conversely, slow decision-making, mismanagement of funds, and organizational dysfunction are frequently invoked when explaining legitimacy difficulties for organizations such as the UN and the EU (Reus-Smit 2007).

The category technocratic performance refers to effectiveness in achieving policy ends. Qualities under this heading include problem-solving (i.e., successfully addressing a policy challenge) and collective gains (i.e., achieving benefits for society) (Keohane 1984; Scharpf 1999). It is commonly claimed that IOs earn their popular legitimacy through the collective advantages they produce for states and societies. In this vein, Lindberg and Scheingold (1970) introduced the term “permissive consensus” to describe a situation where populations enjoy the fruits of international cooperation and support its broad goals, while taking little concrete interest in IO policy-making processes. Likewise, functionalist accounts assume that states and their domestic constituencies support IOs because of the collective benefits they generate (Keohane 1984). In line with these expectations, some recent empirical research concludes that citizen perceptions of successful IO problem-solving constitute a strong base for legitimacy beliefs (Dellmuth and Tallberg 2015; Ecker-Ehrhardt 2016).

The category fair procedure refers to policy-making practices that give equitable treatment to all concerned. This quality is exhibited in impartiality (i.e., decision-taking processes are followed consistently and without discrimination) and proportionality (i.e., members contribute to IO resourcing in accordance with their relative means). For some types of IOs, such as international courts, fair procedure may be the chief institutional source of legitimacy. For instance, low legitimacy for the ICC among many African governments arises from their perception that the ICC imposes double standards between African and other leaders (Helfer and Schowalter 2017).

The category fair performance involves consequences of policy-making in terms of equitable outcomes. This quality can be judged in relation to IO practices that advance human dignity (i.e., norms of basic cultural, economic, and political livelihood) and distributive justice (i.e., equitable sharing of benefits and burdens) (Tyler 1990). For example, global justice protests have often targeted international economic institutions for allegedly producing unacceptable inequalities in society (O’Brien et al. 2000; Scholte et al. 2016). Conversely, IOs with poverty alleviation profiles – such as multilateral development banks and the United Nations Development Programme (UNDP) – often legitimize themselves in terms of promoting fairness for underprivileged people. The expectation is that citizens are more likely to accord legitimacy to IOs that are perceived to reduce injustice within and between countries.

In sum, the typology developed above disaggregates the conventional distinction between procedure and performance to bring out six more specific institutional sources of IO legitimacy. Each of the six qualities gives rise to a hypothesis about effects on legitimacy beliefs when this quality of an IO is communicated to citizens (see Table 6.2). Together, these distinctions permit a more fine-grained assessment of how communicated institutional features shape citizen perceptions of legitimacy in global governance. Social psychology, earlier political research, and anecdotal examples suggest that each of these institutional features may matter for legitimacy beliefs toward IOs. However, it is an empirical question for systematic investigation to establish in what ways and to what extent this is actually the case.

Table 6.2 Hypotheses and corresponding treatments

Hypothesis	Object	Quality	Wording of vignette
Communication about the democratic quality of an IO’s procedures affects its perceived legitimacy.	Procedure	Democratic	As you may have heard, the International Monetary Fund (IMF) holds its meetings about financial crises in public.
Communication about the democratic quality of an IO’s performance affects its perceived legitimacy.	Performance		As you may have heard, International Monetary Fund (IMF) actions on financial crises strengthen democracy in affected countries.
Communication about the technocratic quality of an IO’s procedures affects its perceived legitimacy.	Procedure	Technocratic	As you may have heard, the International Monetary Fund (IMF) responds on time to financial crises.
Communication about the technocratic quality of an IO’s performance affects its perceived legitimacy.	Performance		As you may have heard, International Monetary Fund (IMF) actions on financial crises are effective.
Communication about the fairness quality of an IO’s procedures affects its perceived legitimacy.	Procedure	Fair	As you may have heard, all countries have an equal say in International Monetary Fund (IMF) decisions on financial crises.
Communication about the fairness quality of an IO’s performance affects its perceived legitimacy.	Performance		As you may have heard, International Monetary Fund (IMF) actions on financial crises benefit everyone equally.

Notes: Wording of vignettes for the IMF. The wordings change depending on the IO, as they are active in different issue areas. For exact wordings in the second and third round, see Online Appendix N. After receiving the treatment, people were asked how much confidence they have on IOs: “How much confidence do you personally have in the UN?” Answer categories range from 0 (no confidence at all) to 10 (complete confidence); “don’t know.” The control group receives the question about confidence without a vignette.

Research Design

We assess the institutional grounds for legitimacy beliefs toward IOs through a population-based survey experiment. The purpose of this survey experiment is to establish the effects of communicated institutional qualities on citizen beliefs about IO legitimacy. While such effects could have been assessed with a population sample from a single country, estimating average effects of institutional qualities on legitimacy beliefs in several countries reduces the risk of biases from contextual country factors.

Survey Design

The survey experiment was conducted in Germany, the Philippines, South Africa, and the US. These countries were selected based on two criteria. First, they lie on different continents, thereby reducing the risk of regional biases and covering countries with different experiences of the IOs covered. Second, the four selected countries all have relatively high levels of Internet penetration (almost 90 percent in Germany and the US, over 50 percent in South Africa, and more than 40 percent in the Philippines), thereby increasing the representativeness of the sample to the whole populations of those countries.

The questionnaire was implemented by YouGov from September to November 2016 (see Online Appendix A). A total of 1,586 interviews were conducted in Germany, 1,358 in the Philippines, 1,384 in South Africa, and 1,393 in the US. After respondents accessed the online survey, they filled in a questionnaire in English (except in Germany, where it was translated into German). The survey took about 5 minutes to complete. It started with several “warm-up” questions, then moved to the survey experiment, and ended with several additional questions (see Online Appendix N for the questionnaire). The warm-up and supplementary questions were included to enable balance tests and robustness checks (see below). Questions were sequenced in ways to avoid priming the respondents inappropriately.

Experimental Design

The experimental part of the questionnaire aimed to isolate the causal effects on legitimacy beliefs of communication regarding the six institutional qualities discussed above. To this end, individuals were randomly assigned to groups that received different experimental treatments, as well as to a control group that did not receive any treatment (Mutz 2011, 9). The randomized design consisted of two factors that varied across the respondents of the survey: (a) dimension of policy-making (procedure or performance) and (b) institutional quality (democratic, technocratic, or fair). The resulting 2×3 factorial design yielded six conditions in total. Each combination of factors was presented to a similar number of individuals (around 820).

Table 6.2 summarizes the hypotheses-related treatments tested in the experiment. Taken together, these treatments allowed us not only to identify and compare the respective causal impacts of procedure and performance on legitimacy beliefs but also to disaggregate these dimensions to evaluate the importance of democratic, technocratic, and fair qualities of procedure and performance.

The experimental treatments were operationalized in vignettes. It was vital that respondents should react to the precise prompt in the respective vignettes. We therefore kept vignette formulations as short and straightforward as possible, and similar in strength, since longer and more complex vignettes lead to a greater risk of ineffectual treatments (Mutz 2011, 64–65). The vignettes involved hypothetical rather than actual scenarios. Although using real-world information can increase the credibility of vignettes, using hypothetical vignettes makes it easier to ensure that treatments are of similar strength.

The vignettes exposed respondents to a description of an IO (see Table 6.2). For instance, respondents were presented with the information that “the United Nations (UN) holds its meetings about military conflicts in public,” “United Nations (UN) actions on climate change strengthen democracy in affected countries,” “the International Monetary Fund (IMF) responds on time to financial crises,” or “European Union (EU) actions on regional cooperation benefit everyone equally” (Online Appendix N). The use of framed descriptions as vignettes builds on the assumption in our theory (Chapter 3) that citizens on average are not very well informed about political matters and therefore use communicated information to update their opinions on issues. Survey data suggest that this assumption is reasonable in the present case: Citizens are typically aware of the existence of the IOs used in the experiment, but lack a detailed understanding of their decision-making procedures and policy performances (Gallup 2005; Dellmuth 2016). When respondents are presented with framed descriptions of IOs, the expectation is therefore that they integrate the new information into their opinions. If this new information relates to features of IOs that respondents care about, it should also influence their perceptions of the organization’s legitimacy.

In the treatment groups, people received first a vignette and then a question about their “confidence” in IOs. In contrast, the control group received only the question of how much confidence the respondent has in an IO, without the preceding vignette. Confidence – our preferred measure of legitimacy, as set forth in Chapter 3 – was measured on a scale from no confidence at all (0) to complete confidence (10). Most respondents answered on this scale, although a small minority (close to 10 percent) selected a “don’t know” option.

The experiment used a randomized factorial design that systematically varied vignettes on combinations of the first factor (procedure or performance) and the second factor (democratic, technocratic, or fair). This made it possible to establish the distinct causal effects on IO legitimacy of information regarding each of the six institutional qualities. Each vignette was formulated to operationalize one central institutional quality in each of the six categories summarized in Table 6.1: transparent decision-making (democratic procedure), effects on domestic democracy (democratic performance), efficient decision-making (technocratic procedure), effective problem-solving (technocratic performance), equal say in decision-making (fair procedure), and equal distribution of benefits (fair performance). The effects of information regarding these six institutional qualities were evaluated by comparing mean confidence in the treatment groups with mean confidence in the control group. Since people are psychologically more responsive to negative information than to positive prompts (Kahnemann and Tversky 1979; Baumeister et al. 2001; see also Chapter 4), vignettes were constructed using a positive formulation in order to enable a hard theory test. Positive treatments also speak to IOs’ efforts to appear more democratic, effective, and fair in their procedures and performances.

In sum, we establish whether communication about an institutional quality matters for respondents’ legitimacy beliefs toward an IO by first exposing them to a treatment containing information on this particular quality, then asking them about their confidence in the IO, and finally comparing the average level of confidence among the respondents who receive this particular treatment with that of respondents who receive no treatment (the control group). A statistically significant difference in the average level of confidence between the two groups allows us to conclude that the institutional quality, which is manipulated in the particular treatment, likely contributes to respondents’ confidence in the IO.

The experiment was conducted in four rounds designed to capture institutional sources of legitimacy for seven IOs: three global organizations (IMF, UNFCCC, and UNSC) and four regional organizations (ASEAN, AU, EU, and NAFTA). This design permits the experiment not only to test expectations about the general effects of these six institutional qualities but also to show the occurrence of such effects across countries and IOs. The first round presented vignettes related to the UN’s actions on military conflict, speaking to the UNSC, the primary IO in the area of security. The second round presented vignettes about the UN’s actions on climate change, speaking to the UNFCCC, the central IO on this issue. The third round addressed a prominent economic IO, the IMF. The fourth round focused on one regional organization relevant for each of the four included countries (ASEAN in the Philippines, AU in South Africa, EU in Germany, and NAFTA in the US). Respondents were never exposed to the same treatment more than once. Respondents who were placed in the control group remained there throughout the three rounds. The order of the experimental rounds was randomized in order to avoid potential priming effects.

Results

The results of the experiment are now presented in two steps. The first examines whether and to what extent communication regarding procedure- and performance-related institutional qualities affect respondents’ legitimacy beliefs toward IOs. The second considers whether and to what extent these effects vary across IOs in climate, economic, security, and regional governance. As discussed above, we will present the results in a series of tables that compare the average level of confidence among respondents receiving a particular treatment to those not receiving any treatment at all (the control group).

Similar to Chapter 4, data have been pooled across the four experimental rounds to estimate treatment effects, so that the observations are clustered in individuals. Treatment effects are calculated by using OLS regression analysis using weighted data and robust standard errors clustered at the level of individuals, where confidence is regressed on a treatment dummy (1 = treated, 0 = not treated).

Effects of Communication about Institutional Qualities

The key finding is that communication about all six procedure- and performance-related institutional qualities affected respondents’ legitimacy beliefs. Figure 6.1 sets out differences in average levels of confidence between the control group and the respondents receiving treatments. All of the treatments generated statistically significant effects on respondents’ confidence in IOs, be they about democratic, technocratic, or fair qualities of procedure and performance.

Figure 6.1 Effects of communication about institutional qualities

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O1 for detailed results.

The size of the treatment effects varies from 0.241 to 0.419 on an 11-point scale of confidence. These effects are substantively important. For example, an effect size of 0.419 for “democratic performance” is similar in size to the average difference in confidence between the UNFCCC and the IMF in the control group (Figure 6.2). In addition, the experimental setting likely underestimates the corresponding real-world impact. After all, the experimental effects result from a one-shot treatment, rather than from continuous exposure to a particular institutional quality, as would be the case in actual situations (Gaines and Kuklinski 2011). Moreover, it should be recalled that positive treatments usually generate smaller effects than negative treatments, since people are psychologically less responsive to positive information than to negative prompts.

Figure 6.2 Average confidence in IOs in the control group

Notes: Weighted means, based on data from the control group. N = 676.

We also assess if cues about procedure and performance are more effective among the politically more aware, given that they are more likely to use new information efficiently when updating their opinions, as theorized in Chapter 3. We therefore test if the treatment effects depicted in Figure 6.1 are conditional on our two awareness indicators: education and discussion of politics with friends. The results suggest that more politically aware citizens did not respond differently than less aware citizens when confronted with the treatments (Online Appendices O2–O3).

Taken together, these results demonstrate that both procedure and performance can affect people’s confidence in IOs, and this conclusion holds across democratic, technocratic, and fair qualities of procedure and performance. Hence, claims that legitimacy rests with either procedure or performance, or with either democratic, technocratic, or fair features of IO policy-making, would appear to be misguided.

The finding that democratic, technocratic, and fair qualities all matter speaks to earlier research in interesting ways. First, this result supports previous research which holds that democratic concerns are central to people’s evaluations of IOs (Held 1995; Zürn 2000; Bernstein 2011). Second, this finding shows that the importance of fairness for legitimacy beliefs applies to international as well, ant not only domestic political institutions (Lind and Tyler 1988; Tyler 1990; Gibson et al. 2005; Esaiasson et al. 2019). Third, this outcome cautions against the proposition that people are nowadays less concerned with technocratic aspects of IOs (Scharpf 1999; Hooghe and Marks 2009), instead pointing to the sustained importance of efficiency and effectiveness considerations for legitimacy perceptions.

Disaggregating Treatment Effects across IOs

Next, we examine whether the effects of communication regarding procedure- and performance-related institutional qualities vary across IOs, considering the results from each experimental round separately. Figure 6.3 shows the results for the global organizations, while Figure 6.4 depicts the results for the regional organizations.

Figure 6.3 Effects of communication about institutional qualities, by global organization

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O4 for detailed results.

Figure 6.3 suggests that all treatments are effective in relation to the IMF, with the exception of the treatment on fair procedure. By contrast, in the context of the UNSC and UNFCCC, fewer treatments are statistically significant. In the UNSC, democracy-related qualities matter, and in the UNFCCC, we find no significant effects. Figure 6.4 shows that no significant treatment effects were found for the regional organizations. However, it should be noted the analyses of the regional organizations are based on a considerably smaller sample of respondents – only one country per IO.

Figure 6.4 Effects of communication about institutional qualities, by regional organization

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O4 for detailed results.

These results suggest four observations. First, they confirm that communication regarding both procedural and performance qualities matters for the perceived legitimacy of IOs, also when we break down the effects by IO. In the context of IOs with significant treatment effects (IMF, UNSC), both procedure and performance are important. Second, treatments are most effective in the US, when compared to Germany, the Philippines, and South Africa. Third, the findings suggest that communication regarding institutional qualities is of varying importance for global organizations, mattering most for legitimacy perceptions toward the IMF, less for the UNSC, and least for the UNFCCC. Finally, the varying occurrence of treatment effects at the level of countries and IOs reduces potential concerns that the homogenously positive and significant effects established earlier at the aggregate level would result from a general framing effect in the experiment.

Focusing specifically on the variation in treatment effects across IOs, five interpretations are conceivable. First, variation may reflect the respective missions of the IOs. As previously noted, it may be that some organizational purposes generate legitimacy beliefs more than others because these mandates are perceived to be intrinsically good and uncontestable (Scott 1991; Lenz and Viola 2017). If this logic is at play, it may affect the relative importance of other institutional qualities as sources of legitimacy. For example, the UN’s attention to climate change may generate legitimacy beliefs by itself and thereby reduce the relative importance of procedural and performance features. In contrast, the IMF pursues objectives that are perceived to be more contestable, which may elevate the relative importance of procedural and performance sources of legitimacy.

Second, differences across issue areas could reflect varying levels of authority among the IOs in question. According to one argument, the standards that an IO must reach to be considered legitimate depend on its level of policy competence (Tallberg and Zürn 2019). Institutions that possess higher levels of authority must meet more demanding standards of procedure and performance or else they will suffer a legitimacy deficit. This logic may help to explain why the full range of institutional qualities matter for legitimacy beliefs toward the IMF, which has far-reaching formal authority. Consistent with this interpretation, fewer institutional qualities matter in the case of the UNSC, which possesses medium authority, and none at all in the case of the UNFCCC, which has limited formal authority (Hooghe et al. 2017; Zürn 2018). In Chapter 7, we will systematically examine effects of purpose and authority in conjunction with procedure and performance.

Third, citizens could evaluate some IOs on a broader range of parameters than others. This interpretation ties in well with previous anecdotal evidence on sources of contestation in global governance. Consistent with our experimental findings, public opposition to IOs in the area of economic governance invokes a broad range of criticisms, including democratic shortcomings, perceived injustices of neoliberal doctrine, and ineffective policies (O’Brien et al. 2000; Rauh and Zürn 2020). Similarly, and in keeping with the results here, earlier research on the legitimacy of the UNSC shows that its democratic limitations have attracted particular criticism (Binder and Heupel 2015).

Fourth, the observed pattern of variation across IOs is consistent with the possibility that citizens respond more strongly to new information about IOs they know less well (cf. Chong and Druckman 2007a). As we theorized in Chapter 3, the object of messages may matter for communication effects if citizens have stronger priors toward some IOs than others. Earlier survey data suggest that the UN is more known to citizens in a global sample of countries than the IMF (Gallup International Association 2005; Dellmuth et al. 2022b, ch. 2). This finding is further corroborated by evidence from a question in our survey on factual knowledge about the IMF and the UN (Figure 6.5) (Mondak 1999; Jessee 2017). In addition, the four regional IOs are likely better known among citizens than the global IOs, as suggested by data from the Gallup International Association (2005, 2011) analyzed in Chapter 4. Thus, the broader range of treatment effects for the IMF may partly reflect respondents’ greater sensitivity to new information about this IO, compared to the other two global IOs, and the absence of treatment effects for the regional IOs may partly reflect citizens’ more well-developed priors regarding these organizations.

Figure 6.5 Political knowledge about IOs

Notes: Weighted percentages. Left panel shows responses to a question about knowledge about where the headquarters of the IMF are located. Answer categories: “A) Washington, DC, B) London, C) Geneva, D) Don’t know.” Right panel shows responses to a question about knowledge about which of the following countries does not have a permanent seat on the Security Council of the United Nations. Answer categories: “A) France, B) China, C) India, D) Don’t know.”

Fifth, and related, variation in treatment effects may be shaped by contextual factors, such as whether an IO is subject to intense public debate at a certain point in time. Much like general knowledge about an IO, intense public debate around a particular organization can lead to more developed opinions (cf. Bakaki and Bernauer 2017) and therefore less responsiveness to treatment manipulation. For example, the timing of our survey experiment less than a year after the signing of the Paris Agreement might have contributed to the lack of treatment effects in relation to climate governance. Conversely, contextual events can sensitize people to a particular aspect of an IO’s work, leading to larger treatment effects on that dimension. For instance, the results for problem-solving (technocratic performance) in respect of the UNSC might have been different twenty years ago in the aftermath of its shortfalls in Rwanda and Bosnia. Similarly, the treatment effects in relation to the IMF could potentially have been even stronger in the 1990s when the Fund’s structural adjustment programs were intensely contested.

Disaggregating Treatment Effects across Countries

Next, we examine whether the effects of communication regarding procedure- and performance-related institutional qualities vary across countries. Figure 6.6 calculates the differences in average confidence between the control group and the respondents receiving treatments for each country separately. It suggests several interesting patterns.

Figure 6.6 Effects of communication about institutional qualities, by country

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O for detailed results. Numerical results in Online Appendix O5.

First, the results at the country level are heterogeneous and quite weak. Citizen opinions generally move in the expected directed in two countries (South Africa and US), move little at all in one country (Germany), and move in the opposite direction than expected in one country (Philippines). However, only few treatments are significant. This may partly be the result of lower statistical power, since the N in each category is considerably smaller when we analyze these treatment effects by country (Online Appendix O5).

Second, if we focus on the significant effects, Figure 6.6 reveals that information about technocratic procedure and fair performance leads US citizens to update their opinions about IOs. By contrast, we do not find any treatment effects in Germany and South Africa. In the Philippines, we observe one treatment effect, albeit in the negative direction: Information about technocratic performance appears to weaken legitimacy beliefs in IOs. This deviant result could potentially be understood against the background that public opinion toward IOs in the Philippines is more positive than in other countries (see also Dellmuth et al. 2022a). In our data, mean confidence in IOs in the Philippines is 6.2, while it is much lower in the other three countries (4.8 in South Africa, and 3.9 in Germany and 3.6 the US).² The high degree of confidence in IOs in the Philippines is also a consistent pattern in WVS data over time. WVS data from 1994 through 2019 indicate that respondents in the Philippines on average have particularly high levels of confidence in the UN compared to respondents in other countries.³ Thus, it could be that citizens in the Philippines, because of the high degree of confidence in IOs, have particularly high expectations and therefore are little moved by positive treatments – indeed, may even be disappointed by the additional information.

Validity and Robustness Checks

We conclude this analysis with a number of validity and robustness checks. To test the internal validity of the experiment, we report a series of balance tests. These tests are based on the responses to several additional questions asked in the survey, capturing, inter alia, political knowledge about global governance. The tests check if the randomized allocation of respondents across treatment groups has worked by assessing if there is a statistically significant difference in mean confidence across levels of these variables. The tests reveal only eight imbalances for sixty tests, which should not compromise causal inference (Online Appendix P).

Moreover, we examine if the results could have been moderated by people’s confidence in the domestic government or the interest they have in specific IOs. To begin with, we examine for each country separately if treatment effects depend on confidence in domestic government, as more trusting people might have more fixed opinions about institutions in general and therefore be less swayed by elite communication. However, we do not find evidence for this intuition (Online Appendix Q1). Moreover, the results do not depend on respondent’s interest in the respective global organizations (UN or IMF) (Online Appendix Q2).

Conclusion

This chapter has explored how communicated institutional qualities of IOs affect citizens’ legitimacy beliefs toward these organizations. Moving beyond existing research, it has sought to evaluate the causal significance of procedure and performance, to unpack these two dimensions into specific institutional qualities, and to offer a comparative analysis across IOs in different issue areas. In terms of theory, the chapter has presented a more encompassing and precise sixfold typology of institutional qualities that may affect citizens’ legitimacy beliefs. In terms of empirics, the chapter has presented the findings from a survey experiment among 5,700 respondents in four countries. The diversity of these countries suggests that results from the combined sample have broad applicability.

The central findings are threefold. First, the results indicate that information about both procedure and performance affects citizen legitimacy beliefs about IOs. Second, within procedure and performance, democratic, technocratic, and fair qualities all matter for people’s legitimacy beliefs toward IOs. Third, the effects of communicated institutional qualities vary across IO and country contexts. A broader range of institutional qualities are important for legitimacy beliefs toward IOs in the US and the Philippines compared to Germany and South Africa. Moreover, we find more treatment effects in the context of global than regional IOs, and among the global organizations, in the context of the IMF than the UNFCCC and UNSC.

Our findings suggest four broader implications. First, this chapter corroborates the assumption of earlier research, from Weber to Scharpf, that citizens care about institutional qualities when forming legitimacy beliefs. Arriving at this conclusion from a focus on six procedure- and performance-related qualities does not presume that no other institutional features may matter. Indeed, in Chapter 7, we extend this analysis to include two additional institutional features: an IO’s purpose and authority.

Second, the importance of multiple institutional sources for IO legitimacy invites additional research on their combined effects in elite messages. For example, invoking several institutional features jointly might exert stronger and mutually reinforcing (de)legitimating effects. Thus, an IO could attract greater legitimacy when elites present it as having, say, efficient process together with fair outcome than when messages focus on only one or the other of these two qualities. Alternatively, positive communication on one institutional dimension might be cancelled out by negative communication on another (Bernauer et al. 2020). For instance, democratic procedure might cease to trigger legitimacy beliefs if it is coupled with ineffective problem-solving. This chapter has identified how certain communicated institutional features, taken individually, matter for IO legitimacy; however, future studies can be encouraged to examine institutional factors in combination. In Chapter 7, we take one important step in that direction.

Third, the results suggest that supporters and opponents of IOs may be right to address institutional qualities in their respective strategies to legitimate or delegitimate these organizations. In this context, the existence of multiple institutional sources of legitimacy can present an opportunity for both proponents and critics of IOs. Supporters of an IO need not focus their communication on a single overriding institutional quality. Instead, they can potentially bolster the IO’s public standing by speaking to a variety of features, perhaps concentrating efforts on the qualities that the organization can most easily improve. Likewise, critics can target a wide menu of institutional features in their strivings to delegitimate an IO, possibly focusing their energies on the qualities that are most vulnerable to critique. As a result, legitimators and delegitimators may in their contention around a given IO emphasize different institutional attributes.

Finally, this research has shown that a comparative approach is useful when studying communicated institutional qualities as sources of IO legitimacy. Focusing on single countries and IOs can yield particularistic findings without demonstrating broader patterns and relationships. This chapter has demonstrated how results may vary across countries and IOs, and offered tentative interpretations of these patterns. Future research could deepen and extend such comparisons, for instance, by assessing variation across IOs of different types (e.g., intergovernmental versus hybrid), of different functions (e.g., courts versus executives), and in a larger sample of countries.