Institutional sources of legitimacy for international organisations: Beyond procedure versus performance

Abstract

This article addresses a significant gap in the literature on legitimacy in global governance, exploring whether, in what ways, and to what extent institutional qualities of international organisations (IOs) matter for popular legitimacy beliefs towards these bodies. The study assesses the causal significance of procedure and performance as sources of legitimacy, unpacks these dimensions into specific institutional qualities, and offers a comparative analysis across IOs in three issue areas of global governance. Theoretically, the article disaggregates institutional sources of legitimacy to consider democratic, technocratic, and fair qualities of procedure and performance. Empirically, it examines the effects of these institutional qualities through a population-based survey experiment in four countries in different world regions with respect to IOs in economic, security, and climate governance. The findings demonstrate that both procedure- and performance-related aspects of IO policymaking matter for popular legitimacy beliefs. This result holds across democratic, technocratic, and fair qualities of IO procedure and performance. Disaggregating the results by issue area indicates that a broader scope of institutional qualities are important for legitimacy beliefs in economic governance compared to security governance and, especially, climate governance. These findings suggest that propositions to reduce the institutional sources of IO legitimacy to single specific qualities would be misguided.

Introduction

Recent history has seen international organisations (IOs) acquire substantially enlarged authority, on the premise that increased transnational policy challenges require expanded regional and global governance.¹ However, whether these higher expectations of IOs translate into actual greater problem-solving depends in part on whether these organisations enjoy popular legitimacy. The more that citizens perceive an IO to be legitimate – i.e. to exercise its authority appropriately – the more the institution may be able to obtain resources, take decisions, secure compliance, and, ultimately, mitigate problems.² Conversely, an IO with less popular legitimacy can face greater difficulties to act and impact – and indeed may struggle to maintain its role in competition with other sites of governance.³

The importance of popular legitimacy for IOs requires better understanding of its sources, that is, the conditions under which people view IOs as legitimate. Among the key possible sources of legitimacy – highlighted since the early sociological studies of Max Weber – are institutional features.⁴ In this research tradition, legitimacy beliefs are seen to derive from the workings and outcomes of governing organisations. In this article, we scrutinise this proposition, asking whether, in what ways, and to what extent the institutional qualities of IOs shape citizens’ legitimacy beliefs towards these bodies.

Institutional sources of IO legitimacy are the topic of a growing literature. This research generally revolves around a dichotomy of procedure versus performance.⁵ For advocates of procedure, the legitimacy of an IO derives from the way the institution functions, irrespective of the effects of its policies. For advocates of performance, the legitimacy of an IO derives from its consequences, irrespective of how the institution formulated and executed the relevant policy. Recent research offers numerous examples of both types of accounts, as well as emergent efforts to assess the relative importance of procedure and performance.

Although this literature provides valuable insights into the sources of popular legitimacy for IOs, it confronts three important limitations. First, past research has not produced conclusive evidence on the causal significance of procedure and performance qualities for legitimacy beliefs vis-à-vis IOs. Reliance in earlier work on textual analysis or cross-sectional public opinion surveys has not allowed for isolating the causal effects of these two institutional dimensions. In this regard, experimental designs offer possibilities to study causal relationships under controlled conditions, countering common problems of endogeneity and omitted variable bias, as discussed below. Second, the focus on procedure-versus-performance masks the deeper question of what, more specifically, in these features of IO policymaking generates legitimacy beliefs. Each of the two categories hosts a range of particular qualities that may be important for legitimacy perceptions. Yet existing research has not systematically assessed this issue. Third, comparative analyses of institutional sources of legitimacy across IOs are in short supply. Most existing contributions focus on a single organisation.⁶ Whether and how institutional sources of legitimacy vary in importance across IOs and issue areas remains an open question.

Pushing back these three limitations, this article assesses the causal effect on popular legitimacy beliefs of a broad range of procedure- and performance-related institutional qualities across IOs in multiple fields. To this end, our analysis makes important theoretical and empirical contributions. Theoretically, we disaggregate the institutional sources of IO legitimacy, on the intuition that the categories procedure and performance are by themselves too crude to identify the specific institutional qualities that shape citizens’ legitimacy beliefs. Instead, we develop a more fine-grained typology of institutional sources of legitimacy, distinguishing between democratic, technocratic, and fair qualities of procedure and performance. While this new six-fold classification may not be exhaustive, it provides the most encompassing, precise, and systematic typology to date.

Empirically, we evaluate the effects of institutional qualities on legitimacy beliefs through a population-based survey experiment, conducted in four countries with regard to IOs in three issue areas. The experimental design makes it possible to bypass problems in conventional observational methods through random allocation of respondents to control and treatment groups.⁷ The survey experiment is designed to isolate the causal effects of institutional qualities on legitimacy beliefs towards IOs in three issue areas: security governance (United Nations Security Council, UNSC), climate governance (United Nations Framework Convention on Climate Change, UNFCCC), and economic governance (International Monetary Fund, IMF). To increase the generalisability of the findings, the survey experiment aggregates data from four countries in diverse world regions: Germany, the Philippines, South Africa, and the US. Nationally representative samples were selected within each country, involving a total of about 5,700 respondents.

The survey experiment yields three central findings about the institutional sources of IO legitimacy. First, both procedure and performance matter for citizens’ legitimacy beliefs vis-à-vis IOs. Efforts to privilege the one over the other would therefore seem misguided. Second, within procedure and performance, all three tested qualities (democratic, technocratic, and fair) affect popular perceptions of legitimacy. The importance of both procedure and performance therefore holds across multiple qualities of these dimensions. Third, the extent to which institutional qualities matter for IO legitimacy depends on the issue area at hand. A broader scope of institutional qualities appears to be important for legitimacy beliefs towards IOs in economic governance compared to security governance and, especially, climate governance.

Institutional sources of IO legitimacy

This article adopts a sociological understanding of legitimacy as beliefs or perceptions held by a given population that an IO's authority is appropriately exercised.⁸ This approach is distinct from a normative understanding, where an institution's legitimacy is derived from its conformance to moral principles such as justice and democracy.⁹ In the following theoretical discussion, we first review existing literature on institutional sources of sociological legitimacy for IOs and then introduce an expanded typology that unpacks procedure and performance to capture their multiple qualities.

Of course, sociological legitimacy might also derive from conditions other than institutional qualities. Possible other sources include individual-level factors (such as political awareness or social trust),¹⁰ process factors (such as political communication),¹¹ country-level factors (such as domestic regimes or economic development),¹² and world-order factors (such as global norms or hegemony).¹³ While such further potential sources of IO legitimacy merit close examination in other research, this study focuses on institutional features.

Attention to institutional sources of IO legitimacy is prompted both theoretically and practically. In terms of theory, a long tradition of Weberian sociology has demonstrated that the characteristics of a governing organisation shape the legitimacy beliefs of its subjects. However, existing knowledge about the effects of institutional qualities on popular legitimacy beliefs towards IOs confronts large gaps. In terms of practice, ample observed behaviour around IOs has suggested that institutional features shape audience responses to global governance. For instance, dissatisfaction with institutional qualities of IOs has been a prominent part of mobilisation against global economic organisations.¹⁴ All of this suggests that institutional qualities of IOs can play a key role in respect of legitimacy and therefore warrant specific study.

Procedure versus performance

Recent years have witnessed growing efforts to identify institutional sources of legitimacy for IOs. The starting point for most of this literature has been the distinction between input- and output-based legitimacy, introduced by Fritz Scharpf.¹⁵ This dichotomy originally served to identify two alternative normative grounds for justifying the authority of the European Union (EU). In Scharpf's view, the EU could earn its normative legitimacy either from democratic participation by the people or from problem-solving outcomes for the people. This distinction fed into a broader debate about the normative credentials of European and global governance.¹⁶ Scharpf's dichotomy was also picked up by researchers interested in establishing institutional sources of sociological legitimacy for IOs. Over the past decade, a growing literature has distinguished between procedure (input) and performance (output) as two generic institutional sources of legitimacy for regional and global IOs.¹⁷

The premise of procedural accounts is that process criteria are important for perceptions of IO legitimacy. On this argument, actors support an institution's exercise of authority because of how it is set up and operates. Procedural accounts have an early antecedent in Weber's notion of legal-rational sources of legitimacy.¹⁸ On these lines, governance is regarded as appropriate because properly appointed authorities follow properly formulated decision-taking processes. So, for example, audiences might accord legitimacy to the UNFCCC because its policymaking is perceived to involve a broad range of stakeholders. Alternatively, actors might deny legitimacy to the IMF because its decision-making process is seen to give some states disproportionate weight. For procedural accounts, the legitimacy of an IO derives from the way that the institution functions, irrespective of the consequences of its policies.

Recent research offers several examples of procedural accounts. For instance, Thomas Bernauer and Robert Gampfer focus on whether procedures that allow for greater civil society involvement also translate into greater legitimacy for global environmental governance.¹⁹ They find this to be the case: citizens tend to favour civil society engagement and, therefore, procedures that provide for such participation are rewarded with higher legitimacy. Similarly, Tana Johnson studies how procedures giving certain states particular advantages (for example, through vetoes) influence the legitimacy of IOs.²⁰ She finds that IOs which grant major states such as the US and Russia a special say in decision-making suffer in terms of perceived legitimacy.

In contrast, other accounts emphasise performance as an institutional source of IO legitimacy. On these lines, legitimacy beliefs derive from audience evaluations of a governing institution's outcomes. With a focus on performance, IOs might gain or lose legitimacy depending on whether audiences see them as enhancing or undermining desired conditions in society. For example, the World Health Organization (WHO) might gain legitimacy if actors perceive that it effectively prevents epidemics. Meanwhile, the World Bank might lose legitimacy if subjects believe that this institution fails to reduce poverty. For performance approaches, the legitimacy of an IO derives from its impacts, irrespective of how the institution formulated and executed the relevant policy.

Existing research provides many examples of this type of argument. Multiple studies of public opinion towards the European Union (EU) highlight the importance of policymaking outcomes for people's legitimacy beliefs. These investigations show that citizens evaluate the EU's legitimacy in relation to costs and benefits, both for their personal well-being and for their country.²¹ Martin S. Edwards advances a similar argument to explain public opinion towards the World Trade Organization (WTO), the IMF, and the World Bank.²² He finds that people's legitimacy beliefs towards these IOs are primarily driven by the perceived implications of these organisations for their country's economy.

While this existing work has expanded our understanding of institutional sources of legitimacy in global governance, it suffers from three key limitations. First, efforts to compare the causal significance of procedures and performances for popular legitimacy beliefs are missing. While some recent contributions test factors drawn from both categories, they rely on observational methods such as textual analysis and public opinion surveys that make it difficult to isolate and compare the causal effects of procedures and performances.²³

Second, existing research has not systematically unpacked procedure and performance to consider the more specific qualities of processes and outcomes that generate beliefs in the legitimacy of an IO. Procedure and performance are broad and encompassing categories which, while conceptually useful, evade the deeper question: what is it more particularly about an institution's conduct and results that generates perceptions of legitimacy? Each category encompasses a diversity of more specific qualities that may be important for legitimacy assessments. In terms of procedure, is it civil society involvement, decision-making rules, legality, efficiency, or other features that elicit legitimacy beliefs towards IOs? In terms of performance, is it problem-solving capacity, distributional consequences, implications for democratic governance, or other outcomes that generate perceptions of IO legitimacy?

Third, existing research offers little comparative analysis of institutional sources of legitimacy across different IOs. Only a few contributions have assessed the importance of procedural and performance qualities across several IOs or across different issue areas.²⁴ One reason for this gap lies in existing survey data, which are often fragmented across regional samples (for example, Eurobaromenter) and lack measures for perceptions of procedure and performance (for example, World Values Survey).²⁵ Another reason is that conventional observational methods cannot control for the multitude of factors that vary across IOs, making it hard to isolate the impact of institutional qualities on legitimacy beliefs. As a result, important questions remain unanswered. Are some institutional qualities consistently more significant than others in shaping citizens’ legitimacy beliefs regarding IOs? Does the relative importance of specific institutional features vary across IOs in different issue-areas of global governance?

This article moves beyond these limitations. The remainder of this section develops a conceptualisation that unpacks procedure and performance to identify specific qualities of both that potentially affect legitimacy beliefs. Subsequent sections then describe the design and results of a survey experiment that evaluates the causal effects of specific institutional qualities on popular legitimacy beliefs across IOs in three policy fields.

Unpacking institutional sources of IO legitimacy

Building on the distinction between procedure and performance, this section develops a richer and more systematic typology of institutional sources of legitimacy. As the preceding discussion indicates, there is more to the sources of IO legitimacy than procedure and performance per se. To get at the institutional sources for legitimacy beliefs one must also examine the specific qualities of procedure and performance.

Starting from Scharpf's work, previous research has tended to equate procedure with qualities of democracy and performance with qualities of effectiveness.²⁶ However, this conflation excludes the possibility that democracy and effectiveness could each be features of both procedure and performance.²⁷ For example, democratic governance could be an outcome in cases where an IO's activities (such as election monitoring) generate more transparency and participation in politics. Likewise, effectiveness could be a quality of policy processes if a given institutional procedure (such as use of expert committees) allows an IO to produce more and faster decisions. In addition, it is plausible, for reasons elaborated below, that other qualities of procedure and performance besides democracy and effectiveness, such as fairness, could shape legitimacy perceptions.

Based on these considerations, we introduce a new typology of institutional sources of legitimacy by means of a 2 × 3 matrix (Table 1). In this schema, the two rows make the distinction between procedure and performance that has evolved out of earlier research. Hence, the matrix distinguishes between sources of legitimacy that pertain to the processes and to the consequences of IO policymaking. Meanwhile, the columns in the matrix highlight a threefold distinction between democratic, technocratic, and fair as three generic qualities that may apply to both the procedures and the performances of IOs.

Table 1. Institutional sources of legitimacy.

This new typology uses the category of democratic procedure and performance to cover perceptions that affected publics have due voice in and control over governance arrangements. The category of technocratic procedure and performance is taken to encompass perceptions that a governing authority is effective in the light of best available knowledge and policy instruments. The term ‘technocratic’ is not invoked here with any pejorative connotations and rather seeks to convey a sense of expertise-based problem-solving.²⁸ Finally, the category of fair procedure and performance captures perceptions that process and outcome are just, equitable, and impartial vis-à-vis implicated actors. Fairness is a feature of both procedure and performance that has obtained limited independent attention in previous research and is distinctive from both democratic and technocratic qualities.²⁹

Of course, like any analytical construct, this typology involves certain simplifications, exclusions, and imposed priorities. For example, others might wish to give greater distinctive attention to legality as an institutional source of legitimacy.³⁰ In contrast, the present categorisation suggests that legal sources can be sufficiently covered by technocratic and fair procedure. In addition, the proposed typology could be criticised for inadequately capturing organisational mission as a distinct source of legitimacy, based on the intuition that some purposes and mandates generate legitimacy in and of themselves, irrespective of an organisation's other qualities.³¹ This feature may be covered partly by technocratic performance (that is, realisation of goals) and partly by considering IOs in different policy fields. Likewise, the typology could in principle be expanded to cover potential charismatic sources of legitimacy.³² Yet our judgement is that charismatic leadership does not figure as an institutional source for IOs on a par with the six highlighted here. Similarly, the typology excludes religious rationales as a further possible source of legitimacy beliefs, on the notion that religion is not an important legitimating force for most IOs, duly noting certain exceptions such as the Organization of Islamic Cooperation (OIC).³³ Yet, these potential limitations acknowledged, the typology set out here is broader, more systematic, and more specific than existing alternatives. Much as the present article builds on earlier research to deepen insights and enrich debate, we leave it open for subsequent research to explore additional institutional qualities of IO legitimacy.

The 2 × 3 schema yields six types of institutional features that may generate perceptions of legitimacy for IOs. The expectation that people care about these qualities is anchored in social-psychological research. The right to voice opinions, a key political norm held by individuals in democratic societies,³⁴ has been shown to drive perceptions of legitimacy towards domestic political institutions.³⁵ Similarly, outcome favorability is well known to influence how people evaluate policy decisions and institutions.³⁶ Finally, extensive research has demonstrated that fairness is a fundamental concern in politics and matters to people's perceptions of legitimate institutions.³⁷

The following paragraphs specify and exemplify the six institutional qualities in the context of IOs. First, the category democratic procedure covers frameworks and practices that bring affected people into IO policymaking processes. In this category, one prominent institutional quality is participation: namely, where implicated actors have due involvement in the formulation, implementation, and review of IO decisions.³⁸ Another significant feature of democratic procedure is transparency: namely, where affected publics can access full information about an IO's activities and policies, making it easier to hold the organisation accountable.³⁹ Several studies argue that democratic procedure has become the foremost source of legitimacy in global governance.⁴⁰ More specifically, some research finds that civil society involvement strengthens popular legitimacy in global environmental governance.⁴¹ Other work identifies dissatisfaction with allegedly non-democratic decision-making as a crucial motivation for public contestation of IOs.⁴²

The category democratic performance captures the consequences of policymaking for people's power in national and international governance. Important in this respect are IO activities that safeguard or improve domestic democracy, for instance, by demanding accountability of national governments or by protecting civil rights.⁴³ The expectation that democratic performance matters for IO legitimacy is frequently expressed in the negative: namely, that people withhold support from IOs because of their negative implications for domestic democracy.⁴⁴ Such thinking also figures prominently in populist discourse that argues for a repatriation of powers from IOs to democratic domestic governance.⁴⁵

The category technocratic procedure encompasses practices that bring efficiency and expertise to policymaking processes. Efficiency can lie in the number and speed of an IO's decisions,⁴⁶ while expertise can involve basing IO decisions on the best available knowledge and skills.⁴⁷ Along these lines, Gabriel Chan et al. argue that the legitimacy of the Intergovernmental Panel on Climate Change (IPCC) rests primarily on the scientific expertise that this body brings to global environmental governance.⁴⁸ Conversely, slow decision-making, mismanagement of funds, and organisational dysfunction are frequently invoked when explaining legitimacy difficulties for organisations such as the UN and the EU.⁴⁹

The category technocratic performance refers to effectiveness in achieving policy ends. Qualities under this heading include problem-solving (that is, successfully addressing a policy challenge) and collective gains (achieving benefits for society).⁵⁰ It is commonly claimed that IOs earn their popular legitimacy through the collective advantages they produce for states and societies. In this vein, Leon Lindberg and Stuart Scheingold introduced the term ‘permissive consensus’ to describe a situation where populations enjoy the fruits of international cooperation and support its broad goals, while taking little concrete interest in IO policymaking processes.⁵¹ Likewise, functionalist accounts assume that states and their domestic constituencies support IOs because of the collective benefits they generate.⁵² In line with these expectations, some recent empirical research concludes that citizen perceptions of successful IO problem-solving constitute a strong base for legitimacy beliefs.⁵³

The category fair procedure refers to policymaking practices that give equitable treatment to all concerned. This quality is exhibited in impartiality (that is, decision-taking processes are followed consistently and without discrimination) and proportionality (members contribute to IO resourcing in accordance with their relative means). For some types of IOs, such as international courts, fair procedure may be the chief institutional source of legitimacy. For instance, low legitimacy for the International Criminal Court (ICC) among many African governments arises from their perception that the ICC imposes double standards between African and other leaders.⁵⁴

The category fair performance involves consequences of policymaking in terms of equitable outcomes. This quality can be judged in relation to IO practices that advance human dignity (that is, norms of basic cultural, economic, and political livelihood) and distributive justice (equitable sharing of benefits and burdens).⁵⁵ For example, global justice protests have often targeted international economic institutions for allegedly producing unacceptable inequalities in society.⁵⁶ Conversely, IOs with poverty alleviation profiles – such as multilateral development banks and the United Nations Development Programme (UNDP) – often legitimise themselves in terms of promoting fairness for underprivileged people. The expectation is that citizens are more likely to accord legitimacy to IOs that are perceived to reduce injustice within and between countries.

In sum, the typology developed above disaggregates the conventional distinction between procedure and performance to bring out six more specific institutional sources of IO legitimacy. Each of the six qualities gives rise to a hypothesis about effects on legitimacy beliefs (Table 2). Together, these distinctions permit a more fine-grained assessment of the institutional features that drive citizen perceptions of legitimacy in global governance. Social psychology, earlier political research, and anecdotal examples suggest that each of these institutional features may matter for legitimacy beliefs towards IOs. However, it is an empirical question for systematic investigation to establish in what ways and to what extent this is actually the case.

Table 2. Hypotheses and treatments.

Notes: Wording of vignettes from the first experimental round. For exact wordings in the second and third round, see Appendices A and B. After receiving the treatment, people were asked how much confidence they have on IOs: ‘How much confidence do you personally have in the UN?’ Answer categories range from 0 (no confidence at all) to 10 (complete confidence), as well as ‘don't know’. The control group receives the question about confidence without a vignette.

Research design

We assess the institutional grounds for legitimacy beliefs towards IOs through a population-based survey experiment. This method has clear advantages over other tools in identifying causal effects of institutional qualities on legitimacy beliefs. Certainly, the alternatives of cross-sectional and time-series analysis of survey data on IOs can examine statistical associations between independent variables and a dependent variable. The researcher may then interpret these associations as causal effects in the light of theory. However, cross-sectional survey data analysis often suffers from problems of endogeneity and omitted variables.⁵⁷ Endogeneity implies that causality could flow in the reverse direction, making it difficult to isolate the true effect on outcomes of the variable of interest. Omitted variable bias results when a variable excluded from the model influences the effects of variables included in the analysis. Finally, a weakness of existing survey data is the absence of adequate measurements of the different institutional qualities that our theory leads us to test.

Population-based survey experiments avoid these problems. Like observational surveys, these experiments are administered to randomly selected representative samples of the target population. Unlike observational surveys, however, survey experiments use random assignment to establish the causal effects of different institutional qualities on legitimacy beliefs. Randomisation of individuals to treatment groups and a control group ensures that there is no systematic dependence of the observed treatment effects on uncontrolled influences. Population-based survey experiments thereby combine the internal validity of experiments with the external validity of representative population samples.⁵⁸

Survey design

The purpose of this survey experiment is to establish the effects of institutional qualities on citizen beliefs about IO legitimacy. While such effects could have been assessed with a population sample from a single country, estimating average effects of institutional qualities on legitimacy beliefs in several countries reduces the risk of biases from contextual country factors.

The survey experiment was conducted in Germany, the Philippines, South Africa, and the US. These countries were selected based on two criteria. First, they lie on different continents, thereby reducing the risk of regional biases and covering countries with different experiences of the IOs studied. Second, the four selected countries all have relatively high levels of Internet penetration (almost 90 per cent in Germany and the US, over 50 per cent in South Africa, and more than 40 per cent in the Philippines), thereby increasing the representativeness of the sample to the whole populations of those countries.

The questionnaire was implemented through online panels from YouGov, a well-reputed global survey company.⁵⁹ In contrast to interviewer-administered surveys, online surveys are less prone to social desirability bias, which occurs when survey respondents have inclinations to tailor their answers to please the interviewer.⁶⁰ YouGov uses targeted quota sampling with the aim to achieve representative samples that match the full populations of the four countries in terms of age, education, gender, and political-party identification (as well as ethnicity in the US). From September to November 2016 a total of 1,586 interviews were conducted in Germany, 1,358 in the Philippines, 1,384 in South Africa, and 1,393 in the US.⁶¹ YouGov's methodology has been extensively validated⁶² and delivers representative samples on a par with other probability-based methodologies.⁶³

Respondents accessed the survey online. The questionnaire was in English except in Germany, where it was translated into German. The survey took about five minutes to complete. It started with several ‘warm-up’ questions, then moved to the survey experiment, and ended with several additional questions.⁶⁴ The warm-up and supplementary questions were included to enable balance tests.⁶⁵ Questions were sequenced in ways to avoid priming the respondents inappropriately.

Experimental design

The experimental part of the questionnaire aimed to isolate the causal effects on legitimacy beliefs of the six institutional qualities discussed above. To this end, individuals were randomly assigned to groups that received different experimental treatments, as well as to a control group that did not receive any treatment.⁶⁶ The randomised design consisted of two factors that varied across the respondents of the survey: (a) dimension of policymaking (procedure or performance); and (b) institutional quality (democratic, technocratic, or fair). The resulting 2 × 3 factorial design yielded six conditions in total. Each combination of factors was presented to a similar number of individuals (around 820). Table 2 summarises the hypotheses-related treatments tested in the experiment. Taken together, these treatments allowed us not only to identify and compare the respective causal impacts of procedure and performance on legitimacy beliefs, but also to disaggregate these dimensions to evaluate the importance of democratic, technocratic, and fair qualities of procedure and performance.

The experimental treatments were operationalised in vignettes. Vignettes are short narratives about individuals, situations, or institutions that contain the treatment and precede the survey question of interest. The purpose is to assess what, if any, difference it makes when the factors embedded in the vignettes are systematically varied. It was vital that respondents should react to the precise prompt in the respective vignettes. We therefore kept vignette formulations as short and straightforward as possible, and similar in strength, since longer and more complex vignettes lead to a greater risk of ineffectual treatments.⁶⁷ The vignettes involved hypothetical rather than actual scenarios. Although using real-world information can increase the credibility of vignettes, using hypothetical vignettes makes it easier to ensure that treatments are of similar strength.

The vignettes exposed respondents to a description of an IO (see Table 2). For instance, respondents were presented with the information that ‘the United Nations (UN) holds its meetings about military conflicts in public’, or that ‘the International Monetary Fund (IMF) responds on time to financial crises’. The use of framed descriptions (or cues) as vignettes builds on the assumption that citizens on average are not very well informed about political matters and therefore use communicated information to update their opinions on issues.⁶⁸ Survey data suggest that this assumption is reasonable in the present case: citizens are typically aware of the existence of the IOs used in the experiment, but lack a detailed understanding of their decision-making procedures and policy performances.⁶⁹ When respondents are presented with framed descriptions of IOs, the expectation is therefore that they integrate the new information into their opinions. If this new information relates to features of IOs that respondents care about, it should also influence their perceptions of the organisation's legitimacy.

In the treatment groups, people received first a vignette and then a question about their ‘confidence’ in IOs. In contrast, the control group received only the question of how much confidence the respondent has in an IO, without the preceding vignette. Confidence was measured on a scale from no confidence at all (0) to complete confidence (10). Most respondents answered on this scale, although a small minority (close to 10 per cent) selected a ‘don't know’ option.

Contemporary survey research in comparative politics and international relations commonly uses ‘confidence’ (along with ‘trust’) as a measurement for the legitimacy of political institutions.⁷⁰ Relying on this indicator thus allows us to build on established practices and to relate our findings to results in existing literature.

‘Confidence’ also aligns well with our conceptualisation of legitimacy as the belief that an institution exercises its authority appropriately. Confidence captures the belief – affective or rational – that an institution can be relied upon, and thus taps respondents’ general faith in an organisation.⁷¹ Systems theory makes this link explicit when conceiving of legitimacy as the reservoir of confidence in a political order beyond short-term satisfaction with its distributional outcomes.⁷² In this sense ‘confidence’ is preferable to alternative indicators such as ‘approval’ or ‘support’.

The confidence measure furthermore has specific advantages when studying sources of legitimacy. Contrary to some alternatives, this operationalisation of legitimacy does not incorporate potential institutional sources of legitimacy into the measure itself. ‘Confidence’ is clearly distinct in meaning from ‘democracy’, ‘effectiveness’, and ‘fairness’.

The experiment used a randomised factorial design that systematically varied vignettes on combinations of the first factor (procedure or performance) and the second factor (democratic, technocratic, or fair). This made it possible to establish the distinct causal effects on IO legitimacy of each of the six institutional qualities. Each vignette was formulated to operationalise one central institutional quality in each of the six categories: transparent decision-making (democratic procedure); effects on domestic democracy (democratic performance); efficient decision-making (technocratic procedure); effective problem-solving (technocratic performance); equal say in decision-making (fair procedure); and equal distribution of benefits (fair performance) (see Table 1). The effects of these six institutional qualities on legitimacy beliefs were evaluated by comparing mean confidence in the treatment groups with mean confidence in the control group. Since people are psychologically more responsive to negative information than to positive prompts,⁷³ vignettes were constructed using a positive formulation in order to make it more difficult to find treatment effects. Positive treatments also speak to IOs’ efforts to appear more democratic, effective, and fair in their procedures and performances.

In sum, we establish whether an institutional quality matters for respondents’ legitimacy beliefs towards an IO by first exposing them to a treatment containing this particular quality, then asking them about their confidence in the IO, and finally comparing the average level of confidence among the respondents who receive this particular treatment with that of respondents who receive no treatment (the control group). A statistically significant difference in the average level of confidence between the two groups allows us to conclude that the institutional quality that is manipulated in the particular treatment likely contributes to respondents’ confidence in the IO.

The experiment was conducted in three rounds designed to capture institutional sources of legitimacy for IOs in three issue areas: security, economy, and climate. This design permits the experiment not only to test expectations about the general effects of these six institutional qualities, but also to evaluate if the occurrence and strength of such effects vary across different policy fields. The first round presented vignettes related to the UN's actions on military conflict, implicitly referring to the UNSC, the primary IO in the area of security. The second round presented vignettes about the UN's actions on climate change, speaking to the UNFCCC, the central IO on this issue. The third round addressed a prominent economic IO, the IMF. Respondents were never exposed to the same treatment more than once. Respondents who were placed in the control group remained there throughout the three rounds. The order of the experimental rounds was randomised in order to avoid potential priming effects.

Results

The results of the experiment are now presented in two steps. The first examines whether and to what extent procedure- and performance-related institutional qualities affect respondents’ legitimacy beliefs towards IOs. The second considers whether and to what extent the effects of institutional sources of popular legitimacy vary across IOs in climate, economic, and security governance. As discussed above, we will present the results in a series of tables that compare the average level of confidence among respondents receiving a particular treatment to those not receiving any treatment at all (the control group). We follow a standard convention in statistical practice to interpret as statistically significant only treatment effects that have a 95 per cent chance of being found in the full population (p < 0.05).⁷⁴

Effects of procedure- and performance-related institutional qualities

The key finding is that all six procedure- and performance-related institutional qualities affected respondents’ legitimacy beliefs. Table 3 sets out differences in average levels of confidence between the control group and the respondents receiving treatments. All of the treatments presented in Table 3 generated statistically significant effects on respondents’ confidence in IOs, be they about democratic, technocratic, or fair qualities of procedure and performance.

Table 3. Effects of procedure- and performance-related institutional qualities.

Notes: Significance levels: * p < .05. Difference-in-means tests using OLS regression analysis based on weighted data using design weights. Numbers are unstandardised regression coefficients with t-values in parentheses. The covariance matrices are adjusted for within-person correlations in order to take non-observed individual characteristics into account.

The size of the treatment effects varies from 0.256 to 0.528 on an 11-point scale of confidence. Larger effects are found for treatments about democratic performance (0.528) and for treatments about technocratic and democratic procedure (0.460 and 0.425, respectively). Smaller effects are found for treatments related to fair performance (0.383), technocratic performance (0.368), and fair procedure (0.256).

These effects are substantively important. For example, an effect size of 0.528 on the 11-point scale for ‘democratic performance’ is similar in size to the average difference in confidence between the UNSC and the IMF in the control group (see Appendix E in the supplementary material). In addition, the experimental setting likely underestimates the corresponding real-world impact. After all, the experimental effects result from a one-shot treatment, rather than from continuous exposure to a particular institutional quality, as would be the case in actual situations.⁷⁵ Moreover, it should be recalled that positive treatments usually generate smaller effects than negative treatments, since people are psychologically less responsive to positive information than to negative prompts.

Taken together, these results demonstrate that both procedure and performance can affect people's confidence in IOs, and this conclusion holds across democratic, technocratic, and fair qualities of procedure and performance. Hence claims that legitimacy rests with either procedure or performance, or with either democratic, technocratic, or fair features of IO policymaking, would appear to be misguided.

The finding that democratic, technocratic, and fair qualities all matter speaks to earlier research in interesting ways. First, this result supports previous research, which holds that democratic concerns are central to people's evaluations of IOs.⁷⁶ Second, this finding shows that the importance of fairness for legitimacy beliefs applies to international as well as domestic political institutions.⁷⁷ Third, this outcome cautions against the proposition that people are nowadays less concerned with technocratic aspects of IOs,⁷⁸ instead pointing to the sustained importance of efficiency and effectiveness considerations for legitimacy perceptions.

Effects of institutional qualities across issue areas

To examine whether the effects of procedure- and performance-related institutional qualities on legitimacy beliefs vary across IOs in the three fields of climate, economic, and security governance, the results from each experimental round are now considered separately. For this purpose, Table 4 examines, for each issue area, the differences in the average level of confidence between the control group and the respondents receiving treatments related respectively to democracy, technocracy, and fairness.

Table 4. Effects of institutional qualities across issue areas.

Notes: Significance levels: * p < .05. Difference-in-means tests of treatment groups versus the control group using OLS regression analysis based on weighted data using design weights. Numbers are unstandardised regression coefficients with t-values in parentheses. The covariance matrices are adjusted for within-person correlations in order to take non-observed individual characteristics into account.

Although the lower N in each category makes it more difficult to establish statistically significant treatment effects compared to the aggregate analysis, Table 4 reports several notable findings. In economic governance, all treatments are effective, with the exception of the treatment on fair procedure. In security governance, treatments on democratic procedure and democratic performance yield statistically significant effects. In climate change governance, no treatment is statistically significant.

These results suggest four observations. First, they confirm that both procedural and performance qualities matter for the perceived legitimacy of IOs, also when we break down the effects by issue area. In both issue areas with significant treatment effects (economic and security), both procedure and performance are important. Second, the findings suggest that democratic qualities of procedure and performance may be particularly central for people's legitimacy beliefs. Four of seven significant effects result from treatments invoking democratic qualities. Third, the outcomes suggest that institutional qualities are of varying importance for IOs in the three issue areas, mattering most for legitimacy perceptions regarding economic governance, less in security governance, and least in climate governance. Finally, the varying occurrence of treatment effects at the level of issue area reduces potential concerns that the homogenously positive and significant effects established earlier at the aggregate level would result from a general framing effect in the experiment.

Focusing specifically on the variation in treatment effects across issue areas, five interpretations are conceivable. First, variation may reflect the respective missions of the three IOs. As previously noted, it may be that some organisational purposes generate legitimacy beliefs more than others, because these mandates are perceived to be intrinsically good and uncontestable.⁷⁹ If this logic is at play, it may affect the relative importance of other institutional qualities as sources of legitimacy. For example, the UN's attention to climate change may ipso facto generate legitimacy beliefs and thereby reduce the relative importance of procedural and performance features. In contrast, the IMF pursues objectives that are perceived to be more contestable, which may elevate the relative importance of procedural and performance sources of legitimacy.

Second, differences across issue areas could reflect varying levels of authority among the IOs in question. According to one argument, the standards that an IO must reach to be considered legitimate depend on its level of authority, understood as its right to make binding decisions.⁸⁰ Institutions that possess higher levels of authority must meet more demanding standards of procedure and performance or else they will suffer a legitimacy deficit. This logic may help to explain why the full range of institutional qualities matter for legitimacy beliefs towards the IMF, which has far-reaching authority. Consistent with this interpretation, fewer institutional qualities matter in the case of the UNSC, which possesses medium authority, and none at all in the case of the UNFCCC, which has very limited authority.⁸¹

Third, citizens could evaluate some IOs on a broader range of parameters than others. This interpretation ties in well with previous anecdotal evidence on sources of contestation in global governance. Consistent with our experimental findings, public opposition to IOs in the area of economic governance invokes a broad range of criticisms, including democratic shortcomings, perceived injustices of neoliberal doctrine, and ineffective policies.⁸² Similarly, and in keeping with the results here, earlier research on the legitimacy of the UNSC shows that its democratic limitations have attracted particular criticism.⁸³

Fourth, the observed pattern of variation across IOs is consistent with the possibility that citizens respond more strongly to new information about IOs they know relatively less.⁸⁴ While the vignette strategy was based on the assumption that citizens are not very well informed about IOs, respondents may nevertheless be varyingly familiar with the IOs in the experiment. Earlier survey data suggest that the UN is more known to citizens worldwide than the IMF (Gallup International Association 2005).⁸⁵ This finding is further corroborated by a question in our survey on factual knowledge about the IMF and the UN.⁸⁶ Thus, the broader range of treatment effects for the IMF may partly reflect respondents’ greater sensitivity to new information about this IO. Similar dynamics may explain the variation in treatment effects between the UNFCCC and the UNSC. Over the past decade, exposure to several publicly salient UN climate summits may have rendered citizens on average more aware of UN action on climate change.⁸⁷ During the same period UN actions on security have received generally less public attention, which may have made respondents more sensitive to our treatments in this area. However, this type of knowledge effects cannot account for variation across the six treatments in respect of an individual IO.

Fifth, variation in treatment effects may be shaped by contextual factors, such as whether an IO is subject to intense public attention and debate at a certain point in time. Such contextual influences could either increase or reduce treatment effects compared to the counterfactual baseline. Much like general knowledge about an IO, intense public debate around a particular organisation can lead to more developed opinions and therefore less responsiveness to treatment manipulation.⁸⁸ For example, the timing of our survey experiment less than a year after the signing of the Paris Agreement might have contributed to the lack of treatment effects in relation to climate governance. Conversely, contextual events can sensitise people to a particular aspect of an IO's work, leading to larger treatment effects on that dimension. For instance, the results for problem-solving (technocratic performance) in respect of the UNSC might have been different twenty years ago in the aftermath of its shortfalls in Rwanda and Bosnia. Similarly, the treatment effects in relation to the IMF could potentially have been even stronger in the 1990s when the Fund's structural adjustment programs were intensely contested.

Conclusion

Motivated by the premise that popular legitimacy is important for the effective functioning of IOs, this article has explored the sources of legitimacy beliefs with a particular focus on institutional qualities. Moving beyond existing research, it has sought to evaluate the causal significance of procedure and performance, to unpack these two dimensions into specific institutional qualities, and to offer a comparative analysis across IOs in different issue areas. In terms of theory, the article has developed a novel, more encompassing and more precise six-fold typology of possible institutional sources of IO legitimacy. In terms of empirics, the article has presented the findings from a survey experiment among about 5,700 respondents in four countries. The diversity of these countries suggests that results from the combined sample have broad applicability.

The central findings are three-fold. First, the results indicate that both procedure and performance affect citizen legitimacy beliefs about IOs. Second, within procedure and performance, democratic, technocratic and fair qualities all matter for people's legitimacy beliefs towards IOs. Third, a broader range of institutional qualities are important for legitimacy beliefs towards IOs in economic governance compared to security governance and, especially, climate governance.

The experimental approach has been instrumental in arriving at these findings. While conventional survey data are exposed to problems of endogeneity and omitted variables, the experimental design has made it possible to isolate the causal effects of different institutional qualities on legitimacy beliefs. Although the experimental setting diverges from real-world conditions, systematic comparisons between experimental results and real-world outcomes reveal a high level of external validity.⁸⁹

Our findings suggest four broader implications for knowledge and policy of IOs. First, this study corroborates the assumption of earlier research, from Weber to Scharpf, that institutional qualities constitute an important source of legitimacy. Arriving at this conclusion from a focus on six procedure- and performance-related qualities does not presume that no other institutional features may matter, or that institutional characteristics are all that matter for IO legitimacy. Future research may fruitfully assess the effects of additional institutional qualities of IOs (such as mandate and charismatic leadership) or explore how institutional features interrelate with individual and wider societal sources of IO legitimacy.

Second, the importance of multiple institutional sources for IO legitimacy invites additional research on their combined effects. For example, several institutional features might jointly exert stronger and mutually reinforcing (de)legitimating effects. Thus an IO could attract considerably greater legitimacy beliefs when it is perceived to have, say, efficient process together with fair outcome than when it is seen to have only one or the other of these two qualities. Alternatively, positive legitimacy perceptions on one institutional dimension might be cancelled out by negative assessments on another.⁹⁰ For instance, democratic procedure might cease to trigger legitimacy beliefs if it is coupled with ineffective problem-solving. The present study has identified how certain institutional features, taken individually, matter for IO legitimacy; however, future investigations can be encouraged to examine institutional factors in combination.

Third, the results suggest that supporters and opponents of IOs may be right to address institutional qualities in their respective strategies to legitimate or delegitimate these organisations.⁹¹ In this context, the existence of multiple institutional sources of legitimacy can present an opportunity for both proponents and critics. Supporters of an IO may not be dependent on it fulfilling a single overriding institutional criterion. Instead, they can potentially bolster the IO's public standing by nurturing a variety of features, perhaps concentrating efforts on the qualities that the organisation can most easily improve. Likewise, critics can target a wide menu of institutional features in their strivings to delegitimate an IO, possibly focusing their energies on the qualities that are most vulnerable to critique. As a result, legitimators and delegitimators may in their contention around a given IO emphasise different institutional attributes.

Finally, this research has shown that a comparative approach is useful when studying institutional sources of IO legitimacy. Focusing on single IOs can yield particularistic findings without demonstrating broader patterns and relationships. This article has demonstrated how results may vary across IOs in different issue areas and offered tentative interpretations of these patterns. Future research could deepen and extend such comparisons, for instance, by assessing variation across IOs of different types (for example, intergovernmental vs hybrid), of different functions (for example, courts vs executives), and in different world regions (for example, Africa vs Europe).