Wheeling and dealing behind closed doors: estimating the causal effect of transparency on policy evaluations using a survey experiment

Abstract

The Transatlantic Trade and Investment Partnership between the European Union and the US is highly technical. Still, the negotiations triggered large-scale protests among citizens with very diverse socioeconomic backgrounds. Why has a complex issue such an enormous mobilizing effect although the economic consequences are either unclear or favorable for the participating economies? We argue that the transparency of negotiations is an important consideration for people evaluating the negotiation outcome. Conducting a survey experiment, we show that non-transparent decision-making decreases citizens' appraisal of the agreement independent of its outcome: A non-transparent negotiation is, on average, almost 16 percent less likely to find public approval than a transparent but otherwise identical agreement. Our findings have important implications for democratic decision-making.

“The human mind works in a simple way.

If it doesn't have enough facts it starts making things up.

That's how we ended up with religions, and that's perhaps why

we're turning the TTIP into something it isn't.”

- Bert Wagendorp, De Volkskrant (NL), May 3^rd, 2016¹

In the years of 2015 and 2016, the negotiations of the European Union (EU) on preferential trade agreements (PTAs) triggered huge protest movements all over Europe. Especially the negotiations of the Transatlantic Trade and Investment Partnership (TTIP) with the US and the negotiations between the EU and Canada on the Comprehensive Economic and Trade Agreement (CETA) caused nationwide protests in the member states of the EU. These large-scale protests grew even in countries with a trade surplus like the Netherlands and Germany. One unique feature of these movements is that they were not just composed of a small number of globalization opponents but rather included a broad range of citizens with very diverse social and economic backgrounds. This observation becomes even more puzzling with regard to the expectation that the participating countries generally would benefit from the agreement (e.g., Francois et al. 2013; Aichele et al. 2014; Egger et al. 2015; Felbermayr et al. 2015). Why has such a complex issue this enormous mobilizing effect although the potential consequences are to a large extent either unclear or favorable for the participating regions?

The public discourse in the European media presented some explanations which may account for the broad and emotional opposition against TTIP and CETA. Besides concerns about food safety, labor and environmental standards, the investor protection through an investor-state dispute settlement, and against the backdrop of an increasingly polarized political climate, the negotiations have been criticized for not being open to the public (BBC 2015; Economist 2016; Zeit 2016). Even elected representatives only were allowed to examine many documents in a secure reading room in Brussels (European Commission 2015). Like other PTA negotiations with third countries, the European Commission and expert working groups were in charge of the negotiations. Consequently, the public discourse suggests that, besides other aspects, the perceived obscurity of the negotiations shaped people's evaluation of the utility of an agreement and caused the large-scale protests we witnessed.

Yet, the claim that non-transparency has helped provoke the broad opposition, although in line with common wisdom, is problematic in at least two aspects. First, the effects of transparency remain a point of debate in the political economy literature. While the welfare-enhancing effect of transparency is one motivation behind opening up, influential scholars insist that too much transparency can produce disutilities as well (e.g., Stasavage 2004; de Fine Licht et al. 2014). Second, empirical evidence in support of the claim that secrecy spurs public hostility towards a policy is missing. Hence, the nexus between non-transparency and outcome evaluations, as proclaimed in the public discourse on TTIP and CETA, is by no means obvious.

Against this background, the aim of this article is to empirically evaluate the connection between the perceived level of transparency of a negotiation process and the citizens' evaluations of the policy outcome produced. To this end, we design and implement a randomized survey experiment in a representative panel study in Germany. We find that non-transparency reduces a policy's public support independent of its substance. Hence, our findings support the claim that the opacity of the TTIP and CETA negotiations decreased the public support for these PTAs. Even more, we show that the negative effect of secret negotiations is not limited to the policy area of trade but also holds for environmental agreements.

Our experiment shows that the public's evaluation of policies is partly determined by the political process that brings them about. In modern representative democracies, politicians seeking reelection are interested in a positive evaluation of their policies by the public to whom they are accountable (e.g., Manin et al. 1999). Thus, politicians need to be aware of how the decision-making process is perceived publicly, design the process with respect to the effects on public perceptions, and adapt their communication strategies accordingly.

1. EU trade policy: PTA negotiations with third countries

From its origin, the field of trade policy constitutes the most institutionalized and centralized policy area of the EU. As early as 1958, when the Treaty of Rome came into force, the EU has obtained the exclusive competence in trade policy, formally called the Common Commercial Policy (Art. 207 TFEU).² With the Treaty of Lisbon, these competences were further expanded and the role of the European Parliament (EP) got strengthened so that it has an impact—formally and informally—on all stages of the negotiation process of international agreements (Woolcock 2010; Van den Putte et al. 2014; 2015). As Devuyst (2013, 259) argues, “[…] the enhanced role of the European Parliament in the making of trade agreements is the Lisbon Treaty's most important change in this area.”

The formal negotiation process of trade agreements can be separated into three distinct phases. At the beginning of the negotiation process, the Commission proposes a mandate to the Council which has to approve it so that the Commission can embark on negotiations. Although the EP has no formal power in the opening phase, its position still affects the mandate because the EP ratifies the final outcome by a simple majority (Woolcock 2010). During the negotiations, the Commission has to report regularly to both the Council and the EP on the current state of negotiations (Art. 218 TFEU). After the negotiations are completed, the EP—not the national parliaments—ratifies trade agreements (Van den Putte et al. 2014).³ This formal power enhances the EP's bargaining position during all three phases of the negotiation process because it provides a mechanism through which the EP can credibly threaten to turn down the negotiated agreement.

In 2012, the EP demonstrated that it makes use of the newly obtained powers. It rejected the Anti-Counterfeiting Trade Agreement (ACTA) which had been negotiated since 2006. Designed to enforce intellectual property rights, the main reasons for the EP to reject the agreement were normative ones, namely the secrecy of the negotiations and the possible infringement of individual freedoms and the freedom of expression (Van den Putte et al. 2014). This instance was the first rejection of a trade agreement by the EP and further enhanced the EP's credibility and influence in trade mandates and the negotiations (Van den Putte et al. 2015).

Scholars argue that the EP's increased formal competence on trade policy enhances transparency and democratic accountability in this policy area (Devuyst 2013; Van den Putte et al. 2014). Indeed, the TFEU explicitly highlights the role of the democratically elected EP during the negotiation process. Article 218, which concerns agreements between the EU and third countries (or international organizations), specifies that the EP “shall be immediately and fully informed at all stages of the procedure” (Art. 218 (10) TFEU). Consequently, by requiring the inclusion of the elected assembly during the negotiations, this provision implies that the decision-making process shall be as transparent as possible.

Despite this apparent clarity, there is a disagreement among Members of the European Parliament (MEPs) about whether the Commission fully abides by this rule. Especially left-leaning MEPs consider the disclosure of information as selective since parts of the documents, e.g., concerning the negotiating positions, are not distributed to all MEPs (Van den Putte et al. 2014; 2015). Hence, although the Lisbon Treaty enhances the EP's rights in the area of trade policy and thereby strives to make trade negotiations with third countries as transparent as possible, the vague legal text still leaves room for debate about whether negotiations are transparent or not. In turn, this helps to explain why the transparency of trade agreement negotiations can become an issue of public concern as the elected MEPs can publicly criticize the Commission for withholding information and haggling in smoke-filled rooms.

2. Transparency, policy outcomes, and outcome evaluations

Why could a trade agreement with unclear and complex effects mobilize so many people? According to media reports and expert judgments, part of the answer to this puzzle lies in the public perception of the PTA negotiations being an utterly obscure process (e.g., BBC 2015). While previous work in the political economy tradition extensively debates the effect of transparency on negotiation outcomes, it often does not take into account how people perceive the negotiation process. In the following, we derive and test the expectation that the perceived level of transparency shapes public evaluations of the negotiation agreement independent of the substance of the policy outcome. People discount outcomes when they perceive negotiations to be obscure. Therefore, we argue that the level of transparency has a direct impact on people's evaluations.

2.1. The political economy of transparency

“Transparency,” in a definition that many scholars can subscribe to, “refers to the degree that information is made available about how and why decisions are produced within a certain institution” (de Fine Licht et al. 2014, 112). The political economy literature debates the pros and cons of transparency. Traditionally, the argument has been that transparent institutional designs tighten the connection between the actions of political decision-makers and the interests of their constituents, ideally the public interest. In modern representative democracies, it may be hard for voters to determine whether their representatives act upon their best interest. In situations where the outcome is not entirely under the agent's control, as exogenous shocks may intervene, it is hard for a principal to judge the agent's conduct. The agent can dispel the principal's doubt only by providing information about the base for its actions (Holmström 1979; McCubbins et al. 1987). Consequently, by facilitating the scrutiny of an agent's conduct, transparency forces the agent to become more responsive to the principal.

Indeed, there is evidence that transparent institutional provisions make the agent more responsive to public interests, and bring decisions more closely in line with “a socially desirable course of policy” (Stasavage 2003, 400). Transparency, thus, may ensure accountability and responsiveness of political decision-makers and tighten the connection between citizens' preferences and policy outcomes (Besley and Burgess 2002; Alt and Lassen 2006; Ferraz and Finan 2008). Also, Hagemann and Franchino (2016) argue that publishing voting records aids bargaining efficiency and improves the odds of bargaining success as it clarifies the issue positions of negotiation partners. In more specific contexts, transparency might even help decision-makers attain their policy goals. For central banks, Blinder et al. (2008) argue that transparency helps to manage public expectations, and thus helps to coordinate the people's behavior in line with monetary policy goals (see also Horvath and Katuscakova 2016). Across diverse settings, scholars have convincingly demonstrated that opening up the decision-making process can have considerable benefits.

At the same time, other studies emphasize that transparency comes at a price. This second strand of the political economy literature argues that transparency may commit negotiation partners publicly to incompatible positions, decreasing the likelihood of reaching any agreement even if it is mutually beneficial (Putnam 1988; Stasavage 2004). Transparency may undermine decision-making as bargaining partners are more reluctant to share private information knowing that the public is looking over their shoulders (Stasavage 2007). Using transcripts from the US Federal Open Market Committee, Meade and Stasavage (2008) show that a transparency-enhancing institutional change has suffocated dissenting opinions in internal deliberations (but see Hansen et al. 2017). Naurin (2007) argues that transparency may obstruct bargaining practices the public may consider undesirable, such as exchanging expertise with interest groups or package deals, both of which might help achieve a publicly beneficial outcome. Prat (2005) shows formally that observability can incentivize the agent to act upon the principal's expectations about how a good agent should behave, rather than upon the private information that actually helps achieve the principal's best interest (see also Di Lonardo 2017). Taken together, these studies conclude that transparency might reduce the efficiency of negotiations and negatively affect the quality of the final policy outcome.

2.2. Transparency directly affects public evaluations of negotiation outcomes

Although arriving at opposing conclusions, what both approaches have in common is that both expect that transparency exerts an indirect effect on outcome evaluations. People only care about transparency because it affects the quality of the policy outcomes. Transparency-optimists advocate opening up because they argue that doing so may improve the accountability of political decision-makers and their responsiveness to the public. Transparency-skeptics point toward the side-effects that may even decrease aggregate welfare (see also Fehrler and Hughes 2018). In both approaches, transparency of negotiations and public evaluations of the policy outcomes are connected indirectly: only because it alters negotiation outcomes does transparency affect the evaluation of these outcomes.

In addition to this, we theorize that transparency also directly affects the public evaluation of policies. When people learn whether negotiations have been conducted in secrecy, this may lead them to form expectations about their benefits from the final agreement. Non-transparent negotiations hamper the effectiveness of the EP, as well as the media in their role as watchdogs of the public interest (Schoonvelde 2014). When the MEPs cannot update the public about the state of negotiations, they cannot flag developments suspected to undermine the interests of their constituents. Secret negotiations, without this built-in correction mechanism, may provoke people to discount their expected benefit from the final outcome. Fewer observers, whether parliamentary or public, means fewer safeguards against undesirable clauses entering the negotiated text, and a lower probability that the final agreement will serve the public interest.

The label of negotiations being non-transparent thus may reduce the perceived value of the negotiation outcomes. As elected politicians also serve as important cue-givers, they help citizens to form judgments even if people lack the necessary resources to evaluate negotiation documents themselves (Berelson et al. 1954; Bartels 1996). Hence, the level of transparency may affect how an agreement is evaluated independent of its substantive policy implications: two agreements with identical policy consequences will be evaluated differently when people learn that one of them has been negotiated in secrecy. Figure 1 illustrates this important point: in addition to the indirect effect of transparency that is mediated by the quality of the outcome and has been examined previously, we argue that the transparency of the process directly affects citizens' evaluations as it regulates the flow of information about the outcomes before the agreement enters into force.

Figure 1. Direct and Indirect Causal Effect of Transparency on Outcome Evaluations.

Making this argument, we acknowledge that we cannot directly explore the micro-level mechanism. Also, we make no claims as to whether people personally take advantage of the fact that negotiations are conducted transparently. Given the complexity of many policies, including the TTIP and CETA negotiations, there are strong reasons to believe that ordinary people do not use the opportunity to sift through thousands of pages of legal texts (Ripken 2006; Dickson et al. 2015). We simply submit that transparency comes with enormous evocative weight. Transparency of the process may be inherently valuable to people, as it permits a corrective role of actors within the parliament as well as the media on an ongoing basis (de Fine Licht et al. 2014). That is why transparency should have a direct effect on how negotiation outcomes are evaluated, independent of the substance of the outcome. People appreciate the outcome of negotiations conducted transparently. Conversely, they discount the outcome of negotiations when transparency is missing:

Hypothesis 1:

Upon learning that negotiations have been conducted transparently, people evaluate a policy outcome more favorably than otherwise, ceteris paribus (Control condition).

Hypothesis 2:

Upon learning that negotiations have been conducted in secrecy, people evaluate a policy outcome less favorably than otherwise, ceteris paribus (Treatment condition).

The ceteris paribus condition particularly refers to the negotiation outcomes being held constant. Our goal is to estimate the effect of transparency during the negotiations on public evaluations of an international agreement, while minimizing the confounding impact of different expectations about the agreement's overall value. Using the survey experiment introduced below, we can estimate the direct effect of transparency on the overall evaluation of the agreement.

Besides the political economy literature discussed above, a growing body of research investigates how transparency affects public perceptions. However, this work rarely focuses on the evaluation of policy outcomes. Instead, the outcomes of interest include the perceived procedural fairness and acceptance of the final outcome (de Fine Licht 2014), institutional trust (Bauhr and Grimes 2014; Horvath and Katuscakova 2016) or institutional legitimacy (Bernauer and Gampfer 2013; de Fine Licht et al. 2014; Grimmelikhuijsen and Meijer 2015). While some work does examine the effect of transparency on the evaluation of policy outcomes, our analysis differs in important ways. Often, the supposed mechanism is that transparency helps communicate the benefits of a policy (Cook et al. 2010; Boeri and Tabellini 2012; Porumbescu et al. 2017). In many real-world contexts, however, we would not expect that people could unambiguously determine whether they benefit from a complex policy package or not—the comprehensive legal texts of the TTIP and CETA agreements are only two examples. Even if they could, the information they obtain should not lead them to the same conclusions about the benefits from the agreement—some societal groups profit from liberalizing trade, others do not. Conjecturing a direct transparency effect, we expect that non-transparency leads heterogeneous societal groups to discount the expected outcome. We test this implication of our theoretical argument below.

3. Research design and causal identification

3.1. Experimental setup

In order to causally identify the hypothesized direct effect of transparency on the evaluation of a final policy, we implement a randomized survey experiment with a hypothetical vignette. This design allows us to combine the strengths of a representative survey with a controlled experimental design (Atzmüller and Steiner 2010). Our vignette depicts a hypothetical scenario to ensure that there is no pretreatment effect that might bias our estimates. While a validation study by Hainmueller et al. (2015), which compares different vignette and conjoint survey experimental designs to a behavioral benchmark, finds some variation in the relative performance of these designs, it also confirms that vignette and conjoint survey experiments in general perform well in capturing real-world behavior. The only exception the authors report concerns a non-representative student sample. Hence, Hainmueller et al. (2015) conclude that it is essential for survey samples to match the target population in order to maximize the experiment's external validity about the causal effect in a real-world setting.

Our treatment manipulates the degree of transparency during the negotiation process while holding constant the substance of the policy outcome. Since numerous other factors, including concerns about food safety, labor and environmental standards, and the investor protection through an investor-state dispute settlement, most likely caused public reluctance towards TTIP and CETA, our experimental setup mutes these confounders by holding them constant. The dependent variable is the respondent's assessment of the final policy outcome. We randomly assign the treatment among respondents which minimizes the threat of a selection bias and assures that the treatment and control groups are balanced on observed as well as on unobserved characteristics (Horiuchi et al. 2007).⁴

3.2. Implementation of the survey experiment

The survey experiment was conducted in Wave 30 (July 2017) of the German Internet Panel (GIP) (Blom et al. 2018). The GIP is a probability-based online panel study with a face-to-face recruitment procedure. Although the surveys are conducted via the internet, the study encompasses societal groups that are typically underrepresented online. One key aspect of the GIP is to equip households without the necessary technical devices with access to the internet and a computer so that they are able to participate in the survey. As a result, the GIP is considered to be a representative survey of the entire German population which is essential for the external validity of our findings (Blom et al. 2015; Hainmueller et al. 2015).

As mentioned above, we design a hypothetical vignette to avoid a possible bias due to prior knowledge about existing PTAs, and particularly the TTIP and CETA debates. We present a situation in which respondents are asked to evaluate a policy outcome on a six-point scale. The policy outcome respondents evaluate is the result of a hypothetical negotiation between the EU and Australia and is held constant across all experimental groups. Thereby, we control for the indirect effect of transparency that is mediated by the content of the outcome (the solid lines in Figure 1) and focus on the direct effect (the dashed line in Figure 1) which is of main interest here.

We randomly assign all respondents (N = 2, 575) into two different experimental groups of equal size (see Figure 2). One half of the respondents gets a vignette where the negotiations are about a PTA⁵ (n = 1, 284) while the other half receives a vignette where the EU and Australia negotiate an environmental agreement (n = 1, 291). Thereby, we account for possible policy area effects and evaluate whether or not the effects are confined to PTA negotiations or apply to international agreements more generally. This is important since our theory outlined above is not limited to trade agreements. Within each of these two policy areas, we randomly assign respondents into three experimental groups, resulting in a total of six groups with approximately 430 respondents per group. Each group receives a different stimulus concerning the transparency of the negotiation process.

Figure 2. Random Assignment of Respondents to the Six Experimental Groups.

Groups 1 and 4 (“Baseline”) do not get any information about the negotiations and serve as reference groups. Groups 2 and 5 are the control groups to test the first hypothesis. The vignette they receive states that all proposals were made public on the internet immediately. Groups 3 and 6 receive the treatment in order to test the second hypothesis. The vignette they receive states that the negotiations were secret and that the outcomes were made public on the internet only after the negotiations were completed. In addition to the information about the negotiation process and its policy outcome, which we hold constant across all groups, we also provide information about its potential effects. For this purpose, we include one positive and one negative potential consequence of the policy presented in the vignette. Realistically emphasizing both positive and negative aspects, we give respondents a common anchor for what to expect from the agreement. By doing so, we also aim to avoid flooring and ceiling effects which cause a clustering at the extremes of the response scale. While all respondents receive the same positive and negative statements about the policy's general effect, the order in which we present the statements is randomized in order to safeguard against response-order effects like primacy and recency (e.g., Tversky and Kahneman 1981; Krosnick and Alwin 1987).

Because of the respondents' heterogeneity, it is impossible to specify how the agreement described in the hypothetical vignette affects them personally. Instead, we focus on macroeconomic and environmental effects and assume that each respondent perceives it to be positive for her or him if the German economy grows and the environmental pollution decreases. We cannot control for the fact that some respondents might be more dependent on macroeconomic growth and are thus more affected by the outcome we provide than others. Yet, due to the random assignment, we expect those individual differences to be counterbalanced on average.⁶ We state the vignettes concerning the trade agreement in the following way:⁷

“The European Union seeks to promote trade with other countries by implementing trade agreements. Currently, Australia is a candidate for such a trade agreement. We are interested in your opinion about this.

Imagine the European Union and Australia negotiate a comprehensive trade agreement. Both parties agree on the trade agreement, from which the German economy will probably benefit. However, environmental organizations fear that it might increase air pollution.”

In addition to this baseline vignette Group 1 receives, Group 2 and Group 3 get the control and treatment stimuli, respectively:

“The public had [full/ no] access to information about each step of the negotiations. [All proposals/ The results] were made publicly available on the internet [immediately/ after the negotiations were completed].”

The stimulus that distinguishes the transparent from the non-transparent scenario is designed to capture realistically what (non-)transparency means in the context of international agreements. Importantly, all scenarios depict democratic decision-making processes. Although the difference between the experimental conditions appears to be negligible, its implications are considerable. Learning about the results of an agreement after the negotiations are completed prevents citizens from raising their voices in order to affect the outcome. It can spark skepticism as it suggests that decision-makers might have something to hide from the broad public. Furthermore, the lack of knowledge about the current state of the negotiations might trigger speculations about possible scenarios, including unrealistic worst-case outcomes. Consequently, while appearing minor, the distinction between the experimental conditions can be quite consequential.

As in the concrete case of TTIP, the degree of transparency is characterized by how widely information is disseminated, particularly online (European Commission 2015). The treatment and control statements are phrased in a way that closely resembles how respondents learn about the transparency of negotiations in real-world settings, e.g., by relying on newspaper articles or elected MEPs. Moreover, each respondent easily notices the amount of information the public had during the negotiation process.⁸ The question the respondents answer is: “How do you evaluate the trade agreement?” Respondents select their answer on a six-point scale, ranging from “very bad” (1) to “very good” (6).

Importantly, our vignettes allow us to control for the effect of the outcome's substance. Since all respondents evaluate exactly the same policy outcome irrespective of their assignment to one of the three experimental groups (baseline, control, or treatment), differences between the groups are solely attributable to the manipulation of the negotiations' level of transparency. Hence we can examine the direct effect of transparency on outcome evaluations while controlling for its indirect effect through the outcome's substance.

4. Results

In order to evaluate the causal effect of transparent negotiations on the citizens' evaluations of a particular policy, we follow Freedman's (2008, 241) advice “cross-tabulation before regression” and pursue a twofold strategy. We first evaluate the hypotheses by comparing the means of the experimental groups. Subsequently, we model the citizens' latent propensity to choose each of the six answer categories as a function of the policy field and the treatment assignment with a multinomial logistic regression model. Our rationale for this is that, as Freedman (2008) shows, randomization alone is not a justification for logistic regression models and the resulting estimators might be inconsistent. In contrast, he shows that the simple difference-in-means estimator is consistent. At the same time, by imposing further assumptions, a regression model facilitates the substantive interpretation of the results by enabling us to calculate meaningful quantities of interest. Finally, we use the regression estimates and compute potential outcomes for each respondent to quantify the average treatment effect (ATE).

4.1. Difference in means

Following Freedman (2008), we compare the means of the two baseline scenarios to the means of the treatment and control groups.⁹ Table 1 presents the results of our independent samples t-tests.

Table 1. Comparison of Means Between the Experimental Groups

Note: 95% CIs in parentheses. Signif. codes: ***p < 0.01 (two-tailed).

Before proceeding to the modeling stage, we note some important findings. First, respondents appear to favor the environmental agreement over the trade agreement on average. Given the associations that come with free trade versus environmental protection, not least since TTIP and CETA, this is no surprise. Second, transparency leads to a more positive evaluation compared to the baseline scenario for trade agreements but not for environmental agreements. With the two issue areas combined, however, transparency does exert a significant bonus on the outcome evaluations, which supports Hypothesis 1. Third, as Hypothesis 2 suggests, non-transparency negatively affects the outcome evaluation compared to the baseline scenario in both policy areas. It is also important to note that in every instance, the difference between the treatment group and the control group is statistically significant. In order to further investigate these differences, we now turn to a regression model and employ simulation techniques in order to assess substantive effects.

4.2. Multinomial logistic regression model and simulated quantities of interest

Our analysis is based on a multinomial logistic regression model. Although the outcome variable is measured on an ordinal scale, this model avoids the proportional odds assumption made by ordered logistic regression models and estimates a separate coefficient for each variable and response category. The outcome variable is the respondents' policy evaluation (Rating) measured on a scale from 1 (“very bad”) to 6 (“very good”). Since the treatment is randomly assigned among the respondents, statistically controlling for possible confounders is not necessary. The only covariates entering the model are indicator variables for international trade as policy area (Trade), whether the respondent was assigned to the control group (Control), and whether the respondent was assigned to the treatment group (Treatment), irrespective of the policy area. The model also features two interaction terms indicating whether a respondent was assigned to the control group and the policy area of trade (Trade × Control) and whether the respondent got directed to the vignette for the trade policy treatment group (Trade × Treatment). The reference category is the baseline scenario of an environmental agreement in which the respondents receive no information about the negotiation process.

Table 2 presents the estimates of the regression model. The estimated coefficients vary between the categories which supports our choice to estimate a multinomial logistic regression model. Furthermore, the estimates align with the results obtained by the mean comparison presented above. Therefore, we conclude that the additional modeling assumptions do not invalidate the results. Since the raw coefficients are on a log odds scale, we calculate predicted probabilities and first differences for some selected scenarios. We also simulate the sampling distribution in order to appropriately account for sampling uncertainty.¹⁰ Figure 3 shows the predicted probabilities of a positive policy evaluation (ratings of 4, 5, or 6) for the two policy areas and the three different experimental groups. The left part of the figure depicts these quantities for environmental agreements and the right part for PTAs.

Figure 3. Comparison of Predicted Probabilities for Positive Policy Evaluations Between the Three Experimental Groups for Both Policy Fields.

Table 2. Multinomial Logistic Regression Model Estimates

Note: Reference category is “very bad” (1). Standard errors in parentheses.

Signif. codes: *p < 0.1; **p < 0.05; ***p < 0.01 (two-tailed).

Figure 3 shows that transparency in the negotiations only exerts a positive effect for trade agreements. Whereas there is no detectable difference in the predicted probability of a positive outcome evaluation between the baseline and the control scenario for environmental policies (Pr(y ≥ 4|Trade = 0, Control = 1) − Pr(y ≥ 4|Trade = 0, Control = 0) = −0.11% with 95% CI within [ − 6.22%; 5.57%]), transparency increases this probability by 11.49% [5.21%; 18.39%] in the case of PTAs. Consequently, the positive effect of transparency on policy evaluations is contingent on the issue area. We attribute this finding to the high baseline support for environmental agreements already discussed above. Since the predicted probability for a positive evaluation of the environmental agreement in the baseline scenario is 76.40% [72.38%; 80.15%] (gray area in Figure 3), there is simply not much room for an improvement. In contrast, this probability is only 44.38% [39.53%; 49.18%] for trade agreements which implies that there is some potential for correcting evaluations upward.

With respect to the negative effect of secret negotiations on policy ratings, non-transparent negotiations change the probability of a positive evaluation by − 19.63% [−25.49%; −13.52%] for environmental agreements and by −8.55% [−14.62%; −1.97%] for trade agreements as compared to the baseline scenario. When comparing the control and the treatment groups, we find that non-transparency decreases the probability of a positive evaluation by 19.48% [25.80%; 13.01%] for environmental agreements and by 20.01% [26.44%; 13.59%] for PTAs. Hence, while the effects differ between the two issue areas, non-transparency significantly decreases the respondents' policy evaluations in both instances.¹¹

4.3. Quantifying the average treatment effect

Finally, we turn to the main quantity of interest in this article—the ATE of non-transparent negotiations. In order to quantify this effect, we utilize the estimated coefficients from the multinomial regression model and calculate two potential outcomes for each individual respondent. First, we compute the probability of a positive outcome evaluation given that respondent i received the treatment and second, the same probability given that i did not receive the treatment. All other covariates included in the model are held constant at their observed values.¹² Subtracting the two predicted probabilities for each respondent and averaging over all respondents in the sample yield an estimate of the ATE in the sample. More formally, we calculate the ATE in the following manner:

(1)$$\eqalign{ATE & = \displaystyle{1 \over N}\sum\limits_{i = 1}^N P r(y_i \ge 4 \vert Treatment_i = 1) -Pr(y_i \ge 4 \vert Treatment_i = 0).}$$

This quantity represents the average change in predicted probability of a positive outcome evaluation when the treatment status changes from 0 to 1 for all respondents. Importantly, since we hold the other covariates constant at their observed values, the ATE represents the average effect of receiving the treatment across the different experimental groups and the two policy areas included in the survey. In order to properly incorporate the coefficients' sampling uncertainty, we repeat this procedure for each of the 1,000 sets of simulated coefficients obtained above. Figure 4 graphically shows the empirical distribution of the ATE.

Figure 4. Simulated Density of the ATE in the Sample.

Based on this calculation and conditional on the validity of the model, the probability of a positive policy evaluation changes, on average, by −15.89% [−19.75%; −12.06%] when the decision-making process took place behind closed doors. Recall that the estimated ATE is also independent of the substance of the outcome itself since all respondents evaluate exactly the same policy outcome and the treatment is randomly assigned. Therefore, the analysis provides clear evidence for a sizable direct causal link between the (lack of) transparency of a negotiation process and respondents' evaluations of its outcome. In our sample, respondents are on average almost 16 percent less likely to evaluate a policy outcome favorably when their treatment status changes from 0 to 1.

5. Probing the individual-level mechanism: causal effect heterogeneity

In the analysis above, we relied on providing all respondents with a common frame that highlights both beneficial and adverse consequences of an international agreement in order to control for the indirect effect of transparency. Giving all respondents an identical reference standard by holding constant the expected policy outcomes, we showed that transparency is directly related to the respondents' outcome evaluations. In the next step, we aim to approach the individual-level mechanism underlying the observed aggregate result. To do so, we utilize the fact that the expected benefit from such agreements varies: due to differences in the respondents' socioeconomic backgrounds, some groups of people are more likely to support international agreements, like PTAs or environmental agreements, than others.

An influential strand of research argues that highly educated and mobile people are more likely to support globalization than people with more limited skillsets (Scheve and Slaughter 2001; Mayda and Rodrik 2005). By implication, if transparency only indirectly affects policy evaluations by altering the outcome's substance, different societal groups should evaluate an international agreement differently because they asymmetrically benefit from it. Instead, our theory suggests that transparency has an additional direct effect which is independent of the outcome's quality. Hence, if the mechanism we propose is valid, we would expect respondents to punish secrecy irrespective of the outcome's substance, and irrespective of their likelihood to profit individually from globalization. Although the fundamental problem of causal inference impedes a direct test of the individual-level causal mechanism, our theory enables us to derive and evaluate this observable implication.

By doing so, we utilize information on the respondents' perceived benefits from globalization. In the same wave in which our survey experiment was fielded and before respondents got exposed to our hypothetical vignettes, they were asked whether they profit personally from globalization. We use this information to compare respondents who say they profit from globalization by (fully) agreeing with this statement (1) and respondents who do not (0). Also, respondents record their highest professional qualification. Following the literature on globalization preferences, people with high levels of education are more likely to profit from globalization (Mayda and Rodrik 2005). Accordingly, we distinguish skilled workers and people with university education (1) from people without such skills (0). Following our theoretical discussion, if transparency only matters indirectly, that is, only insofar as it affects the negotiation outcomes people really care about, we would expect substantial differences across these subgroups. People likely to profit from globalization, and hence from an international agreement, should hardly care about whether the negotiations are transparent or not. By contrast, if transparency also exerts a direct effect, we would expect people to punish non-transparency irrespective of the policy outcome's substance, and irrespective of their likelihood to profit from globalization.

Table 3 presents the comparison of group-specific means. It shows that the treatment effect is substantial in all subgroups, both defined subjectively and objectively by the respondents' educational background. Self-identified globalization winners discount the outcome of a secretive international agreement (−0.36 [−0.52; −0.21]) as much as globalization losers do (−0.44 [−0.57; −0.30]). Similarly, we find the negative effect of non-transparent negotiations among respondents with limited education (− 0.27 [−0.41; −0.13]) just as among highly educated respondents (−0.58 [−0.73; −0.42]).

Table 3. Testing for Heterogeneous Treatment Effects Across Subgroups

Note: 95% CIs in parentheses. Signif. codes: **p < 0.05, ***p < 0.01 (two-tailed).

With respect to the TTIP and CETA negotiations, these findings help to explain the broad coalition of heterogeneous societal groups against these PTAs. The negotiation process does not merely indirectly shape public policy evaluations by altering the policy substance. Instead, the characteristics of the negotiation process directly shape public evaluations of the outcome. Secrecy triggers public anxiety. Hence, in contrast to the classical arguments in the political economy literature, the mechanism we propose helps to explain the puzzling empirical finding of a broad public reluctance towards TTIP and CETA. It also helps to explain why citizens who most probably benefit from an increase in international trade activity reject these agreements. Independent of the anticipated content of the agreements and their consequences for specific groups, the public broadly opposes agreements that have been negotiated behind closed doors.

6. Conclusion

In this article, we address the puzzle why international trade agreements such as TTIP and CETA generated such an enormous public backlash in European countries. While the economic consequences of these voluminous and complex agreements were not agreed upon even by pundits, they mobilized a heterogeneous coalition of groups with diverse socioeconomic backgrounds. Besides other factors, a common theme to explain why so many people were drawn into the streets was the purported lack of transparency of the negotiations. Yet, the political economy of transparency is debated and empirical evidence supporting this common wisdom has been absent.

Addressing this theoretical and empirical void, we develop and test the argument that not only the substance of an outcome itself but also the negotiation process directly affects the people's evaluations of a policy. Using a representative sample of the German population, we implement a survey experiment and find support for our argument. Across different policy areas, people evaluate the outcome of an agreement much worse when it has been negotiated behind closed doors. Importantly, we also show that this effect is independent of the expected consequences of the agreement. While much existing work merely expects an indirect effect of transparency on policy evaluations, our experiment provides evidence for a direct causal effect.

In order to explore the individual-level mechanism, we test for heterogeneous treatment effects. If people evaluate an outcome solely on the basis of its substance, as argued in the political economy literature, we would expect that people who benefit from globalization do not care as much about transparency as compared to individuals who do not benefit from a globalized world. Instead, we find a homogeneous treatment effect across these groups which supports the claim that people value transparency irrespective of its effect on the substance of an agreement. While our assessment constitutes an indirect test of the individual-level mechanism at work, this essential finding nevertheless helps to explain the puzzling empirical observation of broad public reluctance towards TTIP and CETA.

With respect to the limitations of our research design, future research can build upon our results in several ways. Although the present study focuses on negotiations at the transnational level, the argument developed here extends to other decision-making processes in representative democracies as well. Further studies may therefore assess the external validity of the results by investigating this causal relationship at the national or even at the intra-party level. With regard to the individual-level mechanism, more scholarly effort is needed in order to learn about the mechanisms at work. While our work causally identifies the aggregate treatment effect, the design presented here cannot evaluate the exact individual-level processes triggered by the treatment. Do citizens use the label of transparency as a heuristic to guide their judgment (e.g., Tversky and Kahneman 1981; Bechtel et al. 2015)? Does the label carry less or more weight when it is attached by the media as opposed to political actors (Schoonvelde 2014)? Future research might identify conditions under which transparency matters more or less for the evaluation of policy outcomes. Having established a direct causal link between transparency and policy evaluations, we consider our work as an important first step in this direction.

Our results are also of practical importance. Given that politicians in modern representative democracies are accountable to the public, their chances of reelection are closely tied to the public's evaluation of the policies they deliver. It is therefore of vital importance for policymakers to consider the effects of the decision-making process as well. Our research shows that citizens not only evaluate policies. They also value politics.