The (in)effectiveness of populist rhetoric: a conjoint experiment of campaign messaging

Abstract

Is populism electorally effective and, if so, why? Scholars agree that populism is a set of people-centric, anti-pluralist, and anti-elitist ideas that can be combined with various ideological positions. It is difficult, yet important, to disentangle populism from its hosting ideology in evaluating populism's effectiveness and its potential conditional effects on the hosting ideology. We conduct a novel US conjoint experiment asking respondents to evaluate pairs of realistic campaign messages with varying populism-related messages and hosting policy positions given by hypothetical primary candidates. Although party-congruent policy positions are expectedly much more popular, we find that none of the populist features have an independent or combined effect on candidate choice.

Over the past two decades, populist parties and candidates have been gaining increasing electoral support across the world. Populism has little programmatic content and is almost always combined with other ideologies in reality (Mudde and Kaltwasser, 2018). As a result, it is easy to conflate the effects of populism with those hosting ideologies or related policy stances when they are employed together (Hunger and Paxton, 2022). Despite the observed success of populist parties and candidates, recent survey experiments find that populist appeals are much less effective than programmatic policy priorities in attracting votes (Neuner and Wratil, 2022; Silva et al., 2022). These results further highlight the importance of disentangling these two distinct concepts when evaluating the effectiveness of populism. So, does combining populism with certain policy positions make populist claims-making or related policies more appealing to voters? How effective is populism compared to non-populist alternatives, such as those related to pluralism? Since populism is multi-dimensional, to what extent is the effect of populism based on each of its components or on their particular combination?

To determine which combinations of populist ideas and various ideological stances are most effective for candidate choice, we conduct a novel conjoint survey experiment. In our study, we ask our respondents from a nationally diverse US online survey to evaluate four pairs of realistic campaign messages by hypothetical primary candidates for their party in the next US House of Representatives elections. The campaign messages have been randomized to include the necessary components of populism, common economic and immigration-related policy stances, and relevant background candidate characteristics. In line with the populism literature, our general expectation has been a positive interaction effect between various components of populist messaging so that they are especially effective when combined together. We have also hypothesized that populist campaign messaging might be especially effective among those voters who hold populist attitudes.

Overall, although party-congruent policy positions are expectedly much more popular among the respondents, we find that none of the necessary components of populism have had an independent or a combined effect on candidate choice, including for those who hold populist attitudes themselves. In addition, we do not find any significant interaction effects of populism and various policy positions. We conclude by discussing the implications of these findings for understanding the role of populism in politics and its apparent (in)effectiveness on vote choice.

1. Populism and its effectiveness

Although populism is still a contested concept, many scholars across disciplines have increasingly adopted a minimum “ideational” definition of populism over the past decade. According to this definition, populism is primarily a set of ideas depicting society as divided into two homogeneous and antagonistic groups—the “good” people and the “corrupt” elites—and emphasizing that politics should reflect the general will of the people (Mudde, 2004; Hawkins et al., 2018; Mudde and Kaltwasser, 2018). Unlike classical ideologies such as socialism or nationalism, populism lacks programmatic content and can be combined with a wide range of “hosting” ideologies from across the left–right spectrum (Mudde and Kaltwasser, 2018).

As populist parties from the left and right have gained significant voter support worldwide in the past decades, there have been significant research efforts to explain the reasons behind the electoral success of populist parties and candidates. Some scholars have investigated various individual attitudes, such as dissatisfaction with representative democracy, and structural factors, such as financial and immigration crises, that might fuel the demand for populism (Akkerman et al., 2013; Bakker et al., 2016; Kustov, 2023). However, since populism is always attached to other ideologies in practice, it has been difficult to disentangle its independent effects in observational studies.

The two recent studies that have managed to separate “thin populism” from the policy positions of populist parties in conjoint experiments find that policy positions, in and of themselves, have much larger effects on respondents’ vote choice than populism (Neuner and Wratil, 2022; Silva et al., 2022). However, we still lack empirical evidence on the interaction between the components of populism and their hosting ideologies. Building on this new strand of experimental research, we seek to further disentangle the effect of populism from the hosting ideology, as well as explicitly test the possible interaction effect between the two.

Although parties or voters can only be considered “populist” if they express or hold all of the core components of populism (Hawkins et al., 2018; Wuttke et al., 2020), when one considers real-world political campaigns, the presence of all dimensions is much less common than the various combinations of two dimensions (Engesser et al., 2017). Similarly, only the most politically sophisticated voters hold consistent populist or anti-populist ideology (Spruyt et al., 2021). In other words, it is possible that it might be beneficial for political parties and candidates to only utilize or emphasize certain combinations of populism's main components.

Several recent experimental studies have investigated the effects of these components of populism on voter support and mobilization. Some scholars find that the morally charged anti-elitist rhetoric is the most appealing aspect of populist candidates, especially among those who are low in agreeableness (Bos et al., 2020; Bakker et al., 2021; Silva et al., 2022). At the same time, other scholars find that people-centrism is the most appealing populist aspect (Neuner and Wratil, 2022). However, it is still unclear whether each of these components is more effective in the presence of other components and certain policy positions. This paper seeks to fill this gap by explicitly testing the interactional effects between the components of populism.

2. Data and methods

In April 2022, we administered a nationally diverse US online survey experiment on Amazon MTurk (N = 1004) with quality controls recommended in the literature (Kennedy et al., 2020). Similar to analogous online surveys, our respondents were slightly more likely to be young, male, white, and educated than the general population. Thirty-six percent of our respondents identified as Republicans and 58 percent as Democrats (including leaners). Fifty-six percent of our respondents also could be categorized as displaying populist attitudes. For full descriptive statistics comparing our sample to a probability-based benchmark, see Table A3.

While the US political system offers much less opportunity for organized populist parties than countries with proportional representation, it still provides ample opportunities for populist candidacies (Lee, 2019). Indeed, populist campaigns have been common among political candidates across all parties in the US (Bonikowski and Gidron, 2016; Hawkins and Kaltwasser, 2018; Dai and Kustov, 2022). Therefore, the US case arguably provides a good opportunity to create realistic candidate profiles with a combination of various components of populism and policy positions in a single experiment.

The survey, which was fielded as a part of the broader omnibus study, first presented our respondents with a conjoint task vignette and four pairs of candidate choices, and, following a few unrelated questions, asked about their demographic characteristics and populist attitudes. Our conjoint experiment asked our respondents to choose their preferred candidate from four realistic US House primary candidate profile pairs for their party with randomized attributes. We focus on hypothetical primary contests since any choice of candidates in the general elections would come down to the candidates’ partisan identity, which we would like to avoid since it is not our primary variable of interest (Kirkland and Coppock, 2018).

Compared to previous conjoint experiments on the topic that introduce populism in the form of candidates’ listed policy priorities (Neuner and Wratil, 2022; Silva et al., 2022), our study embeds the components of populism in the form of candidates’ campaign message features. Populist ideas embedded in campaign messages arguably make the overall conjoint procedure more realistic, and better resemble the information voters receive during elections. Our approach also enables us to better disentangle the potential electoral effects of populism and its components with various policy content from this content itself. We operationalize populism along the dimensions of people-centrism, anti-elitism, and anti-pluralism. In addition to populist messages, we specify the non-populist attributes using the direct ideological opposites of populism, such as elitism and pluralism (for a detailed discussion on the operationalization of populism, see Appendix B).

We also include policy positions (immigration and economy) and background candidate characteristics. All candidate attributes were randomly selected from those discrete predefined levels. The main binary outcome was a forced choice from a given pair of candidate profiles (Hainmueller et al., 2014). The list of all conjoint attributes and their possible levels is presented in Appendix B Table B1. Each theoretically distinct dimension of populist and non-populist elements had four distinct manifestations paraphrased from real-world political campaigns. This helped us ensure the relative realism of our experiment despite the fictitious nature of our candidates (Brutger et al., 2022).

To test our main empirical expectation regarding the effectiveness of populist rhetoric, we follow the conventional empirical approach and estimate average marginal component effects (AMCEs) of various attributes on candidate choice using simple linear regression with robust standard errors clustered by respondent. The AMCE represents the average difference in the probability of being chosen when comparing two different attribute values (e.g., a candidate who worked as a lawyer compared to a candidate who worked as a firefighter) where the average is taken over all other possible attribute combinations. To ensure our results are not driven by our choice of reference categories and to make valid subgroup comparisons, we also provide the estimates of marginal means in addition to AMCEs (Leeper et al., 2020).

To simplify the presentation of our results, our main specification recodes pro-immigration and liberal economic stances as party-congruent for those who identify as Democrats and anti-immigration and conservative economic stances as party-congruent for those who identify as Republicans (with 5 percent of pure Independent respondents dropped from the analysis). For our subgroup analyses, in addition to partisan identification, we measure the respondents’ populist attitudes using a previously validated set of questions from Akkerman et al. (2013) and Wuttke et al. (2020) adjusted for the US context. In line with Wuttke et al. (2020), we define “populists” as only those respondents who at least weakly endorse all core components of populism. For the full list of items, see Appendix C. For power analysis, see Appendix D.

3. Analysis and results

Our main results are summarized in Figures 1–3 below. Table A1 provides the detailed coefficients for the models in Figures 1–3 (model 1). Figures A1–3 provide an original coding of policy positions and an alternative coding of our two complex populist-related conjoint treatments as the four simpler, mutually exclusive binary attributes with no change in the underlying results. For other robustness checks, see Appendix E.

Figure 1. Effects of using various features of populist rhetoric on candidate choice. The plot shows the AMCE and marginal mean estimates of the randomly assigned profile and speech attributes on candidates’ probability of being selected. Estimates are based on the baseline OLS model of the original MTurk sample. Bars represent 95 percent CIs. Robust standard errors are clustered by respondent.

Figure 2. Effects of using various features of populist rhetoric by populist attitudes. The plot shows the AMCE and marginal mean estimates of the randomly assigned profile and speech attributes on candidates’ probability of being selected by respondents’ populist attitudes. Estimates are based on the baseline OLS model of the original MTurk sample. Bars represent 95 percent CIs. Robust standard errors are clustered by respondent.

Figure 3. Effects of using various features of populist rhetoric by partisanship (original coding). The plot shows the AMCE and marginal mean estimates of the randomly assigned profile and speech attributes on candidates’ probability of being selected by respondents’ partisanship. Estimates are based on the baseline OLS model of the original MTurk sample. Bars represent 95 percent CIs. Robust standard errors are clustered by respondent.

Overall, as can be seen from Figures 1 and A1, populist rhetoric does not have a statistically significant effect on candidate choice. At the same time, the effects of policy stances are both statistically and substantively significant. A candidate with a party-congruent stance on immigration and the economy has a respectively 7–8 percentage point higher probability of being selected. Importantly, this holds true regardless of the use of populist rhetorical elements in their campaign message (see Tables A1 and 1). Unlike populist rhetoric, these sizable effects of either economic or immigration issue positioning are also robust to Holm–Bonferroni correction for multiple comparisons across all 128 coefficients and models tested in this paper (p<0.01). The magnitude of these effects is also comparable to the estimates for congruent issue positioning uncovered in the previous candidate choices experiments (e.g., Graham and Svolik, 2020; Silva et al., 2022).

Table 1. Effects of policy and populist rhetoric on vote choice

The table shows the AMCEs of the randomly assigned profile and speech attributes on candidates’ probability of being selected. Robust standard errors clustered by respondent are given in parentheses, *p < 0.05; **p < 0.01; ***p < 0.001.

Of course, the lack of statistical significance or the failure to reject the null hypothesis does not necessarily imply that populist rhetoric has no effect. To test whether there is a practical null effect, we follow the recent methodological literature and consider the “two one-sided test” (TOST) procedure (Lakens et al., 2018). In particular, we test for equivalence against the AMCE interval of ±0.04, which is the minimum effect size that can be considered of substantive political significance given our experimental design. Since the observed confidence intervals for all possible populist features compared to non-populist features are fully contained in this interval, we consider this as evidence for the null practical effect of populist rhetoric.

We then test whether the effects differ among certain subgroups of voters such as among those who hold populist attitudes (Figures 2, A2, and A6). Interestingly, we see that such people are somewhat less sensitive to economic policy. Nonetheless, populists are not more responsive to populist rhetoric than non-populist voters. This holds true across various operationalizations of populist attitudes (see Appendix A).

To further test whether there is an interaction effect between populism and various policy positions, we interact recoded populism variable with the recoded party congruent positions and the original immigration and economic policy positions (see Table 1). Populist messages are the ones that contain all of the anti-pluralist, people-centric, and moralized anti-elitist attributes. We also report results from models with interaction terms between each of the binary components of populism and policy positions in Table A2. We find no significant interaction effects between populist rhetoric and policy positions (regardless of their operationalization). There are also no interaction effects between any of populism's components and policy positions. All in all, the populist rhetoric of any form does not seem to impact the electoral effectiveness or attractiveness of immigration and economic policies.

We also break down our results by partisanship (Figures 3 and A3). Overall, there is some indication that Democrats are relatively more sensitive to policy positions vis-a-́vis populist rhetoric. Specifically, unlike those who lean Democrat, the choices of Republican respondents appear to be almost equally driven by candidates’ economic positions and anti-establishment rhetoric. However, none of these subgroup differences are statistically significant when one accounts for multiple comparisons.

4. Discussion

These findings have important implications for our understanding of the role of populist rhetoric in politics. Fascinated by the rise of populism over the last several decades, many commentators have attributed its success to a particular rhetorical style that may be appealing to a substantial part of the electorate. Our experimental results suggest that a substantial part of the appeal that populists have may actually lie in their substantive policy positions (which are hard to disentangle in available observational data). In that sense, while some attributed the electoral success of Donald Trump in 2016 to his populist style, it is possible that his appeal was more related to his (rhetorical) moderation on economic issues and the emphasis on immigration issues.

Similar to previous conjoint experiments on populism (Neuner and Wratil, 2022; Silva et al., 2022), we also do not find that populist rhetoric is particularly appealing to those who hold populist attitudes. Importantly, our study was conducted during the Biden administration (while these previous studies were conducted during the Trump administration), which indicates that the null effects hold regardless of whether the incumbents themselves are populist (Jungkunz et al., 2021). In line with previous research on democratic attitudes (Graham and Svolik, 2020), our paper also suggests that congruent substantive policy positions are the most important factor for vote choice.

Of course, our research is not without limitations. First, our sample is arguably not large enough to reliably detect three-way and small two-way interaction effects even when they exist. Second, our results may be potentially driven by the conjoint design choices, including the paired task and/or particular speech examples. While it is important to see further replications of our study, the current results do suggest that populist rhetoric is unlikely to be effective in itself, especially compared to candidate issue positioning. In this respect, future research would benefit from considering larger and/or more targeted samples, as well as other possible conjoint and vignette experimental designs.