Null Effects of Pro-Democracy Speeches by U.S. Republicans in the Aftermath of January 6th

Abstract

How can political elites strengthen citizen commitment to democratic norms when democracy is under imminent assault? We report results from a pre-registered survey experiment on the persuasive effects of actual speeches given by prominent Republican politicians (Schwarzenegger, McConnell) shortly after the January 2020 insurrection at the U.S. capitol. Although both speeches were widely considered effective at the time, in a survey experiment among Republican voters, we find no impact of one-time exposure to these speeches on the endorsement of democracy, the acceptance of election losses, the rejection of political violence, or the relevance of democratic norms in hypothetical vote choices.

Introduction

On January 6^th, 2021, after attending a rally with Donald J. Trump to protest the certification of the 2020 U.S. Presidential election results, masses of demonstrators moved to the U.S. capitol, violently overwhelmed the police forces, and entered the congressional chambers. By occupying the buildings that house American democracy’s central institutions, the rioters interrupted the ongoing process of certifying the election results. In the weeks following these events, law enforcement authorities prosecuted the rioters and the House of Representatives impeached Donald Trump for inciting an insurrection.

Commentators first speculated whether this “failed coup attempt” (Solnit 2021) would mark a turning point in the process of American democratic backsliding that had set in after Donald Trump took office (Gricius 2022). There is some evidence that support for the Republican Party declined significantly in the direct aftermath of the Capitol riots, both when measured in Tweets (Eady, Dinesen, and Hjorth 2022), in voter registrations (Loving and Smith 2022) and at the polls (Noort 2021). However, in the weeks following the riots, many prominent Republican lawmakers downplayed the relevance of the incidents at the Capitol and most elected Republicans voted to acquit President Trump in the U.S. Senate, impeding his conviction for inciting an insurrection. Mirroring this elite communication, among ordinary citizens with affinity to the Republican party, acceptance of political violence has increased since the insurrection (CNN 2022). In other words, not all Republican politicians and voters have interpreted the overt attack on democratic principles and procedures on January 6^th, 2021 as a reason to strengthen their often fragile commitment to democratic norms (Arceneaux and Truex 2022).

Fragile commitment to democracy among a significant share of voters or politicians is troubling for the prospect for democracy because in a democracy both elite politicians and ordinary citizens have a role to play in maintaining the rules of the game. By using their ballots strategically, voters can prevent anti-democratic politicians from attaining political power (Graham and Svolik 2020; Simonovits, McCoy, and Littvay 2022). Politicians in office, on the other hand, can strengthen or undermine the quality of democracy by observing or violating democratic norms (Levitsky and Ziblatt 2018). Democratic persistence therefore depends on the dynamic interactions between citizens and elites. The demands of citizens set electoral incentives for politicians who aspire to be re-elected and, by punishing undemocratic behavior, citizens may steer politicians toward adhering to democratic norms. At the same time, through their communicative power, politicians can shape which considerations matter most to voters (Lenz 2012). Hence, democracy is most likely to thrive when citizens sanction violations of democratic norms and politicians persistently endorse these norms.

Recent research suggests that politicians may be effective at using their communicative power to foster support for liberal democracy (Wuttke and Foos 2021), at least in non-emergency, deliberative contexts, where the survival of democracy is not the most prominent issue on the agenda and where a wider elite consensus on this issue exists. This raises the question of what can be done when significant segments of a party’s voters and elites no longer prioritize democratic norms and when democracy is under imminent attack.

In this study, we consider the effectiveness of politicians who are committed to democratic norms and use their voice to speak out in favor of this system of government – even when other elites in their own party send countervailing signals. Specifically, in the immediate context of the salient and contested events at the US capitol in January 2021, we investigate whether exposure to pro-democracy speeches by Republican politicians succeeded in bolstering support for democracy among Republican voters – the segment of the population where commitment to democratic norms was lowest (Bartels 2020; Arceneaux and Truex 2022). Hence, we examine the efficacy of pro-democracy communication at a crucial time in the history of U.S. democracy and on segments of the population that are most relevant to persuade.

We investigate the persuasive impact of two prominent interventions in the service of reinforcing democratic norms that commentators considered highly persuasive in the context of the events on January 6^th, 2021 (Ellefson 2021): a video message by former Governor Schwarzenegger on the value of democracy based on the experience of democratic breakdown in his native Austria and a speech on the Senate floor by Majority Leader Mitch McConnell on the importance of the peaceful transfer of power.

We deliberately selected persuasion attempts by well-known Republicans over alternative interventions by non-partisan actors or politicians from the Democratic party because we considered in-group communication by Republican elites on Republicans as most likely to elicit discernible persuasion effects. It is a well-established fact in public opinion research that citizens are more susceptible to elite cues from their own party (Bullock 2011; Clayton and Willer 2021; Lenz 2012). Moreover, questions surrounding the election results and the Capitol riots were strongly contested and public opinion was divided along partisan lines (Arceneaux and Truex 2022). Hence, attempts by Democratic politicians to win over Republican voters may have backfired (Nyhan 2021). In-party cues also appear to matter for Republican voters’ support of democratic norms (Gidengil, Stolle, and Bergeron-Boutin 2021; Kingzette et al. 2021). Therefore, we reasoned that for a communicative intervention to have persuasive impact on Republican voters even when many of them had already developed strong opinions on these issues, the messages should come from within their own party. Hence, given that we expect persuasive effects of Republican elites on the attitudes of Republican voters, in this study we test the following pre-registered ¹ hypothesis:

Listening to a politician making the case for democracy and democracy’s underlying norms strengthens citizens’ commitment to democracy and its norms.

Experimental design

We fielded a survey experiment in the United States in February 2021. All participants voted for President Trump in the 2020 Presidential Election. Half of the subjects were randomly assigned to a placebo group and watched a political video that was unrelated to the outcomes of interest (Ronald Reagan on the exploration of space, https://youtu.be/yyRMjVV9Gnc). The other half were assigned to one of two democratic norm treatments: an 8-min video by Arnold Schwarzenegger (https://youtu.be/Au1PUXcH7EM) or a speech of similar length by Mitch McConnell on the Senate floor (https://youtu.be/I9_2CRBfYYc). Both treatment videos can be understood as attempts by Republican political elites to strengthen in-partisan commitment to democratic norms and values, although they differ in tone and content.

Schwarzenegger conveys the value of democracy using figurative, emotional language. He references the experience of democratic breakdown in the country of his birth, Austria, in the 1930s. He compares the violence during the assault on the capitol with the “Night of Broken Glass” in 1938 and warns of the “dire consequences” of attempts to overturn the results of fair elections. He closes with a hopeful message of democratic systems becoming stronger the more they are tempered with.

Mitch McConnell focuses on the principles of electoral democracy and makes the case for accepting the 2020 Presidential election results. He warns that “if this election were overturned by mere allegations from the losing side, our democracy would enter a death spiral. We’d never see the whole nation accept an election again. Every four years, there’d be a scramble for power at any cost. Self-governance requires a respect for the truth and for the ground rules of our system.”

Schwarzenegger, the former Republican Governor of California, is seen as a centrist figure within the Republican Party and is known as a long-time critic of Donald Trump who did not vote for him in the 2016 Presidential Election (Politico 2016). McConnell, majority leader in the Senate during Trump’s time in office, eventually endorsed Trump in 2016, voted for him in both elections and was crucial in getting elements of Trump’s agenda enacted during his time in office (NYT 2021). ²

Schwarzenegger and McConnell hence represent a relevant spectrum of opinion among Republican party elites, but it is worth noting that our survey does not include a treatment arm of pro-democratic messaging by one of Trump’s most ardent supporters or by one of the Senators who objected to the certification of Arizona’s electoral votes on January 6th, as none of them prominently and unambiguously spoke out for liberal-democratic principles directly following the insurrection of January 6th. Displaying such a hypothetical intervention would have been technically challenging and would have introduced an important element of deception that we intended to avoid.

Data

On February 19^th, 2021, we fielded an online survey experiment with Prolific. ³ Prior research has shown that in terms of comprehension, attention, and honesty, data quality on Prolific outperforms other survey vendors such as Mturk (Eyal et al. 2021). For the 13-minute survey, participants received the equivalent of 1.64 British Pounds compensation. The survey was restricted to U.S. citizens who had voted for Donald Trump in the 2020 Presidential elections. In line with our a-priori power analysis, we surveyed 660 respondents (also see Appendix). We excluded one respondent for having failed a pre-treatment attention check.

For this sample size, we estimated statistical power of 80% to detect effects larger than Cohen’s d = 0.2 and 94% power to detect effects larger than Cohen’s d = 0.25. As we employ a direct and overt (instead of subtle) manipulation over several minutes, we pre-registered the expectation of effects larger than Cohen’s d = 0.25. For an effect of this size, 90% of the control and experimental group distributions in the outcome variable would still overlap and the probability that a participant in the experimental group would exhibit stronger support for democracy than participants in the control group would be only 57.0%. In such a controlled setting with an overt experimental stimulus, one might consider smaller effect sizes as having negligible practical relevance.

Methods

Before data collection, we pre-registered ⁴ the hypothesis, power analysis, outcome variables, and analysis plan.

We pre-registered a group-sequential testing design (Lakens, Pahlke, and Wassmer 2021), which is a principled approach for flexible research designs with the option for a second round of data collection based on the results of the first peak at the data. ⁵ As explained in the pre-analysis plan, to account for this design choice we set the alpha level to 0.035 as opposed to 0.05 which is the conventional threshold for statistical significance. Hence, we consider results for one-sided tests with a p-value below 0.035 as statistically significant.

To estimate the ATEs, we use the covariate-adjusted Lin estimator (Lin 2013), relying on a pre-registered list of pre-treatment covariates to reduce sampling variability. The analysis was implemented based on the estimatr package (Blair et al. 2019). As pre-registered, we collapse both treatment conditions (Schwarzenegger, McConnell) for greater efficiency, but we also report disaggregated models.

Measures

As commitment to democracy is a multi-faceted concept, we use various approaches to tap into different aspects of citizens’ commitment to democracy.

One self-reported measure captures abstract attitudes toward democracy as a generic concept (“Democracy may have problems but it’s better than any other form of government,” that is, “Churchill attitude”). Yet, prior research has shown that positive sentiment toward that term may reflect meaningless” lip service” (Inglehart 2003). To assess the personal relevance of self-governance to the respondent, we asked “how important is it for you to live in a country that is governed democratically?”

Both of these measures refer to democracy as an abstract concept and do not lay out its meaning. As a consequence, respondents might endorse democracy in the abstract without actually supporting the specific principles that constitute this system of governance (Kirsch and Welzel 2018; Wuttke, Gavras, and Schoen 2020). To account for this problem, we use two self-reported measures that refer to specific norms of electoral democracy. Each concerns one aspect of the peaceful transfer of power which seemed most threatened in the wake of the assault on the U.S. capitol: the acceptance of defeat in fair elections and the rejection of violence as a legitimate political instrument.

To assess support for the principle of non-violence, we borrowed a question from the Democracy Fund Voter Group December 2019 survey and asked “How much do you feel it is justified for Republicans to use violence in advancing their political goals these days?” For the second item, we presented respondents with a hypothetical scenario for the 2022 mid-term elections. In this scenario, facing narrow results in several districts, Republican leaders urge candidates not to concede defeat although courts rule that the ballots have been counted fairly and that any irregularities could not have tipped the outcome of the elections. We presented respondents with a binary choice of whether the Republican candidate in this case should “oppose the certification of the election results despite the court rulings” or “concede defeat despite minor irregularities in the election.”

Finally, to assess the relevance of democratic considerations for voting decisions, we included a conjoint experiment in which respondents had to choose between two candidates with randomly varied characteristics (Graham and Svolik 2020). Next to a photo and information on the candidates’ partisanship and profession, issue stances, and trivia, (some) candidates were described as having supported or violated the norms of non-violence or electoral democracy. Note that we collapse the norm violation dimensions for greater statistical efficiency (see appendix for disaggregated results).

• Election norms, support: “Before a close election, this candidate put country over party and publicly condemned the local mayor of the candidate’s own party who was taped on video seemingly encouraging others to strike out ballots of the competing candidates”

• Election norms, violation 1: “Praised members of congress who voted against certifying the 2020 election results”

• Election norms, violation 2: “Opposed certifying the 2020 election results and tweeted that ‘proud Americans will never accept rigged elections even when corrupt courts say they were free and fair’”

• Non-violence, support: “Strongly condemned anybody who entered the U.S. capitol on January 6th or who incited the hate and anger that led to these events”

• Non-violence, violation 1: “After supporters of the candidate unprovokedly intimated and physically attacked members of the other party, the candidate said in a 2020 interview: ‘What goes around comes around. What can you do when ordinary people are treated so badly? Honestly, I understand every American who is upset about politics and takes it to the street’.”

• Non-violence, violation 2: “Was photographed and later confessed to have illegally entered the buildings on the Capitol after protests against certifying the 2020 election results”

Across two rounds, respondents were asked to choose between two hypothetical candidates with randomly varied attributes. Moreover, respondents were asked to rate how democratic they perceived each candidate to be. Specifically, we asked the following: “Some politicians put democratic principles above everything else. The approach of other politicians to democracy is more strategic. To what extent do you think each candidate is committed to democratic principles?” On a seven-point scale, respondents could choose between: “Not committed to democratic principles – Fully committed to democratic principles”

Overall, this battery of questions uses different techniques and taps into distinct aspects of democratic commitment. Hence, the multiplicity of survey instruments increases our confidence to not overlook potential effects of the treatment on democracy-relevant outcomes.

Results

A necessary condition for an informative experiment is that the respondents attentively watched and processed the speeches they were assigned to. In line with the pre-registered expectation, in the placebo group 99% of respondents (correctly) identified “exploration of space” as the topic of the video they had just watched (experimental group: 0.30%). In contrast, respondents in the treatment condition often identified “2020 U.S. Presidential elections” (82%, Placebo: 0%) or “democratic principles” (73%, Placebo: 5%) as the topic of the videos. We conclude that the vast majority of the respondents watched the video as instructed and correctly inferred what it was about.

Moving forward to the sample’s susceptibility to persuasive influences, we examine agreement with the statement that “the U.S. government should invest much more in the U.S. space program.” This statement reflects the main message of the video that the placebo group was assigned to. Figure 1 shows that respondents in the placebo group were considerably more supportive of the U.S. space program compared to respondents who watched videos on democracy (Cohen’s d = 0.53). These data suggest that respondents in our sample were generally susceptible to persuasive influences on political topics as delivered in a video-taped speech by a Republican politician.

Figure 1. Placebo effects.

Yet, similar persuasive effects did not materialize in relation to any of the outcome variables capturing respondents’ commitment to democratic norms. Figure 2 shows that watching the pro-Democracy speeches by Schwarzenegger or McConnell did not bolster commitment to democracy on any of the outcome variables (see Appendix for tabulated results).

Figure 2. Effects on commitment to democracy.

Panel A reports abstract attitudes toward the generic term of democracy. The pre-registered model specification yields no discernible differences between respondents in the placebo and treatment groups (p = 0.43). The results remain unchanged for a model without covariates (Fig. 2, Panel A, 2nd estimate, p = 0.35). The results also do not change substantially when disaggregating the treatment groups. Neither the Schwarzenegger, nor the McConnell-video resulted in significant effects on any outcome. Hence, no model specification provides evidence of meaningful influence of the two pro-democracy speeches on the recipients’ abstract attitudes toward democracy.

This finding repeats for the other outcome variables shown in Fig. 2. The videos failed to strengthen the importance respondents attached to democracy (p = 0.47) and did not change the viewers’ stance toward political violence (p = 0.49). One methodological reason for the absence of persuasive effects might be ceiling effects of the strongly right-skewed attitudes. Yet, there are also no effects on the willingness to concede defeat in narrow elections (p = 0.17) although a substantial number of respondents express nondemocratic positions and even though this topic was at the heart of both speeches.

What ultimately matters for democratic persistence is whether citizens translate pro-democratic attitudes into behavior, for example, in the voting booth. Hence, a final set of measures captures revealed preferences which are more likely than self-reported attitudes to circumvent expressive reporting or social desirability biases. We analyze whether prior exposure to pro-democracy speeches led to greater consideration of democracy-related factors in a hypothetical vote choice between two candidates with randomly varied attributes.

Replicating previous findings (Graham and Svolik 2020; Simonovits, McCoy, and Littvay 2022), we show in the appendix that support for a candidate varies with their positions toward democracy. Here, we are mainly concerned with the question of whether the experimental stimulus affected the AMCEs of the candidate attributes on hypothetical vote choice. Reporting the difference in marginal effects between experimental conditions, Fig. 3 shows that exposure to pro-democratic messages does not increase the importance of democracy-related candidate attributes compared to the placebo group. Although the estimates were measured with considerable noise, these findings do not provide evidence that the treatment videos made respondents more likely to consider democracy in their hypothetical voting decisions.

Figure 3. Treatment effects on hypothetical vote choice.

Note: Reference categories: Election norms, support (Background), Republican-leaning independent (Partisanship), Teacher (Profession), Minimum wage: opposes raise (Issues).

A final analysis further explores what underlies the attitudes and behaviors we observe in this study. As it is necessary to first recognize a norm violation to then sanction it, Fig. 4 shows what kind of candidate behavior respondents perceived as democratic and undemocratic. The figure again reports the results from the conjoint experiment, but uses a different type of visualization, showing how much each attribute affects perceived democraticness of a candidate, separated by experimental condition.

Figure 4. Relevance of candidate characteristics for perception of democraticness.

As one might expect, respondents rated highly the democratic commitment of candidates who followed democratic norms. Candidate who “put country over party” and called out a party official who engaged in voter fraud (election norms, support) received high democraticness ratings just as candidates who condemned the violence at the Capitol (non-violence, support). Candidates who lacked these pro-democratic credentials were perceived as less committed to democratic norms.

Candidates who violated the norms of electoral integrity (election norms, violation) or non-violence (non-violence, violation) are rated as less democratic than candidates who supported these norms. At first sight, this may attest to the capability of ordinary citizens to spot democratic norms violations. Yet, disturbingly, the level of perceived democraticness of democratic norm violators is similar to the level of perceived democraticness of candidates who had extramarital affairs, underpaid income taxes, or misused campaign funds for private purposes (negative valence issues). In other words, Trump voters on average seemed to recognize the violation of democratic norms but do not perceive democratic norm violations as undermining a candidate’s democratic credentials much more than any other personal or political wrong-doing.

Finally, in the appendix we report results from additional analyses. Equivalence tests (Lakens 2017) show that the reported findings can be considered practically equivalent to zero as we can rule out effects that would exceed the pre-registered smallest effect size of interest. Moreover, analyses with interaction terms and machine learning random forest models (Wager and Athey 2018) on a pre-registered set of pre-treatment variables show no evidence of relevant treatment effects within subgroups of the sample.

Discussion

The experiment reported in this study provides no evidence that Republican politicians who made the case for democratic principles amidst a democratic crisis fostered the commitment of prior Trump voters to these principles. The lack of persuasive effects on democracy-related orientations goes against our pre-registered expectations as the experiment in various respects was designed to maximize treatment effects: We selected co-partisan communicators to exploit source cue effects. We exposed respondents to treatment material for several minutes. To relax dependence on one stimulus, we selected two different messages delivered by speakers with a different history in the Republican party and a different relationship with the former President. Both videos were considered highly persuasive in the broader public debate and by many commentators. Anticipating potential effect decay, we measured attitudes right after the treatment was administered. Finally, we used multiple techniques to measure different facets of commitment to democracy. Yet, despite these precautions, no substantial persuasion effects on the respondents’ democratic commitment were detectable. What could explain the absence of any detectable treatment effects on democracy-relevant outcomes?

One explanation concerns the internal validity of the reported results. Balance tests show that the survey software allocated 361 respondents to the placebo group but only 326 respondents to the control group. Yet, additional analyses (reported in the Appendix) provide no evidence of differential attrition between the experimental groups. Balance tests also show no difference in pre-treatment attitudes between the experimental groups, suggesting that the imbalances in respondents across groups are likely due to chance, given simple random assignment, and do not invalidate the reported findings.

Turning to substantive explanations, it is conceivable that the experiment resulted in null effects because respondents in both control and treatment group had already been treated with the same stimulus in the sense that they had previously been exposed to the treatment material. While it is plausible that some respondents had already seen the treatment videos, given the generally low level of political involvement among average US citizens (Achen and Bartels 2017) and the general tendency to avoid identity-threatening messages (Bullock 2011), we consider it unlikely that a large share of respondents had already watched the 8-minute videos. As pre-treatment exposure was not measured in the sample, the exact level of pre-exposure to the videos remains unknown. But given the size of the US population, even the video’s 6 million views on Schwarzenegger’s YouTube channel and the videos’ availability on multiple other channels and platforms does not change our belief that only a minority of subjects in this experiment would have seen the entire video messages before the experiment.

Still, even if respondents had not previously encountered the specific treatment stimuli, they were likely exposed to numerous messages on the topic and, thus, had been pre-treated in a different sense. We conducted the experiment in the context of a saturated media environment after multiple weeks of intense reporting on the insurrection and the ensuing impeachment procedures. Hence, when the experiment was conducted in mid-February, many respondents could conceivably already have made up their mind and developed crystallized, strong attitudes that are resistant to persuasion (Howe and Krosnick 2017). Perhaps, in this media environment in particular, a single communicative intervention was doomed to fail, in a survey experiment, and beyond. Recent experimental research for instance shows that TV advertisements had minimal effects on political attitudes in a real-world settings (Kalla and Broockman 2021). At the same time, our findings are also consistent with broader, more recent trends in public opinion among a majority of Trump voters and Republican officials (CNN 2022), who continue to support the former President and his decision not to accept the 2020 election results, despite the wide-spread and widely aired condemnation of Trump’s actions during and following the capitol insurrection on January 6^th, 2021.

To the question of whether our survey experiment adds informational value about the effects of real-world communication, it is worth recalling that political campaigns and commercial advertising often aim to hit their audience multiple times with the same message. Our study cannot rule out that crystallized attitudes such as the outcomes measured in our studies are resistant to one-shot persuasion but might bend upon repeated exposure (Foos and Bischof 2022). Similarly, it remains an open question whether treatment effects would have materialized if the study had been conducted at a different time point, for instance in the days directly following the Capitol riots. Moreover, it is possible that a random sample of the US population would have reacted differently to the treatment than our Prolific respondents. However, there is little evidence of systematic treatment effects heterogeneity in persuasion survey experiments across probability and non-probability samples (Coppock, Leeper, and Mullinix 2018). Finally, given the importance of source cues for persuasion effects (Lenz 2012), it is possible and perhaps even likely that unambiguous pro-democracy messages from Donald Trump himself or his ardent supporters might have changed the attitudes of his voters. But in this moment of crisis, many of these politicians decided against speaking out publicly and powerfully for the importance of liberal-democratic principles.

Given these limitations, the reported findings should not be taken as conclusive evidence that Schwarzenegger’s and McConnell’s pro-democracy speeches were in vain and had no communicative effects whatsoever. More narrowly, the study shows that exposing Trump-voting Prolific panelists to these speeches in highly politicized times and in the context of an academic study failed to induce any changes of relevant size on a variety of democracy-related attitudes and behavioral inclinations. To the extent that readers, like us, would have expected such carefully crafted messages to elicit attitude changes even in the specific context of our study, the findings help us to update prior beliefs on the persuasiveness of such messages. They point to potential limitations of persuasive messaging on citizens’ support for democratic principles and norms from in-party politicians at times when those parties are internally divided. More broadly, these findings hint at a research agenda on the conditions under which pro-democracy communication will and will not be successful.