Evaluating methods for examining the relative persuasiveness of policy arguments

Abstract

Survey researchers testing the effectiveness of arguments for or against policies traditionally employ between-subjects designs. In doing so, they lose statistical power and the ability to precisely estimate public attitudes. We explore the efficacy of an approach often used to address these limitations: the repeated measures within-subjects (RMWS) design. This study tests the competing hypotheses that (1) the RMWS will yield smaller effects due to respondents' desire to maintain consistency (the “opinion anchor” hypothesis), and (2) the RMWS will yield larger effects because the researcher provides respondents with the opportunity to update their attitudes (the “opportunity to revise” hypothesis). Using two survey experiments, we find evidence for the opportunity to revise hypothesis, and discuss the implications for future survey research.

1. Introduction

Survey experiments are an essential tool for examining reactions to political stimuli. Researchers have used them to demonstrate the public's bias against racial minority and female candidates (Terkildsen, 1993; Krupnikov and Piston, 2015; Bauer, 2020). They have shown the effect of campaign ads on attitudes toward politicians (Valentino et al., 2002). And they have demonstrated that elite cues affect policy preferences (Lupia and McCubbins, 1998; Guisinger and Saunders, 2017). Survey experiments are also useful for campaigns crafting persuasive messages, activists lobbying for a policy, and pollsters seeking to understand how the public views political issues.

Yet traditional messaging experiments have limitations. In the traditional post-only design, respondents are randomly assigned to receive a single message for or against a policy prior to answering questions about that policy. These opinions are compared to those from individuals assigned to a control condition who receive no information about the policy. This approach sacrifices statistical power in order to provide “clean” estimates of public opinion. When effect sizes are small or researchers wish to investigate subgroup heterogeneity, the lack of statistical precision may be problematic.

The alternative is a repeated measures within-subjects (RMWS) design. With RMWS, respondents first provide their opinion on a policy. Next, they receive one or more arguments for or against the policy, re-answering the question after each argument. Organizations commonly use these designs when many arguments circulate in the media and among policy experts for or against a policy (e.g., Blendon et al., 2003; Kirzinger et al., 2021). Unlike post-only designs, RMWS does not practically limit the number of arguments to maintain statistical power, though the potential for bias remains unexplored.

This study evaluates whether post-only and RMWS designs lead to similar conclusions in the context of policy messaging experiments. We consider two competing hypotheses. The first (opinion anchor hypothesis) suggests that the estimated effect of persuasive messages may be smaller in RMWS designs due to pressures respondents feel to appear consistent. The second (opportunity to revise hypothesis) suggests that the estimated effect of persuasive messages may be larger in a RMWS design because the researcher is explicitly offering respondents additional information and the opportunity to change their minds. We conducted experiments comparing post-only and RMWS designs using persuasive messages for two topics: (1) Washington D.C. statehood; and (2) negotiations to combat high prescription drug prices. We focus on these policy debates for two reasons. First, they represent debates in which the RMWS was previously employed, allowing us to speak to whether those findings were sensitive to the methodological approach. Second, public opinion on these issues has not calcified, thus providing the possibility for persuasion.

We find evidence that RMWS designs amplify treatment effects. Because we estimate the effect of persuasive messages, these findings are directly applicable to political practitioners (e.g., issue advocacy groups, pollsters, campaigns). Although social scientists have long worried whether measuring an outcome multiple times might lead to biased conclusions (e.g., Mutz, 2011), the present research represents one of the first attempts to empirically evaluate the magnitude of that bias (but see Clifford et al., 2021).

2. Anchoring

Competing theories make it difficult to predict how a within-subjects approach might yield different results from a between-subjects design. On one hand, we know that individuals associate consistency with a positive self-image (Cialdini, 1984; Cialdini et al., 1995). Thus, if someone answers the same question multiple times, they may be unwilling to express a change in opinion, lest expressing a different opinion reveal that the respondent's initial opinion was wrong. Consistency pressures are thought to stem from cognitive dissonance (Festinger, 1957), or the notion that individuals prefer to hold harmonious preferences. A respondent who is asked about a policy may, in that moment, develop a preference. Once confronted with an argument against the preference they just expressed, they may resist updating their beliefs. Indeed, they may reject the argument or interpret it in such a way that they do not need to arrive at a new opinion (Taber and Lodge, 2006). Stated formally:

Opinion anchor hypothesis: Because individuals who express an initial opinion will feel pressure to maintain consistency, when asked the same question multiple times, policy arguments will appear less persuasive in an RMWS design when compared to a post-only design. That is, respondents will tend to stick with their initial opinion when presented multiple arguments and given multiple opportunities to answer the question.

3. Revising

Despite consistency pressures, some features of the American electorate run counter to the anchoring hypothesis. First, most matters of policy are not central to American political decision-making (Zaller, 1992; Converse, 2006). Even if respondents express an attitude to an initial question, they may not hold that opinion deeply and might update without the discomfort of cognitive dissonance.

Clifford et al. (2021) discuss demand effects as a possible competing hypothesis to what we refer to as the “opinion anchor hypothesis.” According to this logic, a respondent who divines the purpose of a study may alter their response in accordance with the inferred wishes of the researcher. Yet, as Clifford et al. (2021) note, the literature provides scant evidence for demand effects (Mummolo and Peterson, 2019) in survey research.

There are three reasons, however, to believe that something akin to a demand effect may exist in RMWS designs. First, influence may occur because respondents are not only likely to learn the purpose of the research, but with RMWS designs the researcher allows respondents to update their views and provides additional information to facilitate the process. Research on acquiescence bias (Krosnick and Presser, 2010) suggests that some people find disagreeing unpleasant, so they try to agree with an interviewer whenever possible. Someone seeking to be agreeable and lacking strong opinions on a policy may update their attitude after being offered an opportunity in light of new information.

Second, RMWS designs may encourage contrast effects (Schwarz and Bless, 1992). If a question is asked initially, and then re-asked in light of new information, the researcher is implicitly asking the respondent to down-weight the information they used in answering the initial question and up-weight new information. If respondents reason, “the survey would not ask for the same information twice,” they may interpret the question as asking how they feel about the argument rather than the policy.

Third, Yair and Huber (2020) and Graham and Coppock (2021) suggest that respondents engage in “response substitution.” That is, respondents answer the question they want to answer rather than the one that is asked. This may occur, as Gal and Rucker (2011) note, when respondents have opinions they want to convey but are not asked about. With RMWS designs, surveys provide respondents with new information but never ask respondents directly to rate that information. Instead, the questions ask respondents for their overall opinion about the policy. These theories lead to an expectation that respondents will update their reported views in an exaggerated fashion. Stated formally:

Opportunity to revise hypothesis: Because individuals may perceive that answering a question more than once invites them to update their views based on new information, policy arguments will appear more persuasive in RMWS designs.

These hypotheses act in opposition to one another. It is therefore possible that one will negate the effect of the other, yielding similar results. That said, the weight of the evidence from the literature leads us to favor the opportunity to revise hypothesis.

We use two studies to test these hypotheses. Each employs parallel experiments, comparing the effects of persuasive arguments using either post-only or RMWS designs. Study 1 explored DC statehood while study 2 examined prescription drug price negotiations.¹

4. Study 1

4.1 Design

Study 1 was conducted from June 16–July 5, 2021, using a Qualtrics convenience sample of 1969 online panelists. The sample was designed to be representative of the US adult population with the aid of quotas for age, race, and party identification. Respondents first answered questions about their political views and news habits. In the preamble to the experiment, respondents were informed, “As you may know, Washington, D.C. is not a state and therefore does not have voting members in the U.S. Congress.” They were then randomly assigned to one of five conditions. Three conditions were associated with the traditional post-only approach while two were associated with the RMWS approach.² Respondents in conditions 1–3 took part in the post-only design, while conditions 4–5 took part in the RMWS design. The conditions were:³

1. (1) Control. Respondents answered whether they favored or opposed DC statehood.⁴

2. (2) DC taxes only argument. Respondents read that DC residents pay federal taxes before providing their opinion on DC statehood.

3. (3) DC corruption only argument. Respondents read that DC officials have had corruption scandals before providing their opinion on DC statehood.

4. (4) –(5) RMWS. ⁵ Respondents first answered whether they favored or opposed DC statehood. They then read the same arguments described in conditions 2 and 3, in random order, and re-answered the question about DC statehood after each.

With both studies, we examine experimental differences in two ways. First, we consider whether the effect of an argument is substantively and statistically different from the baseline in each approach. That is, we consider what conclusions we might draw about the effect of an argument if we only had the results of one research design and not both. Second, our overarching experimental assignment to the post-only or RMWS approaches allows us to evaluate the results from each design against one another. That is, we examine whether the differences in results across approaches are substantively and statistically different from one another. We recognize that substantive significance is vague and context dependent. When looking at the designs separately, we evaluate the estimated effect sizes keeping in mind whether politicians, political activists, and scholars of public opinion would care about an effect of the estimated magnitude. We also put those results into further context, evaluating whether researchers would draw the same substantive and statistical conclusions with either design. We emphasize the substantive evaluation because relying just on the statistical outcome of the difference in effect size to make a determination as to whether the differences between the two designs are important fails to recognize that a statistically insignificant difference is not the same as supporting the null hypothesis of no difference (Achen, 1982).

4.2 Results

In study 1, we find evidence consistent with the opportunity to revise hypothesis, i.e., the RMWS approach provides sufficient inducement for people to change their mind that overall support for the policy shifts in a meaningful way. As expected, across each approach, the baseline measure of support for DC statehood is similar (38.8 percent in the post-only versus 39.1 percent in the RMWS, see Supplementary Figure A1).⁶ Yet upon reading the argument provided in favor of DC statehood (DC residents pay federal taxes), support goes up 5.6 percentage points (p = 0.11) in the traditional experiment and 9.5 points (p < 0.01) in the RMWS design (Figure 1).⁷

Figure 1. Effect of arguments on support for DC statehood by research design approach.

Note: Error bars represent 95 percent confidence intervals.

Similarly, the argument that DC sometimes elects corrupt officials dropped support for DC statehood by 2.2 percentage points (p = 0.53) in the traditional experiment, but was 4 times larger, at 9.2 points (p < 0.01) in the RMWS design.

If we only had access to one of these two experiments, we would reach very different conclusions about the arguments' effects (see Figure 1). Relying on the post-only design, we would conclude that both arguments are relatively ineffective. Although the argument that DC residents pay taxes is relatively large and nearly achieves conventional levels of statistical significance, it would not be sufficient to reject the null of no effect. If we only had the RMWS design, however, we would determine that both arguments are very effective and by similar, nearly 10-point margins.

Examining the difference in effect size and focusing on statistical significance, however, we find only partial evidence that the two designs yielded different results. Even though the taxes argument was nearly 4-points more persuasive in the RMWS design, that difference is not statistically significant (p = 0.31). For the corruption argument, which had a treatment effect of 2.2 points in the post-only design compared to 9.2 points in the RMWS design, the difference is statistically significant at p = 0.07. For researchers holding to the p < 0.05 threshold, that would not pass, but many would find it compelling (Achen, 1982). Nonetheless, as noted earlier, this misses several important points. First, researchers generally would choose one approach rather than run both experimentally. Here, that choice would influence the substantive conclusions they would draw. Second, analyses involving a difference in effect size require immense statistical power, since an interaction is added to estimate the difference in differences between treatment and control, both across groups and within subjects. Especially since it is reasonable to think the sample might not be large enough, focusing on substantive significance is most appropriate.

5. Study 2

5.1 Design

Study 2 was conducted from November 11–12, 2021, using a convenience sample of 2002 Lucid Theorem panelists. The sample was designed to be representative of the US adult population with the aid of quotas for age, race, gender, and region. Study 2 was similar to study 1, but with a different area of policy debate—prescription drug prices. We chose a design originally used by the Kaiser Family Foundation (KFF), an organization that frequently uses the RMWS design.⁸ This study has the advantage of having one argument that KFF found to be especially effective in shaping public opinion. The “limit access” argument described below was found to reduce support for the policy by more than 50 percentage points. As in study 1, respondents were randomly assigned to five conditions—three associated with post-only and two associated with RMWS:⁹

1. (1) Control. Respondents answered whether they favored or opposed allowing the federal government to negotiate with drug companies for lower prescription drug prices, which would apply to Medicare and private insurance.

2. (2) Save money only argument. Respondents first read that allowing the government to negotiate for lower prescription drug prices could save people money before providing their opinion on allowing such negotiations.

3. (3) Limit access only argument. Respondents first read that allowing the government to negotiate for lower prescription drug prices could limit access to new prescription drugs before providing their opinion on allowing such negotiations.

4. (4) –(5) RMWS. ¹⁰ Respondents first answered whether they favored or opposed allowing the government to negotiate the price of prescription drugs. They then heard the same arguments described in conditions 2 and 3, in random order, and re-answered the item on support for allowing the negotiations.

As in study 1, we examine differences both in terms of the statistical difference and the substantive conclusions we might draw if we only had the results of one research design.

5.2 Results

We find that allowing the government to negotiate drug prices had exceedingly high initial support: 95.2 percent in the post-only baseline condition, and 93.5 percent in the RMWS (see Supplementary Figure A2). Support is little changed after reading the argument that such negotiations could save individuals money, likely due to a ceiling effect.¹¹

Important differences emerge when examining the argument that such negotiations would limit access to newer prescription drugs. Although the post-only design finds that the argument is effective, reducing support for the policy by 23 percentage points to 72.1 percent, the RMWS design shows a much larger effect, similar to the effect initially discovered by KFF. Compared to the baseline, support drops all the way down to 42.2 percent—more than 51 percentage points (Figure 2).¹² This is consistent with the opportunity to revise hypothesis.

Figure 2. Effect of argument on support for drug price negotiation by research design approach.

Note: Error bars represent 95 percent confidence intervals.

Again, we consider the differences between the two approaches with an eye toward both statistical and substantive significance. Here, both approaches would lead researchers to conclude that the positive argument had little effect and the negative argument had a strong effect. Yet the RMWS approach would suggest that the limit access argument is so persuasive that, upon hearing it, a minority of Americans would support government intervention on prescription drug prices. This is far different from the 72 percent support using the traditional approach. Whether there is a solid majority (72 percent) versus a minority (42 percent) in support is substantively meaningful as it could lead to differences in how policymakers would approach the issue as well as media coverage. And whereas study 1 found only marginal statistical significance between the two approaches, the difference (23 points versus 51 points) here is also statistically significant (p < 0.01).

6. Discussion

This study yielded two important findings. First, contrary to concerns that commitment and consistency pressures would make respondents unwilling to change their mind, we find many respondents are willing to update their views. That we find this in the context of both DC statehood and prescription drug prices suggests that this result is not especially sensitive to policy domain. Second, we find evidence that RMWS designs may increase the estimated effect of persuasive arguments when compared to traditional post-only designs. In sum, the findings are more consistent with the opportunity to revise hypothesis than the opinion anchor hypothesis.

Of course, our approach to the RMWS did not space out the repeated measures over the course of the survey to minimize the effect of one question on the next. Our conclusions, then, are more applicable to campaign professionals, journalists, and scholars who seek to estimate the effect of a broad range of arguments but worry that a post-only design will limit the number of arguments they can explore.

Our findings conform to expectations derived from studies of acquiescence bias (Krosnick and Presser, 2010), contrast effects (Schwarz and Bless, 1992), and response substitution (Graham and Coppock, 2021; Yair and Huber, 2020). The RMWS reduces variance but does so at the expense of accuracy. This trade-off would be unacceptable in most cases, so we advise researchers wishing to estimate accurately the effect of any individual argument to avoid this approach. Despite this, there may be some contexts where presenting respondents with multiple arguments could be appropriate. In some cases, researchers may want to provide respondents with several arguments for or against a policy simultaneously before gauging support in order to measure public opinion in an information-rich environment. This approach would require researchers to be aware of the full universe of popular arguments for and against a policy to be applicable to real-world public opinion. In other cases, researchers may want to employ the RMWS in pilot studies to identify relatively weak versus strong arguments. Future research should explore the most ecologically valid ways of examining opinions in light of multiple competing arguments.