The effect of incentives on motivated numeracy amidst COVID-19

Abstract

How does political ideology affect the processing of information incongruent with one’s worldview? The disagreement in prior research about this question lies in how one’s ideology interacts with cognitive ability to shape motivated numeracy or the tendency to misinterpret data to confirm one’s prior beliefs. Our study conceptually replicates and extends previous research on motivated numeracy by testing whether monetary incentives for accuracy lessen motivated reasoning when high- and low-numeracy partisans interpret data about mask mandates and COVID-19 cases. This research leverages the ongoing COVID-19 crisis, as Americans are polarized along party lines regarding an appropriate government response to the pandemic.

Introduction

How does political ideology affect the processing of data incongruent with one’s worldview? When individuals face uncongenial information, they tend to interpret it inaccurately, engaging in motivated reasoning (Bénabou and Tirole, 2006; Cohen, 2003; Kunda, 1990); this phenomenon is also known as “the congeniality effect/bias” (Khanna and Sood, 2018). Motivated numeracy, a type of motivated reasoning, arises when individuals interpret uncongenial data. Recent studies find that the congeniality bias is a politically neutral phenomenon, as both conservatives and liberals experience discomfort when confronting worldview conflict, and both enjoy consuming congenial information (Brandt et al., 2014, 2019). ¹

This study conceptually replicates and extends Kahan et al. (2017) by testing whether monetary incentives for accuracy lessen the congeniality bias when high- and low-numeracy partisans interpret data about mask mandates and COVID-19 cases. This research leverages the ongoing COVID-19 crisis, as Americans are polarized along party lines regarding an appropriate government response to the pandemic. For example, states that voted Republican in 2016 are 40 percentage points less likely to adopt mask-wearing mandates (Makridis and Rothwell, 2020).

Motivation

Do incentives for accuracy reduce motivated reasoning among numerate partisans?

Scholars disagree about how worldview conflict interacts with scientific literacy to shape motivated numeracy. On the one hand, subjects highest in numeracy appear to exhibit more bias when interpreting uncongenial data (Kahan et al., 2017), consistent with findings that cognitive ability does not impede in-group bias (Stanovich et al., 2013) and that receiving new evidence exacerbates biases in experts (Baekgaard et al., 2017).

On the other hand, Mérola and Hitt (2016) demonstrate that more scientifically literate subjects are more persuaded by data sponsored by an opposing party, suggesting that higher numeracy may provide immunity against the congeniality bias. ² This is consistent with experimental research on persuasion (Redlawsk et al., 2010) and studies on citizens’ updating from observing economic reality (Parker-Stephen, 2013), which suggest voters may update from uncongenial information.

This latter research is also indirectly reinforced by two sets of findings around incentives for accuracy and “motivated responding,” which indicate that the congeniality effect may have been overstated. First, the congeniality bias drops if individuals are motivated to answer accurately by monetary payments or appeals to accuracy (Bullock et al., 2015; Prior et al., 2015). Partisans’ biased reasoning appears to be driven (in part) by the lack of stimulus for accuracy. These studies do not, however, test whether incentives lessen motivated numeracy, ³ instead focusing on bias in recalling facts or interpreting verbal information. This research, by contrast, tests whether incentives reduce motivated interpretation of data, a critical question to answer if we are to understand how individuals form evidence-based opinions.

Second, other studies differentiate “motivated learning” from “motivated responding” (Bisgaard, 2019; Khanna and Sood, 2018), where the former describes absorbing congenial facts more readily than uncongenial information and the latter – giving incorrect but congenial answers even after correctly processing the worldview-conflicting information. ⁴ Our study speaks to the difference in these concepts as follows. To measure motivated responding, Khanna and Sood (2018) ⁵ announce incentives for accuracy after respondents were shown numerical tasks to examine motivated responding. Our research announces incentives before respondents are shown in table. This design allows us to answer to what extent incentives reduce motivated learning when interpreting uncongenial data.

In summary, evidence indicates that incentives for accuracy should lessen the congeniality bias. We now examine how incentives may interact with cognition and ideology to shape motivated numeracy.

Hypotheses

Replication of prior results is key to understanding contradictory findings. Replications may be direct (i.e., testing whether the original findings are true under the same conditions, using as-similar-as-possible design and analysis) or conceptual (i.e., retesting the underlying ideas to specify the conditions under which the same results would be found, potentially using different design or analysis) (Chambers, 2017). Our study conceptually replicates and extends Kahan et al. (2017) by modifying treatments from and adding incentives for accuracy.

Conceptual replication

Prior research replicated Kahan et al.’s (2017) findings directly and conceptually. First, Kahan and Peters (2017) directly retested their own results in a new sample and found the same results. Second, Khanna and Sood (2018) found partially consistent results in a conceptual replication; they did not, however, revisit the initial question of how numeracy interacts with ideology. Third, Baker et al.’s (2020) conceptual replication finds that motivated numeracy depends only on partisanship, not on scientific literacy; this result contradicts both Kahan et al.’s (2017) and Mérola and Hitt’s (2016) contrasting findings. ⁶

These mixed findings emphasize the need for conceptual replication that will revisit the questions of (1) whether accuracy is higher when interpreting congenial data, and (2) whether this effect is amplified among numerate individuals.

Hypothesis 1: Among unincentivized respondents, the rate of correct data interpretation increases as the data become more congenial to one’s ideological beliefs. (The congeniality bias exists.)

Hypothesis 2: Among unincentivized respondents, the congeniality bias increases with one’s numeracy.

Extension

How do incentives for accurate learning influence motivated numeracy? Prior research reveals that rewarding accuracy reduces the congeniality bias (Bullock et al., 2015; Prior et al., 2015), when respondents are asked to recall facts or interpret passages.

But do incentives reduce motivated processing of numerical data rather than verbal information? Khanna and Sood (2018) employed incentives for accuracy when solving numerical tasks and found mixed results: monetary rewards reduced the congeniality bias among some but not all partisans. Given the limitations of Khanna and Sood’s (2018) sample, ⁷ it is difficult to determine how incentives shaped motivated numeracy in partisans vs. moderates and in high- vs. low-numeracy subjects. We thus test the impact of incentives for accuracy in a sample with sufficient variation in numeracy and ideology.

Additionally, Khanna and Sood (2018) focus on motivated responding as opposed to learning, announcing rewards for accuracy after respondents were given the numerical task. In our extension of Kahan et al. (2017), we focus on the impact of incentives on motivated learning from numerical data; therefore, we announce incentives for accuracy before respondents are shown the task.

Our expectations for introducing an incentives condition are twofold. First, if incentives lessen motivated numeracy, then slightly higher stakes of giving a correct answer may alleviate motivated learning from data. Second, theoretically, incentives should increase the accuracy of high-numeracy individuals in both congenial and uncongenial conditions. This means that with incentives for accuracy, the congeniality bias, that is, the difference between correct answers in congenial and uncongenial treatment conditions for a given ideology, should not increase with numeracy.

Hypothesis 3: Relative to unincentivized respondents, those incentivized will exhibit greater accuracy in all conditions.

Hypothesis 4: The congeniality bias among incentivized respondents increases at a lower rate with one’s numeracy, compared to the rate of bias increase among unincentivized respondents.

Research design

Registration

The research design was peer-reviewed in this journal from August 2020–January 2021, and the embargoed OSF pre-registration (https://osf.io/935xb) was released in August 2021. No explorative tests were done on the data.

Deviations from pre-registration

The post-analysis peer-review process uncovered two mistakes in the pre-registered regression equations, and these deviations from the pre-registered design do not substantively alter the results. Table 1 lists said changes, section “Deviations from pre-registration” of the appendix also includes the original analysis that follows the pre-registered design without deviations.

Table 1. Deviations from Pre-registration

Case

From March 2020–August 2021, the U.S. (4.2% of the world population) experienced 18% of COVID-19 cases and 14% of COVID-19-related deaths (Ritchie et al. 2021). Despite the evidence that masks curb the spread of the virus (Ingraham, 2020), in 2020, then President Trump’s and his allies’ dismissive statements about masks politicized masking further (Miller et al., 2020), such that Republicans reported most reluctance to wear masks (Pew Research Center, 2020). Out of nineteen “much more Republican” states based on the 2016 election margin, seventeen did not introduce mask mandates (Markowitz, 2020).

This survey was fielded in May 2021, when mask mandates remained politicized. Republican-led states opted to lift mask mandates earlier, notwithstanding low vaccination rates in those states. In May, eleven Republican governors lifted mask mandates, while House Speaker Pelosi indicated that the chamber would keep its mandate despite new CDC guidelines, sparking a GOP backlash (Elfrink, 2021). In May–August 2021, eight states (all with Republican governors) prohibited local governments from issuing mask mandates, while ten states (all with Democratic governors) required masks in schools (Vestal 2021).

Experimental conditions

The study randomly assigned participants to either the no-incentives or incentives treatment. Within each group, respondents were randomly assigned to one of two data tasks shown in Figure 1. Participants were asked to judge, based on the table, whether cities that had a mask mandate were more likely to experience an increase/decrease in COVID-19 cases. The description of the tables clarified that the numbers represent cases since the mandate was implemented. The correct answers for conditions 1 and 2 are mutually opposite. Cell values remained the same across conditions, which differed only in their column headings. This covariance-detection task represents a difficult numeracy problem because one needs to consider ratios (Baker et al., 2020).

Figure 1. Experimental Conditions.

Participants’ numeracy was captured by six questions, randomized to appear either before or after the contingency table; the analysis section discusses order effects.

Ethics

The study was approved by the Institutional Review Board (#IRB_00132903). Participants were asked to provide consent before the survey and could withdraw at any time during the survey. The survey specified that the data were hypothetical and did not use deception.

Incentives

Participants received $1 for correctly answering the data tasks in Figure 1 in the incentives condition. Additionally, all respondents were motivated to answer the numeracy questions correctly: one of six numeracy questions was randomly selected for each participant and if the participant’s answer was correct in that question, they received $1. Qualtrics sent rewards to respondents as gift cards.

Operationalization

The appendix includes the survey instrument.

Dependent variable

Correct _i equals 1 when a respondent correctly answered the contingency table question, 0 otherwise.

Independent variables

Numeracy _i is the number of correct answers the participant i gives to the six questions aimed at measuring numeracy (Lipkus et al., 2001; Peters et al., 2007; Schwartz et al., 1997). ⁸ Items on disease/infection were removed considering their relevance to the treatment.

Congenial _i is a continuous measure of attitude-consistent message that captures the degree of congeniality of the contingency table’s data with one’s ideology. This variable is constructed using, first, the continuous measure of ideology (Conservative _i ) and, second, two binary indicators of the condition to which a respondent was assigned (Covid_decreases _i or Covid_increases _i ). Using Cronbach’s α, two 7-point Likert scales of party affiliation and ideology form an aggregate scale, Conservative _i , where negative values indicate liberal Democrats and positive values indicate conservative Republicans (α = 0.795 which is similar to the α value of 0.83 in Kahan et al. 2017). Conservative _i is identical to the measure of ideology in Kahan et al. (2017). Second, conservatives received uncongenial data in Covid_decreases _i and liberals – in Covid_increases _i . The Congenial _i variable follows the formula:

$$Congenial = {{\;Conservative*\left( {Covid\_increases - Covid\_decreases} \right)\;\;\;\;{\rm{if}}\;Conservative \ge 0} \over { - Conservative*\left( {Covid\_decreases - Covid\_increases} \right)\;\;{\rm{if}}\;Conservative < {\rm{ }}0}}$$

For a conservative respondent (Conservative _i >0), the Congenial _i variable is positive in the Covid_increases _i condition and negative in the Covid_decreases _i condition, while for a liberal (Conservative _i <0), Congenial _i is positive in Covid_decreases _i and negative in Covid_increases _i condition. The absolute magnitude of Congenial _i increases with the strength of one’s ideology.

Incentive _i is a binary indicator of whether a participant is assigned to that condition.

Controls

Indicators of age, ethnicity, gender, education, and voting behavior were collected to ensure that treatment groups are balanced on observable covariates.

Statistical model

A linear probability model estimates the parameters of equations (1)–(4). Equations (1) and (2) are estimated on unincentivized observations to test hypotheses 1 and 2; equations (3) and (4) test hypotheses 3 and 4, utilizing the full sample. The appendix also includes the logistic regression models that estimate these equations.

(1) $${\rm{Correc}}{{\rm{t}}_i} = {\beta _0} + {\beta _1}{\rm{Congenia}}{{\rm{l}}_i} + {\beta _2}{\rm{Numerac}}{{\rm{y}}_i} + {\beta _3}{\rm{Numeracy}}_i^2 + {\varepsilon _i}$$

(2) \begin{align}{\rm{Correc}}{{\rm{t}}_i} = {\beta _0}{\rm{ }} &+ {\beta _1}{\rm{Congenia}}{{\rm{l}}_i} + {\beta _2}{\rm{Numerac}}{{\rm{y}}_i} + {\beta _3}{\rm{Numerac}}{{\rm{y}}_i} \times {\rm{Congenia}}{{\rm{l}}_i}{\rm{ }}\\ & + {\beta _4}{\rm{Numeracy}}_i^2 \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _5}{\rm{Numeracy}}_i^2 + {\varepsilon _i}\end{align}

(3) $${\rm{Correc}}{{\rm{t}}_i} = {\beta _0} + {\beta _1}{\rm{Incentiv}}{{\rm{e}}_i} + {\varepsilon _i}$$

(4) \begin{align}{\rm{Correc}}{{\rm{t}}_i} &= {\rm{ }}{\beta _0} + {\beta _1}{\rm{Numerac}}{{\rm{y}}_i} + {\beta _2}{\rm{Congenia}}{{\rm{l}}_i} + {\beta _3}{\rm{Numerac}}{{\rm{y}}_i} \times {\rm{Congenia}}{{\rm{l}}_i}{\rm{ }}\\ &\quad + {\beta _4}{\rm{Numeracy}}_i^2 + {\beta _5}{\rm{Numeracy}}_i^2 \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _6}{\rm{Incentiv}}{{\rm{e}}_i}{\rm{ }}\\ &\quad + {\beta _7}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _8}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numerac}}{{\rm{y}}_i}{\rm{ }}\\ &\quad + {\beta _9}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numeracy}}_i^2 + {\beta _{10}}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numerac}}{{\rm{y}}_i}{\rm{ }}\\ &\quad \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _{11}}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numeracy}}_i^2 \times {\rm{Congenia}}{{\rm{l}}_i} + {\varepsilon _i}\end{align}

Sample

No pilot data were collected. Qualtrics fielded the survey.

Per the pre-registered power analysis, the required sample size is 3,000 with 1/3 of observations unincentivized and 2/3 – incentivized; Qualtrics collected 1,016 (512 and 504 in each contingency table condition) unincentivized responses and 2,034 incentivized (992 and 1,042 in each table condition), a total of 3,050 responses.

Simulations for power calculations

The power was calculated using Monte Carlo (MC) simulations with 1,000 repetitions; the appendix details all assumptions.

Data inclusion criteria

Qualtrics assembled a nonprobability online sample of U.S. residents of 18+ years old. To avoid common biases in online samples (Kennedy et al., 2016; Rivers, 2016), we calculated nested quotas. Qualtrics was unable to fulfill all planned nested quotas; the appendix describes the differences between planned and collected quotas. Nevertheless, the resultant sample largely avoids underrepresenting nonvoters and less educated individuals (see the appendix).

Data exclusion criteria based on quality checks

After the treatment, respondents answered the factual manipulation check question (Kane and Barabas, 2019); those who failed it were removed from the sample. Qualtrics removed additional low-quality responses, as the appendix describes.

Termination of data collection

Qualtrics terminated data collection after 3,050 quality responses were reached, conducting quality checks after 10%, 75%, and 100% of the data were collected.

Analysis

Balance on covariates

Per the pre-registered power calculations, respondents were randomly assigned to either a no-incentive or incentive treatment with a probability of 1/3 and 2/3, respectively. Respondents were also assigned randomly to conditions Covid Increases vs. Covid Decreases with a probability of 1/2. While assignment was random, some respondents were removed after failing the manipulation check and/or other quality controls outlined in the “Data exclusion […]” section. To verify that passing the exclusion criteria did not correlate with any observable attributes, we obtained differences-in-means between each pair of treatments (see the appendix). None of the differences-in-means is statistically discernible from 0 at the 5% level; the treatment groups are, therefore, balanced on observable attributes.

Multivariate analyses

The impact of congeniality on accuracy among unincentivized respondents (Hypothesis 1)

To test hypothesis 1, models 1–2 of Table 1 estimate equation (1), using a linear probability model on unincentivized responses; model 2 also adds the controls.

Hypothesis 1 states that, among unincentivized respondents, accuracy is higher when interpreting more congenial data. The coefficient estimate on Congenial _i implies that one-unit increase in Congenial _i (ranges from −1.88 to 1.88, mean = 0, SD = 1) generates a 4.4–4.8 percentage point (pp) increase in accuracy (statistically discernible from 0). Section “Inferiority tests” of the appendix shows that we cannot conclude that a meaningful effect of Congenial _i on Correct _i is absent. We thus reject the first null of no effect.

The impact of congeniality and numeracy on accuracy among unincentivized respondents (Hypothesis 2)

Hypothesis 2 posits that, among unincentivized respondents, the congeniality bias increases with one’s numeracy. The coefficient estimate for the multiplicative interaction term Numeracy×Congenial is positive (only marginally significant in model 3), indicating that increases in numeracy improve accuracy even as congeniality increases. Following Kahan et al. (2017), we also include the squared Numeracy term; the coefficient estimate for the interaction Numeracy ² ×Congenial is not statistically different from zero. We use predicted probability densities (Figure 2) and differences in the predicted congeniality bias for various numeracy levels (Table 3) to further understand if Numeracy _i has a range of values for which Congenial _i generates a statistically meaningful impact among unincentivized respondents.

Figure 2. Predicted Probabilities of Correctly Interpreting the Data.

Note: Density distributions derived via MC simulation from logistic regression that estimates equation 4 (output is shown in the appendix), when Congenial is set at –1 SD (i.e., respondents facing data contradicting their beliefs), at mean (i.e., ideological moderates on the ideology-party affiliation spectrum), and +1 SD (i.e., data are consistent with beliefs), and numeracy set at -1SD (1 out of six correct questions) for “low numeracy” and +1SD (4.35 out of six correct questions) for “high numeracy.” Numeracy is centered at “0” for ease of interpretation. +/-1SD Numeracy value is +/-1.654, and numeracy squared term is 2.736.

Section “Inferiority tests” of the appendix shows that we cannot conclude that a meaningful effect of the interaction terms on accuracy is absent.

The impact of incentives on accuracy (Hypothesis 3)

Hypothesis 3 expects that incentivized participants are more likely to correctly interpret the contingency table compared to those unincentivized. Models 5 and 6 of Table 2 test this expectation, estimating parameters in equation (3), using all observations. The coefficient estimates on Incentive _i in models 3 and 4 are substantively and statistically negligible; we thus fail to reject the null hypothesis of monetary incentives having no impact on accuracy.

Table 2. The Impact of Numeracy and Congeniality on Accuracy (Unincentivized Participants)

Standard errors in parentheses.

Note: Linear Probability Model with heteroscedasticity robust standard errors.

Control variables in the regression are age, gender, race, education, and voting 2016.

* p <0.10, ** p <0.05, *** p <0.01

Table 3. The Impact of Incentives, Numeracy, and Congeniality on Accuracy (All Participants)

Standard errors in parentheses.

Note: Linear Probability Model with heteroscedasticity robust standard errors.

Control variables in the regression are age, gender, race, education, and voting 2016.

* p <0.10, ** p <0.05, *** p< 0.01.

Table 4. Differences in the Predicted Congeniality Bias Between Less and More Numerate Individuals at Various Levels of Numeracy

Note: Congeniality Bias = Pr(correct = 1|congenial = −1 SD) − Pr(correct = 1|congenial = 1 SD); Numeracy is set at either +/−1 SD or +/−1.5 SD or +/−2 SD.

The impact of incentives, numeracy, and congeniality on accuracy (Hypothesis 4)

Hypothesis 4 posits that the congeniality bias among incentivized respondents increases at a lower rate with one’s numeracy, compared to the rate of the congeniality bias increase among unincentivized respondents. To test hypothesis 4, linear probability models 7 and 8 of Table 2 estimate parameters in equation (4). The interaction of numeracy and congeniality is positive, while the interaction of the squared numeracy term and congeniality is negative; however, neither interaction term reaches statistical significance at the 0.05 level. Both three-way interaction coefficients have statistically and substantively negligible effects on correct answers. While this suggests that the increase in the congeniality bias with numeracy is not different between incentivized and unincentivized respondents, we use visualizations to understand if there are ranges of values of Congenial _i and Numeracy _i for which Incentive _i has a statistically meaningful impact.

Visualization of the interaction effects

We supplement the linear probability model results with logistic regression results (in the appendix) to estimate the out-of-sample predicted probabilities of correct answer (based on MC simulations). Figure 2 graphs these probability densities, visualizing how incentives, numeracy, and congeniality interact to shape motivated numeracy. ⁹

Figure 2 demonstrates that the congeniality bias increases with numeracy. The predicted accuracy rates for respondents facing uncongenial, neutral, and congenial data constitute (1) 41%, 43%, and 45%, that is, the congeniality bias equals 4pp for the less numerate unincentivized respondents (top left), (2) 34%, 41%, and 49%, that is, the congeniality bias is 15pp for the more numerate unincentivized (top right), (3) 40%,40%, and 39%, that is, the bias equals -1pp for the less numerate incentivized (bottom left), and (4) 40%, 45%, and 49%, that is, the bias constitutes 9pp for the more numerate incentivized (bottom right).

Table 3 uses differences-in-means to test whether the differences in the congeniality bias between less and more numerate individuals are statistically distinct. To test hypothesis 2, consider respondents in the no-incentive condition first. Highly numerate respondents exhibit greater congeniality bias than less numerate subjects by 12pp, 18pp, and 23pp for respondents with numeracy of −/+1SD, −/+1.5SD, and +/−2SD above/below the mean, however these differences are not statistically distinct at 0.05 level. Our data, therefore, indicate that numeracy does not have an impact on the congeniality bias of unincentivized respondents shown by Kahan et al. (2017); we thus fail to reject the second null hypothesis of no effect of numeracy on congeniality bias among unincentivized respondents.

Table 4 shows that incentives shrink the gap in motivated numeracy from −4pp to 1pp (5pp reduction) among less numerate (+/−1SD) and from 15pp to 9pp (6pp reduction) among more numerate individuals. Similar reductions in the congeniality bias are observed at other levels of numeracy. Hypothesis 4 expected that incentives would generate larger reductions in the bias among more numerate individuals; however, the changes we observe are not statistically distinct between less and more numerate individuals; we fail to reject the fourth null hypothesis of no effect.

Order effects

In summary, this paper tested four hypotheses. With exception of hypothesis 1, we fail to reject the nulls of no effect.

Section “Order effects” of the appendix reruns all analyses on split samples by order (this robustness check was pre-registered). Split sample analysis reveals that one-unit increase in congeniality increases accuracy by 7–8pp when numeracy questions appear after treatment. The congeniality, however, has no effect on accuracy when numeracy questions precede treatment, implying that our rejection of the first null hypothesis in the pooled sample is driven by the subsample that received numeracy questions after treatment.

Additionally, split sample analysis reveals that – consistent with the pooled sample – incentives generate no reductions in the congeniality bias among more numerate respondents when numeracy questions follow treatment. However, in the subsample “numeracy before treatment” we observe that incentives increase the congeniality bias among the more numerate individuals – contrary to the expectation in hypothesis 4.

The results from split samples do not alter our conclusions regarding hypotheses 2 and 3.

Conclusion

The experiment yields five takeaways.

First, unincentivized respondents on average exhibit the congeniality bias (see section “order effects” for more details).

Second, while more numerate individuals exhibit on average greater congeniality bias than less numerate respondents, the differences between less and more numerate unincentivized respondents are not distinguishable at 0.05 level. We therefore fail to reject the second null hypothesis of no effect. We note, however, that the differences of 10pp and 15pp (for +/−1SD and +/−1.5SD above/below mean in numeracy) between less and more numerate incentivized respondents are statistically discernible at 0.05 level. Additionally, the effect size of the congeniality bias in our data is smaller than that reported in Kahan et al. (2017) and is comparable to that found in Khanna and Sood (2018).

Third, incentives do not increase accuracy. This finding contradicts prior scholarship on incentives reducing motivated reasoning (e.g., Bullock et al. 2015; Prior et al. 2015). Our fourth finding provides a potential interpretation as to why incentives did not impact accuracy.

Fourth, reductions in the congeniality bias from incentives between less and more numerate individuals are on average not statistically distinct (see section “order effects” for more details). Given the difficulty of a numerical task, our design is most comparable to Khanna and Sood’s (2018), who find diverging effects of incentives among highly numerate individuals by political ideology: incentives do not reduce bias among supporters of concealed carry (conservatives) but do so among its opponents (liberals). We, therefore, obtained differences-in-means (not planned in the pre-registered design, included in the appendix) separating more and less numerate respondents into presumed opponents and supporters of mask mandates. Incentives reduced the congeniality bias among both more and less numerate conservatives by 24–30pp and among less numerate liberals by 6pp but increased the bias among more numerate liberals by 15pp.

This asymmetry clarifies why we find that incentives do not increase accuracy, suggesting that incentives may generate an asymmetric reduction in the congeniality effect among partisans on certain issues. The asymmetric reduction in bias implies that a uniform communication strategy to reduce motivated reasoning may not succeed. Future research should systematically test whether incentives generate asymmetric changes in the congeniality bias among supporters and opponents of a given policy.

Fifth, the pre-registered robustness check for order effects reveals that the results differ for hypotheses 1 and 4 in split samples (see section “order effects”), implying that the order of numeracy and treatment may shape how respondents perceive the data interpretation task.