Ideological and Partisan Bias in the Canadian Public

Abstract

Partisan and affective polarization should have observable consequences in Canada, such as bias in political information search and processing. This article presents the results of three studies that test for partisan and ideological bias using the Digital Democracy Project's study of the 2019 Canadian election. Study 1 uses a conjoint experiment where respondents choose from pairs of hypothetical news stories where the slant of the source and headline are both randomized. Study 2 tests for partisan-motivated responsiveness to elite cues with a policy vignette that manipulates the presence of party elite cues and a motivational prime. Study 3 requires respondents to solve a randomly assigned numeracy task that is either political or nonpolitical in nature. Results suggest that Canadians (1) select politically congenial information, though not sources of such information, (2) follow elite cues when partisan motivation is primed and (3) evaluate evidence in ways that are biased by their ideological beliefs.

Résumé

La polarisation partisane et affective devrait avoir des conséquences observables au Canada, tel qu’un parti pris dans la recherche et le traitement de l'information politique. Cet article présente les résultats de trois études qui mesurent les préjugés partisans et idéologiques à partir de l'Étude sur l’élection canadienne 2019 du Projet de démocratie numérique. La première de ces études s’appuie sur une expérience conjointe où les répondants opèrent un choix parmi des paires d'articles de presse hypothétiques dans lesquels l’orientation de la source et du titre sont tous deux randomisés. L'étude 2 teste la sensibilité de la motivation partisane aux indices de l'élite avec une vignette politique qui manipule la présence d’indices émanant des dirigeants du parti et d’un raisonnement motivé. L'étude 3 demande aux répondants de résoudre une tâche de calcul, de nature politique ou non politique, assignée au hasard. Les résultats suggèrent que 1) les Canadiens sélectionnent des informations conformes à leur penchant politique, mais pas les sources de ces informations ; 2) ils suivent les indices de l'élite lorsque la motivation partisane est amorcée ; et 3) évaluent les données probantes selon des critères biaisés par leurs croyances idéologiques.

Recent scholarship has illustrated a trend toward polarization in Canada that in many ways mirrors patterns found in the United States. Although there is little evidence that Canadians are becoming more extreme in their policy and ideological beliefs (Johnston, 2014; Merkley, 2020), they are becoming more sorted, in that partisanship and policy beliefs are increasingly intertwined—sometimes called partisan polarization (Kevins and Soroka, 2018; Merkley, 2020). They are also becoming more affectively polarized, where supporters of the major parties increasingly dislike opposing parties (Cochrane, 2015; Johnston, 2019) and even their supporters (Owen et al., 2020). These patterns are very similar to what has been found in the United States (Abramowitz, 2010; Baldassarri and Gelman, 2008; Iyengar et al., 2012; Lelkes, 2016; Levendusky, 2009a), though evidence is more mixed in other national contexts (Adams et al., 2012a, 2012b; Munzert and Bauer, 2013). Canadian politics is now more affectively charged, and left/right ideological conflict reinforces partisan political divisions to an extent not seen before in Canadian history. There are apt to be consequences.

Partisan and affective polarization heightens directional motivation leading to partisan and ideological bias in both the search for, and processing of, political information in the Canadian mass public. A voluminous literature in the United States highlights both of these phenomena. Research on selective exposure has found that Americans choose information (Klapper, 1960), or sources of information, that are aligned with their beliefs (Iyengar and Hahn, 2009). The rapid fragmentation of the media environment and rise of partisan media have heightened the consequences of such a tendency (Prior, 2007, 2013; Stroud, 2008). Work on biased information processing—known as partisan or ideological motivated reasoning—is even larger (see Ditto et al., 2019, for a meta-analysis). The crux of this theory is that citizens will reflexively accept or reject information that is aligned with or undermines their worldviews and identities (Kahan, 2013; Lodge and Taber, 2013).

In this article, I present three studies that shed light on biased information search and processing in the Canadian mass public. These studies were included in the Digital Democracy Project's study of the 2019 Canadian election. Study 1 involves a conjoint experiment that asks respondents to choose from several pairs of hypothetical news articles whose attributes (that is, the source, headline, date and author) are randomized. Scholars have noted a tendency for citizens to be attracted to congenial information (Klapper, 1960) or sources of information (Iyengar and Hahn, 2009), which tend to be correlated in practice. This design allows me to tease out whether respondents are responsive to the congeniality of the source or the content itself (as expressed through the headline).

Study 2 is a party cue experiment that randomly assigns respondents in-party or out-party cues on a recent policy debate and motivational primes toward direction or accuracy. Study 3 assigns respondents a numeracy task that is nonpolitical for some and politically charged for others, with results that are congenial to left- or right-wing ideological beliefs. This design lets me evaluate the ability of respondents to correctly complete the task when the results are aligned or opposed to their ideological beliefs.

The results of these three studies provide compelling evidence of partisan and ideological bias in the Canadian public. I begin by outlining evidence for polarization in Canada and key debates surrounding selective exposure and motivated reasoning, which underlie my research questions. I then describe the design, hypotheses and results for each of my three studies in turn. I end by discussing the limitations of these studies and questions for future research.

Canadian Polarization and Partisan and Ideological Bias

There is considerable debate in the United States about the nature of mass polarization. Scholars have identified conceptually distinct dimensions of this phenomenon that have become conflated in popular discourse. There is only minimal evidence that Americans are gravitating toward the ideological poles, known as ideological polarization (Fiorina and Abrams, 2008; Fiorina et al., 2008). Rather, partisans have developed increasingly dissimilar ideological beliefs. This is also known as partisan sorting or partisan polarization (Abramowitz, 2010; Baldassarri and Gelman, 2008; Levendusky, 2009b). In addition, there appears to have been considerable affective polarization where Democrats and Republicans increasingly dislike one another's parties and their supporters (Iyengar et al., 2012; Druckman and Levendusky, 2019).

Historically there has been a tendency in literature on Canadian political behaviour to emphasize differences between Canadian and American public opinion, which are often thought to stem from each country's unique foundational circumstances and resulting institutions (Horowitz, 1966). Americans, from a country born of revolution, are more receptive to values of individualism, populism and egalitarianism, while Canadians are more elitist and collectivist owing to their counter-revolutionary heritage (Lipset, 1990). Early scholarship also drew attention to the less stable nature of Canadian partisanship and vote choice, which was seen as a contrast to the United States (Elkins, 1978; LeDuc et al., 1984). More recent scholarship shows that these differences are often overblown. Most importantly for my purposes here, partisanship is just as stable and influential in Canada as the United States after accounting for differences in measurement (Bélanger and Stephenson, 2014; Blais et al., 2001; Green et al., 2002; Johnston, 1992, 2006; Merolla et al., 2008, 2016).

Notwithstanding this shift in Canadian public opinion research, to date there has been only sporadic interest in studying mass polarization in the Canadian context and its causes and implications. The evidence accumulated so far indicates that Canadians are not becoming more ideologically extreme (Johnston, 2014; Merkley, 2020). There is some evidence that Canadians are sorting, such that New Democratic party (NDP) and Liberal partisans, on the one hand, and Conservative partisans, on the other, are becoming more ideologically distinct (Merkley, 2020). Kevins and Soroka (2018) find that public attitudes toward redistribution are becoming more strongly correlated with partisanship and vote choice over time.

At the same time, Liberal and NDP partisans increasingly dislike the Conservative party and vice versa—evidence of affective polarization (Cochrane, 2015; Johnston, 2019). Cochrane (2015) observes that this form of polarization was occurring as the parties themselves were diverging ideologically, as evidenced by platform coding done by the Comparative Manifestos Project. Johnston (2019) uses multilevel modelling to show that there is indeed an association between left/right manifesto scores and out-party affect in the Canadian Election Study cumulative file, though perhaps with weaker effects in Quebec. More recently, work by the Digital Democracy Project has revealed that affective polarization extends to how partisans view supporters of other parties and to how comfortable they feel with out-party supporters in their close social proximity—known as social distance (Owen et al., 2020). Canadian politics are more affectively charged now than in the past.

The rise of affectively charged politics and the increasingly intertwined nature of partisanship and ideology are bound to have important implications. My focus here is how they relate to the search and processing of political information. Partisan and affective polarization heightens directional motivation, making citizens more likely to choose and accept congenial political information. Canadian public opinion research has been relatively slow in picking up on the polarization of the Canadian mass public, so research on the consequences of polarized politics has been underemphasized.

Selective exposure

The first observable implication of sharpened lines of mass polarization is partisan or ideological selective exposure: the tendency to choose information (Klapper, 1960) or sources of such information (Iyengar and Hahn, 2009) that are congenial to one's partisan identity and worldviews. This concept has implications that extend beyond the search of political information, but the rise of partisan news and the fragmentation of the media environment in the United States have increased the relevance of this concept in political science (Prior, 2007, 2013; Stroud, 2008). Some scholars sound a further note of alarm that exposure to congenial information and news sources may reinforce one's prior views and thus further exacerbate polarization (Feldman et al., 2014; Stroud, 2010).

There are a few caveats in order. One expectation, building on theories of cognitive dissonance pioneered by Leon Festinger (1957), is that individuals actively avoid information that is inconvenient for their social identities or beliefs because they aim to minimize the cognitive discomfort that would be generated by such information. However, in lab-based settings there is limited evidence that people avoid dissonant information compared to neutral information. People do, however, selectively engage with congenial information that aligns with their identities and beliefs over neutral information. This selective engagement may be due to biased perceptions of quality and credibility of the information provided when it is aligned with their worldviews (Fischer et al., 2008; Metzger et al., 2020). The implication is that biased engagement with political information may not be driven by a need to avoid uncomfortable information, thus curbing the threat to political discourse posed by echo chambers.

Further, outside of the lab, people report high levels of consumption from congenial sources but not to the exclusion of outlets that tend to carry content opposed to their beliefs (Stroud, 2008). It is most often the case that, say, heavy consumers of Fox News also consume news from other mainstream national or local sources, rather than avoiding these sources entirely and residing in echo chambers. Consequently, the audiences for partisan news in the United States are resoundingly dwarfed by mainstream news and there is substantial overlap in the distribution of news consumption by Democrats and Republicans (Eady et al., 2019; Guess, 2021)—a finding that holds in Canada as well (Owen et al., 2020). In short, there is an important conceptual difference between gravitating toward favourable information and avoiding hostile information in both sources and content. The consumption of, and preference for, congenial information does not automatically go hand in hand with the avoidance of dissonant information.

Partisan media have a trivial foothold in Canada, but this does not mean Canadians lack demand for news content that is aligned with their beliefs. It could be that the partisan-congenial sources scholars have identified are simply unknown to most Canadians or are lacking in source credibility. Moreover, individuals can satisfy their demand for partisan-congenial information by simply selecting news stories that support their beliefs. We need experimental evidence illustrating how Canadians choose the news and specifically whether their choice is shaped by the congeniality of the source or the content of the news article. We would expect as much with polarized politics, which leads to my first research question:

RQ1: Do Canadians prefer politically congenial news sources or content?

Partisan or ideological motivated reasoning

Analytically distinct from selective exposure is bias in information processing. Polarization yields a clear prediction that citizens will engage in partisan or ideological bias where they reflexively accept (confirmation bias) and reject (disconfirmation bias) information depending on its implications for their partisan or ideological identities (Bartels, 2002; Kahan, 2013; Kahan et al. 2011; Lodge and Taber, 2013; Taber and Lodge, 2006). This scholarship borrows from psychological theories of directional motivated reasoning, which is a broader phenomenon where individuals process information to maintain their priors rather than in the pursuit of accuracy (Ditto and Lopez, 1992; Kunda, 1990).

A wide range of studies claim to provide evidence of partisan or ideological motivated reasoning. There are two paradigmatic designs to evaluate such bias (see Ditto et al., 2019, and Tappin et al., 2020): party cue designs that manipulate in-party and out-party cues in favour or opposed to certain policies to influence attitudes (see, for example Bullock, 2011; Kam, 2005; Malka and Lelkes, 2010; Merkley and Stecula, 2020; Mullinix, 2016; Nicholson, 2012) and outcome switching designs that manipulate the congeniality of information given to respondents to examine its effect on attitudes (see for example Kahan, 2013; Kahan et al., 2011; MacCoun and Paletz, 2009). There is widespread agreement among these studies that congenial, rather than dissonant, information facilitates persuasion and that people follow the lead of party elites, which leads to two further research questions:

RQ2: Do Canadians follow party elite cues as a result of partisan bias?

RQ3: Does ideological bias shape how Canadians interpret and evaluate information?

Unfortunately, party cue and outcome switching designs often fail to provide direct causal evidence of motivated reasoning by violating the excludability assumption: the treatments influence other factors independent of motivation that may, in turn, influence reasoning. For example, congenial sources or information can be seen as more credible, and this credibility assessment influences respondents’ reasoning, rather than their directional political motivation (see Tappin et al., 2020, for a great discussion). It is possible that their credibility assessment is entirely a product of their directional political motivation, but this is a strong assumption. Relatedly, the observable implications of many of these designs are identical to Bayesian updating. We should not expect attitudinal change in response to experimental exposure when respondents have strong priors (Green et al., 2002; Druckman and McGrath, 2019).

This does not mean that it is impossible to observe motivated reasoning but rather that we want to take measures to minimize the importance of prior beliefs by directly observing biased reasoning, exogenously manipulating motivation independent of information, or by focusing on outcome measures unrelated to beliefs themselves. This point will be expanded on below. The two studies related to motivated reasoning that I describe are carefully designed to avoid these landmines of causal inference.

Study 1: Selective Exposure

All three studies were included in wave nine of the Digital Democracy Project's study of the 2019 Canadian election, fielded from October 24 to November 4, 2019. This wave included a sample of 2,068 Canadian citizens of voting age collected through the online panel sample provider Qualtrics. It used national quotas for age, gender, region and language to ensure representativeness. The data was further weighted within region by age and gender based on population benchmarks provided by the 2016 Canadian census. The recruited sample provides a reasonable partisan balance. Thirty-four per cent of respondents who reported voting in the 2019 election reported voting for the Liberal party, compared to their received 33 per cent, while 32 per cent reported voting Conservative, compared to 34 per cent on election day.

I used a paired conjoint design to evaluate selective exposure in the Canadian public. Conjoint experiments are a technique used in market research to shed light on which attributes of a product appeal to consumers. In this case, I am assessing whether certain features of a news article lead to increases or decreases in its likelihood of selection.

Respondents were given a table with the attributes of two hypothetical news stories, including its source, headline, author and date. They were provided four pairs of news stories and were asked to choose one story from each pair. Every news article had its source randomized across 14 different outlets. They could be assigned stories from the CBC, CTV, Global, the Globe and Mail or the National Post.¹ I call these sources national news. They could also receive news articles from what are identified as a “local newspaper,” “local radio station” or a “local television news station,” which I label local news. Finally, respondents could receive articles from PressProgress, National Observer or Rabble, which I treat as left-congenial news, or from Rebel Media, Quillette or True North News, which I treat as right-congenial news. The classification of outlets is based on research done by the Digital Democracy Project's social media team that evaluated whether certain sources were more likely to be selectively shared or followed by partisans on Twitter (see Owen et al., 2020, for more details).² National news is used as the baseline for comparison. I also randomized the headline. Respondents were exposed to a series of headlines that were either left-congenial or right-congenial and either positive or negative in their tone and content.³

I further randomized the author, so that some were male and others were female, and I randomized the date, so that some articles were closer to the fielding date than others. I have no substantive interest in these last two randomizations. They were included to enhance the realism of the experiment by giving respondents two additional characteristics of news articles that are first apparent when respondents read a news story in the real world. They also allowed me to mask the randomizations of substantive interest and prevent possible carryover effects precipitated by experimental demand. An example of the layout can be found in Figure A1 of the Appendix.

My design leads to the following three hypotheses:

H1: Partisanship/Ideology conditions the effect of headline congeniality on the likelihood of story selection. Right-congenial headlines are more likely to be selected than left-congenial headlines by right-leaning respondents. The reverse is true for left-leaning respondents.

H2A: Partisanship/Ideology conditions the effect of left-congenial sources on the likelihood of story selection with a higher likelihood of selection compared to other news outlets among left-leaning respondents.

H2B: Partisanship/Ideology conditions the effect of right-congenial sources on the likelihood of story selection with a higher likelihood of selection compared to other news outlets among right-leaning respondents.

I test these hypotheses using ordinary least squares (OLS) regression where I estimate the average marginal component effects (AMCEs) of local news (local), left-congenial sources (leftsource), right-congenial sources (rightsource; reference category = national news), negative headlines (negativeheadline; reference category = positive news), right-congenial headlines (rightheadline; reference category = left-congenial headline), author gender (male; reference category = female), and article date (October 7, September 30 and September 23; reference category = October 14) on the likelihood of article selection. I cluster standard errors by respondent. The AMCEs provide the average causal effect of a given attribute on story selection averaged across the joint distributions of the other remaining attributes. Estimating the AMCEs requires including every randomized attribute in the model, so author and date are included even though they are not of substantive interest here. Balance tests for all three studies can be found in the supplementary materials.

In this case, my theory necessitates the inclusion of an interaction between the source or headline attributes, on the one hand, and political predispositions, on the other (predisposition). I focus here on ideology and partisanship. Partisanship is a stable social identity (Green et al., 2002), while ideology is an interconnected set of beliefs characterized by consistency and constraint (Converse, [1964] 2006). Though there has been an increasing relationship between these concepts over time in Canada (Merkley, 2020), this relationship is not automatic. There is considerable scholarly disagreement about whether ideology causes partisanship or vice versa. I am agnostic on this point here, so I estimate two models using each as the moderator of interest.

I measure policy-based ideology using a 15-item policy battery in the Digital Democracy Project to construct a measure of ideological consistency ranging from −15 as most consistently left-wing to 15 as most consistently right-wing (mean = −1.71, standard deviation = 4.25). The distribution of ideology is unimodal with 90 per cent of the distribution falling between −9 and 5. Descriptive statistics for each of the moderators used in these three studies can be found in Table A1. Using the standard 0–10 ideological self-placement scale does not fundamentally change the result. For partisanship, I use a 7-point strength of partisanship scale from strong partisans of the left-leaning parties (Liberals, NDP and Greens) to strong Conservative partisans. Ideology and partisanship, as expected, are highly correlated (0.50). This leads to the estimation of the following equation:

(1)$$\eqalign{& pr( {selection} ) = \alpha + \beta _1local + \beta _2leftsource + \beta _3rightsource + \beta _4negativeheadline \cr & \quad + \beta _5rightheadline + \beta _6male + \beta _7oct7 + \beta _8sept30 + \beta _9sept23 \cr & \quad + \beta _{10}predisposition + \beta _{11}predisposition\;\ast\; leftsource \cr & \quad + \beta _{12}predisposition\;\ast \;rightsource + \beta _{13}predisposition\ast rightheadline + \varepsilon } $$

We should expect β₁₁ to be negative and significant in support of H2A and β₁₂ and β₁₃ to be positive and significant in support of H2B and H1, respectively. I present the interactive effects as the predicted marginal means in Figure 1. The full results, including estimates for the other attributes, can be found in Table A2 of the Appendix.

Figure 1 Predicted Likelihood of Story Selection by Congeniality of the Headline (top) or Source (bottom) across Levels of Partisanship (left) and Policy-based ideology (right)

Note: 95 per cent confidence intervals.

Study 1 results

The effect of the partisan sources and headlines should be conditional on the ideology or partisanship of the respondent. This appears to be the case for the headlines, but not for the sources. Among strong supporters of the left parties, we would expect 57 per cent to choose the left-congenial headline, compared to 46 per cent for strong Conservative partisans. In contrast, 43 per cent of strong left partisans are expected to choose the right-congenial headline, compared to 54 per cent of strong Conservative partisans. These results are shown in the top left panel of Figure 1. An even stronger pattern holds using a −15 to 15 scale of policy-based ideology, which is shown in the top right panel of Figure 1. Both interactions are significant (p < .001), providing strong support for H1. Author gender and date had null effects on story selection, as is shown in Table A2, though respondents exhibited a slight preference for negative news (p < .05 and p < .01 in each model).

In sharp contrast, the bottom panels of Figure 1 show that there is no such effect when evaluating news sources, providing little support for H2A or H2B. Left- and right-congenial outlets pay a penalty relative to national news outlets regardless of the ideology or partisanship of the respondent. Canadians are more likely to choose news content that is aligned with their partisan and ideological identities, even if they do not, at the moment, choose sources that cater to that propensity.⁴

Study 2: Party Cues

Study 1 provides some evidence that Canadians choose information that is congenial with their political beliefs. Quite separately, people may be inclined to accept or reject information based on its alignment with their political views—known as partisan or ideological motivated reasoning. One observable implication of partisan bias in information processing is the tendency of people to follow the lead of their own party's elites (Cohen, 2003; Kam, 2005; Mondak, 1993) or to be repelled by signals sent by the opposing party's officials (Merkley and Stecula, 2020; Nicholson, 2012). Some studies have found this process in Canada as well, though perhaps weaker in magnitude than in the United States (Merolla et al., 2008, 2016).

However, responsiveness to party cues is not necessarily evidence of partisan bias. Many respondents may have limited information on a policy question and use party cues as a low-information shortcut to form an opinion that is generally in line with their partisan or ideological interests (Kam, 2005; Mondak, 1993). It is not then much of a surprise that party cues are weaker when people are given policy information (Bullock, 2011). Further, party cues may activate source credibility assessments that are at least partially independent of political motivation (Tappin et al., 2020). Nonetheless, responsiveness to elite cues is a central observation in a wide range of public opinion research, and such responsiveness could very well be—at least in part—because of partisan-motivated reasoning.

Study 2 evaluates directionally motivated responsiveness to party elite cues using wave nine of the Digital Democracy Project. I follow a strategy used by Bolsen et al. (2014) to experimentally manipulate not just party cues but motivation as well. The exogenous manipulation of motivation prevents the conflation of prior beliefs and directional motivation. This intervention is also orthogonal to the source credibility assessments that might be activated when respondents are exposed to party cues. Respondents who indicated they are supporters of the three major parties in Canada (that is, Liberals, Conservatives and NDP) were given a description of a bill passed in 2019:

We are going to ask you what you think about parts of the Bill C-48 that was passed in the previous Parliament. The Act contains the following provisions:

• • It prohibits oil tankers that are carrying more than 12 500 metric tons of crude oil or persistent oil as cargo from stopping, or unloading crude oil or persistent oil, at ports or marine installations located along British Columbia's north coast from the northern tip of Vancouver Island to the Alaska border.

• • It prohibits vessels and persons from transporting crude oil or persistent oil between oil tankers and those ports or marine installations for the purpose of aiding the oil tanker to circumvent the prohibitions on oil tankers.

• • It establishes an administration and enforcement regime that includes requirements to provide information and to follow directions and that provides for penalties of up to a maximum of five million dollars.

They were randomly assigned into three groups: one with no party cues, one with in-party cues and one with out-party cues. For example, a Conservative partisan in the in-party condition would receive a cue signalling Conservative party opposition to the bill, while NDP and Liberal partisans would receive a cue signalling their own party's support in Parliament. For the out-party condition, I combine Liberal and NDP cues for Conservative respondents and use the Conservative party for both Liberal and NDP partisans because partisan bias in responsiveness in elite cues is unlikely to manifest when Liberals observe NDP cues or vice versa. I make this choice because partisan sorting and affective polarization in Canada have been characterized by divergence between the NDP and Liberals, on the one hand, and Conservatives, on the other. There is an added benefit of preserving statistical power by limiting the number of treatment arms.

I crossed this randomization with a motivational prime. Respondents could either receive no motivational prime, an accuracy prime or a directional prime, leaving me with nine treatment conditions. Individuals can have a number of different motivational goals, with direction and accuracy drawing the most attention from scholars.

Accuracy refers to a motivation to reach correct, or at least relatively accurate, opinions or beliefs (Kunda, 1990). Individuals may achieve this by paying attention to pertinent information and ignoring less relevant pieces of information (Bolsen et al., 2014)—in this case, by focusing on policy details and ignoring party endorsements. Pursuing accuracy goals, however, requires more cognitive effort and the use of more complex decision rules (Kunda, 1990). Observing accuracy goals in practice is difficult, but social psychologists have been able to study accuracy motivation through experimental manipulation. Here, I use the following prime, following Bolsen et al. (2014):

When thinking about your opinion, please try to view the policy in an even-handed way and from various perspectives. We will later ask that you justify the reasons for your judgment—that is, why the policy's content is more or less appealing.

In contrast to accuracy, directional motivation refers to the desire to reach opinions and beliefs that are directionally aligned, such as with one's partisanship or ideology. People may feel a need to support positions of one's own party and to maximize differences with parties that they oppose, which I prime explicitly the following way, again following Bolsen et al. (2014):

When thinking about your opinion, please consider the bill was passed amidst heated conflict between conservative and progressive activists, resulting in a stark divide between parties in Parliament. We will later ask you about your party and why you affiliate with it (or why you choose to not affiliate with a party).

Respondents were asked their level of support for the bill (strongly support to strongly oppose, 5-point) after the vignette. I recoded the variable into a binary measure so that 1 was equal to a partisan-congenial response and 0 was otherwise. I have a clear expectation that party cues will induce partisan-congenial responses when directional motivation is primed.

H3: Exposure to party cues is associated with a higher share of party-congenial responses when directional motivation is primed.

In contrast, an accuracy prime should negate the influence of partisan cues by encouraging respondents to carefully consider the particulars of a policy. We should find no positive effect of party cues on the share of partisan-congenial responses among respondents given this prime. The prediction for respondents in the control condition, in contrast, is unclear. If the default mode of information processing involving party cues is toward direction, we should expect party cues to induce congenial responses at the similar magnitude as the condition with the directional prime. The control condition offers a useful baseline to situate the results from respondents given either the accuracy or directional primes (see Bolsen et al., 2014, for more discussion).

I use OLS to estimate the following model:

(2)$$pr( {congenial} ) = {\rm \;}\alpha + \beta _1cue + \beta _2accuracy + \beta _3direction{\rm \;} + \beta _4cue\; {\rm \ast }\; accuracy + \beta _5cue\; {\rm \ast }\; direction + \varepsilon {\rm \;}$$

My results here are robust to alternative modelling strategies as shown in Table S1 of the supplementary materials.

Study 2 results

Among respondents given the directional motivational prime, congenial responses were nine points higher for those given in- or out-party cues compared to those who were given none (p ~ .02, one-tailed), in support of H3. On average, there appears to be little difference in the effects of in-party and out-party cues among those who are directionally primed. In contrast, we see no evidence of an increase in congeniality precipitated by party cues among those given the accuracy prime (−0.07, p ~ .93, one-tailed). Interestingly, we similarly see no evidence of an increase in congeniality among those in the control condition (−0.02, p ~ .72, one-tailed), suggesting that party cues have little influence on policy attitudes unless partisan motivation is explicitly primed. The estimates are provided in Table A3 of the Appendix.

Replication

I replicated study 2's experiment using a survey conducted by the Media Ecosystem Observatory on a nationally representative sample of 2,499 Canadian citizens of voting age provided by Dynata. It used national quotas for age, gender, region and language to ensure representativeness. The data was further weighted within region by age and gender using the Canadian census. The experiment in this survey was identically constructed, except for the fact that it eliminated conditions featuring the accuracy prime. The lack of difference between the control and accuracy condition in the original study suggests there is little responsiveness to party cues without an explicit priming of partisan identity, at least with this policy issue, so the control condition alone can serve as a baseline for comparison to the directional prime. This choice also allows me to conserve statistical power.

I find congenial responses to be five points higher among those given party cues but only when given the directional prime (p ~ .08, one-tailed). In-party cues (+8, p ~ .03) appear to be more effective than out-party cues (+2, p ~ .60), though they are not significantly different from one another.

In short, this study provides evidence of partisan-motivated responsiveness to party cues in the Canadian context. Citizens follow the lead of parties when their partisan identity is primed, though we find little evidence here that they do so when this identity is not primed. Previous studies in Canada have illustrated a responsiveness to party cues (Merolla et al., 2008, 2016), but this design allows for a more confident attribution of at least some of this responsiveness to partisan bias.

Study 3: Ideologically Motivated Numeracy

Study 2 is based on a party cue design that is specifically tailored to identify partisan bias in the acceptance of source cues. Another approach is to manipulate the congeniality of political information and observe its effect on policy attitudes. As noted, the problem with this design is that the provided information may affect reasoning due to one's prior beliefs, which may or may not be informed by one's political motivation (Tappin et al., 2020). Observing belief change (or not) in response to congenial (or uncongenial) information is not necessarily indicative of partisan or ideological motivated reasoning.

I use Kahan et al.'s (2017) numeracy task to overcome these causal inference challenges. Respondents were given one of four versions of a problem that required them to provide the correct interpretation of data presented in a 2 × 2 table. Respondents in two conditions received a nonpolitical version of the table illustrating whether the application of a skin cream made rashes better or worse among a set of study subjects. They were then asked whether the data provided suggested people who used the skin cream were more likely to get better or worse than those who did not. One group received a version of the study suggesting the cream made the rash better on average. The other group saw the same numeracy task except with the numbers flipped, so that the skin cream appeared to make the rash worse. An example of the nonpolitical task can be found in Figure A2 of the Appendix.

The second pair of conditions included a study related to the effect of a handgun ban on crime rates. They included a table with the exact same numbers as the skin cream study, but in this case, it indicated how many cities that adopted (and failed to adopt) a handgun ban experienced increases (and decreases) in crime afterward. One of these conditions included numbers suggesting that cities that implemented the handgun ban were more likely to experience decreases in crime, while the other condition included numbers that implied the opposite—in the exact same manner as the nonpolitical version of the task. An example of the political task can be found in Figure A3 of the Appendix.

The main advantage to this design is that I am not asking respondents their opinion about gun control. I am asking about their interpretation of the data. This constitutes a direct observation of biased reasoning brought on by respondents' ideological worldviews. The numeracy task is tough. Most respondents will use a simple heuristic, such as comparing cells 1 and 2 without accounting for patients who did not receive the skin cream. Respondents who have strong quantitative aptitude should be more able to answer correctly than those who do not. In the absence of bias, this should be the case in both the political and nonpolitical conditions.

The expectation, however, is that the congeniality of the information will influence the propensity of respondents to get the correct answer among those given the political variant of the task:

H4: Ideological congeniality will predict correct responses more strongly for those given the political numeracy task.

I use a measure of policy-based ideology from study 1 as the primary moderating variable. I rescale it from 0 to 1 such that 0 represents respondents for whom the information in the numeracy task is maximally dissonant and 1 identifies those for whom the information is maximally congenial. I call this measure their ideological congeniality. So, for example, a person would score 1 if they were perfectly consistent in their policy responses as a left- or right-leaning individual and their stimulus suggested the skin rash or crime rate improved or worsened, respectively, as a result of the intervention. I estimate the following model with OLS where β₃ should be positive and significant in support of H4:

(3)$$pr( {correct} ) = \alpha + \beta _1political\;treatment + \beta _2congeniality + \beta _3political\;treatment \;\ast \; congeniality + \varepsilon \;$$

Conversely, I expect numerical aptitude to predict correct answers among those given the nonpolitical task but not for those given the political variant:

H5: Numerical aptitude will predict correct responses more strongly for those given the nonpolitical numeracy task.

I operationalize numerical aptitude using a numeracy index validated in previous literature (Liberali et al., 2012; Peters et al., 2006). I rescale this index from 0 to 1, where 1 represents those who had a perfect score across all items and are thus are strong in their quantitative ability. I estimate the following model where β₂ should be positive and significant and β₃ should be negative and significant in support of H5:

(4)$$pr( {correct} ) = \alpha + \beta _1political\;treatment + \beta _2numeracy + \beta _3political\;treatment \; \ast \; numeracy + \varepsilon \;$$

Ideological bias will also affect people's evaluation of information quality. The prediction here is tricky with this design. Normally we would expect congenial information to generate more positive evaluations of evidence quality compared to uncongenial information. But in this case, not everyone draws the correct inference from the presented data. Here I am instead interested in perceived ideological congeniality:

H6: Perceived ideological congeniality will predict positive evaluations of evidence quality more strongly for those given the political numeracy task.

After completion of the numeracy task, I ask respondents their perception of the strength of the evidence presented by the study (very strong to not strong at all, 4-point), which was rescaled from 0 to 1, where 1 represents respondents with the most positive evaluation of the evidence. Perceived ideological congeniality is based on the same congeniality measure as in equation 3 but with the scale reversed for respondents that arrived at the wrong answer. I estimate the following model where β₃ should be positive and significant in support of H6:

(5)$$evaluation = \alpha + \beta _1political{\rm \;}treatment + \beta _2perceived{\rm \;}congeniality + \beta _3political{\rm \;}treatment\; {\rm \ast } \; perceived{\rm \;}congeniality + \varepsilon $$

Study 3 results

I illustrate the results using linear predictions, but the estimates can be found in Table A4 of the Appendix. The left panel of Figure 2 shows a powerful relationship between ideological congeniality and correct responses among those given the political numeracy task. Eighty-three per cent of respondents were expected to get the correct answer when the treatment was the most aligned with their ideological outlook (for example, a perfectly consistent conservative given a variant in which gun control increases crime), while the same was true of only 13 per cent of respondents when the treatment was maximally dissonant with their worldviews. In sharp contrast, there was no relationship between ideological congeniality and correct responses among those respondents who were given the otherwise identical nonpolitical variant of the numeracy task. The interaction is significant (p < .001), providing strong support for H4.

Figure 2 Predicted Share of Respondents Getting Correct Answer across Levels of Ideological Congeniality (left) and Numerical Ability (right) for Political and Nonpolitical Conditions

Note: 95 per cent confidence intervals.

Ideological congeniality also crowds out numeracy in predicting correct answers in the politically charged conditions. The right panel of Figure 2 illustrates this effect. Among respondents who were given the nonpolitical task, those with the lowest score on the numeracy index were able to get the answer correct only 31 per cent of the time, compared to 66 per cent for those with the highest level of numeracy. Strikingly, there was no relationship between numeracy and correct responses among those given the political numeracy task. The interaction is again significant (p < .001), providing support for H5.

Ideological bias seeps into the numeracy task, leading many respondents astray when the result does not conform to their prior beliefs. It also plays a role in how they evaluate the quality of the evidence presented, as shown in Figure 3. Among respondents who were given the political numeracy task, those who perceived themselves to be most ideologically aligned with the study's findings were more impressed with the quality of the evidence (0.66) than those who perceived the information to be the most dissonant (0.48), a relationship that is significant (p < .001). There was no such relationship among those given the nonpolitical task. This interaction is significant (p < .05), providing support for H6. The relationship presented resembles confirmation bias, rather than disconfirmation bias. People who perceived the study results to be uncongenial with their preferences had similar evaluations of the quality of evidence as those who were given the nonpolitical task.

Figure 3 Predicted Evaluation of Evidence across Levels of Perceived Ideological Congeniality for Political and Nonpolitical Conditions

Note: 95 per cent confidence intervals.

The upshot is that ideology disables faculties we typically use to reason about and evaluate evidence—in this case, respondents’ quantitative aptitude. Respondents facing a task revealing uncongenial ideological information were less likely to get the answer correct than similar respondents in a nonpolitical task, and they were more likely to do so when the information was aligned with their beliefs. In sharp contrast, respondents’ level of numerical aptitude predicted correct responses in the nonpolitical task, but this relationship disappeared entirely for those in the political treatment conditions.

Discussion

The three studies above present a largely consistent picture of ideological and partisan-motivated bias in the political information search and processing of ordinary Canadians. My survey respondents were more likely to choose information that was aligned with their worldviews (H1). They were more likely to follow party cues when their partisan identities were primed (H3) and were more likely to evaluate evidence correctly (and more positively) when it was (perceived to be) aligned with their beliefs (H4 and H6), while their numerical aptitude mattered less in politicized contexts (H5).

I find no evidence, however, that individuals choose news sources that are aligned with their political beliefs over mainstream outlets, or even over those that oppose their views (contra H2A/H2B). Ideologically right-wing and left-wing respondents both prefered mainstream and local news sources over aligned partisan media. This pattern could be the result of the partisan sources having lower levels of source credibility or less familiarity. Digital Democracy Project data provide evidence for both contentions.⁵ This null finding is also consistent with recent work on the media diets of Canadians: the reach of partisan media is very limited (Owen et al., 2020). Although I find null effects at the moment for selective engagement with these partisan-congenial sources, this does not mean this observation will carry into the future if these outlets or others gain more prominence and credibility.

Each study, however, has important limitations. Study 1 presents a choice of news content in the highly artificial environment of a survey experiment. Ultimately, the decisions people make when choosing the news are more complex and organic. The absence of a neutral headline condition also leaves me unable to tease out whether people are being attracted by congenial headlines, repelled by dissonant ones, or some combination. Prior research outside of Canada strongly suggests the former, however (Fischer et al., 2008; Metzger et al., 2020). This limitation does not take away from the central finding that there is some congeniality bias in the selection of news content, though there is no such evidence in source selection. Future research should tease out the precise mechanism.

Another limitation of conjoint experiments is that the AMCEs reflect an average of the direction and intensity of respondent preferences. We cannot make claims about whether or not there is majority preference for a given attribute (Abramson et al., 2019). This problem is analogous to the issues of interpretation of the average treatment effect for traditional experiments; finding such an effect does not mean that there is a positive effect for most respondents (Bansak et al., 2021). My interest, however, is not in illustrating majority preference for any given attribute, as is common in vote choice conjoints, so this problem is less applicable here.

Studies 2 and 3 provide compelling evidence of partisan and ideologically motivated reasoning. They avoid many of the pitfalls found in prior work that make it impossible to disentangle politically motivated cognition from responsiveness to source credibility or prior beliefs. But they also have their limits.

The effects we observe in study 2 are substantively small. And, in fact, I find no responsiveness to party cues without an explicit partisan identity prime. The small effects may be the result of using a salient energy-related topic where parties are polarized and have well-known positions. These small effects limit my ability to tease out heterogeneous effects. For instance, scholars have drawn attention to the role of both in-party (Kam, 2005) and out-party cues (Merkley and Stecula, 2020) or the moderating role of political sophistication (Bakker et al., 2020). More broadly, party cue effects vary considerably across issue, being generally weaker on “easy” issues such as crime and social issues and stronger on more complex domestic and foreign policy issues (see Tappin, 2020). We need a more comprehensive accounting of party cue effects in the Canadian context to fully understand how it varies across issues and individuals and how much of it can be attributed to partisan bias rather than a rational cue-taking process.

For study 3, the gains from the numeracy task in terms of experimental control and interval validity come at a cost of external validity. Future studies should attempt to build in more realism and, in doing so, better approximate how citizens evaluate information in the real world. These studies should not, however, lose sight of the advantages of either exogenously manipulating political motivation or focusing on outcome measures that minimize the importance of credibility assessments and prior beliefs.

These concerns aside, all three studies point in the same direction. Canadians choose and process information in ways that are biased by their partisan identities and ideological worldviews. The results across these experiments are similar to what is found in the United States—perhaps the quintessential case of a country with a polarizing mass electorate.

The implications of these findings are considerable. What do they mean for how Canadians evaluate the economy (Bartels, 2002; Evans and Pickup, 2010) or their beliefs on highly charged issues such as climate change and energy policy? A political environment contaminated by partisan or ideological bias is one in which neutral, objective information matters less in persuasion, and one in which finding frames or sources that can reach across political divides matters much more. It is one in which demand for congenial news content increases at a time when mainstream outlets come under increasing financial pressure.

Perhaps the most striking findings are provided by study 3. Respondents systematically evaluate evidence in ways that conform to their ideological beliefs. The effects are large, and this dynamic crowds out cognitive sophistication, consistent with Kahan et al.'s (2017) work in the American context. This bodes ill for efforts to improve the quality of democratic discourse by providing more information or facts to the mass public. Left to their own devices, partisans and ideologues can observe the exact same information and reach diametrically opposing conclusions about its implications. Future work should examine whether additional communication interventions can reduce this bias, such as by priming accuracy motivation (Bolsen et al., 2014) or by rewarding people for correct responses in case these results emerge through some form of expressive responding (Bullock et al., 2015).

And if ideological and partisan bias exists in Canada, what should we make of US-centric explanations for the existence of this bias? Future research should implement experiments cross-nationally to shed light on how partisan and ideological bias varies across contexts. Perhaps motivated reasoning and partisan bias are simply a reality of democratic politics? Much more work remains to be done in understanding the consequences of both political polarization and partisan or ideological bias in Canada and cross-nationally.