The relationship between cognitive style and political orientation depends on the measures used

Abstract

The present research investigated the reason for mixed evidence concerning the relationship between analytic cognitive style (ACS) and political orientation in previous research. Most past research operationalized ACS with the Cognitive Reflection Test (CRT), which has been criticized as relying heavily on numeracy skills, and operationalized political orientation with the single-item self-placement measure, which has been criticized as masking the distinction between social and economic conservatism. The present research recruited an Amazon Mechanical Turk sample and, for the first time, simultaneously employed three separate ACS measures (CRT, CRT2, Baserate conflict problems), a measure of attitudes toward self-critical and reflective thinking (the Actively Open-Minded Thinking Scale; AOT), and separate measures of social and economic conservatism, as well the standard measure of political orientation. As expected, the total ACS score (combination of the separate measures) was negatively related to social, but not economic, conservatism. However, the CRT by itself was not related to conservatism, in parallel with some past findings, while the two other measures of ACS showed the same pattern as the combined score. Trait reflectiveness (AOT) was related negatively to all measures of political conservatism (social, economic, and general). Results clearly suggest that the conclusion reached regarding the ACS-political orientation relationship depends on the measure(s) used, with the measure most commonly employed in past research (CRT) behaving differently than other measures. Future research must further pursue the implications of the known differences (e.g., reliance on numeracy vs. verbal skills) of ACS measures and distinguish different senses of reflectiveness.

1 Introduction

The dual-process model of the mind has inspired a surge of recent research attempting to explain moral judgment (Greene, Sommerville, Nystrom, Darley & Cohen, 2001; Greene, Morelli, Lowenberg, Nystrom & Cohen, 2008; Paxton, Ungar & Greene, 2012; Trémolière, De Neys & Bonnefon, 2012) and religious belief (Gervais & Norenzayan, 2012; Pennycook, Cheyne, Seli, Koehler & Fugelsang, 2012; Pennycook, Ross, Koehler & Fugelsang, 2016; Shenhav, Rand & Greene, 2012; Yilmaz, Karadöller & Sofuoglu, 2016). The present research is concerned with the application of this model to the domain of political ideology, which is also an increasingly popular research topic. For instance, individual differences in political orientation and ideological beliefs have been associated with cognitive style (e.g., Jost, Glaser, Kruglanski & Sulloway, 2003; Jost, Sterling & Stern, in press). More specifically, there is an emerging literature on the analytic-intuitive thought differences across ideological groups (Brandt, Evans & Crawford, 2015; Deppe et al., 2015; Eidelman, Crandall, Goodman & Blanchar, 2012; Iyer, Koleva, Graham, Ditto & Haidt, 2012; Kahan, 2013; Landy, 2016; Piazza & Sousa, 2014; Pennycook et al., 2012; Sterling, Jost & Pennycook, 2016; Talhelm, Haidt, Oishi, Zhang, Miao & Chen, 2015; Van Berkel, Crandall, Eidelman & Blanchar, 2015; Yilmaz & Saribay, 2016, 2017).

A general observation in this line of research has been the negative correlation between the single-item self-placement political orientation measure (where typically 1 = liberal/left and 7 = conservative/right) and scores on the Cognitive Reflection Test (CRT; Frederick, 2005), a test used to measure analytic thinking tendency (Deppe et al., 2015; Pennycook et al., 2012; Iyer et al., 2012; Talhelm et al., 2015; Yilmaz & Saribay, 2016). However, using these measures, Kahan (2013) was not able to replicate this finding in a representative sample of U.S. participants. More recently, Landy (2016) replicated this finding in only one of three samples (see also Baron, 2017; Kahan & Corbin, 2016; Piazza & Sousa, 2014). What explains these contradictory findings?

The dual-process model originally argues that human mental operations are supported by two different set of mental processes. The evolutionarily older set (Type 1) provides automatic, intuitive, and fast responses whereas the newer set (Type 2) provides analytic, controlled, and slower responses (Evans, 2003; Evans & Stanovich, 2013; Morewedge & Kahneman, 2010). However, this model has been criticized because it assumes that the automatic activations of intuitive processes are blocked sequentially by reflection (e.g., Baron, Scott, Fincher & Metz, 2015; Bialek & De Neys, 2016; Klein, 2011; Thompson, 2009; Trémolière & Bonnefon, 2014; Sinayev & Peters, 2015). This suspect assumption is also evoked in discussions of the relationship between CRT performance and various psychological tendencies. For instance, Baron et al. (2015) argued that this sequential assumption cannot explain the relationship between CRT performance and moral judgment. In the present research, we refrain from drawing inferences regarding this theoretical discussion. However, this critique bears importance for investigations of the cognitive style-conservatism relationship because most past studies of the latter relationship have relied on the CRT to operationalize cognitive style.

There have been other criticisms directed at the CRT. For instance, Sinayev and Peters (2015) found that higher scores on the CRT have more to do with numeracy skills than high-effort thought. Other researchers have shown that there is wide familiarity with the CRT in test populations and this may affect scores (Haigh, 2016; Stieger & Reips, 2016). These criticisms have led to the development of alternative measures of ACS, such as CRT–2 (Thomson & Oppenheimer, 2016).

In addition to overreliance on the CRT, most past studies have used the single-item self-placement measure of political orientation. The limitations of this measure have been discussed extensively (see Feldman & Johnson, 2014). For instance, people who are both socially and economically conservative; socially conservative but economically liberal; and socially liberal but economically conservative all tend to place themselves toward the conservative side of the single-item scale. When given the chance, some of these people identify with ideological labels other than conservative. The best known example may be libertarians: Feldman and Johnson (2014) have shown that approximately 20% of self-identified conservatives are actually libertarians who are conservative economically but not socially. Taking these observations into account, Pennycook et al. (2012) distinguished fiscal and social orientations and showed that self-placement on only the social dimension was significantly related to cognitive style. Likewise, Deppe et al. (2015) measured reactions to various social and economic policy issues in addition to general orientation and showed that socially liberal participants relied more on reflection while economic orientation generally appeared unrelated to cognitive style. Yilmaz and Saribay (2016) replicated these findings using similar measures in a non-Western sample.

Contrary to the findings just summarized, some studies associate a neoliberal orientation (economic conservatism) with intuitive cognitive style. For instance, Sterling et al. (2016) found that participants who gave stronger support for neoliberal policies tend to be higher on bullshit receptivity, which is linked to intuitive thought. In addition, a recent meta-analysis (Jost et al., in press) reported a significant and negative relationship between CRT scores and economic attitudes, even though its effect size was smaller than that for the relationship between CRT scores and social conservatism.

As seen above, the literature offers mixed findings. It is clear that there is need for further research using operationalizations of ACS other than the CRT as well as more nuanced measures of political orientation. On the basis of this observation, we asked an Amazon Mechanical Turk sample to complete measures of ACS that included the CRT, two other problem-solving tasks, a measure of attitudes toward self-critical and reflective thinking (Actively Open Minded Thinking Scale; AOT), and measures of social and economic conservatism. We expected that social conservatism would be related (negatively) to ACS while economic conservatism would not. However, in light of recent criticisms of the CRT, we suspected that it may fail to reveal this expected pattern or otherwise behave differently than other measures of ACS that do not suffer from the same limitations.

2 Method

2.1 Participants

Five hundred and twenty-three participants were recruited via Amazon Mechanical Turk. Participants who did not complete the full battery and those with an IP outside of the U.S.A. were excluded from analyses. This resulted in 426 participants (M_age = 38.67, SD_age = 13.81, 235 female, 160 male). 37.1% of the participants identified themselves as Democrat (n = 158). Eighty five were Republican (20%), 96 (22.5%) Independent, 12 Libertarian (2.8%), nine Socialist (2.1%), four Anarchist (0.9%), 18 Other (4.2%), nine “Don’t know/prefer not to say” (2.1%), and 35 missing.

2.2 Materials and Procedure

Participants were directed to an online survey that took approximately 25 minutes to complete. Except for demographic questions (which were always presented at the end in fixed order), the order of the measures listed below and the order of the items within each measure were randomized. The survey was implemented using PsyToolkit (www.psytoolkit.org; Stoet, 2010, 2017).

ACS.

We used three different measures of ACS (see Appendix A for the entire set of items). The first one was the CRT (Frederick, 2005), which has been used in many other studies for the same purpose. Participants were given three test items each with an intuitive (but wrong) answer. Arriving at the correct answer requires relatively high-effort thinking. Those with a stronger tendency to think analytically can override the intuitive response and produce the logically correct answer. Open-ended responses were coded as correct (1) or incorrect (0) and averaged to produce a total CRT score (Cronbach’s α = .75).

The second ACS measure was CRT–2, developed by Thomson and Oppenheimer (2016) as an alternative to CRT to overcome the problems of growing participant familiarity and emphasis on numeracy (see the Introduction). Unlike original CRT items, CRT–2 items are based on verbal questions. Open-ended responses were coded as correct (1) or incorrect (0) and averaged to produce a total CRT–2 score (Cronbach’s α = .64).

As the third measure of ACS, we used Baserate-conflict problems (BRC; see Pennycook et al., 2012). In these problems, as in the CRT, there is an intuitive (wrong) and analytic (correct) answer. Each problem presents a conflict between baserate information that appeals to Type 2 thinking (and helps to produce the correct answer) and misleading stereotypic content that appeals to Type 1 thinking. Analytic thought presumably enables participants to override the appeal of the misleading stereotypic information and take into account the baserate thereby helping to produce the correct answer. Participants chose among two answers and responses were recorded as correct (1) or incorrect (0) and averaged to produce the BRC score (Cronbach’s α = .75).

All measures of ACS were converted to POMP (percent of maximum possible) scores (Cohen, Cohen, Aiken & West, 1999) to standardize them into a common scale ranging from 0 to 100. We then combined CRT, CRT–2, and BRC POMP scores into a single composite ACS score (Cronbach’s α = .76). We also report separate analyses for three different ACS measures.

Actively Open-Minded Thinking Scale.

This seven item scale was used by Haran, Ritov and Mellers (2013) and measures the value placed on to be open to revisiting one’s beliefs when contradictory evidence is available (see Appendix B). Responses were collected on a 9-point Likert-type scale (1 = strongly disagree; 9 = strongly agree). This scale does not measure reflectiveness in the sense of measures like the CRT. It concerns only the direction of thinking (Cronbach’s α for this study = .69)¹. AOT responses were averaged and converted to a POMP score.

Conservatism.

We used several measures of conservatism. We asked participants their self-reported political (i.e., general), social, and economic orientations with the single-item self-placement question (“How would you place your political views generally speaking?” “When it comes to social issues, how liberal or conservative are you?”, “When it comes to economic issues, how liberal or conservative are you?”). The response format for these three questions was a Likert-type scale ranging from 0 (extremely liberal) to 10 (extremely conservative). In addition, we used a 12-item Social (Cronbach α for this study = .85) and Economic (Cronbach α for this study = .62) Conservatism scale developed by Everett (2013; see Appendix C). In this scale, participants rate how positive or negative they feel about different policy issues (such as Abortion or Limited government) on a feeling thermometer (0 = negative, 100 = positive). Social and Economic items were averaged separately. All the questions were also converted to POMP scores and averaged into two composite variables representing social (social orientation self-placement and Everett’s social conservatism subscale) and economic conservatism (economic orientation self-placement and Everett’s economic conservatism subscale). The general political orientation question was analyzed separately and we also report correlations for the separate scales.

Demographic Variables.

Participants were asked to report their gender (0 = male, 1 = female), age (in years), education level (1 = less than a high school degree, 6 = Graduate degree), income (total earned in 2016: 1 = $15,000 or less, 7 = over $100,000), and political affiliation. Education and income were converted to POMP scores.

3 Results

As expected, composite ACS was negatively correlated with the composite social conservatism score (r = –.166, p = .001) and general political orientation (r = –.116, p = .019), but not with the composite economic conservatism score (r = –.065, p = .183). Among demographics, female gender (r = –.101, p = .046), education level (r = .249, p < .001), and income (r = .113, p = .027) are significantly correlated with ACS.²

Table 1 displays the correlations between the separate cognitive style measures and conservatism. When we separately analyzed three ACS measures, CRT did not significantly correlate with any of the conservatism measures including social conservatism (r = –.061, p = 218). However, both CRT–2 and BRC did significantly correlate with social conservatism (r = –.141, p = .004; r = –.167, p = .001, respectively) and general political orientation (r = –.129, p = .010; r = –.103, p = .039, respectively), but not with economic conservatism (r = –.066, p = .188; r = –.070, p = .159, respectively).

Table 1. Correlations between the cognitive style and conservatism measures.

Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. Polit. Orient. = Political Orientation; Social Orient. = Social Orientation; SECS Social Conserv. = Social Conservatism subscale of Everett’s (2013) scale; Econ. Orient. = Economic Orientation; SECS Econ. Conserv. = Economic Conservatism subscale of Everett’s (2013) scale; BRC = Baserate conflict; CRT = Cognitive Reflection Test; CRT–2 = Thomson & Oppenheimer’s (2016) newer Cognitive Reflection Test; AOT = Actively Open-Minded Thinking Scale.

On the other hand, AOT was negatively correlated with all conservatism measures including social (r = –.483, p < .001) and economic conservatism (r = –.292, p < .001; see Table 1). In other words, these results show that conservatism correlates negatively with cognitive style. However, economic conservatism correlates less well (if at all) than social conservatism. In addition, the original CRT correlates less well than other ACS measures whereas the AOT correlates more strongly.

For descriptive purposes, we provide in Tables 2 and Tables 3 the correlations among demographic variables and conservatism and cognitive style variables.

Table 2. Correlations between the demographic and conservatism measures.

Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. See Table 1 for abbreviations.

Table 3. Correlations between the demographic and cognitive style measures.

Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. See Table 1 for abbreviations.

One may think that the lack of correlations between CRT and political variables is due to the low reliability of the CRT. However, CRT did not fare particularly low in reliability compared to the other ACS measures (see Method). In addition, we examined the correlations between the cognitive style measures and found that the CRT correlated reasonably well with the other ACS measures and the AOT (see Table 4). This suggests that it is unlikely that the CRT in particular suffered from a reliability issue. Finally, we corrected the correlations between the cognitive style measures and political variables for measurement error (see Spearman, 1904)³. These figures are shown in Table 5 and, while the disattenuated correlations are higher than the raw correlations shown in Table 1, they also indicate the separate ACS measures and economic conservatism are negatively but weakly related, if at all. Social conservatism, however, is more strongly related to ACS measures.

Table 4. Correlations amongst the cognitive style measures.

Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. See Table 1 for abbreviations.

Table 5. Disattenuated correlations between the cognitive style and conservatism measures.

Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed.

4 Discussion

As far as we know, our study was the first to use multiple measures of all the key constructs involved: ACS and political orientation. The present findings contrast somewhat with those from Jost et al.’s (in press) meta-analysis in that they revealed a solid relationship between ACS and social conservatism but only a weak relationship with economic conservatism. Moreover, of the three ACS measures — CRT, CRT–2 and BRC — the original CRT was the weakest, with no significant correlations with political measures. Thus, the conclusion reached about the relationship between ACS and political orientation on the basis of our data depended largely on the measure used.

Unfortunately, there is no way to check whether the same would be true of earlier studies, since most of them used only the CRT. This is evident from Jost et al.’s (in press) meta-analysis, which is dominated by the CRT. This situation warrants caution as reliance solely on the CRT has recently been criticized on an empirical basis (Baron et al., 2015). Taking into account such criticism, we employed a more verbally-oriented version of CRT (CRT–2 and partially BRC problems), as well, and it indeed yielded qualitatively different results.

Additionally, we used the AOT, a measure of attitudes toward self-critical and reflective thinking concerning the value placed on revising existing beliefs in light of incoming information, and found that it was negatively related to both social and economic conservatism. This suggests that economic conservatism is related to reflectiveness but mostly in the sense operationalized by the AOT rather than the sense of reflection/impulsivity focal to CRT, CRT–2, and BRC problems.

Overall, the findings are in line with some past research and contrary to others and thus, point to the need for continued research on the topic. For instance, the present findings contrast with analyses reported by Deppe et al. (2015) using only the CRT showing generally negative relationships between CRT scores and social conservatism. Deppe et al. (2015) attempted (but failed) to activate analytic thought experimentally whereas the correlations above are comparable only to their control condition. When we focus only on the control conditions in Deppe et al. (2015), there is a significant negative correlation between the original CRT and economic conservatism only in one of four studies. On further analyses of the same data by Baron (2015), including both the control condition and the (ineffective) experimental condition, there was a relatively strong and significant positive correlation between economic conservatism and the CRT in what was probably the most representative of the U.S. population, yet still a (smaller) negative correlation in one of the studies using Amazon Mechanical Turk. It thus seems that populations also differ in the relationship between the CRT and conservatism.

Similarly, the present findings are also not in line with our previous findings from Turkey showing a negative correlation between CRT scores and general political orientation (Yilmaz & Saribay, 2016). However, they are consistent with Kahan’s (2013) and Landy’s (2016) research, showing a lack of relationship between CRT scores and self-placement on the single-item political orientation scale⁴. Finally, there is also partial overlap between our findings and those of Sterling et al. (2016) in that they found a negative relationship between the Need for Cognition–which is similar to AOT–and fiscal (vs. social) conservatism; and a negative relationship between their Heuristics and Biases measurement–which is similar to our ACS measures– and social (vs. fiscal) conservatism. Thus, it appears so far from these efforts that ACS and trait reflectiveness (e.g., AOT, Need for Cognition) may have different relationships with political attitudes. Baron et al. (2015) argue that CRT-type measures and AOT-type measures tap different cognitive styles (see also Baron, 2017). Baron et al. (2015; see also Baron, 1995) specifically claim that thinking can be characterized in two dimensions–amount and direction–and that CRT measures amount of thinking. CRT performance can be characterized in terms of the desire to spend more time on a task in order to produce more accurate answers. In contrast, AOT concerns the desire to spend time considering arguments countering a preferred position and openness on that position (see Baron, 2017). Baron (2017) argues that the desire to spend time for increased accuracy on CRT-type measures is related to the desire to spend time proving one’s position is superior to countering positions on political and moral issues. Those scoring high on the AOT can instead be characterized as being more willing to spend time processing counter-arguments. This may explain the stronger correlation between political conservatism and AOT versus other ACS measures. Future research should further probe this possibility and the underlying reasons for these differential relationships between social/economic conservatism as well as aim for increased clarity regarding the different senses of reflectiveness.

All in all, the present findings suggest that ACS is related to social, but not economic, conservatism. Two out of three operationalizations of ACS supported this pattern of relationship. The exception was the widely-used CRT which had no significant relationship with political orientation, either social or economic. This adds to previous criticisms of the CRT and suggests that, at the least, whenever it is used in this domain, it should be supplemented with other ACS measures not based on numeracy. In addition, trait reflectiveness is seen to show a distinct pattern of responses compared to ACS. Thus, different conceptualizations of reflectiveness should be distinguished and, in this sense, more sophisticated analyses of the cognitive style-political orientation link should be conducted in future research.