Political and Social Discussion Network Survey Items Are Not Interchangeable

Abstract

Experimentalists and survey researchers regularly measure the makeup and size of respondent personal discussion networks to learn about the social context in which citizens make political choices. When measuring these personal networks, some scholars use question prompts that specifically ask respondents about whom they discuss “politics” with, while others use more general prompts that ask respondents about whom they discuss “important matters” with. Prior research suggests that “political” discussion network prompts create self-reported networks that are substantively similar to “important matters” prompts. We conduct a nationally representative survey experiment to re-evaluate this question. Our results suggest that, although the size of networks generated by the two questions may be similar on average, the two questions generate different response distributions overall. In particular, respondents interested in politics report larger political discussion networks than general discussion networks, and respondents uninterested in politics report smaller political discussion networks than general discussion networks.

Introduction

Personal social networks play an important role in citizens’ political behaviors and attitudes (Pietryka and DeBats, 2017; Reilly, 2017; Ryan, 2011; Santoro and Beck, 2017; Settle, Bond, and Levitt 2011; Sinclair, 2012; Sokhey and McClurg, 2012; Song and Eveland Jr, 2015). As a result, social network measures have been widely used in surveys to measure respondents’ personal communication contexts. Among scholars who analyze political discussion, however, there is a long-running debate about whether important differences exist between measuring a citizen’s “political” discussion networks and their more general “social” discussion networks. For instance, some scholars suggest that a citizen’s political discussion network will be more politically homogenous than their general social discussion network due to political conflict avoidance on the part of citizens (Mutz, 2002). Others, however, suggest that most citizens do not meaningfully distinguish between political and social discussion partners, discussing both political and nonpolitical matters in a single core discussion network of close acquaintances (Huckfeldt and Sprague, 1995).

This debate yields challenges for the construction of survey items. Should researchers ask respondents about their “political” discussion networks or their general “social” discussion networks? Klofstad, McClurg, and Rolfe (2009) argue that the matter is moot: they find that political discussion survey questions prompt largely indistinguishable responses to more general “important matters” social discussion questions on a variety of dimensions, including the size of the discussion network, characteristics of network members, and discussion frequency.

Yet questions remain about the interchangeability of the two discussion measures. Responding to survey questions is a cognitively complex process of memory search, retrieval, consolidation, and finally, mapping judgments to a response option (Krosnick, 1999; Schwarz and Oyserman, 2001). Moreover, many people have different ideas about what constitutes “political” discussion in the first place (Eveland Jr, Morey, and Hutchens 2011; Settle, 2018). Politically interested respondents, in particular, are not only more enthusiastic about discussing politics with others but also have a significantly broader view of what counts as “political” discussion than do the politically uninterested (Fitzgerald, 2013). Simultaneously, increased partisan animosity in recent years (Mason, 2018) has made less interested citizens more inclined to disengage from contentious and rancorous political discussion entirely (Klar and Krupnikov, 2016; Settle and Carlson, 2019).

Accordingly, in the modern political environment, we might expect different kinds of responses from citizens now than in the past. In particular, not only might highly politically interested citizens report larger political discussion networks, when compared to garden-variety social discussion networks, but also the politically uninterested may report smaller political discussion networks. The two survey measures may not be as interchangeable as previously thought.

Study

To evaluate whether political discussion and general social discussion prompts are interchangeable on surveys, we embedded a survey experiment in a module of the 2020 Cooperative Election Study (Reilly, 2022; Schaffner, Ansolabehere, and Luks 2021). In the module, we randomly assigned every respondent to receive one of two network prompts:

• Treatment A (“Politics”): How many friends and family members would you say you regularly talk to about politics? (n = 427)

• Treatment B (“Social”): How many friends and family members would you say you regularly talk to about important matters in your life? (n = 415)

Questions were phrased to be brief but still closely match other survey questions regularly used in political research, including the choice of “politics” and “important matters” as the two key phrases (see, for instance, Huckfeldt, Johnson, and Sprague (2004); Klofstad, McClurg, and Rolfe (2009)). Respondents could indicate that they talked to 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or 10+ people. ¹

Table 1. Distribution of network size across two measures

On average, “politics” respondents reported 4.3 discussion partners, while “social” respondents reported 4.0 discussion partners. However, while the two resulting response distributions are centered in roughly the same place, they have different shapes. The “politics” group provided more responses on the tails of the distribution (0 or 10+ discussion partners) while the “social” group reported comparably fewer, resulting in a distribution with more central clustering, peakedness, and a smaller overall standard deviation ( $sd_{pol}=3.2$ , $sd_{soc}=2.5$ ; $kurtosis_{pol}=2.6$ , $kurtosis_{soc}=4.0$ ; see Figure 1). Furthermore, although a t-test yields a negligible and statistically insignificant difference between sample means, a Kolmogorov–Smirnov test reveals a statistically significant difference in the distribution of overall responses (Table 1). Additional analysis confirms that the probability of response is significantly different in tail categories 0 and 10+, where more people from the “politics” condition respond, and in central categories 3 and 4, where more people from the “social” condition respond. ²

Figure 1. Distribution of network size across two measures.

To evaluate whether these differences were related to political interest, as speculated, we fit a censored poisson regression model, predicting reported discussion network sizes by the experimental condition, the respondent’s level of political interest, and the interaction of the two variables. We operationalized “political interest” using a survey question asking how often a respondent would “follow what’s going on in government and public affairs”, with four response options varying from “hardly at all” to “most of the time”. ³

Figure 2 plots the predicted network size of respondents based on their experimental condition and level of political interest. Our results reveal that respondents who were most interested in politics responded with higher network sizes when in the “politics” condition than when in the “social” condition. Accordingly, respondents who were least interested in politics were more likely to respond with lower network sizes in the “politics” condition when compared with the “social” condition. Perhaps most tellingly, there was little overall variation in network size across political interest for those in the “social” condition, but significant variation across political interest for those in the “politics” condition.

Figure 2. Predicted discussion network size response varies by political interest for “politics” respondents but not for “social” respondents (95% confidence intervals).

Discussion

Our results suggest that “politics” and general “social” prompts are not fully interchangeable when measuring the size of respondent-reported interpersonal discussion networks in surveys. A “politics” generator results in more responses with smaller or larger networks than does an “important matters” social network generator. Furthermore, respondent levels of political interest are strongly correlated with discussion network size in the “politics” generator, but not in the “social” generator, indicating these distributions are generated by different survey response processes in our respondents.

These findings have implications for how we measure social and political discussion networks and for how we think of discussion network formation generally. Methodologically, researchers should consider whether they want to generate estimates of personal networks that are more political in nature or more general in nature. Substantively, our results suggest that political discussion networks are, in fact, different in nature from general social discussion networks, in contrast to earlier findings (Huckfeldt and Sprague, 1995; Klofstad, McClurg, and Rolfe 2009). Accordingly, some changes may be required in the way we think about social communication in politics in the modern polarized age.