Evaluating Policy Mood Measures in the American States

Abstract

The scholarly exchange over approaches to measuring public preferences in the American states dates back several years. This introduction to the debate attempts to provide broad perspective on how scholars have conceptualized and measured policy mood among state mass publics in the past and the implications of those choices for theoretical and empirical questions in state politics. We frame the discussion around two questions: (1) what is the concept to be measured? and (2) how does the concept fit with the research question? We conclude with some insight on the future of the debate and its implications for state politics research.

State politics scholars have long been interested in the measurement and analysis of indicators of the mass public’s preferences in the states. Such measures hold scholarly interest in their own right but are also useful to researchers studying policymaking, responsiveness, and other outcomes related to democratic accountability at the state level. A central concern for these studies is the strategic choices for measuring mass preferences. Scholars have devoted substantial attention to this question for decades and the available options range widely: simulation-based approaches (Weber et al. 1972), disaggregation of survey data (e.g., Carsey and Harden 2010; Erikson, Wright, and McIver 1993), proxy measures (Berry et al. 1998), and modeling and/or poststratification of national surveys (e.g., Caughey and Warshaw 2018; Enns and Koch 2013; Pacheco 2011). While such an array of choices is generally useful to the state politics community, the various measures and the discussions over them that have taken place in the last several years may ultimately create confusion, especially for new scholars in the field.

The debate over measuring one such indicator of public preferences – state policy mood – continues in this issue, with contributions from Berry, Fording, Hanson, and Crofoot (BFHC) and Lagodny, Jones, Koch, and Enns (LJKE). Their articles provide detailed comparisons of the measurement strategies for state mood proposed by Berry et al. (1998) and Enns and Koch (2013). Our aim in this introduction to the debate is to step back, assess the context, and evaluate some important issues for researchers choosing between these measures or any other measures of state preferences. In particular, we focus on two related questions:

1. 1. What is the concept to be measured?

2. 2. How does the concept fit with the research question?

We hope that our answers to these questions can help frame the discussion moving forward and provide useful insight to scholars studying the role of the mass public in American state politics.

What is the concept to be measured?

Our first task is to reflect on the concept these indices purport to measure and to suggest some implications of the very different operationalizations these researchers use. The concept of interest is state policy mood, which is the predisposition of state electorates for more or less government activity on an array of economic policy areas. Both the BRFH and the LJKE measures seek to replicate Stimson’s (1999) measure of national mood at the state level. BFHC’s effort in this issue offers correlations of their measure with a variety of combinations of items drawn from the General Social Survey (GSS) to make the case for its validity. In LJKE’s update of the Enns and Koch (2013) work on state policy mood, the primary data used to construct the measure are the same GSS items Stimson uses to measure national policy mood. These items are restricted to a limited set of economic and related policies, including health, education, and welfare. They explicitly exclude cultural issues such as race, abortion, and guns, which means that these mood measures do not capture that underlying dimension. This measurement strategy explicitly excludes state opinions toward cultural issues. If that is the case, then state policy mood as a primarily economic indicator may not be appropriate for analyses involving morality policies.

Another potential problem with the validity of both the BRFH and LJKE measures is that they have, at best, an indirect connection to factors that scholars often want to explain, such as patterns of state policy or mass attitudes toward state policy. Stimson’s measure clearly assumes that survey respondents are thinking about the policy efforts of the federal government when they answer GSS questions. So if either index is able to replicate Stimson’s measure at the state level, we still have the problem of using a state-level measure of national policy mood to explain changes in state policies and election outcomes.

One way to theoretically address this problem is to assume that citizens do not differentiate between levels of government in their policy preferences. If so, a national policy mood would be roughly the same as a state mood measure and the problem is mitigated. However, just a bit of reflection suggests that this is not a defensible way around the problem. It is only too easy to imagine sets of citizens who would have quite different preferences for policy changes at the state versus the national level due to variations in the status quo at each level. Take, for example, the group of respondents who might have moderate (“spending enough”) responses to national welfare or education expenditures. Those living in Alabama or Mississippi might see the stinginess of their state governments’ efforts as “too little,” whereas their counterparts living in higher-spending states like New York or California could feel that “too much” is being spent by their state governments. We can imagine parallel scenarios in other policy areas.

If either of the measures seems not to “work” in an analysis, we would have to question whether state governments do indeed ignore the preferences of their electorates, or whether we actually have measures of something else (national policy mood) that is not systematically related to the key concept of state policy mood. In fact, it would be surprising if public reactions to national economic policies – a constant at any time point – could capture the variety of citizen reactions to the increasingly divergent patterns of a subset of state policies. If the patterns of “leapfrog representation” identified by Bafumi and Herron (2010) hold for state governments, then we might expect wildly divergent reactions of state electorates as the blue states race toward more liberal policy regimes and the red states adopt increasingly conservative policies. Our simple, but for state politics researchers quite important, point is that we have lots of reasons to suspect that measures of national policy mood, even measured at the state level, do not consistently measure citizens’ evaluations of the policies in their states.

How does the concept fit with the research question?

Decisions on how to measure state public preferences depend on how central the concept is to answer the research question. For some lines of inquiry, state preferences may be a control variable included to capture a state’s tendency toward a particular policy or political outcome. Take the vast literature on policy diffusion as an example. Here, scholars are primarily interested in understanding why a state adopts a policy at a particular time and how policies spread across the states. While public opinion may matter for these processes (Pacheco 2012), it is often secondary to the role of elites (Mintrom 1997), pressures within and across states (e.g., Berry and Berry 1990; Desmarais, Harden, and Boehmke 2015), and influence from local governments and the federal government (e.g., Karch and Rose 2019; Shipan and Volden 2006).

For research questions where public opinion is a secondary concern, researchers might do well to follow the lead of early state opinion scholars (e.g., Brace et al. 2002; Erikson, Wright, and McIver 1993) and include direct aggregate measures of partisanship, ideology, or vote choice. These measures may be drawn from single national surveys with large sample sizes, such as the Cooperative Election Study (CES) or the National Annenberg Election Surveys (NAES), pooled cross-sectional surveys, such as the American National Election Survey (ANES), or state exit polls. All of these options are simplified ways to categorize the broad political leanings of state residents. Regional measures of geography can sufficiently capture meaningful variations in state culture. Of course, the indicators discussed here could also be used, but researchers should be careful to accurately describe what and why they include particular control variables over others.

Scholars must be more thoughtful when state opinion is the dependent variable. A scholar might ask whether state opinion is changing – is state opinion stable or dynamic? If state opinion changes over time, does it change gradually or quickly? Related, are states generally trending in similar or divergent ways? Are there parallel publics across the states (e.g., Page and Shapiro 1992)? These are important questions to ask because the combination of the rate of change (stable, gradual, or abrupt) and the patterns of change (homogeneous or heterogeneous) provide insight into the determinants of aggregate state opinion. Thus, these types of descriptive analyses are useful for understanding theories in political science. They are also informative to journalists, electoral strategists, and policymakers, all of whom are likely especially interested in the practical implications of what determines general state opinions and, specifically, state policy mood.

A guiding principle for these research questions is whether we are interested in describing broad political patterns or narrowly defined issue attitudes. For the former, scholars essentially have three choices: partisanship, ideology, or policy mood. State macropartisanship and macroideology move slowly over time where the relative rankings of the states are fairly consistent (Enns and Koch 2013; Pacheco 2012; Wright and Birkhead 2014). State policy mood, however, is more dynamic, regardless of whether one uses the measure from LJKE or BFHC’s indicator. The implication is that shifts in the partisan or ideological leanings of states are likely caused by gradual population changes such as generational replacement, migration, immigration, and differential birth and death rates (Brace et al. 2004; Carmines and Stimson 1989). Abrupt movements in opinion, such as is the case with state policy mood, are likely caused by reactions to current events, the economy, or other transient outcomes (Erikson, MacKuen, and Stimson 2002; Stimson 2004).

The rate and patterns of opinion change across states on specific issues vary depending on the issue. In the most comprehensive analysis of the dynamic properties of specific state opinions, Pacheco (2014) finds that state attitudes on consumer sentiment and presidential approval are highly dynamic and exhibit homogenous trends. State attitudes toward education spending, welfare spending, and the death penalty exhibit more change over time with heterogeneous trends (Brace et al. 2002; Pacheco 2014). State attitudes toward abortion are essentially stable. More recent studies find that state attitudes toward the Affordable Care Act (ACA) exhibit various patterns, depending on whether residents responded to a general question about ACA favorability, the impact of the ACA, or the future impact of the ACA (Pacheco and Maltby 2019; see also Brodie, Deane, and Cho 2011). Finally, there is evidence that state racial resentment changes slowly over time with heterogenous trends (Smith, Kreitzer, and Suo 2020). Continuing this type of research to understand the descriptive characteristics of state public opinion is a fruitful avenue for the future.

Thoughtfully considering how to measure state preferences is equally important when it is the main independent variable of interest, which is the case for studies asking how closely state policies align with the mass public’s policy attitudes. For these studies, a scholar must be purposeful in making sure that the measure of state preferences reasonably matches the dependent variable. If the dependent variable is capturing broad policy tendencies of the states, for instance, state policy liberalism (e.g., Caughey and Warshaw 2018; Erikson, Wright, and McIver 1993), then it is reasonable to measure state opinion along a similar ideological dimension (see Caughey and Warshaw 2016). When the dependent variable is a specific policy outcome; for instance, on gay rights policies (Lax and Phillips 2009), the ACA (Pacheco and Maltby 2019), or antismoking legislation (Pacheco 2012), it makes more sense for scholars to measure specific state attitudes. Linking concepts to data is a crucial aspect of any research design, especially in cases such as this one where there are multiple options from which scholars can choose.

Conclusions

The study of American public preferences carries a great deal of theoretical and empirical nuance to which scholars must be attentive. We commend the authors in this years-long debate on state policy mood for their careful consideration of these details and encourage researchers employing their data to do the same. The measurement of public preferences in the states is relevant to many research agendas, each with their own unique challenges. Thus, readers should not necessarily look to this debate for guidance on which measure is the “best,” because that distinction is heavily dependent on the context. Rather, we encourage scholars to consider the concept to be measured based on their research question, select an indicator, then assess the sensitivity of their results to that choice. Indeed, none of these measures is perfect and we will never learn “true” state preferences of any kind. It is important for researchers to acknowledge that the available options are proxy measures that contain an error.

Of course, we certainly do not intend to persuade researchers to shy away from using the indicators discussed in this issue. They are the best measures we have (to date) and the state policy–public opinion relationship is too central to the state politics and policy subfield and political science more generally to ignore. But it is equally important to be aware of the shortcomings of even our best measures and to temper our conclusions, especially if they do not reinforce one’s democratic images of state government responsiveness. As in a great deal of state politics research, the continued expansion and increasing availability of new data will no doubt provide additional advances in this area in years to come.