To reduce strategic misreporting on sensitive topics, survey researchers increasingly use list experiments rather than direct questions. However, the complexity of list experiments may increase nonstrategic misreporting. We provide the first empirical assessment of this trade-off between strategic and nonstrategic misreporting. We field list experiments on election turnout in two different countries, collecting measures of respondents’ true turnout. We detail and apply a partition validation method which uses true scores to distinguish true and false positives and negatives for list experiments, thus allowing detection of nonstrategic reporting errors. For both list experiments, partition validation reveals nonstrategic misreporting that is: undetected by standard diagnostics or validation; greater than assumed in extant simulation studies; and severe enough that direct turnout questions subject to strategic misreporting exhibit lower overall reporting error. We discuss how our results can inform the choice between list experiment and direct question for other topics and survey contexts.

How can we elicit honest responses in surveys? Conjoint analysis has become a popular tool to address social desirability bias (SDB), or systematic survey misreporting on sensitive topics. However, there has been no direct evidence showing its suitability for this purpose. We propose a novel experimental design to identify conjoint analysis’s ability to mitigate SDB. Specifically, we compare a standard, fully randomized conjoint design against a partially randomized design where only the sensitive attribute is varied between the two profiles in each task. We also include a control condition to remove confounding due to the increased attention to the varying attribute under the partially randomized design. We implement this empirical strategy in two studies on attitudes about environmental conservation and preferences about congressional candidates. In both studies, our estimates indicate that the fully randomized conjoint design could reduce SDB for the average marginal component effect (AMCE) of the sensitive attribute by about two-thirds of the AMCE itself. Although encouraging, we caution that our results are exploratory and exhibit some sensitivity to alternative model specifications, suggesting the need for additional confirmatory evidence based on the proposed design.

Conjoint experiments are quickly gaining popularity as a vehicle for studying multidimensional political preferences. A common way to explore heterogeneity of preferences estimated with conjoint experiments is by estimating average marginal component effects across subgroups. However, this method does not give the researcher the full access to the variation of preferences in the studied populations, as that would require estimating effects on the individual level. Currently, there is no accepted technique to obtain estimates of individual-level preferences from conjoint experiments. The present paper addresses this gap by proposing a procedure to estimate individual preferences as respondent-specific marginal component effects. The proposed strategy does not require any additional assumptions compared to the standard conjoint analysis, although some changes to the task design are recommended. Methods to account for uncertainty in resulting estimates are also discussed. Using the proposed procedure, I partially replicate a conjoint experiment on immigrant admission with recommended design adjustments. Then, I demonstrate how individual marginal component effects can be used to explore distributions of preferences, intercorrelations between different preference dimensions, and relationships of preferences to other variables of interest.

We introduce a fine-grained measure of the extent to which electoral districts combine and split local communities of co-partisans in unnatural ways. Our indicator—which we term Partisan Dislocation—is a measure of the difference between the partisan composition of a voter’s geographic nearest neighbors and that of her assigned district. We show that our measure is a good local and global indicator of district manipulation, easily identifying instances in which districts carve up clusters of co-partisans (cracking) or combine them in unnatural ways (packing). We demonstrate that our measure is related to but distinct from other approaches to the measurement of gerrymandering, and has some clear advantages, above all as a complement to simulation-based approaches, and as a way to identify the specific neighborhoods most affected by gerrymandering. It can also be used prospectively by district-drawers who wish to create maps that reflect voter geography, but according to our analysis, that goal will sometimes be in conflict with the goal of partisan fairness.

Internet-based surveys have expanded public opinion data collection at the expense of monitoring respondent attentiveness, potentially compromising data quality. Researchers now have to evaluate attentiveness ex-post. We propose a new proxy for attentiveness—response-time attentiveness clustering (RTAC)—that uses dimension reduction and an unsupervised clustering algorithm to leverage variation in response time between respondents and across questions. We advance the literature theoretically arguing that the existing dichotomous classification of respondents as fast or attentive is insufficient and neglects slow and inattentive respondents. We validate our theoretical classification and empirical strategy against commonly used proxies for survey attentiveness. In contrast to other methods for capturing attentiveness, RTAC allows researchers to collect attentiveness data unobtrusively without sacrificing space on the survey instrument.

We provide an introduction of the functioning, implementation, and challenges of convolutional neural networks (CNNs) to classify visual information in social sciences. This tool can help scholars to make more efficient the tedious task of classifying images and extracting information from them. We illustrate the implementation and impact of this methodology by coding handwritten information from vote tallies. Our paper not only demonstrates the contributions of CNNs to both scholars and policy practitioners, but also presents the practical challenges and limitations of the method, providing advice on how to deal with these issues.

Topic models, as developed in computer science, are effective tools for exploring and summarizing large document collections. When applied in social science research, however, they are commonly used for measurement, a task that requires careful validation to ensure that the model outputs actually capture the desired concept of interest. In this paper, we review current practices for topic validation in the field and show that extensive model validation is increasingly rare, or at least not systematically reported in papers and appendices. To supplement current practices, we refine an existing crowd-sourcing method by Chang and coauthors for validating topic quality and go on to create new procedures for validating conceptual labels provided by the researcher. We illustrate our method with an analysis of Facebook posts by U.S. Senators and provide software and guidance for researchers wishing to validate their own topic models. While tailored, case-specific validation exercises will always be best, we aim to improve standard practices by providing a general-purpose tool to validate topics as measures.

In pairwise randomized experiments, what if the outcomes of some units are missing? One solution is to delete missing units (the unitwise deletion estimator, UDE). If attrition is nonignorable, however, the UDE is biased. Instead, scholars might employ the pairwise deletion estimator (PDE), which deletes the pairmates of missing units as well. This study proves that the PDE can be biased but more efficient than the UDE and, surprisingly, the conventional variance estimator of the PDE is unbiased in a super-population. I also propose a new variance estimator for the UDE and argue that it is easier to interpret the PDE as a causal effect than the UDE. To conclude, I recommend the PDE rather than the UDE.

Political scientists routinely face the challenge of assessing the quality (validity and reliability) of measures in order to use them in substantive research. While stand-alone assessment tools exist, researchers rarely combine them comprehensively. Further, while a large literature informs data producers, data consumers lack guidance on how to assess existing measures for use in substantive research. We delineate a three-component practical approach to data quality assessment that integrates complementary multimethod tools to assess: (1) content validity; (2) the validity and reliability of the data generation process; and (3) convergent validity. We apply our quality assessment approach to the corruption measures from the Varieties of Democracy (V-Dem) project, both illustrating our rubric and unearthing several quality advantages and disadvantages of the V-Dem measures, compared to other existing measures of corruption.

Party systems, that is, the number and the size of all the parties within a country, can vary greatly across countries. I conduct a principal component analysis on a party seat share dataset of 17 advanced democracies from 1970 to 2013 to reduce the dimensionality of the data. I find that the most important dimensions that differentiate party systems are: “the size of the biggest two parties” and the level of “competition between the two biggest parties.” I use the results to compare the changes in electoral and legislative party systems. I also juxtapose the results to previous party system typologies and party system size measures. I find that typologies sort countries into categories based on variation along both dimensions. On the other hand, most of the current political science literature use measures (e.g., the effective number of parties) that are correlated with the first dimension. I suggest that instead of these, indices that measure the opposition structure and competition could be used to explore problems pertaining to the competitiveness of the party systems.

This paper proposes a Bayesian alternative to the synthetic control method for comparative case studies with a single or multiple treated units. We adopt a Bayesian posterior predictive approach to Rubin’s causal model, which allows researchers to make inferences about both individual and average treatment effects on treated observations based on the empirical posterior distributions of their counterfactuals. The prediction model we develop is a dynamic multilevel model with a latent factor term to correct biases induced by unit-specific time trends. It also considers heterogeneous and dynamic relationships between covariates and the outcome, thus improving precision of the causal estimates. To reduce model dependency, we adopt a Bayesian shrinkage method for model searching and factor selection. Monte Carlo exercises demonstrate that our method produces more precise causal estimates than existing approaches and achieves correct frequentist coverage rates even when sample sizes are small and rich heterogeneities are present in data. We illustrate the method with two empirical examples from political economy.

Name-based treatments have been used in observational studies and experiments to study the differential effect of identity—commonly race or ethnic minority status. These treatments are typically assumed to signal only a single characteristic. If names unintentionally signal other characteristics, then the treatment can violate information equivalence, and estimated treatment effects cannot be attributed to the desired characteristic alone. Using results from a name perception study paired with an original correspondence audit experiment of U.S. state legislators, we show that names manipulate perceptions of minority status, socioeconomic status (SES), and migrant status. Our audit study shows that low SES status is related to reply rates both across and within each racial category. These results provide evidence that discrimination cannot be easily attributed singularly to the intended treatment of minority status but rather reflect a more multifaceted form of discrimination. More generally, our results provide an example of how name-based treatments manipulate more than the intended characteristic, which means that estimated treatment effects cannot be interpreted as being manipulated solely by the desired characteristic. Future studies with name-based or other informational treatments should account for the potential violation of information equivalence in their research design and interpretation of results.

A single dataset is rarely sufficient to address a question of substantive interest. Instead, most applied data analysis combines data from multiple sources. Very rarely do two datasets contain the same identifiers with which to merge datasets; fields like name, address, and phone number may be entered incorrectly, missing, or in dissimilar formats. Combining multiple datasets absent a unique identifier that unambiguously connects entries is called the record linkage problem. While recent work has made great progress in the case where there are many possible fields on which to match, the much more uncertain case of only one identifying field remains unsolved: this fuzzy string matching problem, both its own problem and a component of standard record linkage problems, is our focus. We design and validate an algorithmic solution called Adaptive Fuzzy String Matching rooted in adaptive learning, and show that our tool identifies more matches, with higher precision, than existing solutions. Finally, we illustrate its validity and practical value through applications to matching organizations, places, and individuals.

Punctuated Equilibrium Theory posits that policy-making is generally characterized by long periods of stability that are interrupted by short periods of fundamental policy change. The literature converged on the measure of kurtosis and L-kurtosis to assess these change patterns. In this letter, we critically discuss these measures and propose the Gini coefficient as a (1) comparable, but (2) more intuitive, and (3) more precise measure of “punctuated” change patterns.

Imputing missing values is an important preprocessing step in data analysis, but the literature offers little guidance on how to choose between imputation models. This letter suggests adopting the imputation model that generates a density of imputed values most similar to those of the observed values for an incomplete variable after balancing all other covariates. We recommend stable balancing weights as a practical approach to balance covariates whose distribution is expected to differ if the values are not missing completely at random. After balancing, discrepancy statistics can be used to compare the density of imputed and observed values. We illustrate the application of the suggested approach using simulated and real-world survey data from the American National Election Study, comparing popular imputation approaches including random forests, hot-deck, predictive mean matching, and multivariate normal imputation. An R package implementing the suggested approach accompanies this letter.

Participants that complete online surveys and experiments may be inattentive, which can hinder researchers’ abilityto draw substantive or causal inferences. As such, many practitioners include multiple factualor instructional closed-ended manipulation checks to identify low-attention respondents. However, closed-ended manipulation checks are either correct or incorrect, which allows participants to more easily guess and it reduces the potential variation in attention between respondents. In response to these shortcomings, I develop an automatic and standardized methodology to measure attention that relies on the text that respondents provide in an open-ended manipulation check. There are multiple benefits to this approach. First, it provides a continuous measure of attention, which allows for greater variation between respondents. Second, it reduces the reliance on subjective, paid humans to analyze open-ended responses. Last, I outline how to diagnose the impact of inattentive workers on the overall results, including how to assess the average treatment effect of those respondents that likely received the treatment. I provide easy-to-use software in R to implement these suggestions for open-ended manipulation checks.

Racial identification is a critical factor in understanding a multitude of important outcomes in many fields. However, inferring an individual’s race from ecological data is prone to bias and error. This process was only recently improved via Bayesian improved surname geocoding (BISG). With surname and geographic-based demographic data, it is possible to more accurately estimate individual racial identification than ever before. However, the level of geography used in this process varies widely. Whereas some existing work makes use of geocoding to place individuals in precise census blocks, a substantial portion either skips geocoding altogether or relies on estimation using surname or county-level analyses. Presently, the trade-offs of such variation are unknown. In this letter, we quantify those trade-offs through a validation of BISG on Georgia’s voter file using both geocoded and nongeocoded processes and introduce a new level of geography—ZIP codes—to this method. We find that when estimating the racial identification of White and Black voters, nongeocoded ZIP code-based estimates are acceptable alternatives. However, census blocks provide the most accurate estimations when imputing racial identification for Asian and Hispanic voters. Our results document the most efficient means to sequentially conduct BISG analysis to maximize racial identification estimation while simultaneously minimizing data missingness and bias.

Bayesian analysis has emerged as a rapidly expanding frontier in qualitative methods. Recent work in this journal has voiced various doubts regarding how to implement Bayesian process tracing and the costs versus benefits of this approach. In this response, we articulate a very different understanding of the state of the method and a much more positive view of what Bayesian reasoning can do to strengthen qualitative social science. Drawing on forthcoming research as well as our earlier work, we focus on clarifying issues involving mutual exclusivity of hypotheses, evidentiary import, adjudicating among more than two hypotheses, and the logic of iterative research, with the goal of elucidating how Bayesian analysis operates and pushing the field forward.