Nobody needs another summary of mass media research right now; there are plenty of fine, current surveys of the field (Kinder 2003). Our chapter discusses specifically how experimental research provides insight into the relationship between the media and the political world. We are especially interested in important questions that experimentation is well suited to address. Experimentation has been vital to the development of scholarship in this area, but we should also recognize when it is best to step away and choose another method.

Causation and experimentation go together hand in glove, and questions of causation are paramount in both lay and scholarly thought about the media (Iyengar 1990). Questions about the social, economic, and organizational factors that determine mass media content are certainly fascinating and relevant in their own right. One can argue, however, that such questions eventually beget questions about the ultimate impact of that content on individuals and political processes and institutions.

As an example of the promise and limitation of experimentation in mass media research, consider the media's constant bugbear: public perceptions of ideological bias in the news. The usual form of this complaint is that media organizations subtly stump for liberal causes (Goldberg 2001). Although this complaint often amounts to little more than strategic bluster, it is conceivable that the increasing differentiation of the media marketplace will encourage news organizations to become more forthright in displaying overt liberal or conservative commentary (Dalton, Beck, and Huckfeldt 1998).

Trust and its complement, trustworthiness, are key concepts in political science. Trust is seen as critical for the existence of stable political institutions, as well as for the formation of social capital and civic engagement (Putnam 1993, 2000; Stolle 1998). It also serves as a social lubricant that reduces the cost of exchange, whether in reaching political compromise (Fenno 1978; Bianco 1994) or in daily market and nonmarket exchange (Lupia and McCubbins 1998; Sztompka 1999; Knight 2001). Researchers in this area face three key challenges. First, the concept of trust has been used in a multiplicity of ways, leaving its meaning unclear. Second, it is used to refer both to trust in government and trust among individuals (interpersonal trust). Third, it is sometimes seen as a cause and sometimes as an effect of effective political institutions, leaving the causal relationship between trust and institutions unclear.

The definition of trust is muddied by the fact that two distinct research methods have been used to explore it. Early research treats trust as a perception of norms in a society, assessed using survey questions about the trustworthiness or fairness of others. For forty years, the General Social Survey (GSS), World Values Survey (WVS), and American National Election Studies (ANES) have relied on the same questions to evaluate trust. In contrast, recent research has turned to behavioral assessments of trust using incentivized, economics-style laboratory experiments; this work is the focus of this chapter.

Researchers have been interested in the distribution of prejudice in the population, as well as its effects on policy preferences, vote choice, and economic and social outcomes for racial minorities, since the dawn of the social sciences. One of the earliest scholars to address these questions was W. E. B. DuBois (1899) in his classic work, The Philadelphia Negro. Referring to prejudice in chapter 16 of his book, DuBois wrote, “Everybody speaks of the matter, everyone knows that it exists, but in just what form it shows itself or how influential it is few agree” (322). Although these words were written more than 100 years ago, this observation still does a good job of summarizing our understanding of prejudice. There have been, to be sure, significant advances in this literature since the time of DuBois, but social scientists continue to disagree about the influence of racial prejudice in modern American politics. The aim of this chapter is to explore and adjudicate some of these differences, paying particular attention to the strengths, limitations, and contributions provided by the use of experimental methods.

Before examining the ways that scholars have studied prejudice, it is important that we define this term. What is prejudice? Social psychologists were among the first to answer this question.

Mediators are variables that transmit causal effects from treatments to outcomes. Those who undertake mediation analysis seek to answer “how” questions about causation: how does this treatment affect that outcome? Typically, we desire answers of the form “the treatment affects a causally intermediate variable, which in turn affects the outcome.” Identifying these causally intermediate variables is the challenge of mediation analysis.

Conjectures about political mediation effects are as old as the study of politics. But codification of procedures by which to test hypotheses about mediation is a relatively new development. The most common procedures are now ubiquitous in psychology (Quiñones-Vidal et al. 2004) and increasingly popular in the other social sciences, not least political science.

Unfortunately, the most common procedures are not very good. They call for indirect effects – the portions of treatment effects that are transmitted through mediators – to be estimated via multiequation regression frameworks. These procedures do not require experimental manipulation of mediators; instead, they encourage the study of mediation with data from unmanipulated mediators (MacKinnon et al. 2002, 86; Spencer, Zanna, and Fong 2005). The procedures are therefore prone to producing biased estimates of mediation effects. Warnings about this problem have been issued for decades by statisticians, psychologists, and political scientists.

In recent decades, formal models have become common means of drawing important inferences in political science. Best practice formal models feature explicitly stated premises, explicitly stated conclusions, and proofs that are used to support claims about focal relationships between these premises and conclusions. When best practices are followed, transparency, replicability, and logical coherence are the hallmarks of the formal theoretic enterprise.

Formal models have affected a broad range of scholarly debates in political science – from individual-level inquiries about why people vote as they do, to large-scale studies of civil wars and international negotiations. The method's contributions come when answers to normative and substantive questions require precise understandings about the conditions under which a given political outcome is, or is not, consistent with a set of clearly stated assumptions about relevant perceptions, motives, feelings, and contexts. Indeed, many formal modelers use mathematics to sort intricate and detailed statements about political cause and effect by the extent to which they can be reconciled logically with basic premises about the people and places involved.

Formal models in political science have been both influential and controversial. Although no one contends that explicitly stated assumptions or attention to logical consistency are anything other than good components of scientific practice, controversy often comes from the content of formal models themselves. Many formal models contain descriptions of political perceptions, opinions, and behaviors that are unrealistic. Some scholars, therefore, conclude that formal models, generally considered, are of little value to political science.

Interest in how social group attachments translate into political attitudes and behavior motivated much of the early attention to social identity. According to Tajfel (1981), the underlying foundation in the development of political and social beliefs was “the shared perceptions of social reality by large numbers of people and of the conditions leading to these shared perceptions” (15), as opposed to personality and environmental characteristics. Although Tajfel was not referencing the forces that make one's racial identity relevant to an individual, but instead focused on the development of social identity around the prejudice toward Jews, racial identity (at least in regard to political and social behavior research) has been among the most powerful explanations of behavior. Despite the multitude of identities a person may possess and the events that make such identities more or less salient, social identity theory has had special insight and significance into the connection between racial identity and political behavior. In the field of political behavior, no form of identity has received nearly the amount of attention and scrutiny as racial identity (e.g., group consciousness, racial consciousness, linked fate, race identification).

At the same time, however, research on racial identity and political behavior could benefit from adherence to the conceptual foundations of social identity theory, as well as from greater reliance on experimental research. Several problems exist in the conceptual development and measurement of racial identity research that beg out for a reexamination of racial identity.

An experiment entails randomly assigning participants to various conditions or manipulations. Given common consent requirements, this means experimenters need to recruit participants who, in essence, agree to be manipulated. The ensuing practical and ethical challenges of subject recruitment have led many researchers to rely on convenience samples of college students. For political scientists who put particular emphasis on generalizability, the use of student participants often constitutes a critical, and according to some reviewers, fatal problem for experimental studies.

In this chapter, we investigate the extent to which using students as experimental participants creates problems for causal inference. First, we discuss the impact of student subjects on a study's internal and external validity. In contrast to common claims, we argue that student subjects do not intrinsically pose a problem for a study's external validity. Second, we use simulations to identify situations when student subjects are likely to constrain experimental inferences. We show that such situations are relatively limited; any convenience sample poses a problem only when the size of an experimental treatment effect depends on a characteristic on which the convenience sample has virtually no variance. Third, we briefly survey empirical evidence that provides guidance on when researchers should be particularly attuned to taking steps to ensure appropriate generalizability from student subjects. We conclude with a discussion of the practical implications of our findings.

People are embedded in networks, neighborhoods, and relationships. Understanding the nature of our entanglements and how they shape who we are is fundamental to social sciences. Networks are likely to explain important parts of personal development and contemporary decision making. Researchers have found social networks to be important in activities as disparate as voting (Berelson, Lazarsfeld, and McPhee 1954), immigration patterns (Sanders, Nee, and Sernau 2002), finding a job (Nordenmark 1999), recycling (Tucker 1999), deworming (Miguel and Kremer 2004), cardiovascular disease and mortality (Kawachi et al. 1996), writing legislation (Caldeira and Patterson 1987), and even happiness (Fowler and Christakis 2008). A wide range of political outcomes could be studied using social networks; the only limitation is that the outcome be measurable. Ironically, the very ubiquity and importance of social networks make them difficult to study. Isolating causal effects is always difficult, but when like-minded individuals cluster together, share material incentives, are exposed to common external stimuli, and simultaneously influence each other, the job of reliably estimating the importance of social ties becomes nearly impossible. Rather than offering a comprehensive overview of the wide number of topics covered by social networks, this chapter focuses on the common empirical challenges faced by studies of social networks by considering the challenges faced by observational studies of social networks, discussing laboratory approaches to networks, and describing how network experiments are conducted in the field. The chapter concludes by summarizing the strengths and weaknesses of the different approaches and considers directions for future work.

It can be complicated to attempt to understand how election mechanisms and other variables surrounding an election determine outcomes. This is because the variables of interest are often intertwined; thus, it is difficult to disentangle them to determine the cause and effect that variables have on each other. Formal models of elections are used to disentangle variables so that cause and effect can be isolated. Laboratory election experiments are conducted so that the causes and effects of these isolated variables from these formal models can be empirically measured. These types of experiments are conducted within a single location, where it is possible for a researcher to control many of the variables of the election environment and thus observe the behavior of subjects under different electoral situations. The elections are often carried out in computer laboratories via computer terminals, and the communication between the researcher and subjects occurs primarily through a computer interface. In these experiments, subjects are assigned as either voters or candidates, and, in some cases, they are given both roles. Voters are rewarded based on a utility function that assigns a preference for a particular candidate or party. Candidates are typically rewarded based on whether they win the election, but sometimes their rewards depend on the actions they take after the election.

The title promises a chapter about methods, so a confession is in order. Here, as everywhere, my concerns are substantive, not methodological. Still, what one wants to learn and how one ought to go about learning it are intertwined. So, I propose to bring out the logic of the survey experiment by presenting a classification of survey experiment designs. Specifically, I distinguish three designs: manipulative, permissive, and facilitative. The distinctions among the designs turn on the hypotheses being tested, not the operations performed, and, above all, on the role of predispositions. The first design aims to get people to do what they are not predisposed to do; the second to allow them to do what they are predisposed to do, without encouraging them; and the third to provide them with a relevant reason to do what they already are predisposed to do. Against the background of this threefold classification, I want to comment briefly on some issues of causal inference and external validity and then conclude by offering my own view on the reasons for the explosive growth in survey experiments in the study of public opinion.

The modern survey experiment is the biggest change in survey research in a half century. There is some interest in how it came about, I am told. So I begin by telling how I got the idea of computer-assisted survey experiments.

Experimentalists want precise estimates of treatment effects and nearly always care about how treatment effects may differ across subgroups. After data collection, concern may turn to random imbalance between treatment groups on substantively important variables. Pursuit of these three goals – enhanced precision, understanding treatment effect heterogeneity, and imbalance adjustment – requires background information about experimental units. For example, one may group similar observations on the basis of such variables and then assign treatment within those blocks. Use of covariates after data have been collected raises extra concerns and requires special justification. For example, standard regression tables only approximate the statistical inference that experimentalists desire. The standard linear model may also mislead via extrapolation. After providing some general background about how covariates may, in principle, enable pursuit of precision and statistical adjustment, this chapter presents two alternative approaches to covariance adjustment: one using modern matching techniques and another using the linear model – both use randomization as the basis for statistical inference.

What Is a Manifest Effect?

A manifest effect is one we can distinguish from zero. Of course, we cannot talk formally about the effects of an experimental treatment as manifest without referring to probability: a scientist asks, “Could this result have occurred merely through chance?” or “If the true effect were zero, what is the chance that we'd observe an effect as large as this?”

Congratulations, intrepid reader! You have come a long way. After thirty-five chapters, many hundreds of pages of text, and cascades of footnotes, what more is there to say on the subject? Exactly. My remarks are thus brief.

The conference that led to this volume began with remarks by Jamie Druckman, who reminded us how right and proper it was that Northwestern University host our gathering. Northwestern was Donald Campbell's home for more than three decades, and no one was more important than Campbell in bringing the philosophy, logic, and practice of experimental methods to the social sciences. In graduate school, I was trained in experimental methods by Barry Collins, a student of Campbell's. For more than thirty years, first at Yale and later at Michigan, I have been teaching a graduate seminar on research design that draws heavily on Campbell's work. Those who survived this experience have taught similar courses elsewhere. Alterations and additions there have been along the way, of course, but the core ideas and the heart of the syllabus go back to Campbell. Throughout the conference, Campbell was never far from my mind, and he is back again now.

This is presumptuous of me to say, but I am going to go ahead and say it anyway: I believe that Campbell would have been thrilled with both the conference proceedings and the handbook that you are doing your best to keep balanced in your hands.

One of the challenges in conducting interdisciplinary work, or in attempting to communicate across disciplinary boundaries, relates to the implicit norms that infuse different fields. Much like trying to speak across cultures, it often becomes frustrating to translate or make explicit differing assumptions underlying appropriate inferential methods and strategies. Status differentials often exacerbate these divergences, privileging one set of disciplinary norms over another, so that decisions about ideal methods do not always rest entirely on the appropriateness of a particular technique for a given project.

Such differences clearly affect the implementation of experimental methodology across the fields of psychology, economics, and political science. One of the areas in which these biases inflict misunderstanding surrounds issues related to internal and external validity. In political science, concerns with external validity often border on the monomaniacal, leading to the neglect, if not the complete dismissal, of attention to the important issues involved in internal validity. In psychology, the reverse emphasis predominates. In behavioral economics, the focus depends more on the primary function of the experiment. Because both internal and external validity remain important in assessing the quality, accuracy, and utility of any given experimental design, it facilitates optimal experimental design to concentrate on attempting to maximize both, but the nature of the enterprise often requires explicit consideration of the trade-offs between them.