Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-669899f699-qzcqf Total loading time: 0 Render date: 2025-04-29T18:35:42.893Z Has data issue: false hasContentIssue false

33 - Design and Analysis of Experiments in Multilevel Populations

Published online by Cambridge University Press:  05 June 2012

By
Betsy Sinclair
Edited by
James N. Druckman ,
Donald P. Greene ,
James H. Kuklinski  and
Arthur Lupia
Betsy Sinclair
Affiliation:
University of Chicago
James N. Druckman
Affiliation:
Northwestern University, Illinois
Donald P. Greene
Affiliation:
Yale University, Connecticut
James H. Kuklinski
Affiliation:
University of Illinois, Urbana-Champaign
Arthur Lupia
Affiliation:
University of Michigan, Ann Arbor
Get access

Summary

Randomized experiments, the most rigorous methodology for testing causal explanations for phenomena in the social sciences, are experiencing a resurgence in political science. The classic experimental design randomly assigns the population of interest into two groups, treatment and control. Ex ante these two groups should have identical distributions in terms of their observed and unobserved characteristics. Treatment is administered based on assignment, and by the assumptions of the Rubin causal model, the average effect of the treatment is calculated as the difference between the average outcome in the group assigned to treatment and the average outcome in the group assigned to control.

Randomized experiments are often conducted within a multilevel setting. These settings can be defined at the level at which the randomization occurs, as well as at the level at which the treatment is both directly and indirectly administered. These indirect effects most often occur as a result of social transmission of the treatment, which is particularly likely when the treatment consists of information. This chapter explores the implications of these multilevel settings to highlight the importance of careful experimental design with respect to random assignment of the population and the implementation of the treatment. There are potential problems with analysis in this context, and this chapter suggests strategies to accommodate multilevel settings. These problems are most likely to occur in field settings where control is lacking, although they can sometimes occur in other settings as well.

Type
Chapter
Information
Cambridge Handbook of Experimental Political Science , pp. 481 - 493
Publisher: Cambridge University Press
Print publication year: 2011

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Adato, Michelle, Coady, David, and Ruel, Marie. 2000. An Operations Evaluation of PROGRESA from the Perspective of Beneficiaries, Promotoras, School Directors and Health Staff. Washington, DC: International Food Policy Research Institute.Google Scholar
Arceneaux, Kevin. 2005. “Using Cluster Randomized Field Experiments to Study Voting Behavior.” Annals of the American Academy of Political and Social Science 601: 169–79.CrossRefGoogle Scholar
Arceneaux, Kevin, and Nickerson, David W.. 2009. “Modeling Certainty with Clustered Data: A Comparison of Methods.” Political Analysis 17: 177–90.CrossRefGoogle Scholar
Berelson, Bernard, Lazarsfeld, Paul F., and McPhee, William N.. 1954. Voting. Chicago: The University of Chicago Press.Google Scholar
Besley, Timothy, and Case, Anne. 1993. “Modeling Technology Adoption in Developing Countries.” American Economic Review 83: 396–402.Google Scholar
Brock, William A., and Durlauf, Steven N.. 1999. “A Formal Model of Theory Choice in Science.” Economic Theory 14: 113–30.CrossRefGoogle Scholar
Cacioppo, John T., Fowler, James H., and Christakis, Nicholas A.. 2009. “Alone in the Crowd: The Structure and Spread of Loneliness in a Large Social Network.” Journal of Personality and Social Psychology 97: 977–91.CrossRefGoogle Scholar
Coleman, James S., Katz, Elihu, and Menzel, Herbert. 1966. Medical Innovation. Indianapolis: Bobbs-Merrill Press.Google Scholar
Conley, Timothy G., and Udry, Christopher R.. 2010. “Learning about a New Technology: Pineapple in Ghana.” American Economic Review 100: 35–69.CrossRefGoogle Scholar
DeMarzo, Peter M., Vayanos, Dimitri, and Zwiebel, Jeffrey. 2003. “Persuasion Bias, Social Influence and Unidimensional Opinions.” Quarterly Journal of Economics 118: 909–68.CrossRefGoogle Scholar
Duflo, Esther, Kremer, Michael, and Robinson, Jonathan. 2006. “Understanding Technology Adoption: Fertilizer in Western Kenya.” Unpublished manuscript, Massachusetts Institute of Technology.
Fowler, James H., and Christakis, Nicholas A.. 2008. “Dynamic Spread of Happiness in a Large Social Network: Longitudinal Analysis over 20 Years in the Framingham Heart Study.” British Medical Journal 337: a2338.CrossRefGoogle Scholar
Gerber, Alan S., and Green, Donald P.. 2008. Get Out the Vote! 2nd ed. Washington, DC: Brookings Institution Press.Google Scholar
Gerber, Alan S., Green, Donald P., and Larimer, Christopher W.. 2008. “Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment.” American Political Science Review 94: 653–63.CrossRefGoogle Scholar
Green, Donald P., Gerber, Alan S., and Nickerson, David W.. 2003. “Getting Out the Vote in Local Elections: Results from Six Door-to-Door Canvassing Experiments.” Journal of Politics 65: 1083–96.CrossRefGoogle Scholar
Green, Donald P., and Vavreck, Lynn. 2008. “Analysis of Cluster-Randomized Experiments: A Comparison of Alternative Estimation Approaches.” Political Analysis 16: 138–52.CrossRefGoogle Scholar
Hansen, Ben B., and Bowers, Jake. 2008. “Attributing Effects to a Cluster Randomized Get-Out-the-Vote Campaign.” Journal of the American Statistical Association 104 (487): 873–85.CrossRefGoogle Scholar
Huckfeldt, Robert, Johnson, Paul E., and Sprague, John. 2004. Political Disagreement. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Imai, Kosuke, King, Gary, and Nall, Clayton. 2009. “The Essential Role of Pair Matching in Cluster-Randomized Experiments, with Application to the Mexican Universal Health Insurance Evaluation.” Statistical Science 24: 29–53.CrossRefGoogle Scholar
King, Gary, Gakidou, Emmanuela, Ravishankar, Nirmala, Moore, Ryan T., Lakin, Jason, Vargas, Manett, Tellez-Rojo, Martha Maria, Eugenio, JuanAvila, Hernandez, Avila, Mauricio Hernandez, and Llamas, Hector Hernandez. 2007. “A ‘Politically Robust’ Experimental Design for Public Policy Evaluation, with Application to the Mexican Universal Health Insurance Program.” Journal of Policy Analysis and Management 26: 479–509.CrossRefGoogle Scholar
Krueger, Alan B. 1999. “Experimental Estimates of Education Production Functions.” Quarterly Journal of Economics 114: 497–532.CrossRefGoogle Scholar
Lazarsfeld, Paul, Berelson, Bernard, and Gaudet, Hazel. 1948. The People's Choice. New York: Columbia University Press.Google Scholar
Manski, Charles. 1993. “Identification of Exogenous Social Effects: The Reflection Problem.” Review of Economic Studies 60: 531–42.CrossRefGoogle Scholar
McClurg, Scott. 2006. “Political Disagreement in Context: The Conditional Effect of Neighborhood Context, Discussion, and Disagreement on Electoral Participation.” Political Behavior 28: 349–66.CrossRefGoogle Scholar
Miguel, Edward, and Kremer, Michael. 2004. “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities.” Econometrica 72: 159–217.CrossRefGoogle Scholar
Munshi, Kaivan. 2004. “Social Learning in a Heterogeneous Population: Technology Diffusion in the Indian Green Revolution.” Journal of Development Economics 73: 185–213.CrossRefGoogle Scholar
Nair, Harikesh, Manchanda, Puneet, and Bhatia, Tulikaa. 2008. “Asymmetric Social Interactions in Physician Prescription Behavior: The Role of Opinion Leaders.” SSRN Working Paper 937021.
Nickerson, David. 2008. “Is Voting Contagious? Evidence from Two Field Experiments.” American Political Science Review 102: 49–57.CrossRefGoogle Scholar
Panagopoulos, Costas, and Green, Donald P.. 2006. “The Impact of Radio Advertisements on Voter Turnout and Electoral Competition.” Paper presented at the annual meeting of the Midwest Political Science Association, Chicago.
Rubin, Donald B. 1980. “Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment.” Journal of the American Statistical Association 57 (371): 591–93.Google Scholar
Rubin, Donald B. 1986. “Which Ifs Have Causal Answers? Discussion of Holland's ‘Statistics and Causal Inferences.’” Journal of the American Statistical Association 81: 961–62.Google Scholar
Rubin, Donald B. 1990. “Formal Modes of Statistical Inference for Causal Effects.” Journal of Statistical Planning and Inference 25: 279–92.CrossRefGoogle Scholar
Sinclair, Betsy, McConnell, Margaret A., and Green, Donald P.. 2010. “Detecting Spillover in Social Networks: Design and Analysis of Multilevel Experiments.” Unpublished manuscript, The University of Chicago.
Sinclair, Betsy, McConnell, Margaret A., and Michelson, Melissa R.. 2010. “Strangers vs Neighbors: The Efficacy of Grassroots Voter Mobilization.” Unpublished manuscript, The University of Chicago.
Sinclair, Betsy, and Rogers, Brian. 2010. “Political Networks: The Relationship between Candidate Platform Positions and Constituency Communication Structures.” Unpublished manuscript, The University of Chicago.
Stoker, Laura, and Bowers, Jake. 2002. “Designing Multilevel Studies: Sampling Voters and Electoral Contexts.” Electoral Studies 21: 235–67.CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×