Hostname: page-component-586b7cd67f-t7fkt Total loading time: 0 Render date: 2024-11-27T20:23:38.976Z Has data issue: false hasContentIssue false
  • English
  • Français

Mechanical Turk and the “Don’t Know” Option

Published online by Cambridge University Press:  12 April 2021

Adam R. Brown  and
Jeremy C. Pope Open the ORCID record for Jeremy C. Pope [Opens in a new window]
Adam R. Brown
Affiliation:
Brigham Young University
Jeremy C. Pope
Affiliation:
Brigham Young University
Get access Rights & Permissions [Opens in a new window]

Abstract

Luskin and Bullock’s (2011) randomized experiment on live-interview respondents found no evidence that American National Election Studies and Time-Sharing Experiments for the Social Sciences respondents hide knowledge behind the “don’t know” (DK) option. We successfully replicated their finding using two online platforms, the Cooperative Congressional Election Study and Google Surveys. However, we obtained different results on Amazon’s Mechanical Turk (MTurk). We attribute this difference to MTurkers’ experience with attention checks and other quality-control mechanisms, which condition them to avoid errors. This conditioning leads MTurkers to hide knowledge behind DK in ways not observed on other platforms. Researchers conducting political knowledge experiments or piloting surveys on MTurk should be aware of these differences.

Type
Article
Information
PS: Political Science & Politics , Volume 54 , Issue 3 , July 2021 , pp. 416 - 420
Copyright
© The Author(s), 2021. Published by Cambridge University Press on behalf of the American Political Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Achen, Christopher H., and Bartels, Larry M.. 2016. Democracy for Realists: Why Elections Do Not Produce Responsive Government. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Ansolabehere, Stephen, and Schaffner, Brian. 2014. “Does Survey Mode Still Matter? Findings from a 2010 Multimode Comparison.” Political Analysis 22:285–303.CrossRefGoogle Scholar
Berelson, Bernard R., Lazarsfeld, Paul F., and McPhee, William N.. 1954. Voting: A Study of Opinion Formation in a Presidential Campaign. Chicago: University of Chicago Press.Google Scholar
Berinsky, Adam J., Huber, Gregory A., and Lenz, Gabriel S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon’s Mechanical Turk.” Political Analysis 20:351–68.CrossRefGoogle Scholar
Clifford, Scott, and Jerit, Jennifer. 2016. “Cheating on Political Knowledge Questions in Online Surveys: An Assessment of the Problem and Solutions.” Public Opinion Quarterly 80:858–87.CrossRefGoogle Scholar
Converse, Philip. 1964. “The Nature of Belief Systems in Mass Publics.” In Ideology and Discontent, ed. Apter, David E., 206–61. New York: The Free Press of Glencoe.Google Scholar
Carpini, Delli, Michael, X., and Keeter, Scott. 1996. What Americans Know about Politics and Why It Matters. New Haven, CT: Yale University Press.Google Scholar
Franco, Annie, Malhotra, Neil, Simonovitz, Gabor, and Zigerell, L. J.. 2017. “Developing Standards for Post-Hoc Weighting in Population-Based Survey Experiments.” Journal of Experimental Political Science 4:161–72.CrossRefGoogle Scholar
Hauser, David J., and Schwarz, Norbert. 2016. “Attentive Turkers: MTurk Participants Perform Better on Online Attention Checks Than Do Subject Pool Participants.” Behavior Research Methods 48 (1): 400–407.CrossRefGoogle ScholarPubMed
Huff, Connor, and Tingley, Dustin. 2015. “‘Who Are These People?’ Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents.” Research and Politics 2:1–15.CrossRefGoogle Scholar
Krosnick, Jon A., Holbrook, Allyson L., Berent, Matthew K., Carson, Richard T., Michael Hanemann, W., Kopp, Raymond J., Mitchell, Robert Cameron, Presser, Stanley, Ruud, Paul A., Kerry Smith, V., Moody, Wendy R., Green, Melanie C., and Conaway, Michael. 2002. “The Impact of ‘No Opinion’ Response Options on Data Quality.” Public Opinion Quarterly 66:371–403.CrossRefGoogle Scholar
Luskin, Robert C., and Bullock, John G.. 2011. “‘Don’t Know’ Means ‘Don’t Know’: DK Responses and the Public’s Level of Political Knowledge.” Journal of Politics 73 (2): 547–57.CrossRefGoogle Scholar
Mondak, Jeffery J. 2000. “Reconsidering the Measurement of Political Knowledge.” Political Analysis 8:57–82.CrossRefGoogle Scholar
Mondak, Jeffery J. 2001. “Developing Valid Knowledge Scales.” American Journal of Political Science 45:224–38.CrossRefGoogle Scholar
Mondak, Jeffery J., and Anderson, Mary R.. 2004. “The Knowledge Gap: A Reexamination of Gender-Based Differences in Political Knowledge.” Journal of Politics 66:492–512.CrossRefGoogle Scholar
Mondak, Jeffery J., and Davis, Belinda Creel. 2001. “Asked and Answered: Knowledge Levels When We Will Not Take ‘Don’t Know’ for an Answer.” Political Behavior 23:199–224.CrossRefGoogle Scholar
Nie, Norman, Verba, Sidney, and Petrocik, John R.. 1979. The Changing American Voter. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Peer, Eyal, Vosgerau, Joachim, and Acquisti, Alessandro. 2014. “Reputation as a Sufficient Condition for Data Quality on Amazon Mechanical Turk.” Behavior Research Methods 46 (4): 1023–31.CrossRefGoogle ScholarPubMed
Pietryka, Matthew T., and MacIntosh, Randall C.. 2013. “An Analysis of ANES Items and Their Use in the Construction of Political Knowledge Scales.” Political Analysis 21 (4): 407–29.CrossRefGoogle Scholar
Pope, Jeremy. 2021. “Replication Data for: Mechanical Turk and the “Don’t Know” Option.” Harvard Dataverse. doi: 10.7910/DVN/KAPQWT.CrossRefGoogle Scholar
Popkin, Samuel L. 1991. The Reasoning Voter: Communication and Persuasion in Presidential Campaigns. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Prior, Markus, and Lupia, Arthur. 2008. “Money, Time, and Political Knowledge: Distinguishing Quick Recall and Political Learning Skills.” American Journal of Political Science 52 (1): 169–83.CrossRefGoogle Scholar
Sturgis, Patrick, Allum, Nick, and Smith, Patten. 2008. “An Experiment on the Measurement of Political Knowledge in Surveys.” Public Opinion Quarterly 72:90–102.CrossRefGoogle Scholar
Tourangeau, Roger, Maitland, Aaron, and Yan, H. Yanna. 2016. “Assessing the Scientific Knowledge of the General Public: The Effects of Question Format and Encouraging or Discouraging Don’t Know Responses.” Public Opinion Quarterly 80:741–60.CrossRefGoogle Scholar
Supplementary material: PDF

Brown and Pope supplementary material

Brown and Pope supplementary material

Download Brown and Pope supplementary material(PDF)
PDF 211.5 KB