Hostname: page-component-77c89778f8-5wvtr Total loading time: 0 Render date: 2024-07-21T18:29:30.288Z Has data issue: false hasContentIssue false
  • English
  • Français

Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations

Published online by Cambridge University Press:  28 July 2015

Sang Eun Woo ,
Melissa Keith  and
Meghan A. Thornton
Sang Eun Woo*
Affiliation:
Purdue University
Melissa Keith
Affiliation:
Purdue University
Meghan A. Thornton
Affiliation:
Purdue University
*
Correspondence concerning this article should be addressed to Sang Eun Woo, 703 Third Street, West Lafayette, IN 47907. E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Extract

We are in almost full agreement with Landers and Behrend's (2015) thoughtful and balanced critiques of various convenience sampling strategies focusing on the four most frequently used data sources in our field. In this commentary, we expand on Landers and Behrend's discussions specifically around Mechanical Turk (MTurk) by providing further supporting voice and/or clarity to the four potential concerns and relative advantages associated with MTurk. We also raise a few additional concerns and challenges to which the current literature does not yet offer definitive answers. We conclude with some practical guidelines summarizing the relative advantages and unique challenges of using MTurk.

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 8 , Issue 2 , June 2015 , pp. 171 - 179
Copyright
Copyright © Society for Industrial and Organizational Psychology 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Arnett, J. J. (2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist, 63, 602–614.CrossRefGoogle ScholarPubMed
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavioral Research Methods, 43 (3), 1–14. doi:10.3758/s13428-011-0081-0CrossRefGoogle ScholarPubMed
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20, 351–368.Google Scholar
Bloxham, E. (2014, September 12). The Fortune 500 can easily afford to give low-wage workers a healthy raise. Fortune. Retrieved from http://fortune.comGoogle Scholar
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5.Google Scholar
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavioral Research, 46, 112–130. doi:10.3758/s13428-013-0365-7Google Scholar
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLos ONE, 8 (3). doi:10.1371/journal.pone.0057410Google Scholar
Fort, K., Gilles, A., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37 (2), 413–420.Google Scholar
Holden, C. J., Dennie, T., & Hicks, A. D. (2013). Assessing the reliability of the M5–120 on Amazon's Mechanical Turk. Computers in Human Behavior, 29, 1749–1754. doi:10.1016/j.chb.2013.02.020Google Scholar
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399–425.Google Scholar
International Labour Organization. (2011). Statistical update on employment in the informal economy. Geneva, Switzerland: ILO Department of Statistics.Google Scholar
Ipeirotis, P. G. (2010). Demographics of Mechanical Turk (Technical Report CeDER-10–01). New York: New York University.Google Scholar
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310–314.Google Scholar
Lelkes, Y., Krosnick, J. A., Marx, D. M., Judd, C. M., & Park, B. (2012). Complete anonymity compromises accuracy of self-reports. Journal of Experimental Social Psychology, 48, 1291–1299.Google Scholar
Maynard, D. C., & Ferdman, B. M. (2009). The marginalized workforce: How I-O psychology can make a difference. The Industrial–Organizational Psychologist, 46 (4), 25–29.Google Scholar
Ong, A. D., & Weiss, D. J. (2000). The impact of anonymity on responses to sensitive questions. Journal of Applied Social Psychology, 30, 1691–1708.Google Scholar
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5 (5), 411–419.Google Scholar
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46 (4), 1023–1031.CrossRefGoogle ScholarPubMed
Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179.Google Scholar
Ross, J., Zaldivar, A., Irani, L., & Tomlinson, B. (2010). Who are the Turkers? Worker demographics in Amazon Mechanical Turk. Proceedings from CHI ’10: The 28th ACM Conference on Human Factors in Computing Systems. Atlanta, GA.Google Scholar
Simons, D. J., & Chabris, C. F. (2012). Common (mis)beliefs about memory: A replication and comparison of telephone and Mechanical Turk survey methods. PLoS ONE, 7 (12), 1–5. doi:10.1371/journal.pone.0051876Google Scholar
Van Zant, A. B., & Kray, L. J. (2014). “I can't lie to your face”: Minimal face-to-face interaction promotes honesty. Journal of Experimental Social Psychology, 55, 234–238.Google Scholar
Wolfson, S. N., & Bartkus, J. R. (2013). An assessment of experiments run on Amazon's Mechanical Turk. Mustang Journal of Business Ethics, 5, 119–129.Google Scholar