Book contents
- The Cambridge Handbook of Consumer Psychology
- Cambridge Handbooks in Psychology
- The Cambridge Handbook of Consumer Psychology
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Introduction
- 1 Consumer Psychology of Individuals
- 2 Consumer Psychology of Groups and Society
- 3 Methods for Understanding Consumer Psychology
- 19 Field Experimentation in Consumer Research
- 20 MTurk and Online Panel Research
- 21 Meta-analysis: Assessing Heterogeneity Using Traditional and Contemporary Approaches
- 22 Netnography for Consumer Psychologists
- 23 A Recipe for Honest Consumer Research
- Index
- References
20 - MTurk and Online Panel Research
from 3 - Methods for Understanding Consumer Psychology
Published online by Cambridge University Press: 30 March 2023
Book contents
- The Cambridge Handbook of Consumer Psychology
- Cambridge Handbooks in Psychology
- The Cambridge Handbook of Consumer Psychology
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Introduction
- 1 Consumer Psychology of Individuals
- 2 Consumer Psychology of Groups and Society
- 3 Methods for Understanding Consumer Psychology
- 19 Field Experimentation in Consumer Research
- 20 MTurk and Online Panel Research
- 21 Meta-analysis: Assessing Heterogeneity Using Traditional and Contemporary Approaches
- 22 Netnography for Consumer Psychologists
- 23 A Recipe for Honest Consumer Research
- Index
- References
Summary
Online platforms such as Amazon’s Mechanical Turk (MTurk), CloudResearch, and Prolific have become a common source of data for behavioral researchers and consumer psychologists alike. This chapter reviews contemporary issues associated with online panel research, discussing first how the COVID-19 pandemic impacted the extent to which researchers use online panels and the workers participating on certain online panels. The chapter explores how factors like a TikTok video can impact who uses these online panels and why. A longitudinal study of researcher perceptions and data quality practices finds that many practices do not align with current recommendations. The authors provide several recommendations for researchers to conduct high-quality behavioral research online, including the use of appropriate prescreens before data collection, data analysis preregistration practices, and avoiding post-screens after data collection that are not preregistered. Finally, the authors recommend researchers thoroughly report details on recruitment, restrictions, completion rates, and any differences in dropout rates across conditions.
Keywords
- Type
- Chapter
- Information
- The Cambridge Handbook of Consumer Psychology , pp. 548 - 571Publisher: Cambridge University PressPrint publication year: 2023
References
Aguinis, H., Villamor, I., & Ramani, R. S. (2021). MTurk research: Review and recommendations. Journal of Management, 47(4), 823–837.Google Scholar
Arechar, A. A., & Rand, D. G. (2021). Turking in the time of COVID. Behavior Research Methods, 53(6), 2591–2595.Google Scholar
Arndt, A. D., Ford, J. B., Babin, B. J., & Luong, V. (2022). Collecting samples from online services: How to use screeners to improve data quality. International Journal of Research in Marketing, 39(1), 117–133.Google Scholar
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Current Directions in Psychological Science, 13(2), 149–154.Google Scholar
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130.Google Scholar
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.Google Scholar
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51(5), 2022–2038.Google Scholar
Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12.Google Scholar
Charalambides, N. (2021, August 24). We recently went viral on TikTok - here’s what we learned. Prolific Blog, www.blog.prolific.co/we-recently-went-viral-on-tiktok-heres-what-we-learnedGoogle Scholar
Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464–473.Google Scholar
Couper, M. (2008). Designing Effective Web Surveys, Vol. 75. Cambridge University Press.Google Scholar
Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2020). Online worker fraud and evolving threats to the integrity of MTurk data: A discussion of virtual private servers and the limitations of IP-based screening procedures. Behavioral Research in Accounting, 32(1), 119–134.Google Scholar
Dholakia, U. (2015, July 20). My Experience as an Amazon Mechanical Turk (MTurk) Worker. www.linkedin.com/pulse/my-experience-amazon-mechanical-turkmturk-worker-utpal-dholakiaGoogle Scholar
Dreyfuss, E. (2018, August 17). A bot panic hits Amazon’s Mechanical Turk. Wired, www.wired.com/story/amazon-mechanical-turk-bot-panicGoogle Scholar
Eyal, P., David, R., Andrew, G., Zak, E., & Ekaterina, D. (2022). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54, 1643–1662.Google Scholar
Freitas, A. L., Gollwitzer, P., & Trope, Y. (2004). The influence of abstract and concrete mindsets on anticipating and guiding others’ self-regulatory efforts. Journal of Experimental Social Psychology, 40(6), 739–752.Google Scholar
Garcia, T. (2021, June 15). Consumers find shortages and higher prices as COVID-impacted supply chains shift for recovery. MarketWatch. www.marketwatch.com/story/consumers-find-shortages-and-higher-prices-as-covid-impacted-supply-chains-shift-for-recovery-11623425796Google Scholar
Goodman, J. K., Cryder, C. E., Cheema, A. A. (2013). Data collection in a flat world: Strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26 (July), 213–224.Google Scholar
Goodman, J. K., & Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research, 44(1), 196–210.Google Scholar
Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902.Google Scholar
Hauser, D., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2021). Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk. https://psyarxiv.com/48yxj/Google Scholar
Hauser, D. J., Paolacci, G., & Chandler, J. (2019). Common concerns with MTurk as a participant pool: Evidence and solutions. In Kardes, F. R., Herr, P. M., & Schwarz, N. (Eds.). Handbook of Research Methods in Consumer Psychology (pp. 319–337). Routledge.Google Scholar
Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407.Google Scholar
Hernandez, J. M. da, C., Wright, S. A., & Rodrigues, F. F. (2015). Attributes versus benefits: The role of construal levels and appeal type on the persuasiveness of marketing messages. Journal of Advertising, 44(3), 243–253.Google Scholar
Jacowitz, K. E., & Kahneman, D. (1995). Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21(11), 1161–1166.Google Scholar
Jacquemet, N., James, A. G., Luchini, S., Murphy, J. J., & Shogren, J. F. (2021). Do truth-telling oaths improve honesty in crowd-working? PLoS ONE, 16(1), e0244958.Google Scholar
Janiszewski, C., & van Osselaer, S. M. J. (2021). The benefits of candidly reporting consumer research. Journal of Consumer Psychology, 31(4), 633–646.Google Scholar
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s Mechanical Turk. Journal of Advertising, 46(1), 141–155.Google Scholar
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. (2020). The shape of and solutions to the MTurk quality crisis. Political Science Research and Methods, 8(4), 614–629.Google Scholar
Kochhar, R. (2020, June 11). Unemployment rose higher in three months of COVID-19 than it did in two years of the Great Recession. Pew Research Center. www.pewresearch.org/fact-tank/2020/06/11/unemployment-rose-higher-in-three-months-of-covid-19-than-it-did-in-two-years-of-the-great-recessionGoogle Scholar
Litman, L., Moss, A., Rosenzweig, C., & Robinson, J. (2021, January 28). Reply to MTurk, Prolific or panels? Choosing the right audience for online research. Available at SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3775075Google Scholar
Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavioral Research Methods, 47, 519–528.Google Scholar
Loepp, E., & Kelly, J. T. (2020). Distinction without a difference? An assessment of MTurk Worker types. Research & Politics, 7(1), p. 1–8.Google Scholar
Lourenco, S. F., & Tasimi, A. (2020). No participant left behind: Conducting science during COVID-19. Trends in Cognitive Sciences, 24(8), 583–584.Google Scholar
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2021). Improving data quality using Amazon Mechanical Turk through platform setup. Cornell Hospitality Quarterly, 19389655211025475.Google Scholar
Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014). Being a Turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing, Association for Computing Machinery, 224–235.Google Scholar
Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23.Google Scholar
Moss, A., & Litman, L. (2018). After the bot scare: Understanding what’s been happening with data collection on MTurk and how to stop it. Retrieved February 4, 2019. www.cloudresearch.com/resources/blog/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-itGoogle Scholar
Moss, A. J., Rosenzweig, C., Jaffe, S. N., Gautam, R., Robinson, J., & Litman, L. (2021). Bots or inattentive humans? Identifying sources of low-quality data in online platforms. https://doi.org/10.31234/osf.io/wr8dsGoogle Scholar
Moss, A. J., Rosenzweig, C., Robinson, J., & Litman, L. (2020). Demographic stability on Mechanical Turk despite COVID-19. Trends in Cognitive Sciences, 24(9), 678–680.Google Scholar
Mummolo, J., & Peterson, E. (2019). Demand effects in survey experiments: An empirical assessment. American Political Science Review, 113, 517–529.Google Scholar
Mutikani, L. (2021, September 8). U.S. job openings hit record high as employers struggle to find workers. Reuters, www.reuters.com/business/us-job-openings-rise-record-109-million-july-2021-09-08Google Scholar
Palan, S., & Schitter, C. (2018). Prolific.ac – A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27.Google Scholar
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.Google Scholar
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163.Google Scholar
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031.Google Scholar
Peyton, K., Huber, G. A., & Coppock, A. (2021). The generalizability of online experiments conducted during the COVID-19 pandemic. Journal of Experimental Political Science, 1–16.Google Scholar
Ramos, S. (2021, March 1). COVID-19’s impact felt by researchers. American Psychological Association, www.apa.org/science/leadership/students/covid-19-impact-researchersGoogle Scholar
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLoS ONE, https://doi.org/10.1371/journal.pone.0226394Google Scholar
Sandford, A. (2020, April 2). Coronavirus: Half of humanity on lockdown in 90 countries. Euronews. Archived from the original on May 19, 2020. Retrieved June 15, 2021. www.euronews.com/2020/04/02/coronavirus-in-europe-spain-s-death-toll-hits-10-000-after-record-950-new-deaths-in-24-hou#vuukle-comments-1065562Google Scholar
Shapiro, D. N., Chandler, J., & Mueller, P. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220.Google Scholar
Sheth, J. (2020). Impact of Covid-19 on consumer behavior: Will the old habits return or die? Journal of Business Research, 117, 280–283.Google Scholar
Simon, R. (2021, April 16). Covid-19’s toll on U.S. business? 200,000 Extra closures in pandemic’s first year. The Wall Street Journal, www.wsj.com/livecoverage/covid-2021-04-16/card/QKG4xgwU4HyCqMq6TiyIGoogle Scholar
Simmons, J., Nelson, L. D., & Simonsohn, U. (2021). Pre‐registration: Why and how. Journal of Consumer Psychology, 31(1), 151–162.Google Scholar
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547.Google Scholar
Stokel-Walker, C. (2018, August 10). Bots on Amazon’s Mechanical Turk are ruining psychology studies. New Scientist, www.newscientist.com/article/2176436-bots-on-amazons-mechanical-turk-are-ruining-psychology-studiesGoogle Scholar
Su, I., & Ceci, S. (2021). “Zoom developmentalists”: Home-based videoconferencing developmental research during COVID-19. https://doi.org/10.31234/osf.io/nvdy6Google Scholar
Wessling, K. S., Huber, J., Netzer, O. (2017). MTurk character misrepresentation: Assessment and solutions. Journal of Consumer Research, 44, 211–230.Google Scholar
Wright, S., Manolis, C., Brown, D., et al. (2012). Construal-level mind-sets and the perceived validity of marketing claims. Marketing Letters, 23(1), 253–261.Google Scholar
Wright, S. A., & Goodman, J. K. (2019). Mechanical Turk in consumer research: Perceptions and usage in marketing academia. In Kardes, F. R., Herr, P. M., & Schwarz, N. (Eds.). Handbook of Research Methods in Consumer Psychology (pp. 338–357). Routledge.Google Scholar
Yue, C., Gilbert, B., Zhu, Q., Kleeman, H., & Rea, S. C. (2020). Crowdsourcing as a tool for research: Methodological, fair, and political considerations. Bulletin of Science, Technology & Society, 40(3–4), 40–53.Google Scholar
Zhou, H., & Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111(4), 493–504.Google Scholar