Hostname: page-component-5cf477f64f-2wr7h Total loading time: 0 Render date: 2025-04-04T13:10:59.598Z Has data issue: false hasContentIssue false
  • English
  • Français

Racing the Clock: Using Response Time as a Proxy for Attentiveness on Self-Administered Surveys

Published online by Cambridge University Press:  15 September 2021

Blair Read Open the ORCID record for Blair Read [Opens in a new window] ,
Lukas Wolters Open the ORCID record for Lukas Wolters [Opens in a new window]  and
Adam J. Berinsky Open the ORCID record for Adam J. Berinsky [Opens in a new window]
Blair Read*
Affiliation:
Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. E-mail: [email protected]
Lukas Wolters
Affiliation:
Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. E-mail: [email protected]
Adam J. Berinsky
Affiliation:
Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. E-mail: [email protected]
*
Corresponding author Blair Read
Get access Rights & Permissions [Opens in a new window]

Abstract

Internet-based surveys have expanded public opinion data collection at the expense of monitoring respondent attentiveness, potentially compromising data quality. Researchers now have to evaluate attentiveness ex-post. We propose a new proxy for attentiveness—response-time attentiveness clustering (RTAC)—that uses dimension reduction and an unsupervised clustering algorithm to leverage variation in response time between respondents and across questions. We advance the literature theoretically arguing that the existing dichotomous classification of respondents as fast or attentive is insufficient and neglects slow and inattentive respondents. We validate our theoretical classification and empirical strategy against commonly used proxies for survey attentiveness. In contrast to other methods for capturing attentiveness, RTAC allows researchers to collect attentiveness data unobtrusively without sacrificing space on the survey instrument.

Keywords

response timesurvey attentivenessGaussian mixture model
Type
Article
Information
Political Analysis , Volume 30 , Issue 4 , October 2022 , pp. 550 - 569
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Edited by Jeff Gill

References

Alvarez, R. M., Atkeson, L. R., Levin, I., and Li, Y.. 2019. “Paying Attention to Inattentive Survey Respondents.” Political Analysis 27(2):145–162.CrossRefGoogle Scholar
Anduiza, E., and Galais, C.. 2016. “Answering Without Reading: IMCs and Strong Satisficing in Online Surveys.” International Journal of Public Opinion Research 29(3):497–519.Google Scholar
Ansolabehere, S., and Schaffner, B. F.. 2015. “Distractions: The Incidence and Consequences of Interruptions for Survey Respondents.” Journal of Survey Statistics and Methodology 3(2):216–239.CrossRefGoogle Scholar
Baker, R., et al. 2010. “American Association of Public Opinion Researchers’ Report on Online Panels.” Public Opinion Quarterly 74(4):711–781.Google Scholar
Barge, S., and Gehlbach, H.. 2012. “Using the Theory of Satisficing to Evaluate the Quality of Survey Data.” Research in Higher Education 53(2):182–200.CrossRefGoogle Scholar
Bassili, J. N., and Scott, B. S.. 1996. “Response Latency as a Signal to Question Problems in Survey Research.” Public Opinion Quarterly 60(3):390–399.CrossRefGoogle Scholar
Berinsky, A. J. 2017. “Rumors and Health Care Reform: Experiments in Political Misinformation.” British Journal of Political Science 47(2):241–262.CrossRefGoogle Scholar
Berinsky, A. J., Margolis, M. F., and Sances, M. W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58(3):739–753.CrossRefGoogle Scholar
Berinsky, A. J., Margolis, M. F., Sances, M. W., and Warshaw, C.. 2019. “Using Screeners to Measure Respondent Attention on Self-Administered Surveys: Which Items and How Many?” Political Science Research and Methods 9(2):430–437.CrossRefGoogle Scholar
Bishop, C. M. 2006. Pattern Recognition and Machine Learning. New York: Springer.Google Scholar
Börger, T. 2016. “Are Fast Responses More Random? Testing the Effect of Response Time on Scale in an Online Choice Experiment.” Environmental and Resource Economics 65(2):389–413.CrossRefGoogle Scholar
Callegaro, M., Yang, Y., Bhola, D. S., Dillman, D. A., and Chin, T.-Y.. 2009. “Response Latency as an Indicator of Optimizing in Online Questionnaires.” Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique 103(1):5–25.CrossRefGoogle Scholar
Fazio, R. H. 1990. “A Practical Guide to the Use of Response Latency in Social Psychological Research.” Research Methods in Personality and Social Psychology 11:74–97.Google Scholar
Greszki, R., Meyer, M., and Schoen, H.. 2015. “Exploring the Effects of Removing “Too Fast” Responses and Respondents from Web Surveys.” Public Opinion Quarterly 79(2):471–503.CrossRefGoogle Scholar
Harden, J. J., Sokhey, A. E., and Runge, K. L.. 2019. “Accounting for Noncompliance in Survey Experiments.” Journal of Experimental Political Science 6(3):199–202.CrossRefGoogle Scholar
Hillygus, D. S., Jackson, N., and Young, M.. 2014. “Professional Respondents in Non-probability Online Panels.” Online Panel Research: A Data Quality Perspective 1:219–237.CrossRefGoogle Scholar
Höhne, J. K., Schlosser, S., Couper, M. P., and Blom, A. G.. 2020. “Switching Away: Exploring On-device Media Multitasking in Web Surveys.” Computers in Human Behavior 111:106417.CrossRefGoogle Scholar
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., and DeShon, R. P.. 2012. “Detecting and Deterring Insufficient Effort Responding to Surveys.” Journal of Business and Psychology 27(1):99–114.CrossRefGoogle Scholar
Huckfeldt, R., Levine, J., Morgan, W., and Sprague, J.. 1999. “Accessibility and the Political Utility of Partisan and Ideological Orientations.” American Journal of Political Science 43(3):888–911.CrossRefGoogle Scholar
Imai, K., and Tingley, D.. 2012. “A Statistical Method for Empirical Testing of Competing Theories.” American Journal of Political Science 56(1):218–236.CrossRefGoogle Scholar
Johnson, M. 2004. “Timepieces: Components of Survey Question Response Latencies.” Political Psychology 25(5):679–702.CrossRefGoogle Scholar
Kong, X. J., Wise, S. L., and Bhola, D. S.. 2007. “Setting the Response Time Threshold Parameter to Differentiate Solution Behavior from Rapid-Guessing Behavior.” Educational and Psychological Measurement 67(4):606–619.CrossRefGoogle Scholar
Krosnick, J. A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5(3):213–236.CrossRefGoogle Scholar
Malhotra, N. 2008. “Completion Time and Response Order Effects in Web Surveys.” Public Opinion Quarterly 72(5):914–934.CrossRefGoogle Scholar
McLachlan, G. J., Lee, S. X., and Rathnayake, S. I.. 2019. “Finite Mixture Models.” Annual Review of Statistics and Its Application 6:355–378.CrossRefGoogle Scholar
Mulligan, K., Grant, J. T., Mockabee, S. T., and Monson, J. Q.. 2003. “Response Latency Methodology for Survey Research: Measurement and Modeling Strategies.” Political Analysis 11(3):289–301.CrossRefGoogle Scholar
Olson, K., and Smyth, J. D.. 2015. “The Effect of CATI Questions, Respondents, and Interviewers on Response Time.” Journal of Survey Statistics and Methodology 3(3):361–396.CrossRefGoogle Scholar
Oppenheimer, D. M., Meyvis, T., and Davidenko, N.. 2009. “Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power.” Journal of Experimental Social Psychology 45(4):867–872.CrossRefGoogle Scholar
Read, B., Wolters, L., and Berinsky, A.J.. 2021. “Replication Data for: Racing the Clock: Using Response Time as a Proxy for Attentiveness on Self-Administered Surveys.” Harvard Dataverse, V1. https://doi.org/10.7910/DVN/6OGTHL CrossRefGoogle Scholar
Sendelbah, A., Vehovar, V., Slavec, A., and Petrovčič, A.. 2016. “Investigating Respondent Multitasking in Web Surveys using Paradata.” Computers in Human Behavior 55:777–787.CrossRefGoogle Scholar
Tourangeau, R., Rips, L. J., and Rasinski, K.. 2000. The Psychology of Survey Response. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Tversky, A., and Kahneman, D.. 1981. “The Framing of Decisions and the Psychology of Choice.” Science 211(4481):453–458.CrossRefGoogle ScholarPubMed
Vandenplas, C., Beullens, K., and Loosveldt, G.. 2019. “Linking Interview Speed and Interviewer Effects on Target Variables in Face-to-Face Surveys.” Survey Research Methods 13(3):249–265.Google Scholar
Wise, S. L., and Kong, X.. 2005. “Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests.” Applied Measurement in Education 18(2):163–183.CrossRefGoogle Scholar
Wood, D., Harms, P., Lowman, G. H., and DeSimone, J. A.. 2017. “Response Speed and Response Consistency as Mutually Validating Indicators of Data Quality in Online Samples.” Social Psychological and Personality Science 8(4):454–464.CrossRefGoogle Scholar
Yan, T., and Olson, K.. 2013. “Analyzing Paradata to Investigate Measurement Error.” In Improving Surveys with Paradata: Analytic Uses of Process Information, edited by Kreuter, F., 73–96. Hoboken, NJ: John Wiley & Sons.CrossRefGoogle Scholar
Yan, T., Ryan, L., Becker, S. E., and Smith, J.. 2015. “Assessing Quality of Answers to a Global Subjective Well-Being Question through Response Times.” Survey Research Methods 9(2):101.Google ScholarPubMed
Zandt, T. V. 2002. “Analysis of Response Time Distributions.” In Stevens’ Handbook of Experimental Psychology, edited by Pashler, H., and Wixted, J., 62. Atlanta: American Airlines.Google Scholar
Zhang, C., and Conrad, F.. 2014. “Speeding in Web Surveys: The Tendency to Answer Very Fast and Its Association with Straightlining.” Survey Research Methods 8(2):127–135.Google Scholar
Supplementary material: PDF

Read et al. supplementary material

Read et al. supplementary material

Download Read et al. supplementary material(PDF)
PDF 3.8 MB