Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-745bb68f8f-grxwn Total loading time: 0 Render date: 2025-02-08T10:34:01.362Z Has data issue: false hasContentIssue false

29 - Social Signals of Deception and Dishonesty

from Part IV - Applications of Social Signal Processing

Published online by Cambridge University Press:  13 July 2017

By
Judee K. Burgoon ,
Dimitris Metaxas ,
Thirimachos Bourlai  and
Aaron Elkins
Edited by
Judee K. Burgoon ,
Nadia Magnenat-Thalmann ,
Maja Pantic  and
Alessandro Vinciarelli
Judee K. Burgoon
Affiliation:
University of Arizona
Dimitris Metaxas
Affiliation:
Rutgers University
Thirimachos Bourlai
Affiliation:
West Viriginia University
Aaron Elkins
Affiliation:
San Diego State University
Judee K. Burgoon
Affiliation:
University of Arizona
Nadia Magnenat-Thalmann
Affiliation:
Université de Genève
Maja Pantic
Affiliation:
Imperial College London
Alessandro Vinciarelli
Affiliation:
University of Glasgow
Get access

Summary

Social life is constituted of interactions with others – others whom we must rapidly classify as friend or foe, as trustworthy or not. Gauging another's trustworthiness relies on successfully reading nonverbal signals – signals that have been selected through human evolution to serve precisely such a communicative function. These deeply ingrained signals – some part of our phylogenetic heritage and some part of our socially constructed communication system – form the unwritten “order” for cooperative encounters, enabling both individuals and societies to survive and thrive. Yet paradoxically, the same course of evolution has also remunerated, with greater prospects for survival, those who manipulate and falsely manufacture such signals; in short, those who cheat, dissemble and deceive. Put differently, the course of human development has produced a system of presumably reliable signals of veracity, authenticity, trust and trustworthiness, while simultaneously conferring advantages on sham portrayals of those same signals. The use of dishonest signals is not confined to humans; natural selection has also rewarded sophisticated forms of cheating among all manner of living organisms (Greenfield, 2006). Consequently, these nonverbal signals, many of which are universal and have similarities among other species, are among the most important for humans to produce and read and the most useful for computational methods to detect and track.

In what follows, we foreground those aspects of social signaling related to veracity that have widespread use and recognition. These are the kinds of signals that Burgoon and Newton (1991) identified as corresponding to a social meaning model in that they are recurrent expressions that have consensually recognized meanings within a given community. In this chapter, we first provide the reader background on the nature of the aforementioned signals. Next, we discuss automated methods for human nonverbal communication computing, i.e., methods we used for identifying and tracking such signals. Then, we discuss computer vision technologies using sensors operating in different wavelengths of the infrared band and not only the visible band, which is conventionally used. We conclude with recommendations for promising future research directions where the latest technologies can be applied to this elemental aspect of social signaling.

Type
Chapter
Information
Social Signal Processing , pp. 404 - 428
Publisher: Cambridge University Press
Print publication year: 2017

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abaza, A., Harrison, M. A., Bourlai, T., & Ross, A. (2014). Design and evaluation of photometric image quality measures for effective face recognition.IET Biometrics, 3(4), 314–324.Google Scholar
Adams, G. & Markus, H. R. (2004). Toward a conception of culture suitable for a social psychology of culture. In M., Schaller & C. S., Crandall (Eds), The Psychological Foundations of Culture, 335–360. New York: Springer.
Ao, M., Yi, D. Lei, Z., & Li, S. Z. (2009). Handbook of Remote Biometrics for Surveillance and Security. London: Springer.
Baddeley, A. (1986). Working Memory. Oxford: Clarendon Press.
Baddeley, A. (2000a). Short-term and working memory. In E, Tulving & F. I. M, Craik (Eds), The Oxford Handbook of Memory (pp. 77–92). Oxford: Oxford University Press.
Baddeley, A. (2000b). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4(11), 417–423.Google Scholar
Black, M. J. & Yacoob, Y. (1997). Recognizing facial expressions in image sequences using local parameterized models of image motion.International Journal of Computer Vision, 25(1), 23– 48.Google Scholar
Blanz, V. & Vetter, T. (1999). A morphable model for the synthesis of 3-D faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (pp. 187–194).
Bourlai, T. (2015). Face recognition by using multispectral imagery. In InfraMation, Nashville, Tennessee, May 12–14 (invited paper).
Bourlai, T., Kalka, N., Cao, D., et al. (2011). Ascertaining human identity in night environments. In B, Bhanu, C, Ravishankar, A, Roy-Chowdhury, H, Aghajan, & D, Terzopoulos (Eds), Distributed Video Sensor Networks (pp. 451–468). London: Springer.
Bourlai, T., Narang, N., Cukic, B., & Hornak, L. (2012). SWIR multi-wavelength acquisition system for simultaneous capture of face images. In Proceedings of SPIE Infrared Technology and Applications XXXVIII(vol. 8353).Google Scholar
Buck, R. (1988). Nonverbal communication: Spontaneous and symbolic aspects.American Behavioral Scientist, 31, 341–354.Google Scholar
Buck, R. & VanLear, C. A. (2002). Verbal and nonverbal communication: Distinguishing spontaneous, symbolic and pseudo-spontaneous nonverbal behavior.Journal of Communication, 52, 522–541.Google Scholar
Buller, D. B. & Burgoon, J. K. (1994). Deception: Strategic and nonstrategic communication. In J. A, Daly & J. M, Wiemann (Eds.), Strategic Interpersonal Communication (pp. 191–223). Hillsdale, NJ: Erlbaum.
Burgoon, J. K. (2005). Nonverbal measurement of deceit. In V, Manusov (Ed.), The Sourcebook of Nonverbal Measures: Going Beyond Words (pp. 237–250). Hillsdale, NJ: Erlbaum.
Burgoon, J. K., Buller, D. B., Floyd, K., & Grandpre, J. (1996). Deceptive realities: Sender, receiver, and observer perspectives in deceptive conversations.Communication Research, 23, 724–748.Google Scholar
Burgoon, J. K., Buller, D. B., Floyd, K., & Grandpre, J. (1996). Deceptive realities: Sender, receiver, and observer perspectives in deceptive conversations.Communication Research, 23, 724–748.Google Scholar
Burgoon, J. K., Buller, D. B., Floyd, K., & Grandpre, J. (1996). Deceptive realities: Sender, receiver, and observer perspectives in deceptive conversations.Communication Research, 23, 724–748.Google Scholar
Burgoon, J. K. & Dunbar, N. E. (2006). Dominance, power and influence. In V, Manusov & M, Patterson (Eds.), The SAGE Handbook of Nonverbal Communication (pp. 279–298). Thousand Oaks, CA: SAGE.
Burgoon, J. K. & Floyd, K. (2000). Testing for the motivation impairment effect during deceptive and truthful interaction.Western Journal of Communication, 64, 243–267.Google Scholar
Burgoon, J. K., Guerrero, L., & Floyd, K. (2010). Nonverbal Communication. Boston: Allyn & Bacon.
Burgoon, J. K. & Hale, J. L. (1984). The fundamental topoi of relational communication.Communication Monographs, 51, 193–214.Google Scholar
Burgoon, J. K., Kelley, D. L., Newton, D. A., & Keeley-Dyreson, M. P. (1989). The nature of arousal and nonverbal indices.Human Communication Research, 16, 217–255.Google Scholar
Burgoon, J. K. & Newton, D. A. (1991). Applying a social meaning model to relational messages of conversational involvement: Comparing participant and observer perspectives.Southern Communication Journal, 56, 96–113.Google Scholar
Burgoon, J. K., Schuetzler, R., & Wilson, D. (2014). Kinesic patterns in deceptive and truthful interactions.Journal of Nonverbal Behavior, 39, 1–24.Google Scholar
Burgoon, J. K., Stern, L. A., & Dillman, L. (1995). Interpersonal Adaptation: Dyadic Interaction Patterns. New York: Cambridge University Press.
Coker, D. A. & Burgoon, J. K. (1987). The nature of conversational involvement and nonverbal encoding patterns.Human Communication Research, 13, 463–494.Google Scholar
Cootes, T., Taylor, C., Cooper, D., & Graham, J. (1995). Active shape models: Their training and application.Computer Vision and Image Understanding, 61(1) 38–59.Google Scholar
Dcosta, M., Shastri, D., Vilalta, R., Pavilidis, I., & Burgoon, J. K. (2015). Perinasal indicators of deceptive behavior.Paper presented to IEEE International Conference on Automatic Face and Gesture Recognition, Slovenia.
DeCarlo, D. & Metaxas, D. (1996). The integration of optical flow and deformable models with applications to human face shape and motion estimation. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 231–238).
DeCarlo, D. & Metaxas, D. (2000). Optical flow constraints on deformable models with applications to face tracking.International Journal of Computer Vision, 38(2), 99–127.Google Scholar
DePaulo, B. M., Lindsay, J. J., Malone, B. E., et al. (2003). Cues to deception.Psychological Bulletin, 129, 74–118.Google Scholar
Duchenne, G. B. (1990). The Mechanism Of Human Facial Expression. New York: Cambridge University Press. (Original work published 1862)
Dunbar, N. E., Jensen, M. L., Bessabarova, E., et al. (2014). Empowered by persuasive deception: The effects of power and deception on interactional dominance, credibility, and decisionmaking.Communication Research, 41, 852–876.Google Scholar
Ekman, P. (1985). Telling Lies: Clues to Deceit in the Marketplace, Marriage, and Politics. New York: Norton.
Ekman, P. (1992). Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. New York: Norton.
Ekman, P. (2009). Lie catching and micro-expressions. In C, Martin (Ed.), The Philosophy of Deception (pp. 118–133). New York: Oxford University Press.
Ekman, P., Davidson, R. J., & Friesen, W. V. (1990). The Duchenne smile: Emotional expression and brain physiology: II.Journal of Personality and Social Psychology, 58, 342–353.Google Scholar
Ekman, P. & Friesen, W. V. (1969). Nonverbal leakage and clues to deception.Psychiatry, 32, 88–106.Google Scholar
Ekman, P. & Friesen, W. V. (1982). Felt, false, and miserable smiles.Journal of Nonverbal Behavior, 6, 238–252.Google Scholar
Ekman, P., Friesen, W. V., & Hagar, J. C. (2002). Facial Action Coding System. Salt Lake City, UT: Network Information Research. (Original work published 1976).
Ekman, P., O’Sullivan, M., Friesen, W. V., & Scherer, K. R. (1991). Invited article: Face, voice, and body in detecting deceit.Journal of Nonverbal Behavior, 15, 125–135.Google Scholar
Elkins, A., Zafeiriou, S., Pantic, M., & Burgoon, J. K. (2015). Unobtrusive deception detection. In R, Calvo, S. K.D’, Mello, J, Gratch, & A, Kappas (Eds.), The Oxford Handbook of Affective Computing. Oxford: Oxford University Press.
Fernandez-Dols, J. M., Sanchez, F., Carrera, P., & Ruiz-Belda, M. A. (1997). Are spontaneous expressions and emotions linked? An experimental test of coherence.Journal of Nonverbal Behavior, 21, 163–177.Google Scholar
Floyd, K. & Burgoon, J. K. (1999). Reacting to nonverbal expressions of liking: A test of interaction adaptation theory.Communication Monographs, 66, 219–239.Google Scholar
Frank, M. G. & Ekman, P. (1997). The ability to detect deceit generalizes across different types of high-stake lies.Journal of Personality and Social Psychology, 72, 1429– 1439.Google Scholar
Frank, M. G. & Ekman, P. (2004). Appearing truthful generalizes across different deception situations.Journal of Personality and Social Psychology, 86, 486–495.Google Scholar
Frank, M. G., Ekman, P., & Friesen, W. V. (1993). Behavioral markers and recognizability of the smile of enjoyment.Journal of Personality and Social Psychology, 64, 83–93.Google Scholar
Goldman-Eisler, F. (1968). Psycholinguistics: Experiments in Spontaneous Speech. New York: Academic Press.
Greenfield, H. D. (2006). Honesty and deception in animal signals. In J. R, Lucas and L. W, Simmons (Eds), Essays in Animal Behaviour: Celebrating 50 Years of Animal Behaviour (1st edn, pp. 278–298). New York: Academic Press.
Gunnery, S. D., Hall, J. A., & Ruben, M. A. (2013). The deliberate Duchenne smile: Individual differences in expressive control.Journal of Nonverbal Behavior, 37, 29–41.Google Scholar
Haggard, E. A. & Isaacs, K. S. (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In Methods of Research in Psychotherapy (pp. 154–165). New York: Springer.
Hartwig, M. & Bond, C. F. (2011).Why do lie-catchers fail? A lens model meta-analysis of human lie judgments.Psychological Bulletin, 137, 643–659.Google Scholar
Hartwig, M. & Bond, C. F. (2014). Lie detection from multiple cues: A meta-analysis.Applied Cognitive Psychology, 28, 661–676.Google Scholar
Hatfield, E., Cacioppo, J. T., & Rapson, R. L. (1994). Emotional Contagion. New York: Cambridge University Press.
Hess, U. & Bourgeois, P. (2010). You smile – I smile: Emotion expression in social interaction.Biological Psychology, 84, 514–520.Google Scholar
HSNW (2010). Efficacy of TSA's behavioral threat detection program questioned.Homeland Security News Wire, May 25.
Huang, Y., Liu, Q., & Metaxas, D. (2007). A component based deformable model for generalized face alignment In Proceedings of IEEE 11th International Conference on Computer Vision (pp. 1–8).
Kanaujia, A. & Metaxas, D. (2006). Recognizing facial expressions by tracking feature shapes. In Proceedings of IEEE 18th International Conference on Pattern Recognition(vol. 2, pp. 33–38).Google Scholar
Knapp, M. L. & Comadena, M. E. (1979). Telling it like it isn’t: A review of theory and research on deceptive communications.Human Communication Research, 5, 270–285.Google Scholar
Maccario, C. (2013). Screening of passengers by observation techniques (SPOT). Transportation Security Administration, Department of Homeland Security, May.
Masip, J., Garrido, E., & Herrero, C. (2004). Defining deception.Anales de Psicología, 20, 147– 171.Google Scholar
Metaxas, D. & Zhang, S. (2013). A review of motion analysis methods for human nonverbal communication computing.Image and Vision Computing(special issue on Machine learning in motion analysis: New advances), 31(6–7), 421–433.Google Scholar
Michael, N., Yang, F., Metaxas, D., & Dinges, D. (2011). Development of optical computer recognition (OCR) for monitoring stress and emotions in space. In 18th IAA Humans in Space Symposium.
Miller, J. L. (1994). Principles of Infrared Technology: A Practical Guide to the State of the Art. Boston: Springer.
Mullin, D. S., King, G. W., Saripalle, S. K., et al. (2014). Deception effects on standing center of pressure.Human Movement Science, 38, 106–115.Google Scholar
Narang, N. & Bourlai, T. (2015a). Can we match ultraviolet face images against their visible counterparts? In Proceedings of SPIE, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultra-spectral Imagery XXI, Baltimore, MD.
Narang, N. & Bourlai, T. (2015b). Face recognition in the SWIR band when using single sensor multi-wavelength imaging systems.Journal of Image and Vision Computing, 33, 26–43.Google Scholar
Pentland, S., Burgoon, J. K., & Twyman, N. (2015). Face and head movement analysis using automated feature extraction software. Proceedings of the 48th Hawaii International Conference on System Sciences Credibility Assessment Symposium.
Pentland, S., Twyman, N., & Burgoon, J. K. (2014). Automated analysis of guilt and deception from facial affect in a concealed information test.Presented to the Society for Personality and Social Psychology, Austin.
Porter, S. & Ten Brinke, L. (2013). Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions.Psychological Science, 19, 508–514.Google Scholar
Porter, S., Ten Brinke, L., Baker, A., & Wallace, B. (2011). Would I lie to you? “Leakage” in deceptive facial expressions relates to psychopathy and emotional intelligence.Personality and Individual Differences, 51(2), 133–137.Google Scholar
Porter, S., Ten Brinke, L., & Wallace, B. (2012). Secrets and lies: Involuntary leakage in deceptive facial expressions as a function of emotional intensity.Journal of Nonverbal Behavior, 36, 23– 37.Google Scholar
Rockwell, P., Buller, D. B., & Burgoon, J. K. (1997). The voice of deceit: Refining and expanding vocal cues to deception.Communication Research Reports, 14(4), 451–459.Google Scholar
Schuller, B. (2013). Applications in intelligent speech analysis. In Intelligent audio analysis (pp. 169–223). Berlin: Springer.
Selinger, A. & Socolinsky, D. A. (2004). Face recognition in the dark. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop (pp. 129–134).
Sporer, S. L. & Schwandt, B. (2006). Paraverbal indicators of deception: A meta-analytic synthesis.Applied Cognitive Psychology, 20(4), 421–446.Google Scholar
Tan, X. & Triggs, B. (2010). Enhanced local texture feature sets for face recognition under difficult lighting conditions.IEEE Transactions on Image Processing, 19(6), 1635–1650.Google Scholar
Terzopoulos, D. & Waters, K. (1993). Analysis and synthesis of facial image sequences using physical and anatomical models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(6), pp. 569–579.Google Scholar
Twyman, N.W., Elkins, A., Burgoon, J. K. & Nunamaker, J. F. (2014). A rigidity detection system for automated credibility assessment.Journal of Management Information Systems, 31, 173– 201.Google Scholar
Vogler, C., Li, Z., Kanaujia, A., Goldenstein, S., & Metaxas, D. (2007). The best of both worlds: Combining 3-D deformable models with active shape models. In Proceedings of IEEE 11th International Conference on Computer Vision (pp. 1–7).
Vrij, A. (2008). Detecting Lies and Deceit: Pitfalls and Opportunities (2nd edn). Chichester, UK: John Wiley & Sons.
Wang, L., Hu, W., & Tan, T. (2003). Recent developments in human motion analysis.Pattern Recognition, 36(3), 585–601.Google Scholar
Wang, Y., Huang, X., Lee, C., et al. (2004). High resolution acquisition, learning and transfer of dynamic 3-D facial expressions.Computer Graphics Forum, 23, 677–686.Google Scholar
Warren, G., Schertler, E., & Bull, P. (2009). Detecting deception from emotional and unemotional cues.Journal of Nonverbal Behavior, 33, 59–69.Google Scholar
Whitelam, C. & Bourlai, T. (2014). On designing SWIR to visible face matching algorithms.Intel® Technology Journal(special issue on Biometrics and Authentication), 18(4), 98–118.Google Scholar
Yang, F., Bourdev, L., Shechtman, E., Wang, J., & Metaxas, D. (2012). Facial expression editing in video using a temporally smooth factorization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 861–868).
Yang, F., Huang, J., & Metaxas, D. (2011). Sparse shape registration for occluded facial feature localization. In Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (pp. 272–277).
Yang, F., Wang, J., Shechtman, E., Bourdev, L., & Metaxas, D. (2011). Expression flow for 3-Daware face component transfer.ACM Transactions on Graphics, 30(4), art. 60.Google Scholar
Yang, P., Liu, Q., & Metaxas, D. (2007). Boosting coded dynamic features for facial action units and facial expression recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–6).
Yang, P., Liu, Q., & Metaxas, D. (2009). Rankboost with L1 regularization for facial expression recognition and intensity estimation, In Proceedings of IEEE 12th International Conference on Computer Vision (pp. 1018–1025).
Yang, P., Liu, Q., & Metaxas, D. (2011). Dynamic soft encoded patterns for facial event analysis.Computer Vision and Image Understanding, 115(3), 456–465.Google Scholar
Yu, X., Huang, J., Zhang, S., Yan, W., & Metaxas, D. (2013). Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In Proceedings of IEEE International Conference on Computer Vision (pp. 1944–1951).
Yu, X., Zhang, S., Yan, Z., et al. (2015). Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues.IEEE Transactions on Cybernetics, 45(3), 506–520.Google Scholar
Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions.IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39–58.Google Scholar
Zhang, S., Zhan, Y., Dewan, M., et al. (2011). Sparse shape composition: A new framework for shape prior modeling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1025–1032).
Zuckerman, M., DePaulo, B. M., & Rosenthal, R. (1981). Verbal and nonverbal communication of deception. In L, Berkowitz (Ed.), Advances in Experimental Social Psychology (pp. 1–59). New York: Academic Press.
Bourlai, T. (2016). Face Recognition across the Imaging Spectrum. Cham, Switzerland: Springer.
Bourlai, T., Whitelam, I., & Kakadiaris, I. (2011). Pupil detection under lighting and pose variations in the visible and active infrared bands. IEEE International Workshop on Information Forensics and Security, Iguacu Falls, Brazil.
Buller, D. B. & Burgoon, J. K. (1996). Interpersonal deception theory.Communication Theory, 6, 203–242.Google Scholar
Osia, N. & Bourlai, T. (2014). A spectral independent approach for physiological and geometric based face recognition in the visible, middle-wave and long-wave infrared bands.Journal of Image and Vision Computing, 32, 847–859.Google Scholar
Pfister, T., Li, X., Zhao, G. & Pietikainen, M. (2011). Recognising spontaneous facial microexpressions. In Proceedings of IEEE International Conference on Computer Vision (pp. 1449– 1456).
Whitelam, C. & Bourlai, T. (2015, July). Accurate eye localization in the short waved infrared spectrum through summation range filters.Elsevier Computer Vision and Image Understanding, 139(C), 59–72.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×