Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2024-12-31T15:29:27.289Z Has data issue: false hasContentIssue false
  • English
  • Français

Speaking through the body

Do people associate the body movements of politicians with their speech?

Published online by Cambridge University Press:  27 December 2017

Markus Koppensteiner  and
Greg Siegle
Markus Koppensteiner*
Affiliation:
University of Vienna
Greg Siegle
Affiliation:
University of Pittsburgh Netherlands Institute for Advanced Study
*
Correspondence: Markus Koppensteiner, NIAS/KNAW, Korte Spinhuissteeg 3, 1012 CG Amsterdam, The Netherlands. Email: [email protected]
Get access

Abstract

When people speak, they gesture. However, is the audience watching a speaker who is sensitive to this link? We translated the body movements of politicians into stick-figure animations and separated the visual from the audio channel. We then asked participants to match a selection of five audio tracks (including the correct one) with the stick-figure animations. The participants made correct decisions in 65% of all cases (chance level of 20%). Matching voices with animations was less difficult when politicians showed expansive movements and spoke with a loud voice. Thus, people are sensitive to the link between motion cues and vocal cues, and this link appears to become even more apparent when a speaker shows expressive behaviors. Future work will have to refine and validate the methods applied and investigate how mismatches between communication channels affect the impressions that people form of politicians.

Keywords

Nonverbal communicationmultimodal signalsbodily expressivenessvocal expressivenessgestureimpression formationstick figure animationpolitics
Type
Articles
Information
Politics and the Life Sciences , Volume 36 , Issue 2 , Fall 2017 , pp. 104 - 113
Copyright
© Association for Politics and the Life Sciences 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bente, G. and Krämer, N. C., “Psychologische Aspekte bei der Implementierung und Evaluation nonverbal agierender Interface-Agenten,” in Mensch & Computer, Oberquelle, H., Oppermann, R., and Krause, J., eds. (Berlin: Springer, 2001), pp. 275–285.Google Scholar
Bull, P. and Connelly, G., “Body movement and emphasis in speech,” Journal of Nonverbal Behavior , 1985, 9(3): 169–187.CrossRefGoogle Scholar
Ekman, P. and Friesen, W. V., “The repertoire of nonverbal behavior: Categories, origins, usage, and coding,” Semiotica , 1969, 1(1): 49–98.CrossRefGoogle Scholar
Krauss, R. M., Chen, Y., and Chawla, P., “Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us? ” Advances in Experimental Social Psychology , 1996, 28: 389–450.CrossRefGoogle Scholar
Wagner, P., Malisz, Z., and Kopp, S., “Gesture and speech in interaction: An overview,” Speech Communication , 2014, 57: 209–232.CrossRefGoogle Scholar
Condon, W. S. and Ogston, W. D., “Sound film analysis of normal and pathological behavior patterns,” Journal of Nervous and Mental Disease , 1966, 143(4): 338–347.CrossRefGoogle ScholarPubMed
Loehr, D. P., Gesture and intonation, Unpublished doctoral dissertation. Department of Linguistics, Georgetown University, Washington, DC 2004.Google Scholar
Kendon, A., “Gesticulation and speech: Two aspects of the process of utterance,” in The Relationship of Verbal and Nonverbal Communication, Key, Mary Ritchie, ed. (New  York: Mouton, 1980), pp. 207–227.CrossRefGoogle Scholar
McNeill, D., “So you think gestures are nonverbal? ” Psychological Review , 1985, 92(3): 350.CrossRefGoogle Scholar
Barsalou, L. W., “Grounded cognition,” Annual Review of Psychology , 2008, 59: 617–645.CrossRefGoogle ScholarPubMed
Gentilucci, M. and Volta, R. D., “Spoken language and arm gestures are controlled by the same motor control system,” Quarterly Journal of Experimental Psychology , 2008, 61(6): 944–957.CrossRefGoogle ScholarPubMed
Willems, R. M. and Hagoort, P., “Neural evidence for the interplay between language, gesture, and action: A review,” Brain and Language , 2007, 101(3): 278–289.CrossRefGoogle ScholarPubMed
Goldin-Meadow, S., “The role of gesture in communication and thinking,” Trends in Cognitive Science , 1999, 3(11): 419–429.CrossRefGoogle ScholarPubMed
Bernardis, P. and Gentilucci, M., “Speech and gesture share the same communication system,” Neuropsychologia , 2006, 44(2): 178–190.CrossRefGoogle ScholarPubMed
Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., and Wagner, S., “Explaining math: Gesturing lightens the load,” Psychological Science , 2001, 12(6): 516–522.CrossRefGoogle ScholarPubMed
Kelly, S. D., Özyürek, A., and Maris, E., “Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension,” Psychological Science , 2010, 21(2): 260–267.CrossRefGoogle ScholarPubMed
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., and Vatikiotis-Bateson, E., “Visual prosody and speech intelligibility: Head movement improves auditory speech perception,” Psychological Science , 2004, 15(2): 133–137.CrossRefGoogle ScholarPubMed
Ambady, N., Bernieri, F. J., and Richeson, J. A., “Toward a histology of social behavior: Judgmental accuracy from thin slices of the behavioral stream,” Advances in Experimental Social Psychology , 2000, 32: 201–271.CrossRefGoogle Scholar
Borkenau, P., Mauer, N., Riemann, R., Spinath, F. M., and Angleitner, A., “Thin slices of behavior as cues of personality and intelligence,” Journal of Personality and Social Psychology , 2004, 86(4): 599–614.CrossRefGoogle ScholarPubMed
Zebrowitz, L. A. and Montepare, J. M., “Social psychological face perception: Why appearance matters,” Social and Personality Psychology Compass , 2008, 2(3): 1497–1517.CrossRefGoogle ScholarPubMed
Bänziger, T., Mortillaro, M., and Scherer, K. R., “Introducing the Geneva multimodal expression corpus for experimental research on emotion perception,” Emotion , 2012, 12(5): 1161–1179.CrossRefGoogle ScholarPubMed
Clarke, T. J., Bradshaw, M. F., Field, D. T., Hampson, S. E., and Rose, D., “The perception of emotion from body movement in point-light displays of interpersonal dialogue,” Perception , 2005, 34(10): 1171–1180.CrossRefGoogle ScholarPubMed
Dael, N., Mortillaro, M., and Scherer, K. R., “Emotion expression in body action and posture,” Emotion , 2012, 12(5): 1085–1101.CrossRefGoogle ScholarPubMed
Pollick, F., Paterson, H., Bruderlin, A., and Sanford, A., “Perceiving affect from arm movement,” Cognition , 2001, 82: B51–B61.CrossRefGoogle ScholarPubMed
Scherer, K. R., “Expression of emotion in voice and music,” Journal of Voice , 1995, 9(3): 235–248.CrossRefGoogle ScholarPubMed
Thoresen, J. C., Vuong, Q. C., and Atkinson, A. P., “First impressions: Gait cues drive reliable trait judgements,” Cognition , 124(3): 261–271.CrossRefGoogle Scholar
Koppensteiner, M., Stephan, P., and Jäschke, J. P. M., “More than words: Judgments of politicians and the role of different communication channels,” Journal of Research in Personality , 2015, 58: 21–30.CrossRefGoogle Scholar
Kramer, R. S., Arend, I., and Ward, R., “Perceived health from biological motion predicts voting behavior,” Quarterly Journal of Experimental Psychology , 2010, 63(4): 625–632.CrossRefGoogle Scholar
Klofstad, C. A., Anderson, R. C., and Peters, S., “Sounds like a winner: Voice pitch influences perception of leadership capacity in both men and women,” Proceedings of the Royal Society of London B: Biological Sciences , 2012, 297(1738): 2698–2704.Google Scholar
Tigue, C. C., Borak, D. J., O’Connor, J. J., Schandl, C., and Feinberg, D. R., “Voice pitch influences voting behavior,” Evolution and Human Behavior , 2012, 33(3): 210–216.CrossRefGoogle Scholar
Mehu, M. and van der Maaten, L., “Multimodal integration of dynamic audio-visual cues in the communication of agreement and disagreement,” Journal of Nonverbal Behavior , 2014, 38(4): 569–597.CrossRefGoogle Scholar
Shah, D. V., Hanna, A., and Bucy, E. P. et al. , “Dual screening during presidential debates: Political nonverbals and the volume and valence of online expression,” American Behavioral Scientist , 2016, 60(14): 1816–1843.CrossRefGoogle Scholar
Shah, D. V., Hanna, A., Bucy, E. P., Wells, C., and Quevedo, V., “The power of television images in a social media age: Linking biobehavioral and computational approaches via the second screen,” Annals of the American Academy of Political and Social Science , 2015, 659(1): 225–245.CrossRefGoogle Scholar
Bucy, E. P., “Emotional and evaluative consequences of inappropriate leader displays,” Communication Research , 2000, 27(2): 194–226.CrossRefGoogle Scholar
Bergen, L., Grimes, T., and Potter, D., “How attention partitions itself during simultaneous message presentations,” Human Communication Research , 2005, 31(3): 311–336.CrossRefGoogle Scholar
Lang, A., “Defining audio/video redundancy from a limited-capacity information processing perspective,” Communication Research , 1995, 22(1): 86–115.CrossRefGoogle Scholar
De Gelder, B. and Bertelson, P., “Multisensory integration, perception and ecological validity,” Trends in Cognitive Sciences , 2003, 7(10): 460–467.CrossRefGoogle ScholarPubMed
Navarra, J., Alsius, A., Soto-Faraco, S., and Spence, C., “Assessing the role of attention in the audiovisual integration of speech,” Information Fusion , 2010, 11(1): 4–11.CrossRefGoogle Scholar
Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., and Foxe, J. J., “Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments,” Cerebral Cortex , 2006, 17(5): 1147–1153.CrossRefGoogle Scholar
Johnstone, R. A., “Multiple displays in animal communication: ‘Backup signals’ and ‘multiple messages,”’ Philosophical Transactions of the Royal Society of London B: Biological Sciences , 1996, 351(1337): 329–338.Google Scholar
Rowe, C., “Receiver psychology and the evolution of multicomponent signals,” Animal Behavior , 1999, 58(5): 921–931.CrossRefGoogle ScholarPubMed
Grammer, K., Filova, V., and Fieder, M., “The communication paradox and possible solutions,” in New Aspects of Human Ethology, Atzwanger, K., Grammer, K., Schäfer, K., and Schmitt, A., eds. (New York: Springer, 1996), pp. 91–120.Google Scholar
Mehu, M., “The integration of emotional and symbolic components in multimodal communication,” Frontiers in Psychology , 2015, 6: 7–11.CrossRefGoogle ScholarPubMed
Carli, L. L., LaFleur, S. J., and Loeber, C. C., “Nonverbal behavior, gender, and influence,” Journal of Personality and Social Psychology , 1995, 68(6): 1030–1041.CrossRefGoogle Scholar
Koppensteiner, M., Stephan, P., and Jäschke, J. P. M., “Moving speeches: Dominance, trustworthiness and competence in body motion,” Personal and Individual Differences , 2016, 94: 101–106.CrossRefGoogle Scholar
Koppensteiner, M. and Grammer, K., “Motion patterns in political speech and their influence on personality ratings,” Journal of Research in Personality , 2010, 44(3): 374–379.CrossRefGoogle Scholar
Koppensteiner, M., Stephan, P., and Jäschke, J. P. M., “Shaking takete and flowing maluma: Non-sense words are associated with motion patterns,” PLOS ONE , 2016, 11(3): e0150610, doi:10.1371/journal.pone.0150610.CrossRefGoogle ScholarPubMed
Ma, R., “Parametric speech emotion recognition using neural network,” in Proceedings of the 5th International Symposium on Neural Networks, Sun, F., Zhang, J., Cao, J., and Yu, W., eds. (Berlin: Springer-Verlag, 2008).Google Scholar
Team RC, R: A Language and Environment for Statistical Computing (Vienna: R Foundation for Statistical Computing, 2013).Google Scholar
Johnson, J. W., “A heuristic method for estimating the relative weight of predictor variables in multiple regression,” Multivariate Behavioral Research , 2000, 35(1): 1–19.CrossRefGoogle ScholarPubMed
Krahmer, E. and Swerts, M., “The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception,” Journal of Memory and Language , 2007, 57(3): 396–414.CrossRefGoogle Scholar
Van den Stock, J., Righart, R., and De Gelder, B., “Body expressions influence recognition of emotions in the face and voice,” Emotion , 2007, 7(3): 487–494.CrossRefGoogle ScholarPubMed
Burgoon, J. K., “Interpersonal expectations, expectancy violations, and emotional communication,” Journal of Language and Social Psychology , 1993, 12(1/2): 30–48.CrossRefGoogle Scholar
Awamleh, R. and Gardner, W. L., “Perceptions of leader charisma and effectiveness: The effects of vision content, delivery, and organizational performance,” Leadership Quarterly , 1999, 10(3): 345–373.CrossRefGoogle Scholar
Gong, Z. H. and Bucy, E. P., “When style obscures substance: Visual attention to display appropriateness in the 2012 presidential debates,” Communication Monographs , 2016, 83(3): 349–372.CrossRefGoogle Scholar
Supplementary material: File

Koppensteiner and Siegle supplementary material 1

Koppensteiner and Siegle supplementary material

Download Koppensteiner and Siegle supplementary material 1(File)
File 2.7 MB